Data comparison method and device, storage medium and electronic equipment

Abstract

The invention discloses a data comparison method and device, a storage medium and electronic equipment. The method comprises the following steps: in response to a data comparison request initiated by a client, determining comparison rule data in the data comparison request; analyzing the comparison rule data to obtain query format data; inputting the query format data into a stream processing framework to obtain a data comparison result; wherein the stream processing framework is used for executing a data comparison task; and feeding back a data comparison result to the client. Through the method and the device, the problem that the rule comparison result cannot be flexibly pushed to the user side in real time in the prior art is solved.

Description

Background Art

[0002] In a production environment, the amount of data that needs to be processed every day reaches tens of billions. Users who actually use the data are very concerned about how to compare and extract data that meets the specified rules. In actual business needs, there are often situations where the Source or Sink that comes with FlinkSQL cannot meet business needs, and it cannot guarantee that the rule comparison results can be pushed to the user end in real time and flexibly.

[0003] With regard to the problem in related technologies that the rule comparison results cannot be pushed to the user end in real time and flexibly, no effective solution has been proposed yet. Summary of the invention

[0004] The main purpose of the present application is to provide a data comparison method, device, storage medium and electronic device to solve the problem in the related art that it is impossible to ensure that the rule comparison results can be pushed to the user end in real time and flexibly.

[0005] In order to achieve the above-mentioned purpose, according to the first aspect of the present application, a data comparison method is provided. The method comprises: responding to a data comparison request initiated by a client, determining comparison rule data in the data comparison request; parsing the comparison rule data to obtain query format data; inputting the query format data into a stream processing framework to obtain a data comparison result; wherein the stream processing framework is used to execute the data comparison task; and feeding back the data comparison result to the client.

[0006] Furthermore, in response to a data comparison request initiated by a client, comparison rule data in the data comparison request is determined, including: parsing the data comparison request to obtain rule definition data, wherein the rule definition data includes at least one of the following: data comparison method, data association, and data constraint conditions, and the data comparison method includes at least one of the following: exact comparison, keyword comparison, and logical expression comparison; and using the rule definition data as comparison rule data.

[0007] Furthermore, in response to a data comparison request initiated by the client, after determining the comparison rule data in the data comparison request, the method further includes: receiving the comparison rule data sent by the client through a service interface program; and storing the comparison rule data in a database.

[0008] Furthermore, the query format data is input into the stream processing framework to obtain data comparison results, including: in response to the comparison start command, executing the data comparison task, and reading the configuration items of the query format data in the comparison start command; splitting the query format data based on the configuration items to obtain multiple query format data; parsing the multiple query format data through a statement parser to obtain clause parsing results, creating a syntax tree object based on the clause parsing results, and determining the data comparison results based on the syntax tree object.

[0009] Furthermore, a syntax tree object is created based on the clause parsing result, and a data comparison result is determined based on the syntax tree object, including: building a running environment object of a stream processing framework according to the syntax tree object; registering target information into the running environment object, and executing a statement query method, wherein the target information includes: a custom function method, a registration source, and a target table; determining a stream processing task plan according to the statement query method, matching query format data based on the stream processing task plan, and obtaining a data comparison result.

[0010] Furthermore, the statement parser has at least one of the following functions: checking and converting field types of statements for creating tables; checking and converting field types of SQL statements; parsing statements for creating functions; and parsing statements for creating views.

[0011] Furthermore, the data comparison result is fed back to the client, including: outputting the data comparison result to the target system, wherein the target system at least includes: Kafka; determining the instantiated component corresponding to the target system, and writing the data comparison result to the storage mechanism of the target system based on the instantiated component, so as to display the data comparison result on the client.

[0012] In order to achieve the above-mentioned purpose, according to the second aspect of the present application, a data comparison device is provided. The device includes: a response unit, which is used to respond to a data comparison request initiated by a client and determine the comparison rule data in the data comparison request; a parsing unit, which is used to parse the comparison rule data to obtain query format data; an input unit, which is used to input the query format data into a stream processing framework to obtain a data comparison result; wherein the stream processing framework is used to perform a data comparison task; and a feedback unit, which is used to feed back the data comparison result to the client.

[0013] Furthermore, the response unit includes: a parsing module, which is used to parse the data comparison request to obtain rule definition data, wherein the rule definition data includes at least one of the following: data comparison method, data association, and data constraint conditions, and the data comparison method includes at least one of the following: exact comparison, keyword comparison, and logical expression comparison; a determination module, which is used to use the rule definition data as comparison rule data.

[0014] Furthermore, the device also includes: a sending unit, which is used to respond to a data comparison request initiated by the client, determine the comparison rule data in the data comparison request, and then receive the comparison rule data sent by the client through a service interface program; a storage unit, which is used to store the comparison rule data in a database.

[0015] Furthermore, the input unit includes: a processing module, which is used to respond to the comparison start command, execute the data comparison task, and read the configuration items of the query format data in the comparison start command; a splitting module, which is used to split the query format data based on the configuration items to obtain multiple query format data; a parsing module, which is used to parse the multiple query format data through a statement parser to obtain clause parsing results, create a syntax tree object based on the clause parsing results, and determine the data comparison results based on the syntax tree object.

[0016] Furthermore, the parsing module includes: a construction submodule, which is used to construct a running environment object of a stream processing framework based on a syntax tree object; a registration submodule, which is used to register target information into the running environment object and execute a statement query method, wherein the target information includes: a custom function method, a registration source, and a target table; a determination submodule, which is used to determine a stream processing task plan based on the statement query method, match the query format data based on the stream processing task plan, and obtain a data comparison result.

[0017] Furthermore, the statement parser has at least one of the following functions: checking and converting field types of statements for creating tables; checking and converting field types of SQL statements; parsing statements for creating functions; and parsing statements for creating views.

[0018] Furthermore, the feedback unit includes: an output module for outputting the data comparison results to the target system, wherein the target system includes at least: Kafka; a display module for determining the instantiated component corresponding to the target system, and writing the data comparison results to the storage mechanism of the target system based on the instantiated component, so as to display the data comparison results on the client.

[0019] According to a third aspect of an embodiment of the present application, an electronic device is provided, comprising: a processor; and a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, any one of the above-mentioned data comparison methods is implemented.

[0020] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the data comparison method according to any one of the above items is implemented.

[0021] Through this application, the following steps are adopted: in response to a data comparison request initiated by a client, the comparison rule data in the data comparison request is determined; the comparison rule data is parsed to obtain query format data; the query format data is input into the stream processing framework to obtain the data comparison result; wherein the stream processing framework is used to execute the data comparison task; the data comparison result is fed back to the client. Through this application, the problem that the rule comparison results cannot be guaranteed to be pushed to the user end in real time and flexibly in the related technology is solved. Through this application, the problem that the rule comparison results cannot be guaranteed to be pushed to the user end in real time and flexibly in the related technology is solved. By flexibly configuring the comparison rules and sending the rules to the data stream processing task framework, and without the need for code development for each requirement, the query format data written can be submitted to the stream processing task. Thereby achieving the effect of pushing the rule comparison results to the user end in real time and flexibly. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0023] Figure 1 is a flow chart of a data comparison method provided according to an embodiment of the present application;

[0024] Figure 2 It is an overall schematic diagram of data comparison provided according to the embodiment of the present application;

[0025] Figure 3 is a schematic diagram of a data comparison device provided according to an embodiment of the present application;

[0026] Figure 4 It is a schematic diagram of the network architecture of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0027] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0028] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0029] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0030] According to an embodiment of the present application, a data comparison method is provided.

[0031] Figure 1 is a flow chart of a data comparison method according to an embodiment of the present application. Figure 1 As shown, the method comprises the following steps:

[0032] Step S101, in response to a data comparison request initiated by a client, determining comparison rule data in the data comparison request.

[0033] Exemplarily, a user sends a data comparison request on a smartphone or computer, wherein the data comparison request requires setting some rules for comparing gift lists, and the comparison rule data may be the user's demand data, for example, the user's demand data may be the need to filter out identical gift lists.

[0034] Specifically, in response to a data comparison request initiated by a client, the comparison rule data in the data comparison request is determined, which can be obtained through the following steps: parsing the data comparison request to obtain rule definition data, wherein the rule definition data includes at least one of the following: data comparison method, data association, and data constraint conditions, and the data comparison method includes at least one of the following: exact comparison, keyword comparison, and logical expression comparison; using the rule definition data as comparison rule data.

[0035] Exemplarily, rule definition data may include data comparison methods, data associations, data constraints, and may also include data matching standards, data conversion rules, field priorities, data filtering conditions, etc., wherein data comparison methods include exact comparison, keyword comparison, and logical expression comparison. By using rule definition data as comparison rule data, the accuracy, efficiency, consistency, and repeatability of the comparison operation can be significantly improved, while enhancing the user experience and the maintainability of the system.

[0036] Exemplarily, the data model of the rule definition data can be implemented by the following code:

[0037] db.getCollection("t_resource_alarm_info").drop();

[0038] db.createCollection("t_resource_alarm_info");

[0039] / / Rule id

[0040] db.t_resource_alarm_info.createIndex({"ruleid":1},{"background":true});

[0041] / / Rule type

[0042] db.t_resource_alarm_info.createIndex({"ruletype":1},{"background":true});

[0043] / / Regular expression

[0044] db.t_resource_alarm_info.createIndex({"rulevalue":1},{"background":true});

[0045] / / Data resource id

[0046] db.

[0047] t_resource_alarm_info.createIndex({"resourceid":1},{"background":true});

[0048] / / start date

[0049] db.t_resource_alarm_info.createIndex({"sttm":1},{"background":true});

[0050] / / expiration date

[0051] db.t_resource_alarm_info.createIndex({"entm":1},{"background":true});

[0052] / / Creation date

[0053] db.t_resource_alarm_info.createIndex({"cttm":1},{"background":true});

[0054] / / Update date

[0055] db.t_resource_alarm_info.createIndex({"uptm":1},{"background":true});

[0056] / / Number of hits

[0057] db.t_resource_alarm_info.createIndex({"hitc":1},{"background":true});

[0058] / / state

[0059] db.t_resource_alarm_info.createIndex({"stat":1},{"background":true});

[0060] Among them, the core business processing classes corresponding to the comparison methods are: CompareAlarm and CheckAlarm.

[0061] 1. CompareAlarm provides the implementation logic of exact comparison, keyword comparison, and logical expression comparison

[0062] (1) Accurate comparison

[0063] 1. Obtain all rules that require accurate comparison from the memory cache through the data resource ID and comparison type.

[0064] 2. Traverse the exact matching rules one by one to obtain the matching fields and matching values ​​in the rules.

[0065] 3. Get the value of the corresponding matching field in the data currently being processed in the data stream.

[0066] 4. Compare the value in the data with the value configured in the rule for equality. If the match is successful, the first hit is successful.

[0067] (2) Keyword comparison

[0068] 1. Obtain the keyword comparison rules of the current data resource from the memory cache through the data resource ID and comparison type.

[0069] 2. Traverse the keyword matching rules one by one, obtain the field name that needs fuzzy matching, and add one or more fuzzy matching values ​​to the Set collection.

[0070] 3. According to the fuzzy matched field name, get the value of the field from the data object in the data stream.

[0071] 4. Pass the results of steps 2 and 3 to the AhoCorasik multi-pattern matching algorithm. If the match is successful, the first hit is successful.

[0072] It should be noted that the Aho-Corasik algorithm is a multi-mode matching algorithm that can find all occurrences of multiple pattern strings in a text string.

[0073] (3) Logical expression comparison

[0074] 1. Obtain all rules that require logical expression comparison from the memory cache based on the data resource ID and comparison type.

[0075] 2. Traverse the rules for logical expression comparison one by one to obtain the matching fields, matching values, and expression conditions defined in the rules.

[0076] 3. Get the value of the corresponding matching field in the data.

[0077] 4. Convert the and and or conditions to && and || in Java.

[0078] 5. Use the field values ​​and logical expressions obtained in the above steps to determine whether the data is matched. If the match is successful, the first match is successful.

[0079] 2. The function implemented by the CheckAlarm class is to perform a secondary check on the hit data. The main functions are as follows:

[0080] (1) Determine whether the data hit by the matching rule exceeds the maximum data volume configured by the user.

[0081] (2) Update the amount of data hit by each matching rule in the mongodb table.

[0082] (3) Push the hit data to Kafka configured in the SQL statement source_table.

[0083] In some embodiments, in response to a data comparison request initiated by a client, after determining the comparison rule data in the data comparison request, the method further includes: receiving the comparison rule data sent by the client through a service interface program; and storing the comparison rule data in a database.

[0084] Specifically, the database can be a mongodb library. For example, the interface program exposes a service port to receive the comparison rule parameters sent by the application client and stores the rule data persistently in the defined mongodb library table. There are two types of interfaces for the service interface program provided in this case: issuing rule interface and deactivating rule interface. This case receives the comparison rule data sent by the client through the service interface program and stores the data in the database, which can bring about centralized data management, dynamic update flexibility, and improved comparison efficiency.

[0085] Step S102, parsing the comparison rule data to obtain query format data.

[0086] For example, the comparison rule data can be "the quantity of each product in the order cannot exceed the available quantity in the inventory". The system will parse the logic in the comparison rule. Taking SQL as an example, the query format data obtained is SELECT*FROMinventory WHERE product_id=? AND available_quantity>=?. In this case, parsing the comparison rule data and obtaining the query format data is one of the key steps to realize data comparison and query operations, ensuring that the system can efficiently perform data comparison and query tasks according to the user's intentions and requirements.

[0087] Step S103, inputting the query format data into the stream processing framework to obtain a data comparison result; wherein the stream processing framework is used to execute the data comparison task.

[0088] Among them, the stream processing framework of the present application is the FlinkSQL framework, which supports distributed processing and can horizontally expand the system capacity by adding nodes to handle larger-scale data comparison tasks.

[0089] Specifically, the query format data is input into the stream processing framework to obtain the data comparison result, which can be achieved through the following steps: in response to the comparison start command, the data comparison task is executed, and the configuration items of the query format data in the comparison start command are read; based on the configuration items, the query format data is split to obtain multiple query format data; the multiple query format data are parsed by a statement parser to obtain clause parsing results, a syntax tree object is created based on the clause parsing results, and the data comparison result is determined based on the syntax tree object.

[0090] For example, the main process description of starting the FlinkSQL engine, entry class: com.meiya.whale.engine.SqlEngine;

[0091] The main tasks completed by this entry class are as follows:

[0092] 1. Read and compare the various configuration items in the configuration file specified in the startup command and encapsulate them into the FlinkJob object. The main configuration items include: the jar package of the custom UDF line number, the parallelism parameters of the FlinkSQL task, and the actual SQL statement to be submitted.

[0093] 2. Split the SQL statement into different clauses based on the configuration items, judge the type of each clause, and call the corresponding SQL statement parser. The results of parsing each clause are encapsulated into the SqlTree object (that is, the syntax tree object in this case).

[0094] 3. And determine the data comparison result based on the syntax tree object.

[0095] In some embodiments, a syntax tree object is created based on the clause parsing result, and a data comparison result is determined based on the syntax tree object, which can be obtained through the following steps: constructing an operating environment object of a stream processing framework based on the syntax tree object; registering target information into the operating environment object, and executing a statement query method, wherein the target information includes: a custom function method, a registration source, and a target table; determining a stream processing task plan based on the statement query method, matching query format data based on the stream processing task plan, and obtaining a data comparison result.

[0096] Exemplarily, it is mainly achieved by the following steps:

[0097] 1. Build the instance running environment object tableEnv of the Flink running environment TableEnvironment.

[0098] 2. Get the relevant information of the parsed custom function from the SqlTree object. If the function is defined, register it to the tableEnv object.

[0099] 3. Get the relevant information of the parsed source table and target table from the SqlTree object, and register the source table and target table to the tableEnv object.

[0100] Among them, when registering the table, the corresponding custom Sink object will be instantiated through the reflection technology in Java according to the 'type'='alarm' defined in the SQL statement.

[0101] 4. Get the parsed executable SQL statement from the SqlTree object and submit the SQL statement through the executeSql method of the tableEnv object to implement the stream processing task plan.

[0102] The SQL statement parser has at least one of the following functions: verifying statements for creating tables and converting field types; verifying SQL statements and converting field types; parsing statements for creating functions; and parsing statements for creating views.

[0103] For example, the SQL statement parser can implement the following functions:

[0104] 1. "create table..." statement parser: Verify the statement for creating a table and convert the field type. And record the parsed table name, field name and field type, and parameter information after the WITH keyword.

[0105] 2. "insert into...select..." statement parser: Verify SQL statements and convert field types. Record the parsed source table, target table, target table field name and field type, and execute SQL statements.

[0106] 3. "create function..." statement parser: parses the statement for creating a function and records the parsed function type, function name, and target class that implements the function.

[0107] 4. "create view..." statement parser: parses the statement for creating a view, verifies the SQL statement, and records the parsed view name, view field name and field type, and creates an executable SQL statement for the view.

[0108] This case brings many benefits by calling the corresponding SQL statement parser and encapsulating the parsing results into the SqlTree object, including improving the readability and maintainability of the code, simplifying SQL operations, enhancing the security of SQL statements, facilitating SQL optimization, supporting multiple database systems, and promoting the decoupling of the database access layer.

[0109] Step S104, feeding back the data comparison result to the client.

[0110] Specifically, the data comparison results are fed back to the client through the following steps: outputting the data comparison results to the target system, wherein the target system at least includes: Kafka; determining the instantiated component corresponding to the target system, and writing the data comparison results to the storage mechanism of the target system based on the instantiated component, so as to display the data comparison results on the client.

[0111] For example, the custom Sink implements the above steps:

[0112] According to different output scenario requirements, different instantiation components SinkConnector are implemented, including output to target systems such as kafka, elasticsearch, and hive.

[0113] Here, output to Kafka is used as an example. When the program is running, the parameter 'type' = 'alarm' in SQL is used to load the custom Sink processing class AlarmStreamSink, and instantiate the object of this class. Based on the instantiated component, the data comparison result is written to the storage mechanism of the target system to display the data comparison result on the client, improve the real-time and accuracy of data processing, realize centralized management and efficient access to data, enhance the visualization of data, support multi-client access and data sharing, and improve the scalability and flexibility of the system.

[0114] Among them, the interfaces implemented by AlarmStreamSink in this example are AppendStreamTableSink and RichSinkFunction, and the interface methods consumeDataStream, open, invoke, and close are implemented. The following is an implementation description of the key methods.

[0115] 1. Contents of the open method: This method is executed once when the task is created. It is used to complete some initialization operations, create a MongoDB connection (for reading comparison rules), create a Kafka connection (for outputting hit data), and create a timed thread (for timed synchronization of comparison rules in MongoDB).

[0116] 2. The content of the invoke method: This method is used to process each piece of data in the data stream. It converts the data format in the data stream from the Row object of FlinkSQL to the Map object in Java, and calls the method that implements the comparison logic. The Map object is passed as a parameter to the object method that implements the comparison logic. The comparison method in the CompareAlarm class is declared as follows:

[0117] public static void compareAlarm(Row row,KafkaProducer<String,String> producer,String topic,String redisAddr,Map<String,Object> jsonMap).

[0118] Alternatively, if Figure 2As shown in the figure, the client application initiates a request to send the comparison rule. After receiving the comparison request, the service interface writes the comparison rule to mongodb. Define the SQL processing statement and pass the defined SQL statement to the SQL parsing engine. Register the source table and target table parsed by the SQL engine to the Flink execution environment. Submit the executable statement "insert into select..." to the execution environment through the FlinkSQL API to start the task. The scheduled thread in the FlinkSQL engine program reads the mongodb table and loads the data into the global cache of the memory. The FlinkSQL engine program compares the data in the data stream according to the configured comparison rules. The successfully compared data is pushed to Kafka or elasticsearch, and the successfully compared data is written to mongodb. The comparison will be automatically stopped when the successfully compared data reaches the upper limit. The comparison rule can also be disabled through the interface to disable the comparison rule. This case introduces an SQL task instance as an example. The data structure and connection configuration information of the source table (source_table) and the target table (result_table) are defined in the SQL sample of the job. The "insert into..." statement indicates the processing of mapping the data of the source table to the target table. When the FlinkSQL task is executed, it reads the data in the Kafka defined in the source table source_table and writes the data that meets the comparison rules to the Kafka defined in the target table result_table.

[0119] The data comparison method provided in the embodiment of the present application determines the comparison rule data in the data comparison request by responding to the data comparison request initiated by the client; parses the comparison rule data to obtain query format data; inputs the query format data into the stream processing framework to obtain the data comparison result; wherein the stream processing framework is used to execute the data comparison task; and feeds back the data comparison result to the client. Through this application, the problem that the rule comparison results cannot be guaranteed to be pushed to the user end in real time and flexibly in the related technology is solved. By flexibly configuring the comparison rules and issuing the rules to the data stream processing task framework, and without the need for code development for each requirement, the written query format data can be submitted to the stream processing task. Thereby achieving the effect of pushing the rule comparison results to the user end in real time and flexibly.

[0120] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0121] The embodiment of the present application also provides a data comparison device. It should be noted that the data comparison device of the embodiment of the present application can be used to execute the data comparison method provided in the embodiment of the present application. The data comparison device provided in the embodiment of the present application is introduced below.

[0122] Figure 3 Schematic diagram of a data comparison device according to an embodiment of the present application. Figure 3 As shown, the device 300 includes: a response unit 301, a parsing unit 302, an input unit 303, and a feedback unit 304.

[0123] Specifically, the response unit 301 is used to respond to the data comparison request initiated by the client and determine the comparison rule data in the data comparison request;

[0124] A parsing unit 302, used to parse the comparison rule data to obtain query format data;

[0125] An input unit 303 is used to input the query format data into the stream processing framework to obtain a data comparison result; wherein the stream processing framework is used to perform the data comparison task;

[0126] The feedback unit 304 is used to feed back the data comparison result to the client.

[0127] In summary, the data comparison device provided by the embodiment of the present application is used to respond to the data comparison request initiated by the client through the response unit 301, and determine the comparison rule data in the data comparison request; the parsing unit 302 is used to parse the comparison rule data to obtain the query format data; the input unit 303 is used to input the query format data into the stream processing framework to obtain the data comparison result; wherein the stream processing framework is used to perform the data comparison task; the feedback unit 304 is used to feed back the data comparison result to the client. Through this application, the problem that the rule comparison results cannot be guaranteed to be pushed to the user end in real time and flexibly in the related technology is solved. By flexibly configuring the comparison rules and issuing the rules to the data stream processing task framework, and without the need for code development for each requirement, the query format data written can be submitted to the stream processing task. Thereby achieving the effect of pushing the rule comparison results to the user end in real time and flexibly.

[0128] Optionally, in the data comparison device provided in the embodiment of the present application, the response unit includes: a parsing module, used to parse the data comparison request to obtain rule definition data, wherein the rule definition data includes at least one of the following: data comparison method, data association, and data constraint conditions, and the data comparison method includes at least one of the following: exact comparison, keyword comparison, and logical expression comparison; a determination module, used to use the rule definition data as comparison rule data.

[0129] Optionally, in the data comparison device provided in the embodiment of the present application, the device also includes: a sending unit, used to respond to a data comparison request initiated by the client, determine the comparison rule data in the data comparison request, and then receive the comparison rule data sent by the client through a service interface program; a storage unit, used to store the comparison rule data in a database.

[0130] Optionally, in the data comparison device provided in the embodiment of the present application, the input unit includes: a processing module, which is used to respond to the comparison start command, execute the data comparison task, and read the configuration items of the query format data in the comparison start command; a splitting module, which is used to split the query format data based on the configuration items to obtain multiple query format data; a parsing module, which is used to parse the multiple query format data through a statement parser to obtain clause parsing results, create a syntax tree object based on the clause parsing results, and determine the data comparison results based on the syntax tree object.

[0131] Optionally, in the data comparison device provided in the embodiment of the present application, the parsing module includes: a construction sub-module, which is used to construct a running environment object of a stream processing framework based on a syntax tree object; a registration sub-module, which is used to register target information into the running environment object and execute a statement query method, wherein the target information includes: a custom function method, a registration source, and a target table; a determination sub-module, which is used to determine a stream processing task plan according to the statement query method, match the query format data based on the stream processing task plan, and obtain a data comparison result.

[0132] Optionally, in the data comparison device provided in the embodiment of the present application, the statement parser has at least one of the following functions: checking and converting field types of statements for creating tables; checking and converting field types of SQL statements; parsing statements for creating functions; and parsing statements for creating views.

[0133] Optionally, in the data comparison device provided in the embodiment of the present application, the feedback unit includes: an output module, used to output the data comparison results to the target system, wherein the target system includes at least: Kafka; a display module, used to determine the instantiated component corresponding to the target system, and write the data comparison results to the storage mechanism of the target system based on the instantiated component, so as to realize the display of the data comparison results on the client.

[0134] The data comparison device includes a processor and a memory. The response unit 301, the analysis unit 302, the input unit 303, the feedback unit 304, etc. are all stored in the memory as program units, and the processor executes the program units stored in the memory to realize corresponding functions.

[0135] The processor includes a kernel, which calls the corresponding program unit from the memory. One or more kernels can be set, and data comparison can be performed by adjusting kernel parameters.

[0136] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0137] In the exemplary embodiment of the present application, a computer storage medium capable of implementing the above method is also provided. A program product capable of implementing the above method of this specification is stored thereon. In some possible embodiments, various aspects of the present application can also be implemented in the form of a program product, which includes a program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps described in the above "Exemplary Method" section of this specification according to various exemplary embodiments of the present application, such as the following steps can be performed: in response to a data comparison request initiated by a client, determine the comparison rule data in the data comparison request; parse the comparison rule data to obtain query format data; input the query format data into a stream processing framework to obtain a data comparison result; wherein the stream processing framework is used to perform data comparison tasks; and feed back the data comparison result to the client.

[0138] In an optional implementation: the data comparison request is parsed to obtain rule definition data, wherein the rule definition data includes at least one of the following: data comparison method, data association, data constraint condition, the data comparison method includes at least one of the following: exact comparison, keyword comparison, logical expression comparison; the rule definition data is used as comparison rule data.

[0139] In an optional implementation: in response to a data comparison request initiated by a client, after determining the comparison rule data in the data comparison request, the method further includes: receiving the comparison rule data sent by the client through a service interface program; and storing the comparison rule data in a database.

[0140] In an optional implementation: in response to a comparison start command, a data comparison task is executed, and a configuration item of the query format data in the comparison start command is read; the query format data is split based on the configuration item to obtain multiple query format data; the multiple query format data are parsed by a statement parser to obtain clause parsing results, a syntax tree object is created based on the clause parsing results, and the data comparison result is determined based on the syntax tree object.

[0141] In an optional implementation: construct an operating environment object of a stream processing framework based on a syntax tree object; register target information into the operating environment object, and execute a statement query method, wherein the target information includes: a custom function method, a registration source, and a target table; determine a stream processing task plan based on the statement query method, match query format data based on the stream processing task plan, and obtain a data comparison result.

[0142] In an optional implementation: the statement for creating a table is checked and the field type is converted; the SQL statement is checked and the field type is converted; the statement for creating a function is parsed; and the statement for creating a view is parsed.

[0143] In an optional implementation: outputting the data comparison result to a target system, wherein the target system includes at least: Kafka; determining an instantiated component corresponding to the target system, and writing the data comparison result to a storage mechanism of the target system based on the instantiated component, so as to display the data comparison result on the client.

[0144] In an optional embodiment, the embodiment of the present application may also include a program product for implementing the above method, which may adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present application is not limited thereto, and in this document, a readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, apparatus, or device.

[0145] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0146] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0147] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0148] Program code for performing the operations of the present application may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0149] In addition, in an exemplary embodiment of the present application, an electronic device capable of implementing the above method is also provided.

[0150] Those skilled in the art will appreciate that various aspects of the present application may be implemented as a system, method or program product. Therefore, various aspects of the present application may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to as "circuit", "module" or "system" herein.

[0151] Refer to the following Figure 4 The electronic device 400 according to this embodiment of the present application is described. Figure 4 The electronic device 400 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0152] like Figure 4As shown, the electronic device 400 is in the form of a general computing device. The components of the electronic device 400 may include, but are not limited to: the at least one processing unit 410, the at least one storage unit 420, a bus 430 connecting different system components (including the storage unit 420 and the processing unit 410), and a display unit 440.

[0153] The storage unit stores program codes, which can be executed by the processing unit 410, so that the processing unit 410 executes the steps described in the "Exemplary Method" section of the present specification according to various exemplary embodiments of the present application. For example, the processing unit 410 can execute the following steps: responding to a data comparison request initiated by a client, determining the comparison rule data in the data comparison request; parsing the comparison rule data to obtain query format data; inputting the query format data into the stream processing framework to obtain data comparison results; wherein the stream processing framework is used to perform data comparison tasks; and feeding back the data comparison results to the client.

[0154] In an optional implementation: the data comparison request is parsed to obtain rule definition data, wherein the rule definition data includes at least one of the following: data comparison method, data association, data constraint condition, the data comparison method includes at least one of the following: exact comparison, keyword comparison, logical expression comparison; the rule definition data is used as comparison rule data.

[0155] In an optional implementation: in response to a data comparison request initiated by a client, after determining the comparison rule data in the data comparison request, the method further includes: receiving the comparison rule data sent by the client through a service interface program; and storing the comparison rule data in a database.

[0156] In an optional implementation: in response to a comparison start command, a data comparison task is executed, and a configuration item of the query format data in the comparison start command is read; the query format data is split based on the configuration item to obtain multiple query format data; the multiple query format data are parsed by a statement parser to obtain clause parsing results, a syntax tree object is created based on the clause parsing results, and the data comparison result is determined based on the syntax tree object.

[0157] In an optional implementation: construct an operating environment object of a stream processing framework based on a syntax tree object; register target information into the operating environment object, and execute a statement query method, wherein the target information includes: a custom function method, a registration source, and a target table; determine a stream processing task plan based on the statement query method, match query format data based on the stream processing task plan, and obtain a data comparison result.

[0158] In an optional implementation: the statement for creating a table is checked and the field type is converted; the SQL statement is checked and the field type is converted; the statement for creating a function is parsed; and the statement for creating a view is parsed.

[0159] In an optional implementation: outputting the data comparison result to a target system, wherein the target system includes at least: Kafka; determining an instantiated component corresponding to the target system, and writing the data comparison result to a storage mechanism of the target system based on the instantiated component, so as to display the data comparison result on the client.

[0160] The storage unit 420 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 4201 and / or a cache storage unit 4202 , and may further include a read-only storage unit (ROM) 4203 .

[0161] The storage unit 420 may also include a program / utility 4204 having a set (at least one) of program modules 4205, such program modules 4205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0162] Bus 430 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0163] The electronic device 400 may also communicate with one or more external devices 500 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 400, and / or communicate with any device that enables the electronic device 400 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 440. Furthermore, the electronic device 400 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 460. As shown, the network adapter 460 communicates with other modules of the electronic device 400 via a bus 430. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 400, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0164] Through the description of the above implementation methods, it is easy for those skilled in the art to understand that the example implementation methods described here can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the implementation methods of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation methods of the present application.

[0165] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present application, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0166] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the art that are not disclosed in the present application. The specification and embodiments are to be regarded as exemplary only, and the true scope and spirit of the present application are indicated by the claims.

Claims

1. A data comparison method, characterized in that: include: In response to a data comparison request initiated by a client, determining comparison rule data in the data comparison request; Parsing the comparison rule data to obtain query format data; Inputting the query format data into a stream processing framework to obtain a data comparison result; wherein the stream processing framework is used to perform a data comparison task; The data comparison result is fed back to the client.

2. The method according to claim 1, characterized in that In response to a data comparison request initiated by a client, determining comparison rule data in the data comparison request includes: Parsing the data comparison request to obtain rule definition data, wherein the rule definition data includes at least one of the following: data comparison method, data association, and data constraint conditions, and the data comparison method includes at least one of the following: exact comparison, keyword comparison, and logical expression comparison; The rule definition data is used as the comparison rule data.

3. The method according to claim 1, characterized in that In response to a data comparison request initiated by a client, after determining the comparison rule data in the data comparison request, the method further includes: Receiving the comparison rule data sent by the client through a service interface program; The comparison rule data is stored in a database.

4. The method according to claim 1, characterized in that: The query format data is input into the stream processing framework to obtain data comparison results, including: In response to a comparison start command, executing the data comparison task, and reading the configuration item of the query format data in the comparison start command; Splitting the query format data based on the configuration item to obtain multiple query format data; The plurality of query format data are parsed by a statement parser to obtain clause parsing results, a syntax tree object is created based on the clause parsing results, and a data comparison result is determined based on the syntax tree object.

5. The method according to claim 4, characterized in that Creating a syntax tree object based on the clause parsing result, and determining a data comparison result based on the syntax tree object, including: Constructing the operating environment object of the stream processing framework according to the syntax tree object; Registering target information into the operating environment object and executing a statement query method, wherein the target information includes: a user-defined function method, a registration source, and a target table; A stream processing task plan is determined according to the statement query method, and the query format data is matched based on the stream processing task plan to obtain the data comparison result.

6. The method according to claim 4, characterized in that The statement parser has at least one of the following functions: Verify the statement for creating a table and convert the field type; Verify SQL statements and convert field types; Parse the statement to create the function; Parse the statement for creating a view.

7. The method according to claim 1, characterized in that Feeding back the data comparison result to the client includes: Outputting the data comparison result to a target system, wherein the target system at least includes: Kafka; An instantiated component corresponding to the target system is determined, and based on the instantiated component, the data comparison result is written into a storage mechanism of the target system, so as to realize display of the data comparison result on the client.

8. A data comparison device, characterized in that: include: A response unit, configured to respond to a data comparison request initiated by a client and determine comparison rule data in the data comparison request; A parsing unit, used for parsing the comparison rule data to obtain query format data; An input unit, used to input the query format data into a stream processing framework to obtain a data comparison result; wherein the stream processing framework is used to perform a data comparison task; A feedback unit is used to feed back the data comparison result to the client.

9. A computer-readable storage medium, characterized in that: The storage medium includes a stored program, wherein the program executes the data comparison method according to any one of claims 1 to 7.

10. An electronic device, characterized in that: include: One or more processors, a memory, a display device, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include a method for executing the data comparison method described in any one of claims 1 to 7.

Images