Data verification system, method and electronic device

Abstract

The embodiment of the present application provides a data verification system, method and electronic device, which relates to the field of data verification technology, and is used to realize comprehensive, flexible and efficient batch data verification. In the system, it includes: a data extraction module, a data storage module, a push module and a rule engine; wherein the data extraction module is used to receive a data verification request; the data verification request includes a query condition for the data to be verified; a query request is generated according to the query condition, and the query request is sent to the data storage module; the data storage module is used to query the data to be verified that meets the query condition according to the query request, and send the data to be verified to the push module; the push module is used to receive the data to be verified, and send the data to be verified to the rule engine; the rule engine is used to verify the data to be verified according to the pre-stored verification rules, and output the verification result.

Description

Technical Field

[0001] The present application relates to the technical field of data verification, and in particular to a data verification system, method and electronic device. Background Art

[0002] Data is an important asset of an enterprise. The purpose of online analytical processing is to mine business value from data, improve business operations and business analysis capabilities, and promote effective decision-making for the enterprise. Data verification is essential in data mining.

[0003] Data verification is the process of locating erroneous data through a series of verification rules after the data extraction, cleaning and conversion process is completed, and the abnormal data is pushed to the downstream for processing, ultimately improving the data quality. Since data verification is often performed on batch data, the traditional batch data verification method uses manual or automatic testing methods for verification. However, manual testing is costly and ineffective, and it is almost impossible to find a few anomalies in batch data; automatic testing only extracts representative data from batch data as much as possible, and then compares it by executing the program language. These methods cannot fully proofread batch data and cannot ensure the accuracy of data verification.

[0004] In the era of big data, data is huge and complex, and how to conduct batch data verification comprehensively, flexibly and efficiently is a difficult problem. Summary of the invention

[0005] The embodiments of the present application provide a data verification system, method and electronic device to achieve comprehensive, flexible and efficient batch data verification.

[0006] In a first aspect, an embodiment of the present application provides a data verification system, including: a data extraction module, a data storage module, a push module and a rule engine;

[0007] Among them, the data extraction module is used to receive a data verification request; the data verification request includes a query condition for the data to be verified; a query request is generated according to the query condition, and the query request is sent to the data storage module; the data storage module is used to query the data to be verified that meets the query condition according to the query request, and send the data to be verified to the data extraction module; the data extraction module is also used to send the data to be verified to the push module; the push module is used to receive the data to be verified, and send the data to be verified to the rule engine; the rule engine is used to verify the data to be verified according to a pre-stored verification rule, and output a verification result; the verification result is used to indicate that the verification of the data to be verified has passed, or the verification result is used to indicate that the verification of the data to be verified has failed.

[0008] Based on the above solution, the user's data verification request can be processed through the data extraction module, data storage module, push module and rule engine module to achieve comprehensive, flexible and efficient batch data quality verification when the data is large and complex.

[0009] In a possible implementation, when the data extraction module generates a query request according to the query condition, it is specifically used to: parse the query condition; when the query condition includes the association relationship of the data to be verified, generate an association relationship query request according to the association relationship; and send the association relationship query request and the query request to the data storage module.

[0010] Based on the above solution, by generating an association query request according to the association, the problem that the traditional rule engine cannot implement rule verification across data tables can be solved.

[0011] In a possible implementation, the pre-stored verification rules include a data set and an attribute variable, the data set is used to indicate a data identifier, and the attribute variable is used to indicate data content; wherein at least two data sets have an association relationship, and the association relationship corresponds to the data set and the attribute variable; the data set corresponding to the association relationship includes the at least two data sets; and the attribute variable corresponding to the association relationship is used to indicate that the at least two data sets have an association relationship.

[0012] In a possible implementation manner, the pre-stored verification rule is visually configured.

[0013] Based on the above solution, the verification rules are configured visually, avoiding the previous method of verifying batch business data through hard coding. In addition, business personnel can update the verification rules conveniently and quickly according to changes in demand without the participation of technical personnel, reducing manpower requirements and improving the efficiency of updating verification rules.

[0014] In a possible implementation, the number of the rule engines is at least two; when the push module sends the data to be verified to the rule engines, the push module is specifically used to send the data to be verified to the at least two rule engines by polling.

[0015] Based on the above scheme, the push module uses polling to send the data to be verified to at least two rule engines, which can easily realize rule verification by multiple rule engines at the same time, realize stable operation of the system and improve the efficiency of data verification when the data volume is large.

[0016] In a possible implementation, the data storage module is a clickhouse cluster, and the push module is a rocketMQ cluster.

[0017] Based on the above solution, through the application of clickhouse cluster and rocketMQ cluster technology, the performance of the data verification system execution is improved, the scalability of the data verification system is increased, and the reliability of the data verification system is enhanced.

[0018] In a second aspect, an embodiment of the present application provides a data verification method, comprising: receiving a data verification request; the data verification request includes query conditions for the data to be verified; according to the query conditions, querying the data to be verified that meets the query conditions; according to a pre-stored verification rule, verifying the data to be verified and outputting a verification result; the verification result is used to indicate that the verification of the data to be verified has passed, or the verification result is used to indicate that the verification of the data to be verified has failed.

[0019] In a possible implementation, the receiving of the data verification request specifically includes: parsing the query condition; when the query condition includes the association relationship of the data to be verified, generating an association relationship query request according to the association relationship; and sending the association relationship query request and the query request to the data storage module.

[0020] In a possible implementation, the pre-stored verification rules include a data set and an attribute variable, the data set is used to indicate a data identifier, and the attribute variable is used to indicate data content; wherein at least two data sets have an association relationship, and the association relationship corresponds to the data set and the attribute variable; the data set corresponding to the association relationship includes the at least two data sets; and the attribute variable corresponding to the association relationship is used to indicate that the at least two data sets have an association relationship.

[0021] In a possible implementation manner, the pre-stored verification rule is visually configured.

[0022] In a third aspect, an embodiment of the present application provides an electronic device, including:

[0023] Memory, for storing computer instructions;

[0024] A processor is connected to the memory, and is used to execute computer instructions in the memory, and implement the method as described in any one of the second aspects when executing the computer instructions.

[0025] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, including:

[0026] The computer-readable storage medium stores computer instructions, and when the computer instructions are executed on a computer, the computer is enabled to execute the method as described in any one of the second aspects.

[0027] In a fifth aspect, an embodiment of the present application provides a computer program product, including:

[0028] When the computer program product is called by a computer, the computer executes the method as described in any one of the second aspects.

[0029] For each of the above-mentioned aspects from the second to the fifth aspects and the technical effects that may be achieved by each of the aspects, please refer to the above-mentioned description of the technical effects that can be achieved by the first aspect or various possible schemes in the first aspect, and no further details will be given here. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for describing the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application.

[0031] Figure 1 A schematic diagram of a system for applying the data verification method provided in an embodiment of the present application;

[0032] Figure 2 One of the structural schematic diagrams of a data verification system provided in an embodiment of the present application;

[0033] Figure 3 A schematic diagram of the data extraction module workflow provided in the embodiment of the present application;

[0034] Figure 4 A schematic diagram of data push for multiple rule engines provided in an embodiment of the present application;

[0035] Figure 5 One of the structural schematic diagrams of a data verification system provided in an embodiment of the present application;

[0036] Figure 6 An exemplary flow chart of a data verification method provided in an embodiment of the present application;

[0037] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0038] In order to facilitate understanding of the technical solutions provided by the embodiments of the present application, the professional terms involved in the embodiments of the present application are introduced below.

[0039] (1) Online Analytical Processing (OLAP) is a data analysis and processing technology that supports complex and large-scale data analysis operations, usually batch operations. It explores and mines data value by aggregating data by different dimensions, drilling up and down by dimensions, and focuses on decision support.

[0040] (2) Data validation is a verification operation performed on data quality to ensure the integrity and validity of the data. In the OLAP processing process, in addition to technical verification, due to the business attributes of the business data itself, a series of business check rules can be defined to perform batch verification on the data used or result data in the OLAP analysis.

[0041] (3) Drools is an excellent Java-based rule engine that frees complex and ever-changing business rules from hard coding and stores them in files in the form of rule scripts, so that rule modifications can take effect without changing the code or restarting the business service.

[0042] (4) A cluster is a group of independent computers connected via a high-speed network. They form a group and are managed and provide services to the outside world in a single system mode. Cluster deployment can improve performance, increase scalability, and enhance reliability.

[0043] (5) Rule engine is a component embedded in the application that separates business rules or decisions from application code and uses predefined semantic modules to write business rules, accept data input, interpret business rules, and make business decisions based on business rules.

[0044] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the technical solution of the present application, rather than all of the embodiments. Based on the embodiments recorded in the application documents, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the technical solution of the present application.

[0045] The terms "first" and "second" in the embodiments of the present application are used to distinguish different objects, rather than to describe a specific order. In addition, the term "comprising" and any of their variations are intended to cover non-exclusive protection. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices. "Multiple" in the present application can mean at least two, for example, two, three or more, and the embodiments of the present application are not limited.

[0046] In addition, the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article, unless otherwise specified, generally indicates that the associated objects before and after are in an "or" relationship.

[0047] Traditional batch data verification methods use manual or automatic testing methods for verification. However, manual testing is costly and ineffective, and it is almost impossible to find a few anomalies in batch data. Automatic testing only extracts representative data from batch data as much as possible, and then compares them by executing program languages. These methods cannot fully verify batch data and cannot ensure the accuracy of data verification. In the era of big data, data is huge and complex, and how to conduct batch data verification comprehensively, flexibly and efficiently is an urgent problem to be solved.

[0048] In view of this, the present application provides a data verification system. In the system, a data verification request is received through a data extraction module, and the data storage module queries the data to be verified that meets the query conditions according to the query request generated by the data verification request, and then the data to be verified is sent to the rule engine through a push module to verify the data according to the pre-configured verification rules. It is used to realize comprehensive, flexible and efficient batch data verification.

[0049] See also Figure 1 A schematic diagram of a system for applying the data verification method provided in an embodiment of the present application. System 100 includes: a terminal device 101, a memory 102, and a processor 103. The terminal device 101 may be a device such as a tablet computer, a computer, or a smart phone. The terminal device 101 is used to receive a data verification request from a user. The memory 102 is used to store verification rules and data to be verified. The processor 103 is used to perform rule verification on the data to be verified according to the verification rules.

[0050] See also Figure 2One of the structural diagrams of a data verification system provided in an embodiment of the present application is shown. As shown in the figure, the system may include: a data extraction module 201, a data storage module 202, a push module 203 and a rule engine 204.

[0051] The data extraction module 201 receives the data verification request from the user. Then the data extraction module 201 generates a query request according to the query conditions of the data to be verified in the data verification request, and sends the query request to the data storage module 202. The data verification request includes the query conditions of the data to be verified. The query conditions are used to extract the data to be verified in the data storage module 202 that meets the query conditions.

[0052] When the data extraction module 201 generates a query request according to the query conditions of the data to be verified in the data verification request, the data extraction module 201 parses the query conditions of the data to be verified in the data verification request. If the query conditions of the data include the association relationship of the data to be verified, the data extraction module 201 generates an association relationship query request according to the association relationship. Then the data extraction module 201 sends both the association relationship query request and the query request to the data storage module 202. If the query conditions of the data do not include the association relationship of the data to be verified, the data extraction module 201 will only generate a query request, and then send the generated query request to the data storage module 202.

[0053] Among them, the association relationship of the data to be verified refers to the association relationship between two or more data to be verified stored in the form of data tables. For example, it is assumed that the data to be verified is stored in three data tables, namely data table 1, data table 2 and data table 3. If the data in the first row of data table 3 can be obtained through the data in the first row of data table 1 and the data in the second row of data table 2. Then in this case, it can be considered that there is an association relationship between the three data tables of data table 1, data table 2 and data table 3. Then the data extraction module 201 generates an association relationship query request based on the association relationship, and sends the association relationship query request and the query request to the data storage module 202.

[0054] When the data storage module 202 receives the query request sent by the data extraction module 201, the data storage module 202 will query the data to be verified stored in the data storage module 202 according to the query request sent by the data extraction module 201. Then the data to be verified that meets the query request is sent to the data extraction module 201. When the data storage module 202 receives the association query request and query request sent by the data extraction module 201, the data storage module 202 will query the data to be verified stored in the data storage module 202 according to the association query request and query request sent by the data extraction module 201. Then the data to be verified that meets the association query request and query request is sent to the data extraction module 201.

[0055] After receiving the data to be verified sent by the data storage module 202 , the data extraction module 201 will also send the received data to be verified to the push module 203 .

[0056] See also Figure 3 This is a schematic diagram of the data extraction module workflow provided in the embodiment of the present application. The specific process includes:

[0057] S301, the data extraction module receives a data verification request.

[0058] The data verification request includes query conditions.

[0059] S302: The data extraction module analyzes the query conditions of the data to be verified.

[0060] S303: The data extraction module determines whether the query condition includes the association relationship of the data to be verified.

[0061] If the data extraction module determines that the query condition in the data verification request includes the association relationship of the data to be verified after executing S302, then S304 is executed. If the data extraction module determines that the query condition in the data verification request does not include the association relationship of the data to be verified after executing S302, then S306 is executed.

[0062] S304: The data extraction module generates an association query request and a query request.

[0063] The data extraction module generates an association relationship query request according to the association relationship of the data to be verified parsed in S302, and generates a query request according to the query condition in the data verification request.

[0064] S305: The data extraction module sends both the query request and the association relationship query request to the data storage module.

[0065] S306: The data extraction module generates a query request.

[0066] If, after parsing the query condition in S302 , the query condition of the data verification request does not contain the association relationship of the data to be verified, the data extraction module only needs to generate a query request according to the query condition.

[0067] S307, the data extraction module sends the query request to the data storage module.

[0068] S308, the data extraction module receives the data to be verified that meets the query conditions and is sent by the data storage module.

[0069] After the data extraction module executes S307, the data storage module receives the query request sent by the data extraction module, and then queries the data to be verified stored in the data storage module, and sends the queried data to be verified that meets the query request to the data extraction module. At this time, the data extraction module receives the data to be verified that meets the query request sent by the data storage module.

[0070] S309, the data extraction module sends the data to be verified that meets the query conditions to the push module.

[0071] The data extraction module sends the data to be verified that meets the query request sent by the data storage module to the push module.

[0072] After the data extraction module 201 sends the data to be verified that meets the query conditions to the push module 203, the push module 203 receives the data to be verified and sends the data to be verified to the rule engine 204. The push module 203 mainly plays the role of distributing the data to be verified in the data verification system provided in the embodiment of the present application. Since the unit of the data volume during data verification is usually millions or hundreds of millions, if the batch of data to be verified is directly sent from the data storage module to the data engine, it will cause performance problems in the system, and in severe cases, it will cause the system to crash. Therefore, when the data verification system needs to perform data verification of batch data to be verified, the push module 203 is particularly important.

[0073] In a possible implementation, when there are at least two rule engines 204 , when the push module 203 sends the data to be verified to the rule engines 204 , the data to be verified may be sent to at least two rule engines 204 in a polling manner.

[0074] See also Figure 4, a schematic diagram of data push for multiple rule engines provided in an embodiment of the present application. As shown in the figure, the figure includes a push module 401, and the three rule engines are rule engine A, rule engine B and rule engine C. When the push module 401 needs to send the data to be verified to the rule engine for rule verification, the 3m+1th data to be verified will be sent to rule engine A, the 3m+2th data to be verified will be sent to rule engine B, and the 3m+3th data to be verified will be sent to rule engine C, where m is an integer greater than or equal to 0. For example, when m is 0, the 1st data to be verified will be sent to rule engine A, the 2nd data to be verified will be sent to rule engine B, and the 3rd data to be verified will be sent to rule engine C. And so on, until all the data to be verified are sent to the rule engine for verification.

[0075] Based on the above scheme, the push module uses polling to send the data to be verified to at least two rule engines, which can easily realize rule verification by multiple rule engines at the same time, realize stable operation of the system and improve the efficiency of data verification when the data volume is large.

[0076] The rule engine 204 verifies the data to be verified according to the pre-stored verification rules and outputs the verification result. Specifically, the rule engine 204 matches the conditions of the data to be verified sent to the rule engine 204 according to the pre-configured verification rules. If the data to be verified matches the conditions of the verification rules, the verification result indicating that the data to be verified has passed the verification is output accordingly. If the data to be verified does not match the conditions of the verification rules, the verification result indicating that the data to be verified has failed is output accordingly.

[0077] The pre-stored validation rules are visually configured. The validation rules can include a condition part and a behavior part, and the validation rules can be visually configured through one or more of the multiple rule components. After the business personnel visually configure the validation rules through the rule components, the configured validation rules can be converted into a language in a grammatical form that can be interpreted by the rule engine through a conversion program.

[0078] Among them, the rule component may include a data model definition component, a decision tree component, a decision table component, a rule set component and a rule flow component.

[0079] Data model definition component, since the data model can be divided into data sets and attribute variables, where data sets are used to indicate data identifiers and attribute variables are used to indicate data content. Therefore, the data model can be defined by defining data sets and attribute variables. In other words, the validation rules are configured through the data model definition component, that is, the validation rules are configured by defining the data model to which the validation rules apply. For example, a data model can be defined for each data table, and the table fields are attribute variables. You can also select multiple data tables, specify the association relationship between each data table, and then define data sets and attribute variables based on the results of the association relationship.

[0080] In a possible implementation, at least two of the multiple data sets in the pre-stored verification rules are associated with each other, and the associated relationship corresponds to a data set and an attribute variable. The data set corresponding to the associated relationship includes at least two data sets, and the attribute variable corresponding to the associated relationship is used to indicate that the at least two data sets are associated with each other.

[0081] The decision tree component is a rule setting component that configures a group of related rules together in a tree diagram. The "trunk" part of the tree diagram is the specific bifurcation value of each condition, and the "leaf" part, or the node part, is the result of each condition. Each branch of the decision tree from the "root" to the "leaf" is a complete verification rule. Business personnel can configure the verification rules by setting the conditions and results of the verification rules in the decision tree component.

[0082] The decision table component is a rule setting component that configures a group of rules related to each other in a multi-dimensional chart. It is suitable for describing situations where there are many processing judgment conditions, and the conditions are combined with each other and there are multiple decision plans. From the first "dimension" on the left of the decision table to the last "result" on the right, a complete verification rule and result can be formed. Business personnel can configure the verification rule by setting the "dimension" and "result" of the verification rule in the decision table component.

[0083] A rule set component is a collection of a series of scattered rule components, that is, the rule components in the rule set are not related to each other. It can include one or more of the data model definition component, decision tree component, and decision table component. In the rule set component configuration page, business personnel are guided to configure the conditions of the verification rules and the results of satisfying the conditions for each rule component, and configure the execution order of each verification rule among multiple verification rules.

[0084] The rule flow component can be regarded as an ordered combination of a data model definition component, a decision tree component, a decision table component, a rule set component and a rule flow component, and can include one or more of the above components. The rule flow component can also be embedded in other rule flow components as a rule component. The rule flow component can be used to configure the execution time of each rule component, the execution conditions of each rule component and the execution method of each rule component, so as to realize the process processing of the verification rules configured by multiple rule components. In the rule flow component, two adjacent rule components can use the output of the previous rule component as the input of the next rule component. Through the rule flow component, multiple process-based rule components can be used as a whole to realize complex rule strategies. It should be understood that the above rule component can be any one of the data model definition component, decision tree component, decision table component, rule set component and rule flow component.

[0085] Based on the above solution, the verification rules are configured visually, avoiding the previous method of verifying batch business data through hard coding. In addition, business personnel can update the verification rules conveniently and quickly according to changes in demand without the participation of technical personnel, reducing manpower requirements and improving the efficiency of updating verification rules.

[0086] After the verification rules are configured through the above rule components, they will be saved in the system. Until the next time the business personnel change the verification rules, the rule engine 204 will always perform rule verification on the verification data with the verification rules configured this time.

[0087] Since data verification is usually performed on batch data, and there is generally a correlation between the batch data, data verification can be performed by building a processing cluster, which can not only solve the performance problem of the system, but also improve the efficiency of data verification.

[0088] For example, see Figure 5 This is one of the structural diagrams of a data verification system provided in an embodiment of the present application. As shown in the figure, the data verification system 500 includes a data extraction cluster 501, a data storage cluster 502, a push cluster 503 and a rule engine cluster 504.

[0089] The data extraction cluster 501 may be composed of multiple data extraction modules. In the embodiment of the present application, the data extraction cluster 501 includes two data extraction modules as an example. When receiving a data verification request, the data extraction cluster 501 will use load balancing to send the data verification request to multiple data extraction modules in the data extraction cluster 501. Among them, load balancing refers to balancing the work tasks of the system. In other words, multiple work tasks of the system are distributed to multiple operating units for operation, thereby improving the data processing capability of the system, increasing the throughput of the system, and increasing the availability and flexibility of the system.

[0090] The workflow of each data extraction module in the data extraction cluster 501 can be found in Figure 2 The relevant description will not be repeated here.

[0091] The data storage cluster 502 is composed of multiple data storage modules. In the embodiment of the present application, the data storage cluster 502 includes two data storage modules as an example. Optionally, the data storage cluster 502 can adopt a clickhouse cluster, that is, the clickhouse database is used as a data storage module. Among them, clickhouse is a column-based storage massively parallel processing (Massively Parallel Processing, MPP) database, which uses vector computing to support multi-core parallel processing and multi-server distributed processing, so the query efficiency is very high.

[0092] The push cluster 503 is composed of multiple push modules. In the embodiment of the present application, the push cluster 503 includes two push modules as an example. Optionally, the push cluster 503 can adopt a rocketMQ cluster, that is, a rocketMQ message middleware is used as a push module. Among them, rocketMQ is a message middleware with a message queue model with high performance, high reliability, high real-time and distributed characteristics.

[0093] The rule engine cluster 504 is composed of multiple rule engines. In the embodiment of the present application, the rule engine cluster 504 includes two rule engines as an example. Optionally, the rule engine cluster 504 can adopt a Drools rule engine cluster, that is, the Drools rule engine is used as the rule engine of the data verification system in the embodiment of the present application. The Drools rule engine performs rule verification based on the Rete algorithm. The rule engine cluster 504 uses multi-threading technology to process data in parallel. At the same time, since the rule engine cluster 504 includes multiple rule engines, the data processing capacity of the data verification system can be improved.

[0094] The data extraction cluster 501 in the data verification system 500 receives the user's data verification request, and then sends the query request and the association query request to the data storage cluster 502. The data storage cluster 502 queries the data to be verified that meets the query conditions according to the query request and the association query request, and sends the data to be verified to the data extraction cluster 501. The data extraction cluster 501 then sends the data to be verified to the push cluster 503. The push cluster 503 sends the data to be verified to all the rule engines in the rule engine cluster 504 in a polling manner for data verification. Finally, the rule engine cluster 504 outputs the verification results of all the data to be verified.

[0095] Below, in order to more clearly understand the solution proposed in the embodiment of the present application, a data verification method provided by the present application will be introduced in combination with a specific embodiment.

[0096] The data verification method provided in the embodiment of the present application can be applied to OLAP. Figure 6 , is an exemplary flow chart of a data verification method provided in an embodiment of the present application, specifically comprising:

[0097] S601, the system receives a data verification request.

[0098] The query request includes query conditions.

[0099] S602: The system searches for data to be verified that meets the query conditions in the data verification request.

[0100] For the query method of the data to be verified that meets the query conditions, please refer to Figure 2 The relevant description in the illustrated system embodiment will not be repeated in this application.

[0101] S603, the system verifies the data to be verified according to pre-stored verification rules and outputs the verification result.

[0102] The verification result is used to indicate whether the data to be verified has passed the verification or failed the verification. The verification method and verification rules of the data to be verified can be found in Figure 2 The relevant description in the illustrated system embodiment will not be repeated in this application.

[0103] Based on the same concept of the above method, see Figure 7 , is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. As shown in the figure, the electronic device may include: a processor 701, a memory 702 and a bus interface 703.

[0104] The processor 701 is responsible for managing the bus architecture and general processing, and the memory 702 can store data used by the processor 701 when performing operations.

[0105] The bus architecture may include any number of interconnected buses and bridges, specifically linking together various circuits of one or more processors represented by processor 701 and memory represented by memory 702. The bus architecture may also link together various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and are therefore not further described herein. The bus interface provides an interface. The processor 701 is responsible for managing the bus architecture and general processing, and the memory 702 may store data used by the processor 701 when performing operations.

[0106] The process disclosed in the embodiment of the present application can be applied to the processor 701, or implemented by the processor 701. In the implementation process, each step of the process flow can be completed by the hardware integrated logic circuit or software instructions in the processor 701. The processor 701 can be a general-purpose processor, a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, and can implement or execute the methods, steps and logic block diagrams disclosed in the embodiment of the present application. The general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware processor to execute, or the hardware and software modules in the processor are combined and executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 702, and the processor 701 reads the information in the memory 702, and completes the steps of the method flow in the embodiment of the present application in combination with its hardware.

[0107] Specifically, the processor 701 is used to read the computer instructions in the memory 702 and execute the data verification method in the embodiment of the present application.

[0108] It should be noted here that the above-mentioned communication device provided in the embodiment of the present application can implement all the method steps implemented in the above-mentioned method embodiment, and can achieve the same technical effect. The parts and beneficial effects that are the same as the method embodiment in this embodiment will not be described in detail here.

[0109] The embodiment of the present application further provides a computer-readable storage medium on which computer instructions are stored. When the computer instructions are executed on a computer, the computer executes the steps of any of the above methods.

[0110] The embodiment of the present application also provides a computer program product, which, when called by a computer, enables the computer to execute the steps of any of the above methods.

[0111] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0112] Although the specific embodiments of the present application are described above, it should be understood by those skilled in the art that these are merely examples, and the scope of protection of the present application is limited by the attached claims. Those skilled in the art may make various changes or modifications to these embodiments without departing from the principles and essence of the present application, but these changes and modifications all fall within the scope of protection of the present application. Although the preferred embodiments of the present application have been described, once those skilled in the art are aware of the basic creative concepts, additional changes and modifications may be made to these embodiments. Therefore, the attached claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0113] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A data verification system, characterized in that: include: Data extraction module, data storage module, push module and rule engine; The data extraction module is used to receive a data verification request; the data verification request includes a query condition for the data to be verified; parse the query condition; when the query condition includes an association relationship of the data to be verified, generate an association relationship query request according to the association relationship, generate a query request according to the query condition, and send the association relationship query request and the query request to the data storage module; when the query condition does not include the association relationship of the data to be verified, generate a query request according to the query condition, and send the query request to the data storage module; The data storage module is used to query the data to be verified that meets the query condition according to the query request when receiving the query request; query the data to be verified that meets the query condition according to the query request when receiving the association query request and the query request; and send the data to be verified to the data extraction module; The data extraction module is also used to send the data to be verified to the push module; The push module is used to receive the data to be verified and send the data to be verified to the rule engine; The rule engine is used to verify the data to be verified according to pre-stored verification rules and output a verification result; the verification result is used to indicate that the verification of the data to be verified has passed, or the verification result is used to indicate that the verification of the data to be verified has failed; the pre-stored verification rules include a data set and an attribute variable, the data set is used to indicate a data identifier, and the attribute variable is used to indicate data content; wherein at least two data sets have an association relationship, and the association relationship corresponds to a data set and an attribute variable; the data set corresponding to the association relationship includes the at least two data sets; the attribute variable corresponding to the association relationship is used to indicate that the at least two data sets have an association relationship.

2. The system according to claim 1, characterized in that The pre-stored verification rules are visually configured.

3. The system according to claim 1, characterized in that The number of the rule engines is at least two; When the push module sends the data to be verified to the rule engine, it is specifically used to: The push module sends the data to be verified to the at least two rule engines in a polling manner.

4. The system according to claim 1, characterized in that The data storage module is a clickhouse cluster, and the push module is a rocketMQ cluster.

5. A data verification method, characterized in that: include: Receiving a data verification request; the data verification request includes a query condition for the data to be verified; Parsing the query condition; When the query condition includes the association relationship of the data to be verified, generating an association relationship query request according to the association relationship, and generating a query request according to the query condition; When the query condition does not include the association relationship of the data to be verified, generating a query request according to the query condition; According to the association query request and the query request, query the to-be-verified data that meets the query condition; Alternatively, according to the query request, query the to-be-verified data that meets the query condition; Verify the data to be verified according to the pre-stored verification rules and output the verification result; The verification result is used to indicate that the verification of the data to be verified has passed, or the verification result is used to indicate that the verification of the data to be verified has failed; the pre-stored verification rule includes a data set and an attribute variable, the data set is used to indicate a data identifier, and the attribute variable is used to indicate data content; wherein at least two data sets have an association relationship, and the association relationship corresponds to the data set and the attribute variable; The data set corresponding to the association relationship includes the at least two data sets; and the attribute variable corresponding to the association relationship is used to indicate that the at least two data sets have an association relationship.

6. The method according to claim 5, characterized in that The pre-stored verification rules are visually configured.

7. An electronic device, characterized in that: include: Memory, for storing computer instructions; A processor is connected to the memory, and is used to execute computer instructions in the memory, and implement the method as described in any one of claims 5-6 when executing the computer instructions.

8. A computer-readable storage medium, characterized in that: include: The computer-readable storage medium stores computer instructions, and when the computer instructions are executed on a computer, the computer is enabled to execute the method according to any one of claims 5 to 6.

9. A computer program product, characterized in that include: When the computer program product is called by a computer, the computer executes the method according to any one of claims 5 to 6.

Images