Data quality verification method and device for micro service

Abstract

The invention provides a data quality verification method and device for micro-service. The method comprises the steps of obtaining a data updating operation for micro-service associated data; based on the data updating operation, determining a to-be-updated table in the micro-service associated data and a to-be-updated field in the to-be-updated table; if the to-be-updated table and / or the to-be-updated field are / is the preset verification object, acquiring a data quality verification rule matched with the to-be-updated table and the to-be-updated field from metadata of the micro-service; determining whether a field value of a to-be-updated field in the data updating operation meets a data quality verification rule or not; if the field value meets the data quality verification rule, executing a data updating operation, and if the field value does not meet the data quality verification rule, interrupting the data updating operation. According to the technical scheme, the rule maintenance cost and workload are reduced, the perspectiveness of data quality detection is improved, and the overall data quality and data security of the micro service and the database are effectively protected.

Description

[Technical field]

[0001] The present application relates to the technical field of data quality detection, and in particular to a data quality verification method and device for microservices. [Background technology]

[0002] In related technologies, in order to ensure the quality of microservices, the quality of the data involved in the microservices can be checked in the database linked to the microservices. In general, the data inspection rules are stored in a separate table. When data quality inspection is required, the separately stored table is called to obtain the data inspection rules.

[0003] In this solution, since the object of inspection is the data in the database itself, the verification of data quality is delayed and it is impossible to grasp the timing of data quality risks. At the same time, since the data verification rules are stored separately in the table, the table needs to be maintained separately. Since the data quality requirements change with the actual business, the changes in the data verification rules will lead to multiple maintenance, which increases the maintenance cost. In addition, since this solution needs to directly verify the data in the database, facing a variety of databases, it is necessary to build and maintain tables for different database types separately, which is a large workload and has a high error rate.

[0004] Therefore, how to efficiently and conveniently complete data quality testing for microservices has become a technical problem that needs to be solved urgently. [Summary of the invention]

[0005] The embodiments of the present application provide a data quality verification method and device for microservices, aiming to solve the technical problem in the related art that it is difficult to meet the timeliness and convenience requirements for detecting the data quality of microservices in a database.

[0006] In a first aspect, an embodiment of the present application provides a data quality verification method for a microservice, comprising:

[0007] Get data update operations for microservice-related data;

[0008] Based on the data update operation, determine a table to be updated in the microservice-related data and a field to be updated in the table to be updated;

[0009] If the table to be updated and / or the field to be updated are preset verification objects, a data quality verification rule matching the table to be updated and the field to be updated is obtained in the metadata of the microservice;

[0010] Determining whether the field value of the field to be updated in the data update operation satisfies the data quality verification rule;

[0011] If the field value satisfies the data quality verification rule, the data update operation is performed, and if the field value does not satisfy the data quality verification rule, the data update operation is interrupted.

[0012] In one embodiment of the present application, optionally, before the data update operation for obtaining microservice-related data, the process further includes:

[0013] Setting a table creation statement in the metadata of the microservice, wherein the table creation statement is used to describe the table and / or field in the microservice-related data as the preset verification object;

[0014] Before obtaining the data quality verification rule matching the table to be updated and the field to be updated, the method further includes:

[0015] Determine whether the table creation statement corresponding to the microservice records the code information of the table to be updated and / or the field to be updated;

[0016] If the code information of the table to be updated and / or the field to be updated is recorded in the table creation statement, it is determined that the table to be updated and / or the field to be updated are the preset verification objects.

[0017] In one embodiment of the present application, optionally, before the data update operation for obtaining microservice-related data, the process further includes:

[0018] In response to the startup of the microservice, a DatabaseMetaData object is obtained through the Java language Spring-Jdbc framework, and the DatabaseMetaData object is loaded with the table creation statement in the metadata and the data quality verification rules corresponding to each of the preset verification objects.

[0019] In one embodiment of the present application, optionally, the data update operation for obtaining microservice-related data includes:

[0020] Define the mybatisUpdateInterceptor class to intercept data insertion and modification operations on microservice-related data;

[0021] The determining, based on the data update operation, a table to be updated in the microservice associated data and a field to be updated in the table to be updated includes:

[0022] Parsing the table to be updated, the field to be updated in the table to be updated, and the field value of the field to be updated in the SQL statement in the statement object corresponding to the data update operation;

[0023] The table to be updated and the field to be updated are retrieved from the microservice related data.

[0024] In one embodiment of the present application, optionally, before the data update operation for obtaining microservice-related data, the process further includes:

[0025] Based on the aviator engine grammatical rules and the rule data input by the user, a corresponding rule verification function is established for each of the preset verification objects;

[0026] The step of determining whether the field value of the field to be updated in the data update operation satisfies the data quality verification rule comprises:

[0027] Setting the field value of the field to be updated in the data update operation as the independent variable of the rule verification function corresponding to the field to be updated, solving the rule verification function corresponding to the field to be updated, and obtaining a verification value;

[0028] Determine whether the check value is within a preset reasonable range, wherein:

[0029] If the verification value is within the preset reasonable range, determining that the field value satisfies the data quality verification rule; and

[0030] If the verification value is not within the preset reasonable range, it is determined that the field value does not meet the data quality verification rule.

[0031] In one embodiment of the present application, optionally, the method further includes:

[0032] In response to the acquisition of the data update operation, inserting a first hook function in the process of the microservice, wherein the first hook function is used to obtain the matching data quality verification rule when the table to be updated and the field to be updated in the data update operation are the preset verification objects;

[0033] In response to the acquisition of the data quality verification rule, a second hook function is inserted into the process of the microservice, wherein the second hook function is used to call the aviator engine to check whether the field value of the field to be updated meets the data quality verification rule, and execute the data update operation if it meets the rule, and interrupt the data update operation if it does not meet the rule.

[0034] In one embodiment of the present application, optionally, the method further includes:

[0035] In response to the acquisition of the data update operation, the execution function corresponding to the data update operation in the process of the microservice is replaced by a third hook function, wherein the third hook function is used to obtain the matching data quality verification rule when the table to be updated and the field to be updated in the data update operation are the preset verification objects, and to call the aviator engine to check whether the field value of the field to be updated satisfies the data quality verification rule, and to execute the data update operation if it is satisfied and interrupt the data update operation if it is not satisfied.

[0036] In a second aspect, an embodiment of the present application provides a data quality verification device for microservices, comprising:

[0037] A data update operation acquisition unit, used to acquire data update operations for microservice-related data;

[0038] An update content determination unit, configured to determine, based on the data update operation, a table to be updated in the microservice associated data and a field to be updated in the table to be updated;

[0039] A verification rule matching unit, configured to obtain, in the metadata of the microservice, a data quality verification rule that matches the table to be updated and the field to be updated if the table to be updated and / or the field to be updated are preset verification objects;

[0040] A verification rule verification unit, used to determine whether the field value of the field to be updated in the data update operation satisfies the data quality verification rule;

[0041] An execution unit is used to execute the data update operation if the field value satisfies the data quality verification rule, and interrupt the data update operation if the field value does not satisfy the data quality verification rule.

[0042] In one embodiment of the present application, optionally, the device further includes:

[0043] a table creation statement setting unit, configured to set a table creation statement in the metadata of the microservice before the data update operation acquisition unit acquires the data update operation for the microservice associated data, wherein the table creation statement is used to describe the table and / or field in the microservice associated data as the preset verification object;

[0044] A table creation statement query unit is used to determine whether the code information of the table to be updated and / or the field to be updated is recorded in the table creation statement corresponding to the microservice before the verification rule matching unit obtains the data quality verification rule matching the table to be updated and the field to be updated, wherein if the code information of the table to be updated and / or the field to be updated is recorded in the table creation statement, it is determined that the table to be updated and / or the field to be updated are the preset verification objects.

[0045] In one embodiment of the present application, optionally, the device further includes:

[0046] The initial loading unit is used to obtain a DatabaseMetaData object through the Java language Spring-Jdbc framework in response to the startup of the microservice before the data update operation acquisition unit obtains the data update operation for the microservice-related data, and the DatabaseMetaData object is loaded with the table creation statement in the metadata and the data quality verification rules corresponding to each of the preset verification objects.

[0047] In one embodiment of the present application, optionally, the data update operation acquisition unit includes:

[0048] The interceptor interception unit is used to define the mybatisUpdateInterceptor class to intercept data insertion and data modification operations for microservice-related data;

[0049] The update content determination unit is used to: parse the table to be updated, the field to be updated in the table to be updated, and the field value of the field to be updated in the SQL statement in the statement object corresponding to the data update operation, and retrieve the table to be updated and the field to be updated in the microservice associated data.

[0050] In one embodiment of the present application, optionally, the device further includes:

[0051] A rule verification function setting unit, used to establish a corresponding rule verification function for each of the preset verification objects based on the aviator engine grammar rules and the rule data input by the user before the data update operation acquisition unit acquires the data update operation for the microservice-related data;

[0052] The verification rule verification unit is used to: set the field value of the field to be updated in the data update operation as the independent variable of the rule verification function corresponding to the field to be updated, solve the rule verification function corresponding to the field to be updated, and obtain the verification value; determine whether the verification value is within a preset reasonable range, wherein if the verification value is within the preset reasonable range, it is determined that the field value satisfies the data quality verification rule; and if the verification value is not within the preset reasonable range, it is determined that the field value does not satisfy the data quality verification rule.

[0053] In one embodiment of the present application, optionally, the device further includes:

[0054] A first hook unit, configured to insert a first hook function into the process of the microservice in response to the acquisition of the data update operation, wherein the first hook function is configured to acquire the matched data quality verification rule when the table to be updated and the field to be updated in the data update operation are the preset verification objects;

[0055] The second hook unit is used to insert a second hook function in the process of the microservice in response to the acquisition of the data quality verification rule, wherein the second hook function is used to call the aviator engine to check whether the field value of the field to be updated meets the data quality verification rule, and execute the data update operation if it meets the rule, and interrupt the data update operation if it does not meet the rule.

[0056] In one embodiment of the present application, optionally, the device further includes:

[0057] A third hook unit is used to replace the execution function corresponding to the data update operation in the process of the microservice through a third hook function in response to the acquisition of the data update operation, wherein the third hook function is used to obtain the matching data quality verification rule when the table to be updated and the field to be updated in the data update operation are the preset verification objects, and to call the aviator engine to check whether the field value of the field to be updated satisfies the data quality verification rule, and to execute the data update operation if it is satisfied and interrupt the data update operation if it is not satisfied.

[0058] In a third aspect, an embodiment of the present application provides a computer device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are configured to execute the method described in the first aspect above.

[0059] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute the method described in the first aspect above.

[0060] The above technical scheme aims at the technical problem that it is difficult to meet the timeliness and convenience requirements for detecting the data quality of microservices in the database in the related technology. In the process of data quality detection of microservices, it is only necessary to call the corresponding data quality verification rules in the metadata to perform quality detection on the changed content of the data update operation after the microservice receives the data update operation and before executing the data update operation, so as to realize the real-time detection of data quality in advance. Thus, data quality detection is directly realized by loading the rules in the metadata, and there is no need to build tables and maintain them for different database types respectively, so that different types of databases can be compatible, which reduces the cost and workload of rule maintenance and reduces the error rate. At the same time, the technical scheme determines the timing of data quality detection before the data is changed. In other words, before the data is updated, the quantity quality detection of the update data that may be generated is carried out. In this way, when the changed content of the data update operation does not meet the data quality verification rules, that is, when the data quality is not up to standard, the data update operation can be directly interrupted, so as to avoid the generation of data with poor quality from the root. Thus, the forward-looking nature of data quality detection is improved, the generation of low-quality data caused by the lag of data quality detection is avoided, and the overall data quality and data security of microservices and databases are effectively protected.

Brief Description of the Drawings

[0061] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0062] Figure 1 A flow chart of a data quality verification method for microservices according to an embodiment of the present application is shown;

[0063] Figure 2 A code schematic diagram of loading metadata through the DatabaseMetaData interface according to an embodiment of the present application is shown;

[0064] Figure 3 A code schematic diagram of setting a rule verification function with the aviator engine according to an embodiment of the present application is shown;

[0065] Figure 4 A schematic diagram of a code for parsing field values ​​according to an embodiment of the present application is shown;

[0066] Figure 5 A schematic diagram of a code for verifying whether a field value matches a data quality verification rule according to an embodiment of the present application is shown;

[0067] Figure 6 A flow chart of a data quality verification method for microservices according to another embodiment of the present application is shown;

[0068] Figure 7 A schematic diagram of the architecture of a data quality verification system according to an embodiment of the present application is shown;

[0069] Figure 8 A block diagram of a computer device according to an embodiment of the present application is shown;

[0070] Fig. 9 A block diagram of a computer device according to another embodiment of the present application is shown. [Specific implementation method]

[0071] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0072] Figure 1 A flow chart of a data quality verification method for microservices according to an embodiment of the present application is shown.

[0073] like Figure 1 As shown, the process of the data quality verification method for microservices according to an embodiment of the present application includes:

[0074] Step 102: Obtain data update operations for microservice-related data.

[0075] Microservices are a way to develop a single application using a set of small services. Each small service runs in a separate process, has a low amount of code, and uses lightweight mechanisms to communicate. Each service can be implemented using different programming methods and uses different storage technologies. On this basis, microservices can maintain efficient operation and flexible development mode while being compatible with different computer languages ​​and database types.

[0076] Microservice-related data refers to any data of different storage types involved in the small services within the microservice during its operation. The data update operation on microservice-related data is data of different storage types between multiple small services.

[0077] In a possible design, the data update operation includes a data insertion operation and a data modification operation, and step 102 includes: defining the mybatisUpdateInterceptor class to intercept the data insertion operation and the data modification operation for the microservice associated data. Among them, the mybatisUpdateInterceptor class implements the Interceptor interface of mybatis, which is an interceptor of sql statements, and can intercept sql statements in data update operations, and sql statements in data update operations often carry the changes of data update operations. Thus, the changes in the data update operation can be intercepted, so as to detect the data quality of the upcoming changes in the subsequent steps.

[0078] Step 104: Based on the data update operation, determine the table to be updated in the microservice-related data and the field to be updated in the table to be updated.

[0079] The data update operation is intended to change the microservice-related data, which indicates the location of the data to be updated, that is, the table where the data to be updated is located and the fields in the table. Relatively speaking, the field position of the table in the microservice-related data corresponds to the data to be updated that the data update operation needs to change.

[0080] In a possible design, before step 102, initial settings may be performed, and the initial settings include setting a table creation statement in the metadata of the microservice, and the table creation statement is used to describe the table and / or field in the microservice-related data that is the preset verification object. The metadata of the microservice can reflect the table and / or field in the microservice-related data that can be used as the preset verification object, or in other words, the metadata of the microservice records the tables and / or fields that may be involved in all data update operations that the microservice may face.

[0081] In this way, the process of determining the table to be updated in the microservice-related data and the field to be updated in the table to be updated is actually to retrieve the table to be updated and / or the field to be updated mentioned in the data update operation in the table creation statement of the metadata of the microservice, that is, to determine whether the code information of the table to be updated and / or the field to be updated is recorded in the table creation statement corresponding to the microservice.

[0082] If the table creation statement records the code information of the table to be updated and / or the field to be updated mentioned in the data update operation, it is determined that the table to be updated and / or the field to be updated are indeed the objects that need to be modified in the data update operation, that is, the preset verification objects.

[0083] In one possible design, step 102 includes: parsing the table to be updated, the field to be updated in the table to be updated, and the field value of the field to be updated in the SQL statement in the statement object corresponding to the data update operation; and retrieving the table to be updated and the field to be updated in the microservice-related data.

[0084] The statement object can be understood as a carrier for sending SQL statements to a data block. In combination with the above, it can be seen that by intercepting the SQL statements in the data update operation, the changes in the data update operation can be obtained, so as to detect the data quality of the upcoming changes in the subsequent steps. The SQL statements in the data update operation are often located in the statement object corresponding to the data update operation. Therefore, the table to be updated, the field to be updated in the table to be updated, and the field value of the field to be updated can be parsed in the SQL statement in the statement object, where the field value is the change content.

[0085] Intercept the SQL statements in the data update operation, and the SQL statements in the data update operation often carry the changed content of the data update operation. Therefore, the changed content in the data update operation can be intercepted.

[0086] Step 106: If the table to be updated and / or the field to be updated are preset verification objects, a data quality verification rule matching the table to be updated and the field to be updated is obtained in the metadata of the microservice.

[0087] After determining that the table to be updated and / or the field to be updated are preset verification objects, data quality verification rules corresponding to the table to be updated and the field to be updated may be further obtained.

[0088] For example, the following is a table creation statement that contains the corresponding fields for data verification. The content between [ ] is the verification rule, and the x variable represents the current field value.

[0089] CREATE TABLE employee_info(

[0090] employee_idINTAUTO_INCREMENTPRIMARYKEYCOMMENT'Employee ID, primary key',

[0091] first_nameVARCHAR(50)NOTNULLCOMMENT'first name,[isChinese(x)]', last_nameVARCHAR(50)NOTNULLCOMMENT'last name,[isChinese(x)]',

[0092] genderNOTNULLCOMMENT'gender, [inArray(x, ['male', 'female', 'other']])]',

[0093] birthdateDATECOMMENT'Date of birth, [x>='2020-01-01']',

[0094] emailVARCHAR(100)UNIQUENOTNULLCOMMENT'Email, unique, [isValidEmail(x)]',

[0095] phone_numberVARCHAR(20)COMMENT'Phone number [isValidPhone(x)]',

[0096] emergency_contact VARCHAR(100) COMMENT 'Emergency contact number',

[0097] addressVARCHAR(255)COMMENT'Home address',

[0098] cityVARCHAR(100)COMMENT'city',

[0099] state_province VARCHAR (100) COMMENT 'State / Province',

[0100] country VARCHAR (100) COMMENT 'country',

[0101] zip_code VARCHAR (20) COMMENT 'Postal code',

[0102] department_idINTCOMMENT'Department ID, foreign key points to the departments table',

[0103] position VARCHAR (100) COMMENT 'Position',

[0104] salaryDECIMAL(10,2)COMMENT'Salary, keep two decimal places',

[0105] hire_dateDATECOMMENT'Job Date',

[0106] termination_dateDATECOMMENT'termination date, NULL means in service',

[0107] statusNOTNULLDEFAULT'active'COMMENT'Employee status: employed, resigned, on vacation, [inArray(x, ['employed', 'resigned', 'on vacation']])]',

[0108] remarksTEXTCOMMENT'Remarks information'

[0109] )ENGINE=InnoDBDEFAULTCHARSET=utf8mb4COMMENT='Employee information table'.

[0110] The above DDL statements define the data quality check for the following fields:

[0111] (1) first_name must be in Chinese;

[0112] (2) last_name must be in Chinese;

[0113] (3) Gender must be one of male / female / other;

[0114] (4) birthdate must be greater than or equal to 2020-01-01;

[0115] (5) The email conforms to the mailbox format;

[0116] (6)phone_number conforms to the telephone number format;

[0117] (7) Status must be one of: Employed, Resigned, or On Leave.

[0118] In a possible design, before step 102, initial settings may be performed, and the initial settings include: in response to the startup of the microservice, obtaining a DatabaseMetaData object through the Java language Spring-Jdbc framework, and the DatabaseMetaData object is loaded with the table creation statements in the metadata and the data quality verification rules corresponding to each of the preset verification objects. Figure 2 The code schematic diagram of loading metadata through the DatabaseMetaData interface in the above example is shown. Figure 2 This is only a code example, and this application can use any code that meets the actual quality verification requirements to complete this function. Among them, the DatabaseMetaData interface is a special interface for obtaining metadata.

[0119] Specifically, once the microservice is started, the table creation statements and data quality verification rules, which are required for real-time data quality detection, are loaded into the system memory for use in data quality detection before data update operations during the use of the microservice.

[0120] It can be seen that the data quality verification rules corresponding to the data as the preset verification object in the microservice are located in the metadata of the microservice, rather than being stored in a separate table. At the same time, as long as the microservice is started, the data quality verification rules can be loaded into the memory for real-time data quality detection. It can be seen that the maintenance of data quality verification rules is integrated with the maintenance of the microservice itself, and no specific additional maintenance is required, which can reduce the cost of rule maintenance and achieve low-energy and high-efficiency data quality detection.

[0121] Step 108: Determine whether the field value of the field to be updated in the data update operation satisfies the data quality verification rule.

[0122] That is, it is determined whether the change content required by the data update operation meets the data quality verification rules corresponding to the location to be changed. The data update operation can be performed only when the field value meets the data quality verification rules, otherwise, the data update operation is interrupted.

[0123] Specifically, before step 102, initial settings can be performed, which include: abstracting and establishing corresponding rule verification functions for each preset verification object based on the aviator engine syntax rules and the rule data input by the user. Custom functions such as isChinese, inArray, isValidEmail, and isValidPhone can be defined through the aviator engine. The code for setting the isChinese function with the aviator engine is as follows: Figure 3 Of course, Figure 3 This is just a code example. This application can use any code that meets the actual quality verification requirements to complete the rule verification function setting. Among them, the aviator engine is a lightweight Java expression evaluation engine used for dynamic evaluation of various expressions. On this basis, the rule data can be input as the constant of the expression, and then the rule verification function for calculating the verification value can be established based on the constant and the aviator engine.

[0124] Of course, for each table and each field in the microservice that is a preset verification object, the aviator engine is used to set the corresponding rule verification function for it.

[0125] On this basis, step 108 includes: setting the field value of the field to be updated in the data update operation as the independent variable of the rule verification function corresponding to the field to be updated, solving the rule verification function corresponding to the field to be updated to obtain a verification value; determining whether the verification value is within a preset reasonable range, wherein if the verification value is within the preset reasonable range, determining that the field value meets the data quality verification rule; and if the verification value is not within the preset reasonable range, determining that the field value does not meet the data quality verification rule. Figure 4 The following shows the underlying code used to parse field values ​​in the microservice in the previous example. Figure 4 This is only a code example. This application can use any code that meets the actual quality verification requirements to complete this function.

[0126] Step 110: If the field value satisfies the data quality verification rule, the data update operation is performed; and if the field value does not satisfy the data quality verification rule, the data update operation is interrupted.

[0127] in, Figure 5 The following shows the underlying code in the microservice in the above example that verifies whether the field value matches the data quality verification rules. According to the data quality verification rules and field values ​​of the fields, the assembled parameters are passed to the aviator engine, which performs calculations. If the aviator engine returns True, it is determined that the field value meets the data quality verification rules and the data update operation is performed normally. If the aviator engine returns False, it is determined that the field value does not meet the data quality verification rules and the data update operation is interrupted. Of course, Figure 5 This is only a code example. This application can use any code that meets the actual quality verification requirements to complete this function.

[0128] In the process of data quality detection of microservices, the above technical solution only needs to call the corresponding data quality verification rules in the metadata to perform quality detection on the changed content of the data update operation after the microservice receives the data update operation and before executing the data update operation, so as to realize the real-time detection of data quality in advance. Therefore, data quality detection can be directly realized by loading the rules in the metadata, without the need to build and maintain tables for different database types respectively, and it can be compatible with different types of databases, reducing the cost and workload of rule maintenance and reducing the error rate.

[0129] At the same time, this technical solution determines the timing of data quality detection before the data is changed. In other words, before the data is updated, the quantity and quality of the updated data that may be generated are detected. In this way, when the changes in the data update operation do not meet the data quality verification rules, that is, when the data quality does not meet the standards, the data update operation can be directly interrupted to avoid the generation of data with poor quality from the root. As a result, the forward-looking nature of data quality detection is improved, the generation of low-quality data caused by delayed data quality detection is avoided, and the overall data quality and data security of microservices and databases are effectively protected.

[0130] Figure 6 A flow chart of a data quality verification method for microservices according to another embodiment of the present application is shown.

[0131] like Figure 6 As shown, the process of a data quality verification method for microservices according to another embodiment of the present application includes:

[0132] Step 602: Obtain data update operations for microservice-related data.

[0133] Step 604: In response to the acquisition of the data update operation, insert a first hook function into the process of the microservice, wherein the first hook function is used to obtain the matching data quality verification rule when the table to be updated and the field to be updated in the data update operation are the preset verification objects.

[0134] Step 606: Call the first hook function to obtain data quality verification rules that match the table to be updated and the field to be updated in the metadata of the microservice.

[0135] The hook function uses hook technology and can intercept corresponding information through monitoring, which has similar functions to the interceptor. In this technical solution, the first hook function can be set to monitor and intercept the information of data update operation, and obtain the corresponding data quality verification rules for the table to be updated and the field to be updated when they are the preset verification objects.

[0136] Therefore, the data update operation can be intercepted through hook technology. Before executing the data update operation, the data quality verification rules corresponding to the update content that may be generated are called to complete the data quality prediction of the update content that may be generated before executing the data update operation.

[0137] Step 608, in response to the acquisition of the data quality verification rule, insert a second hook function into the process of the microservice, wherein the second hook function is used to call the aviator engine to check whether the field value of the field to be updated meets the data quality verification rule, and execute the data update operation if it meets the rule, and interrupt the data update operation if it does not meet the rule.

[0138] Step 610, calling the second hook function to determine whether the field value of the field to be updated in the data update operation satisfies the data quality verification rules, wherein if the field value satisfies the data quality verification rules, the data update operation is executed, and if the field value does not satisfy the data quality verification rules, the data update operation is interrupted.

[0139] Similarly, a second hook function can be set to monitor and intercept the action of obtaining data quality verification rules in the microservice, wherein the second hook function also has the function of calling the aviator engine to judge whether the update content that may be generated by the data update operation meets the corresponding data quality verification rules, so as to further execute the data update operation when the data quality verification rules are met and interrupt the data update operation when the data quality verification rules are not met.

[0140] Among them, the first hook function and the second hook function form a hook chain and are called successively.

[0141] In one possible design, a hook chain formed by the first hook function and the second hook function replaces the function in the microservice that originally directly executes the data update operation. Once a data update operation is received, the function that originally directly executes the data update operation is no longer directly called. Instead, the first hook function and the second hook function are called in sequence to realize the data quality prediction of the update content that may be brought about by the data update operation.

[0142] In another possible design, the hook chain formed by the first hook function and the second hook function is hooked to the function in the microservice that originally directly executes the data update operation. The first hook function and the second hook function are used to predict the data quality of the update content that may be brought about by the data update operation. Only when the prediction result shows that the quality of the update content is qualified, the function that originally executes the data update operation is allowed to execute.

[0143] In another possible design, in response to the acquisition of the data update operation, the execution function corresponding to the data update operation in the process of the microservice is replaced by a third hook function, wherein the third hook function is used to obtain the matching data quality verification rules when the table to be updated and the field to be updated in the data update operation are the preset verification objects, and is used to call the aviator engine to check whether the field value of the field to be updated satisfies the data quality verification rules, and is used to execute the data update operation if it is satisfied and interrupt the data update operation if it is not satisfied.

[0144] That is to say, the first hook function and the second hook function are directly integrated into the third hook function, and the third hook function is used alone to predict the data quality of the update content that may be brought about by the data update operation. This can reduce the number of hook calls and the system operating capacity consumption caused by the hook calls.

[0145] The above technical solution monitors and intercepts the data update operation of the microservice through the hook technology, so as to predict the data quality of the update content that may be brought by the data update operation before executing the data update operation. Therefore, the timing of data quality detection is determined before the data is changed. In other words, before the data is updated, the quantity and quality of the update data that may be generated are detected. In this way, when the changed content of the data update operation does not meet the data quality verification rules, that is, when the data quality does not meet the standards, the data update operation can be directly interrupted to avoid the generation of data with poor quality from the root. Therefore, the forward-looking nature of data quality detection is improved, the generation of low-quality data caused by delayed data quality detection is avoided, and the overall data quality and data security of microservices and databases are effectively protected.

[0146] Based on any of the above embodiments, Figure 7 A schematic diagram of the architecture of a data quality verification system according to an embodiment of the present application is shown.

[0147] like Figure 7 As shown, during the startup phase of the microservice, the microservice application connects to the database and loads MetaData information into the DatabaseMetaData object in the database through the Java language Spring-Jdbc framework.

[0148] During the operation phase of the microservice, first, the microservice application obtains a data update request or a data insertion request from the client, and then the data update request or the data insertion request is intercepted by the mybatisUpdateInterceptor class defined in the microservice application.

[0149] At this time, the microservice application obtains the data quality verification rules in the DatabaseMetaData object according to the MetaData information, and calls the aviator engine to perform verification based on the data quality verification rules. Finally, if the data update request or data insertion request passes the verification, the microservice application can write the data change content corresponding to the data update request or data insertion request into the database.

[0150] The present application provides a data quality verification device for microservices, including:

[0151] A data update operation acquisition unit, used to acquire data update operations for microservice-related data;

[0152] An update content determination unit, configured to determine, based on the data update operation, a table to be updated in the microservice associated data and a field to be updated in the table to be updated;

[0153] A verification rule matching unit, configured to obtain, in the metadata of the microservice, a data quality verification rule that matches the table to be updated and the field to be updated if the table to be updated and / or the field to be updated are preset verification objects;

[0154] A verification rule verification unit, used to determine whether the field value of the field to be updated in the data update operation satisfies the data quality verification rule;

[0155] An execution unit is used to execute the data update operation if the field value satisfies the data quality verification rule, and interrupt the data update operation if the field value does not satisfy the data quality verification rule.

[0156] In one embodiment of the present application, optionally, the device further includes:

[0157] a table creation statement setting unit, configured to set a table creation statement in the metadata of the microservice before the data update operation acquisition unit acquires the data update operation for the microservice associated data, wherein the table creation statement is used to describe the table and / or field in the microservice associated data as the preset verification object;

[0158] A table creation statement query unit is used to determine whether the code information of the table to be updated and / or the field to be updated is recorded in the table creation statement corresponding to the microservice before the verification rule matching unit obtains the data quality verification rule matching the table to be updated and the field to be updated, wherein if the code information of the table to be updated and / or the field to be updated is recorded in the table creation statement, it is determined that the table to be updated and / or the field to be updated are the preset verification objects.

[0159] In one embodiment of the present application, optionally, the device further includes:

[0160] The initial loading unit is used to obtain a DatabaseMetaData object through the Java language Spring-Jdbc framework in response to the startup of the microservice before the data update operation acquisition unit obtains the data update operation for the microservice-related data, and the DatabaseMetaData object is loaded with the table creation statement in the metadata and the data quality verification rules corresponding to each of the preset verification objects.

[0161] In one embodiment of the present application, optionally, the data update operation acquisition unit includes:

[0162] The interceptor interception unit is used to define the mybatisUpdateInterceptor class to intercept data insertion and data modification operations for microservice-related data;

[0163] The update content determination unit is used to: parse the table to be updated, the field to be updated in the table to be updated, and the field value of the field to be updated in the SQL statement in the statement object corresponding to the data update operation, and retrieve the table to be updated and the field to be updated in the microservice associated data.

[0164] In one embodiment of the present application, optionally, the device further includes:

[0165] A rule verification function setting unit, used to establish a corresponding rule verification function for each of the preset verification objects based on the aviator engine grammar rules and the rule data input by the user before the data update operation acquisition unit acquires the data update operation for the microservice-related data;

[0166] The verification rule verification unit is used to: set the field value of the field to be updated in the data update operation as the independent variable of the rule verification function corresponding to the field to be updated, solve the rule verification function corresponding to the field to be updated, and obtain the verification value; determine whether the verification value is within a preset reasonable range, wherein if the verification value is within the preset reasonable range, it is determined that the field value satisfies the data quality verification rule; and if the verification value is not within the preset reasonable range, it is determined that the field value does not satisfy the data quality verification rule.

[0167] In one embodiment of the present application, optionally, the device further includes:

[0168] A first hook unit, configured to insert a first hook function into the process of the microservice in response to the acquisition of the data update operation, wherein the first hook function is configured to acquire the matched data quality verification rule when the table to be updated and the field to be updated in the data update operation are the preset verification objects;

[0169] The second hook unit is used to insert a second hook function in the process of the microservice in response to the acquisition of the data quality verification rule, wherein the second hook function is used to call the aviator engine to check whether the field value of the field to be updated meets the data quality verification rule, and execute the data update operation if it meets the rule, and interrupt the data update operation if it does not meet the rule.

[0170] In one embodiment of the present application, optionally, the device further includes:

[0171] A third hook unit is used to replace the execution function corresponding to the data update operation in the process of the microservice through a third hook function in response to the acquisition of the data update operation, wherein the third hook function is used to obtain the matching data quality verification rule when the table to be updated and the field to be updated in the data update operation are the preset verification objects, and to call the aviator engine to check whether the field value of the field to be updated satisfies the data quality verification rule, and to execute the data update operation if it is satisfied and interrupt the data update operation if it is not satisfied.

[0172] The device uses any of the solutions described in the above embodiments, and therefore has all the above technical effects, which will not be repeated here.

[0173] In addition, in one embodiment, the present application provides a computer device, which may be a server, and its internal structure diagram may be as shown in FIG. Figure 7 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client via a network connection. When the computer program is executed by the processor, the method described in any of the above embodiments can be implemented.

[0174] In one embodiment, the present application further provides a computer device, which may be a client, and its internal structure diagram may be as follows: Figure 8 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, the method described in any of the above embodiments can be implemented.

[0175] Any of the above-mentioned computer devices in the embodiments of the present application may exist in various forms, including but not limited to:

[0176] (1) Mobile communication devices: These devices are characterized by their mobile communication functions and their main purpose is to provide voice and data communications. These terminals include: smart phones (such as iPhone), multimedia phones, functional phones, and low-end phones.

[0177] (2) Ultra-mobile personal computer devices: These devices fall into the category of personal computers, have computing and processing capabilities, and generally also have mobile Internet access features. These terminals include: PDA, MID and UMPC devices, such as iPad.

[0178] (3) Portable entertainment devices: These devices can display and play multimedia content. They include audio and video players (such as iPods), handheld game consoles, e-books, as well as smart toys, wearable devices, and portable car navigation devices.

[0179] (4) Server: A device that provides computing services. The server consists of a processor, hard disk, memory, system bus, etc. The server is similar to a general computer architecture, but because it needs to provide highly reliable services, it has higher requirements in terms of processing power, stability, reliability, security, scalability, and manageability.

[0180] (5) Other electronic devices with data interaction functions.

[0181] In addition, an embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to perform the following steps:

[0182] Get data update operations for microservice-related data;

[0183] Based on the data update operation, determine a table to be updated in the microservice-related data and a field to be updated in the table to be updated;

[0184] If the table to be updated and / or the field to be updated are preset verification objects, a data quality verification rule matching the table to be updated and the field to be updated is obtained in the metadata of the microservice;

[0185] Determining whether the field value of the field to be updated in the data update operation satisfies the data quality verification rule;

[0186] If the field value satisfies the data quality verification rule, the data update operation is performed, and if the field value does not satisfy the data quality verification rule, the data update operation is interrupted.

[0187] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant description in the aforementioned method embodiment. In order to avoid repetition, they will not be described one by one here.

[0188] The technical solution of the present application is described in detail above in conjunction with the accompanying drawings. Through the technical solution of the present application, in the process of data quality detection of microservices, it is only necessary to call the corresponding data quality verification rules in the metadata to perform quality detection on the changed content of the data update operation after the microservice receives the data update operation and before executing the data update operation, so as to realize the real-time detection of data quality in advance. Thus, data quality detection is directly realized by loading the rules in the metadata, and there is no need to build tables and maintain them for different database types respectively, so that different types of databases can be compatible, which reduces the cost and workload of rule maintenance and reduces the error rate. At the same time, the technical solution determines the timing of data quality detection before the data is changed. In other words, when the data has not been updated, the quantity quality detection of the update data that may be generated is performed. In this way, when the changed content of the data update operation does not meet the data quality verification rules, that is, when the data quality is not up to standard, the data update operation can be directly interrupted, and the generation of data with poor quality can be avoided from the root. Thus, the forward-looking nature of data quality detection is improved, the generation of low-quality data caused by the lag of data quality detection is avoided, and the overall data quality and data security of microservices and databases are effectively protected.

[0189] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0190] It should be understood that although the terms first, second, etc. may be used to describe hook functions in the embodiments of the present application, these hook functions should not be limited to these terms. These terms are only used to distinguish hook functions from each other. For example, without departing from the scope of the embodiments of the present application, the first hook function may also be referred to as the second hook function, and similarly, the second hook function may also be referred to as the first hook function.

[0191] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0192] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the present application. The singular forms "a", "said" and "the" used in the embodiments of the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0193] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0194] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0195] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0196] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A data quality verification method for microservices, characterized in that: include: Get data update operations for microservice-related data; Based on the data update operation, determine a table to be updated in the microservice-related data and a field to be updated in the table to be updated; If the table to be updated and / or the field to be updated are preset verification objects, a data quality verification rule matching the table to be updated and the field to be updated is obtained in the metadata of the microservice; Determining whether the field value of the field to be updated in the data update operation satisfies the data quality verification rule; If the field value satisfies the data quality verification rule, the data update operation is performed, and if the field value does not satisfy the data quality verification rule, the data update operation is interrupted.

2. The method according to claim 1, characterized in that Before the data update operation for obtaining microservice-related data, the following step is also included: Setting a table creation statement in the metadata of the microservice, wherein the table creation statement is used to describe the table and / or field in the microservice-related data as the preset verification object; Before obtaining the data quality verification rule matching the table to be updated and the field to be updated, the method further includes: Determine whether the table creation statement corresponding to the microservice records the code information of the table to be updated and / or the field to be updated; If the code information of the table to be updated and / or the field to be updated is recorded in the table creation statement, it is determined that the table to be updated and / or the field to be updated are the preset verification objects.

3. The method according to claim 2, characterized in that Before the data update operation for obtaining microservice-related data, the following step is also included: In response to the startup of the microservice, a DatabaseMetaData object is obtained through the Java language Spring-Jdbc framework, and the DatabaseMetaData object is loaded with the table creation statement in the metadata and the data quality verification rules corresponding to each of the preset verification objects.

4. The method according to claim 1, characterized in that: The data update operation for obtaining microservice-related data includes: Define the mybatisUpdateInterceptor class to intercept data insertion and modification operations on microservice-related data; The determining, based on the data update operation, a table to be updated in the microservice associated data and a field to be updated in the table to be updated includes: Parsing the table to be updated, the field to be updated in the table to be updated, and the field value of the field to be updated in the SQL statement in the statement object corresponding to the data update operation; The table to be updated and the field to be updated are retrieved from the microservice related data.

5. The method according to claim 4, characterized in that Before the data update operation for obtaining microservice-related data, the following step is also included: Based on the aviator engine grammatical rules and the rule data input by the user, a corresponding rule verification function is established for each of the preset verification objects; The step of determining whether the field value of the field to be updated in the data update operation satisfies the data quality verification rule comprises: Setting the field value of the field to be updated in the data update operation as the independent variable of the rule verification function corresponding to the field to be updated, solving the rule verification function corresponding to the field to be updated, and obtaining a verification value; Determine whether the check value is within a preset reasonable range, wherein: If the verification value is within the preset reasonable range, determining that the field value satisfies the data quality verification rule; and If the verification value is not within the preset reasonable range, it is determined that the field value does not meet the data quality verification rule.

6. The method according to claim 1 or 5, characterized in that: Also includes: In response to the acquisition of the data update operation, inserting a first hook function in the process of the microservice, wherein the first hook function is used to obtain the matching data quality verification rule when the table to be updated and the field to be updated in the data update operation are the preset verification objects; In response to the acquisition of the data quality verification rule, a second hook function is inserted into the process of the microservice, wherein the second hook function is used to call the aviator engine to check whether the field value of the field to be updated meets the data quality verification rule, and execute the data update operation if it meets the rule, and interrupt the data update operation if it does not meet the rule.

7. The method according to claim 1 or 5, characterized in that: Also includes: In response to the acquisition of the data update operation, the execution function corresponding to the data update operation in the process of the microservice is replaced by a third hook function, wherein the third hook function is used to obtain the matching data quality verification rule when the table to be updated and the field to be updated in the data update operation are the preset verification objects, and to call the aviator engine to check whether the field value of the field to be updated satisfies the data quality verification rule, and to execute the data update operation if it is satisfied and interrupt the data update operation if it is not satisfied.

8. A data quality verification device for microservices, characterized in that: include: A data update operation acquisition unit, used to acquire data update operations for microservice-related data; An update content determination unit, configured to determine, based on the data update operation, a table to be updated in the microservice associated data and a field to be updated in the table to be updated; A verification rule matching unit, configured to obtain, in the metadata of the microservice, a data quality verification rule that matches the table to be updated and the field to be updated if the table to be updated and / or the field to be updated are preset verification objects; A verification rule verification unit, used to determine whether the field value of the field to be updated in the data update operation satisfies the data quality verification rule; An execution unit is used to execute the data update operation if the field value satisfies the data quality verification rule, and interrupt the data update operation if the field value does not satisfy the data quality verification rule.

9. A computer device, characterized in that: include: at least one processor; and, a memory communicatively coupled to the at least one processor; The memory stores instructions executable by the at least one processor, wherein the instructions are configured to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: Computer executable instructions are stored, and the computer executable instructions are used to execute the method according to any one of claims 1 to 7.

Images