Data quality detection method and device, server and storage medium

Abstract

This disclosure provides a data quality detection method, apparatus, server, and storage medium, belonging to the field of Internet technology. The method includes: determining the target state node corresponding to the data to be detected based on the field values ​​of each field in the data to be detected; obtaining a data detection model corresponding to the business scenario in which the data to be detected exists, the data detection model being used to detect the data quality under the business scenario, and the data detection model representing the state change relationship between N legal state nodes under the business scenario; performing legality detection on the target state node based on the data detection model; and determining the data quality of the data to be detected based on the detection results. This disclosure selects a data detection model corresponding to the business scenario in which the data to be detected exists, and then detects the data quality of the data to be detected. Because this detection method conforms to the specific business form, the detection result is more accurate.

Description

Technical Field

[0001] This disclosure relates to the field of Internet technology, and in particular to a method, apparatus, server, and storage medium for detecting data quality. Background Technology

[0002] In the closed loop of food delivery consisting of riders, merchants, and users, delivery service is the fulfillment link. Every day, tens of millions of food orders are delivered from merchants to users via delivery services. As the core data carrier of the fulfillment process, waybills are influenced by users, merchants, and business models, requiring the recording of a vast amount of data. If an error occurs in the data of a waybill and is not detected and corrected in a timely manner, it will lead to a very poor experience for riders, merchants, or users. To improve the quality of delivery services, data quality testing is necessary.

[0003] When performing data quality testing, the relevant technologies mainly employ the following methods: determining the testing strategy corresponding to the dataset to be tested, which is used to test whether the data to be tested in the dataset meets the preset data quality requirements; testing the data to be tested using the testing strategy to obtain the testing results corresponding to the data to be tested; and obtaining the data quality testing results of the dataset to be tested based on the testing results.

[0004] However, the detection strategies set by the relevant technologies are fixed and detached from specific business scenarios, resulting in inaccurate detection results. Summary of the Invention

[0005] This disclosure provides a data quality detection method, apparatus, server, and storage medium. It enables the selection of a data detection model corresponding to the business scenario in which the data to be detected exists, thereby improving the accuracy of the detection results. The technical solution is as follows:

[0006] Firstly, a method for detecting data quality is provided, the method comprising:

[0007] Based on the field values ​​of each field in the data to be detected, determine the target state node corresponding to the data to be detected;

[0008] Based on the business scenario in which the data to be detected is located, a data detection model corresponding to the business scenario is obtained. The data detection model is used to detect the data quality in the business scenario. The data detection model is used to represent the state change relationship between N legal state nodes in the business scenario, where N is a positive integer.

[0009] Based on the data detection model, the legality of the target state node is detected;

[0010] Based on the test results, the data quality of the data to be tested is determined.

[0011] In another embodiment of this disclosure, the data detection model is represented by an N*N detection matrix consisting of N valid state nodes. The element value of each element in the detection matrix is ​​used to indicate whether the state change process between the valid state node in the column and the valid state node in the row is valid.

[0012] The legality detection of the target state node based on the data detection model includes:

[0013] Based on the detection matrix, it is determined whether the target state node is a valid state node;

[0014] If the target state node is a valid state node and there is a previous state node of the target state node, check whether the state change process of the target state node is valid;

[0015] If the state change process of the target state node is legal, the target state node is determined to be legal.

[0016] In another embodiment of this disclosure, detecting whether the target state node is a valid state node based on the detection matrix includes:

[0017] When the target state node is the initial state node in the business scenario, it is detected whether there is a specified column in the detection matrix. The specified column refers to a column in which all elements have the value of a first value. The first value is used to indicate that the state change process between two legal state nodes in the detection matrix is ​​illegal.

[0018] When the target state node is the final state node in the business scenario, it is detected whether there is a specified row in the detection matrix. The specified row refers to a row in which all elements have the first value.

[0019] When the target state node is an intermediate state node in the business scenario, the data detection model is checked to see if there is a valid state node that is the same as the target state node.

[0020] In another embodiment of this disclosure, the process of detecting whether the state change of the target state node is legal includes:

[0021] Obtain the element value of the target element corresponding to the target state change process in the detection matrix, wherein the target state change process is the state change process of the data to be detected from the previous state node to the target state;

[0022] When the element value of the target element is the second value, the state change process of the target state node is determined to be legal. The second value is used to indicate that the state change process between two legal state nodes in the detection matrix is ​​legal.

[0023] In another embodiment of this disclosure, before obtaining the data detection model corresponding to the business scenario based on the business scenario in which the data to be detected exists, the method further includes:

[0024] Based on the field values ​​of M fields in multiple historical business data under the business scenario, N finite state nodes are generated. Each finite state machine node is represented by an M-dimensional vector, where M is a positive integer.

[0025] The data detection model is constructed based on the N finite state nodes.

[0026] In another embodiment of this disclosure, generating N finite state nodes based on the field values ​​of M fields from multiple historical business data in the business scenario includes:

[0027] Based on the M fields included in each historical business data under the aforementioned business scenario, construct an M-dimensional vector space, with each dimension representing a field;

[0028] Based on the field values ​​of M fields in the multiple historical business data, determine all possible values ​​of elements in each dimension of the vector space;

[0029] Based on all the values ​​of the elements in each dimension of the vector space, the N finite state nodes are generated.

[0030] In another embodiment of this disclosure, determining all values ​​of elements in each dimension of the vector space based on the field values ​​of M fields in the plurality of historical business data includes:

[0031] For any field, when the value of the field is a discrete value, all discrete values ​​are determined as all values ​​of the elements of the field in the target dimension in the vector space.

[0032] When the field value of the field is a continuous range of values, the range of values ​​is divided into multiple sub-ranges of values, and each sub-range of values ​​is mapped one-to-one with a function value; the function value corresponding to each sub-range of values ​​is determined as all the values ​​of the element on the target dimension.

[0033] In another embodiment of this disclosure, constructing the data detection model based on the N finite state nodes includes:

[0034] Based on the N finite state nodes, construct an N*N matrix structure, where each row and column of the matrix structure corresponds to a finite state node.

[0035] Based on the state change relationships among the N finite state machine nodes, determine the element value of each element in the matrix structure;

[0036] The data detection model is determined based on the matrix structure and the element value of each element in the matrix structure.

[0037] Secondly, a data quality detection device is provided, the device comprising:

[0038] The determination module is used to determine the target state node corresponding to the data to be detected based on the field values ​​of each field in the data to be detected;

[0039] The acquisition module is used to acquire a data detection model corresponding to the business scenario based on the business scenario in which the data to be detected is located. The data detection model is used to detect the data quality in the business scenario. The data detection model is used to represent the state change relationship between N legal state nodes in the business scenario, where N is a positive integer.

[0040] The detection module is used to perform legality detection on the target state node based on the data detection model;

[0041] The determining module is further configured to determine the data quality of the data to be detected based on the detection results.

[0042] In another embodiment of this disclosure, the data detection model is represented by an N*N detection matrix consisting of N valid state nodes. The element value of each element in the detection matrix is ​​used to indicate whether the state change process between the valid state node in the column and the valid state node in the row is valid.

[0043] The detection module is used to detect whether the target state node is a valid state node based on the detection matrix; when the target state node is a valid state node and there is a previous state node of the target state node, it detects whether the state change process of the target state node is valid; when the state change process of the target state node is valid, it determines that the target state node is valid.

[0044] In another embodiment of this disclosure, the detection module is configured to: when the target state node is an initial state node in the business scenario, detect whether a specified column exists in the detection matrix, wherein the specified column is a column in which all elements have a first value, and the first value is used to indicate that the state change process between two legal state nodes in the detection matrix is ​​illegal; when the target state node is a final state node in the business scenario, detect whether a specified row exists in the detection matrix, wherein the specified row is a row in which all elements have the first value; and when the target state node is an intermediate state node in the business scenario, detect whether the data detection model has a legal state node that is the same as the target state node.

[0045] In another embodiment of this disclosure, the detection module is used to obtain the element value of the target element corresponding to the target state change process in the detection matrix, wherein the target state change process is the state change process of the data to be detected from the previous state node to the target state; when the element value of the target element is a second value, the state change process of the target state node is determined to be legal, wherein the second value is used to indicate that the state change process between two legal state nodes in the detection matrix is ​​legal.

[0046] In another embodiment of this disclosure, the apparatus further includes:

[0047] The generation module is used to generate N finite state nodes based on the field values ​​of M fields in multiple historical business data under the business scenario. Each finite state machine node is represented by an M-dimensional vector, where M is a positive integer.

[0048] A construction module is used to construct the data detection model based on the N finite state nodes.

[0049] In another embodiment of this disclosure, the generation module is configured to construct an M-dimensional vector space based on the M fields included in each historical business data in the business scenario, with each dimension representing a field; determine all values ​​of elements in each dimension of the vector space based on the field values ​​of the M fields in the multiple historical business data; and generate the N finite state nodes based on all values ​​of elements in each dimension of the vector space.

[0050] In another embodiment of this disclosure, the generation module is configured to, for any field, when the value of the field is a discrete numerical value, determine all discrete numerical values ​​as all values ​​of the elements of the field in the target dimension in the vector space; when the field value of the field is a continuous numerical interval, divide the numerical interval into multiple numerical sub-intervals, each numerical sub-interval being mapped one-to-one with a function value; and determine the function value corresponding to each numerical sub-interval as all values ​​of the elements in the target dimension.

[0051] In another embodiment of this disclosure, the construction module is configured to construct an N*N matrix structure based on the N finite state nodes, wherein each row and column of the matrix structure corresponds to a finite state node; determine the element value of each element in the matrix structure based on the state change relationship between the N finite state machine nodes; and determine the data detection model based on the matrix structure and the element value of each element in the matrix structure.

[0052] Thirdly, a server is provided, the server including a processor and a memory, the memory storing at least one piece of program code, the at least one piece of program code being loaded and executed by the processor to implement the data quality detection method as described in the first aspect.

[0053] Fourthly, a computer-readable storage medium is provided, wherein at least one piece of program code is stored therein, the at least one piece of program code being loaded and executed by a processor to implement the data quality detection method as described in the first aspect.

[0054] Fifthly, a computer program product is provided, including a computer program that, when executed by a processor, implements the data quality detection method described in the first aspect.

[0055] The beneficial effects of the technical solutions provided in this disclosure are:

[0056] Based on the business scenario in which the data to be tested exists, a data detection model corresponding to that business scenario is selected, and then the data quality of the data to be tested is detected. Since this detection method conforms to the specific business form, the detection result is more accurate. Attached Figure Description

[0057] To more clearly illustrate the technical solutions in the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0058] Figure 1This is a flowchart of a data quality detection method provided in an embodiment of this disclosure;

[0059] Figure 2 This is a flowchart of another data quality detection method provided in this embodiment of the disclosure;

[0060] Figure 3 This is a schematic diagram of a data change process provided in an embodiment of this disclosure;

[0061] Figure 4 This is a schematic diagram of a directed topology graph provided in an embodiment of this disclosure;

[0062] Figure 5 This is a schematic diagram of a detection matrix provided in an embodiment of this disclosure;

[0063] Figure 6 This is a schematic diagram of a data quality detection device provided in an embodiment of this disclosure;

[0064] Figure 7 This is a server for detecting data quality, as illustrated in an exemplary embodiment. Detailed Implementation

[0065] To make the objectives, technical solutions, and advantages of this disclosure clearer, the embodiments of this disclosure will be described in further detail below with reference to the accompanying drawings.

[0066] It is understood that the terms "each," "multiple," and "any," etc., used in the embodiments of this disclosure, include "multiple" (two or more), "each" (each of the corresponding multiples), and "any" (any one of the corresponding multiples). For example, multiple words include 10 words, and "each word" refers to each of the 10 words, while "any word" refers to any one of the 10 words.

[0067] This disclosure provides a method for detecting data quality, see [link to relevant documentation]. Figure 1 The method flow provided in this disclosure includes:

[0068] 101. Determine the target state node corresponding to the data to be detected based on the field values ​​of each field in the data to be detected.

[0069] 102. Based on the business scenario in which the data to be detected is located, obtain the data detection model corresponding to the business scenario.

[0070] Among them, the data detection model is used to detect the data quality in the business scenario. The data detection model is used to represent the state change relationship between N legal state nodes in the business scenario, where N is a positive integer.

[0071] 103. Based on the data detection model, perform legality detection on the target state node.

[0072] 104. Based on the test results, determine the data quality of the data to be tested.

[0073] The method provided in this disclosure selects a data detection model corresponding to the business scenario in which the data to be detected is located, and then detects the data quality of the data to be detected. Since this detection method conforms to the specific business form, the detection result is more accurate.

[0074] In another embodiment of this disclosure, the data detection model is represented by an N*N detection matrix consisting of N legal state nodes. The element value of each element in the detection matrix is ​​used to indicate whether the state change process between the legal state node in the column and the legal state node in the row is legal.

[0075] Based on the data detection model, the legality of the target state node is checked, including:

[0076] Based on the detection matrix, determine whether the target state node is a valid state node;

[0077] If the target state node is a valid state node and there is a previous state node of the target state node, check whether the state change process of the target state node is valid;

[0078] If the state change process of the target state node is legal, the target state node is determined to be legal.

[0079] In another embodiment of this disclosure, detecting whether a target state node is a legitimate state node based on a detection matrix includes:

[0080] When the target state node is the initial state node in the business scenario, check whether there is a specified column in the detection matrix. The specified column is a column in which all elements have the first value. This first value is used to indicate that the state change process between two legal state nodes in the detection matrix is ​​illegal.

[0081] When the target state node is the final state node in the business scenario, check whether there is a specified row in the detection matrix. The specified row refers to the row in which all elements have the first value.

[0082] When the target state node is an intermediate state node in the business scenario, the detection data detection model checks whether there is a valid state node that is the same as the target state node.

[0083] In another embodiment of this disclosure, detecting whether the state change process of the target state node is legal includes:

[0084] Obtain the element value of the target element in the detection matrix corresponding to the target state change process. The target state change process is the state change process of the data to be detected from the previous state node to the target state.

[0085] When the element value of the target element is the second value, the state change process of the target state node is determined to be legal. The second value is used to indicate that the state change process between two legal state nodes in the detection matrix is ​​legal.

[0086] In another embodiment of this disclosure, before obtaining the data detection model corresponding to the business scenario based on the business scenario in which the data to be detected exists, the method further includes:

[0087] Based on the field values ​​of M fields in multiple historical business data under the business scenario, N finite state nodes are generated. Each finite state machine node is represented by an M-dimensional vector, where M is a positive integer.

[0088] Construct a data detection model based on N finite state nodes.

[0089] In another embodiment of this disclosure, N finite state nodes are generated based on the field values ​​of M fields from multiple historical business data in a business scenario, including:

[0090] Based on the M fields included in each historical business data in the business scenario, construct an M-dimensional vector space, with each dimension representing a field;

[0091] Based on the field values ​​of M fields from multiple historical business data, determine all possible values ​​for elements in each dimension of the vector space;

[0092] Based on all possible values ​​of the elements in each dimension of the vector space, generate N finite state nodes.

[0093] In another embodiment of this disclosure, based on the field values ​​of M fields from multiple historical business data, all possible values ​​for elements in each dimension of the vector space are determined, including:

[0094] For any field, when the field takes discrete values, all discrete values ​​are determined as all values ​​of the elements of the field in the target dimension in the vector space.

[0095] When the field value is a continuous range of values, the range is divided into multiple sub-ranges, and each sub-range is mapped to a function value. The function value corresponding to each sub-range is determined as all possible values ​​of the element in the target dimension.

[0096] In another embodiment of this disclosure, a data detection model is constructed based on N finite state nodes, including:

[0097] Based on N finite state nodes, construct an N*N matrix structure, in which each row and column corresponds to a finite state node.

[0098] Based on the state change relationships between N finite state machine nodes, determine the element value of each element in the matrix structure;

[0099] The data detection model is determined based on the matrix structure and the element value of each element in the matrix structure.

[0100] All of the above-mentioned optional technical solutions can be combined in any way to form optional embodiments of this disclosure, and will not be described in detail here.

[0101] This disclosure provides a data quality detection method. Taking a server executing this disclosure as an example, the server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, and big data and artificial intelligence platforms. See also Figure 2 The method flow provided in this disclosure includes:

[0102] 201. The server builds a data detection model.

[0103] The data detection model is used to detect data quality. Considering that the content and fields of data differ across business scenarios, this embodiment of the disclosure can divide the acquired historical business data according to the business scenario to obtain historical business data for different business scenarios. Then, based on the historical business data for different business scenarios, data detection models for different business scenarios can be constructed. For example, for historical orders in a waybill system, historical orders can be divided into historical food delivery orders and historical errand-running orders according to different business scenarios. A data detection model for detecting food delivery orders can be constructed based on historical food delivery orders, and a data detection model for detecting errand-running orders can be constructed based on historical errand-running orders.

[0104] For any given business scenario, the process of constructing the corresponding data detection model is as follows:

[0105] 2011. The server generates N finite state nodes based on the field values ​​of M fields in multiple historical business data under this business scenario.

[0106] This embodiment employs massive data storage technology to save snapshots of each change in historical business data, serving as state nodes for the historical business data. To facilitate understanding the state change process of historical business data, the server will also assign different version numbers to the historical business data according to the chronological order of these changes. For example, for any historical business data, when it first enters the system, version number V1 can be assigned. As the business processing progresses, the values ​​of certain fields in the historical business data will change, and different version numbers will be assigned to the changed historical business data. For example, version number V2 can be assigned to the historical business data that changes for the first time, version number Vn can be assigned to the historical business data that changes for the second time, until the state node corresponding to the historical business data is in a completed state.

[0107] By assigning different version numbers to different states of historical business data, different version numbers can be used to describe the state change path of the corresponding state nodes in the historical business data. For example, for two historical business data points identified as 1 and 2, see [link to relevant documentation]. Figure 3 In the left figure, the historical business data of Identifier 1 includes fields A, B and C, and the version numbers set for the historical business data of Identifier 1 are V1, V2 and V3. The historical business data of Identifier 2 includes fields A, B and C, and the version numbers set for the historical business data of Identifier 2 are V1 and V2. Figure 3 The right figure shows the status change path of historical business data based on the version number set in the historical business data in the left figure. The status change path of historical business data with identifier 1 is: V1-V2-V3, and the status change path of historical business data with identifier 2 is: V1-V2.

[0108] In this embodiment of the disclosure, each historical business data includes M fields, and each finite state machine node is represented by an M-dimensional vector, where M is a positive integer. When the server generates N finite state nodes based on the field values ​​of M fields from multiple historical business data in a business scenario, the following method can be used:

[0109] 20111. The server constructs an M-dimensional vector space based on the M fields included in each historical business data in the business scenario.

[0110] In this embodiment of the disclosure, the server constructs an M-dimensional vector space based on the number M fields included in each historical business data, with each dimension representing a field. The state of the historical business data is then described using an M-dimensional vector. For example, state 1 is represented by [X1, X2, X3, ..., Xm], and state 2 is represented by [X1', X2', X3', ..., Xm'].

[0111] 20112. The server determines all possible values ​​for each element in each dimension of the vector space based on the field values ​​of M fields from multiple historical business data.

[0112] For any field in any historical business data, when the server determines all possible values ​​of the element in the target dimension corresponding to that field in the vector space, it includes the following two cases:

[0113] In the first case, when the value of the field is a discrete numerical value, the server obtains all the discrete numerical values ​​of the field and determines all the discrete numerical values ​​as all the values ​​of the elements of the field in the target dimension in the vector space.

[0114] In the second scenario, when the field value is a continuous range of values, the server divides the range into multiple sub-ranges, each sub-range being mapped to a function value. The function value corresponding to each sub-range is then used to determine all possible values ​​for the element in the target dimension.

[0115] For a field A with a value x, if the range of x is too large to be exhaustively listed (e.g., [0, +∞]), the corresponding state node for this historical business data is an infinite state node. To transform this infinite state node into a finite state node with limited values, a non-linear amplitude-limiting mapping process is needed for the historical business data. When performing non-linear amplitude-limiting on historical business data, data standardization functions can be used, such as min-max normalization, log function transformation, atan function transformation, z-score normalization (zero-mean normalization), etc., to divide the field's values ​​into multiple numerical sub-intervals. Then, a function y = f(x) with business meaning is constructed, making the range of y finite and exhaustively listable.

[0116] For example, in terms of business, the value of the "estimated delivery time" field on a waybill may be within [0, +∞], so the values of this field cannot be exhausted. However, after combining with business analysis, it can be found that the value rule of estimated delivery time can be described as follows: for the creation time ctime of the waybill, when the estimated delivery time is less than ctime, it can be considered invalid. Therefore, [0, ctime] can be mapped to the same fixed value through a function. Similarly, when the estimated delivery time is more than 24 hours later than the creation time, the data is also considered inaccurate, so [ctime+24*60*60, +∞] can also be mapped to the same fixed value through a function. Combined with specific business scenarios, the value range of estimated delivery time between [ctime, ctime+24*60*60] can be divided into several sub-intervals, and their meanings are then interpreted from a business perspective. In business practices, it is usually not necessary to pay attention to whether the waybill is delivered in 20 minutes or 25 minutes, but only whether it is delivered within 1 hour, 1 to 3 hours, or more than 3 hours. The above rules can be represented by the following pseudo-code:

[0117] y = f(estimated delivery time)

[0118] When (estimated delivery time - ctime) < 0, y = 1, this scenario is impossible in business

[0119] When (estimated delivery time - ctime) > 24*60*60, y = 2, this scenario is impossible in business

[0120] When 0 < (estimated delivery time - ctime) < 3600 (business semantic: delivery within 1 hour), y = 1

[0121] When 3600 < (estimated delivery time - ctime) < 10800 (business semantic: delivery within 3 hours), y = 2

[0122] When (estimated delivery time - ctime) > 10800 (business semantic: delivery after more than 3 hours), y = 3

[0123] By adopting the above processing method, a value with an infinite range can be converted into a value with a finite range according to business scenarios. After performing the above processing on each field in the waybill, all valid states of the waybill can finally be exhausted.

[0124] 20113. The server generates N finite state nodes based on all values of elements in each dimension of the vector space.

[0125] Where N is a positive integer. The server enumerates all possible values ​​of the elements in each dimension of the M-dimensional vector space, for example, S1 = [X1,X2,X3,......Xm], S2 = [X1',X2',X3'......Xm'], ..., Sn = [X1n,X2n,X3n,........Xmn].

[0126] 2012. The server constructs a data detection model based on N finite state nodes.

[0127] When a server constructs a data detection model based on N finite state nodes, the following method can be used:

[0128] 20121. The server constructs an N*N matrix structure based on N finite state nodes.

[0129] Based on the N finite state nodes obtained, the server constructs an N*N matrix structure with N rows and N columns. Each row in this matrix structure corresponds to a finite state node, and each column also corresponds to a finite state node.

[0130] 20122. The server determines the element value of each element in the matrix structure based on the state change relationships between N finite state machine nodes.

[0131] When determining the value of each element in a matrix structure, the server can determine the value of each element according to the state change process from row to column, or according to the state change process from column to row. Previously, the server could set a first value to indicate that the state change process for obtaining the element was invalid, and set a second value to indicate that the state change process for obtaining the element was valid. The first value could be 0, and the second value could be 1.

[0132] Based on the above, the server determines the value of each element in the matrix structure according to the state change process of the state node corresponding to each element. To visually demonstrate the state change relationships among N finite state nodes, the server can draw a directed acyclic topology graph based on the version number set for each state node and the state change relationships between each state node in the actual business scenario. The drawn directed acyclic topology graph can be found in [reference needed]. Figure 4 Based on the connection relationships between the state nodes in the directed acyclic topology, the state change relationships between N finite state nodes are determined.

[0133] 201223. The server determines the data detection model based on the matrix structure and the element value of each element in the matrix structure.

[0134] The server fills the position of each element in the matrix structure with the element value of each element to obtain the data detection model. This data detection model is used to represent the state change relationship between N legal state nodes in the business scenario. The element value of each element in the detection matrix is ​​used to indicate whether the state change process between the legal state node in the column and the legal state node in the row is legal. This detection matrix is ​​used to detect the data quality in the business scenario.

[0135] The constructed data detection model can also undergo the following update and iteration processes:

[0136] Firstly, for another different order system, different data standardization functions can be customized based on the business characteristics of each system. Other than that, no modifications are required, and the method of this disclosure embodiment can be ported to the new system for monitoring.

[0137] The second approach is to manually identify the change paths of newly added status nodes before the new process goes live, or to adjust the data standardization function according to the new process to adapt to the new process.

[0138] The third method involves regenerating the detection matrix based on historical snapshots after each adjustment of the data standardization function. This regeneration process is fully automated and requires no human intervention.

[0139] 202. The server determines the target state node corresponding to the data to be detected based on the field values ​​of each field in the data to be detected.

[0140] In real-world business scenarios, to monitor whether any data to be tested has encountered an error at a certain point in time, the server needs to perform real-time quality checks on the data to be tested and take timely measures to process it, preventing the error from persisting and causing adverse effects on users. Before checking the data quality, the server can map the data to be tested to target state nodes based on the field values ​​of each field in the data to be tested and the mapping method of finite state nodes used in constructing the data detection model.

[0141] 203. Based on the business scenario in which the data to be detected is located, the server obtains the data detection model corresponding to the business scenario.

[0142] Since the content and type of data differ in different business scenarios, in order to better detect the data to be detected, the server can obtain the data detection model corresponding to the business scenario in which the data to be detected is located, and then detect the data based on the data detection model in subsequent processes.

[0143] 204. Based on the data detection model, the server performs legality checks on the target state nodes.

[0144] Research on the data revealed that errors occurring at a given moment mainly fall into two categories: one is that such data is impossible to exist in the business context, meaning the state node corresponding to the data is not a valid state node; the other is that such data may exist in the business context, but the data change scenario is incorrect. For example, a waybill can change from "created" to "scheduled," but cannot change from "cancelled" to "scheduled," meaning the state change process corresponding to the data is invalid.

[0145] In this embodiment of the disclosure, the data detection model is represented by a detection matrix. The element value of each element in the detection matrix indicates whether the state change process between the valid state node in the column and the valid state node in the row is valid. When the server performs validity detection on the target state node based on the data detection model, it can use the following method:

[0146] The server uses a detection matrix to determine whether a target state node is a valid state node.

[0147] When the server detects whether a target state node is a valid state node based on the detection matrix, the following situations apply:

[0148] In the first case, when the target state node is the initial state node in the business scenario, the server checks whether a specified column exists in the detection matrix. This specified column refers to a column in which all elements have the first value. This first value is used to indicate that the state change process between two legal state nodes in the detection matrix is ​​illegal. If the specified column exists in the detection matrix, the target state node is determined to be a legal state node.

[0149] In the second scenario, when the target state node is the final state node in the business scenario, the server checks whether a specified row exists in the detection matrix. The specified row refers to the row in which all elements have the first value. If the specified row exists in the detection matrix, the target state node is determined to be a valid state node.

[0150] In the third case, when the target state node is an intermediate state node in the business scenario, the server detects whether there is a valid state node in the detection model that is the same as the target state node. If there is a valid state node in the detection matrix that is the same as the target state node, then the target state node is determined to be a valid state node.

[0151] When the target state node is a valid state node and there is a previous state node for the target state node, the server checks whether the state change process of the target state node is valid.

[0152] When the server detects whether the state change process of the target state node is legal, it can obtain the element value of the target element corresponding to the target state change process in the detection matrix. The target state change process is the state change process of the data to be detected from the previous state node to the target state. When the element value of the target element is the second value, the state change process of the target state node is determined to be legal.

[0153] If the state change process of the target state node is legal, the server determines that the target state node is legal.

[0154] The data detection model constructed in this embodiment can not only detect whether the data itself is legal and whether the data state change process is legal, but also view the shortest path, longest path and hottest path of the data state flow process, thereby optimizing the business process based on the viewing results.

[0155] 205. The server determines the data quality of the data to be tested based on the test results.

[0156] If the data to be tested is determined to be valid based on the above steps, it can be determined that the data to be tested has not been erroneous and the quality of the data to be tested is good.

[0157] The method provided in this disclosure selects a data detection model corresponding to the business scenario in which the data to be detected exists, and then detects the data quality of the data to be detected. Because this detection method conforms to the specific business form, the detection result is more accurate. Furthermore, this disclosure supports the legality verification of the state change process of the data to be detected, expanding the scope of data verification and further ensuring the accuracy of the detection results. In addition, this disclosure supports rapid business iteration, eliminating the need for manual formulation and interpretation of verification rules, resulting in lower maintenance costs.

[0158] See Figure 6 This disclosure provides a data quality detection device, which includes:

[0159] The determination module 601 is used to determine the target state node corresponding to the data to be detected based on the field values ​​of each field in the data to be detected;

[0160] The acquisition module 602 is used to acquire the data detection model corresponding to the business scenario based on the business scenario in which the data to be detected is located. The data detection model is used to detect the data quality in the business scenario. The data detection model is used to represent the state change relationship between N legal state nodes in the business scenario, where N is a positive integer.

[0161] The detection module 603 is used to perform legality detection on the target state node based on the data detection model;

[0162] The determination module 601 is also used to determine the data quality of the data to be tested based on the test results.

[0163] In another embodiment of this disclosure, the data detection model is represented by an N*N detection matrix consisting of N legal state nodes. The element value of each element in the detection matrix is ​​used to indicate whether the state change process between the legal state node in the column and the legal state node in the row is legal.

[0164] The detection module 603 is used to detect whether a target state node is a valid state node based on the detection matrix; when the target state node is a valid state node and there is a previous state node of the target state node, it detects whether the state change process of the target state node is valid; when the state change process of the target state node is valid, it determines that the target state node is valid.

[0165] In another embodiment of this disclosure, the detection module 603 is configured to: when the target state node is an initial state node in a business scenario, detect whether a specified column exists in the detection matrix, wherein the specified column is a column in which all elements have a first value, and the first value is used to indicate that the state change process between two legal state nodes in the detection matrix is ​​illegal; when the target state node is a final state node in a business scenario, detect whether a specified row exists in the detection matrix, wherein the specified row is a row in which all elements have a first value; and when the target state node is an intermediate state node in a business scenario, detect whether the data detection model has a legal state node that is the same as the target state node.

[0166] In another embodiment of this disclosure, the detection module 603 is used to obtain the element value of the target element corresponding to the target state change process in the detection matrix. The target state change process is the state change process of the data to be detected from the previous state node to the target state. When the element value of the target element is a second value, the state change process of the target state node is determined to be legal. The second value is used to indicate that the state change process between two legal state nodes in the detection matrix is ​​legal.

[0167] In another embodiment of this disclosure, the device further includes:

[0168] The generation module is used to generate N finite state nodes based on the field values ​​of M fields in multiple historical business data under the business scenario. Each finite state machine node is represented by an M-dimensional vector, where M is a positive integer.

[0169] The building module is used to construct a data detection model based on N finite state nodes.

[0170] In another embodiment of this disclosure, the generation module is used to construct an M-dimensional vector space based on the M fields included in each historical business data in the business scenario, with each dimension representing a field; determine all values ​​of the elements in each dimension of the vector space based on the field values ​​of the M fields in multiple historical business data; and generate N finite state nodes based on all values ​​of the elements in each dimension of the vector space.

[0171] In another embodiment of this disclosure, the generation module is configured to, for any field, when the value of the field is a discrete numerical value, determine all discrete numerical values ​​as all values ​​of the elements of the field in the target dimension in the vector space; when the field value of the field is a continuous numerical interval, divide the numerical interval into multiple numerical sub-intervals, each numerical sub-interval being mapped one-to-one with a function value; and determine the function value corresponding to each numerical sub-interval as all values ​​of the elements in the target dimension.

[0172] In another embodiment of this disclosure, the construction module is configured to construct an N*N matrix structure based on N finite state nodes, wherein each row and column of the matrix structure corresponds to a finite state node; determine the element value of each element in the matrix structure based on the state change relationship between the N finite state machine nodes; and determine the data detection model based on the matrix structure and the element value of each element in the matrix structure.

[0173] In summary, the apparatus provided in this disclosure selects a data detection model corresponding to the business scenario in which the data to be detected exists, and then detects the data quality of the data to be detected. Because this detection method conforms to the specific business form, the detection result is more accurate. Furthermore, this disclosure supports the legality verification of the state change process of the data to be detected, expanding the scope of data verification and further ensuring the accuracy of the detection results. In addition, this disclosure supports rapid business iteration, eliminating the need for manual formulation and interpretation of verification rules, resulting in lower maintenance costs.

[0174] Figure 7 This is a server for data quality detection, illustrated according to an exemplary embodiment. (Refer to...) Figure 7 The server 700 includes a processing component 722, which further includes one or more processors, and memory resources represented by memory 732 for storing instructions, such as applications, that can be executed by the processing component 722. The applications stored in memory 732 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 722 is configured to execute instructions to perform the functions performed by the server in the aforementioned data quality detection method.

[0175] Server 700 may also include a power supply component 726 configured to perform power management of server 700, a wired or wireless network interface 750 configured to connect server 700 to a network, and an input / output (I / O) interface 758. Server 700 can operate on an operating system, such as Windows Server, stored in memory 732. TM Mac OSX TM Unix TM Linux TM FreeBSD TM Or similar.

[0176] This disclosure provides a computer-readable storage medium storing at least one line of program code, which is loaded and executed by a processor to implement a data quality detection method. The computer-readable storage medium can be non-transitory. For example, it can be a read-only memory (ROM), a random access memory (RAM), a CD-ROM, magnetic tape, a floppy disk, or an optical data storage device.

[0177] This disclosure provides a computer program product, including a computer program that, when executed by a processor, implements a method for detecting data quality.

[0178] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0179] The above description is merely an optional embodiment of this disclosure and is not intended to limit this disclosure. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the protection scope of this disclosure.

Claims

1. A method for detecting data quality, characterized in that, The method includes: Based on the field values ​​of M fields in multiple historical business data under the business scenario, N finite state nodes are generated. Each finite state node is represented by an M-dimensional vector, where M is a positive integer. Specifically, a snapshot of each change in the historical business data is saved as a state node of the historical business data. Different version numbers are set for the historical business data according to the order of state changes. The version number is used to describe the state change path of the state node corresponding to the historical business data. Based on the N finite state nodes, a data detection model corresponding to the business scenario is constructed. The data detection model is represented by an N×N detection matrix composed of N legal state nodes. The element value of each element in the detection matrix is ​​used to indicate whether the state change process between the legal state node in the column and the legal state node in the row is legal. Based on the field values ​​of each field in the data to be detected, determine the target state node corresponding to the data to be detected; Based on the data detection model, the legality of the target state node is checked, including: Based on the detection matrix, it is determined whether the target state node is a valid state node; If the target state node is a valid state node and there is a previous state node of the target state node, check whether the state change process of the target state node is valid; If the state change process of the target state node is legal, the target state node is determined to be legal. Based on the test results, the data quality of the data to be tested is determined.

2. The method according to claim 1, characterized in that, The step of detecting whether the target state node is a valid state node based on the detection matrix includes: When the target state node is the initial state node in the business scenario, it is detected whether there is a specified column in the detection matrix. The specified column refers to a column in which all elements have the value of a first value. The first value is used to indicate that the state change process between two legal state nodes in the detection matrix is ​​illegal. When the target state node is the final state node in the business scenario, it is detected whether there is a specified row in the detection matrix. The specified row refers to a row in which all elements have the first value. When the target state node is an intermediate state node in the business scenario, the data detection model is checked to see if there is a valid state node that is the same as the target state node.

3. The method according to claim 1, characterized in that, The process of detecting the legality of the state change of the target state node includes: Obtain the element value of the target element corresponding to the target state change process in the detection matrix, wherein the target state change process is the state change process of the data to be detected from the previous state node to the target state; When the element value of the target element is the second value, the state change process of the target state node is determined to be legal. The second value is used to indicate that the state change process between two legal state nodes in the detection matrix is ​​legal.

4. The method according to claim 1, characterized in that, The generation of N finite state nodes includes: Based on the M fields included in each historical business data under the aforementioned business scenario, construct an M-dimensional vector space, with each dimension representing a field; Based on the field values ​​of M fields in the multiple historical business data, determine all possible values ​​of elements in each dimension of the vector space; Based on all the values ​​of the elements in each dimension of the vector space, the N finite state nodes are generated.

5. The method according to claim 4, characterized in that, The step of determining all possible values ​​for each element in each dimension of the vector space based on the field values ​​of M fields from the plurality of historical business data includes: For any field, when the value of the field is a discrete value, all discrete values ​​are determined as all values ​​of the elements of the field in the target dimension in the vector space. When the field value of the field is a continuous range of values, the range of values ​​is divided into multiple sub-ranges of values, and each sub-range of values ​​is mapped one-to-one with a function value. The function value corresponding to each numerical sub-interval is determined as all the values ​​of the element in the target dimension.

6. The method according to claim 1, characterized in that, The construction of the data detection model includes: Based on the N finite state nodes, construct an N×N matrix structure, in which each row and each column of the matrix structure corresponds to a finite state node; Based on the state change relationships among the N finite state nodes, determine the element value of each element in the matrix structure; The data detection model is determined based on the matrix structure and the element value of each element in the matrix structure.

7. A data quality detection device, characterized in that, The device includes: The generation module is used to generate N finite state nodes based on the field values ​​of M fields in multiple historical business data under the business scenario. Each finite state node is represented by an M-dimensional vector, where M is a positive integer. The generation module is further configured to save a snapshot of each change in the historical business data as a state node of the historical business data, and to set different version numbers for the historical business data according to the order of state changes of the historical business data. The version number is used to describe the state change path of the state node corresponding to the historical business data. The construction module is used to construct a data detection model corresponding to the business scenario based on the N finite state nodes. The data detection model is represented by an N×N detection matrix composed of N legal state nodes. The element value of each element in the detection matrix is ​​used to indicate whether the state change process between the legal state node in the column and the legal state node in the row is legal. The determination module is used to determine the target state node corresponding to the data to be detected based on the field values ​​of each field in the data to be detected; The detection module is used to perform legality detection on the target state node based on the data detection model, specifically for: Based on the detection matrix, it is determined whether the target state node is a valid state node; If the target state node is a valid state node and there is a previous state node of the target state node, check whether the state change process of the target state node is valid; If the state change process of the target state node is legal, the target state node is determined to be legal. The determining module is further configured to determine the data quality of the data to be detected based on the detection results.

8. A server, characterized in that, The server includes a processor and a memory, the memory storing at least one piece of program code, which is loaded and executed by the processor to implement the data quality detection method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The storage medium stores at least one piece of program code, which is loaded and executed by a processor to implement the data quality detection method as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the data quality detection method according to any one of claims 1 to 6.

Images