Data checking method and device, electronic equipment and storage medium

Abstract

The application provides a data checking method and device, electronic equipment and a storage medium, and relates to the field of data processing. The method comprises the following steps: obtaining a numerical consistency rule based on historical data by using a numerical consistency algorithm; taking the numerical consistency rule that passes the artificial checking as the numerical consistency rule; and checking the current data by using the numerical consistency rule to obtain abnormal data. The above scheme can comprehensively generate the numerical consistency rule before the quality problem of the current data occurs, and the abnormal data is checked, so that the effect of data checking is improved.

Description

Technical Field

[0001] This application relates to the field of data processing, and more particularly to a data inspection method, apparatus, electronic device, and storage medium. Background Technology

[0002] Before using data, it is necessary to check the data quality in a timely manner to identify data with quality problems and handle them in a targeted manner to avoid the quality problems affecting the use of the data.

[0003] In related technologies, data inspection involves formulating data quality rules based on historical data issues or experience, and then using these rules to inspect the data and identify quality problems. This method has drawbacks: the rules lag behind the data issues, and experience-based approaches cannot enumerate all rules, resulting in a limited scope of data inspection and affecting its effectiveness. Summary of the Invention

[0004] This application provides a data inspection method, apparatus, electronic device, and storage medium to improve the effectiveness of data inspection.

[0005] In a first aspect, this application provides a data inspection method, comprising: obtaining numerical consistency patterns based on historical data using a numerical consistency algorithm; using manually verified numerical consistency patterns as numerical consistency rules; and inspecting current data using the numerical consistency rules to obtain abnormal data.

[0006] In one possible implementation, obtaining numerical consistency rules based on historical data and using a numerical consistency algorithm includes: establishing a matrix according to the quantity of historical data and the quantity of attributes of the historical data; performing linear decomposition on the matrix to obtain numerical consistency rules; wherein, the numerical consistency rules characterize the numerical consistency rules between different attributes, and the numerical consistency rules include numerical consistency vectors.

[0007] In one possible implementation, before establishing the matrix based on the quantity of historical data and the quantity of attributes of the historical data, the method further includes: removing historical abnormal data from the historical data.

[0008] In one possible implementation, before removing historical outlier data from the historical data, the method further includes: establishing a matrix based on the quantity of historical data and the quantity of attributes of the historical data; performing iterative linear fitting on the matrix to obtain a sparse matrix including the historical outlier data.

[0009] In one possible implementation, the step of checking the current data through the numerical consistency rules to obtain abnormal data includes: assembling the numerical consistency rules into a data quality check statement based on a structured query language; and executing the data quality check statement on the current data to obtain the abnormal data.

[0010] Secondly, this application provides a data inspection device, comprising: an algorithm module for obtaining numerical consistency patterns based on historical data using a numerical consistency algorithm; a verification module for using manually verified numerical consistency patterns as numerical consistency rules; and an inspection module for inspecting current data using the numerical consistency rules to obtain abnormal data.

[0011] In one possible implementation, the algorithm module is specifically used to establish a matrix based on the quantity of historical data and the quantity of attributes of the historical data; the algorithm module is further used to perform linear decomposition on the matrix to obtain numerical consistency rules; wherein, the numerical consistency rules characterize the numerical consistency rules between different attributes, and the numerical consistency rules include numerical consistency vectors.

[0012] In one possible implementation, the apparatus further includes a filtering module for removing historically abnormal data from the historical data.

[0013] In one possible implementation, the apparatus further includes: an iteration module for establishing a matrix based on the quantity of historical data and the quantity of attributes of the historical data; the iteration module is further configured to perform iterative linear fitting on the matrix to obtain a sparse matrix including the historical abnormal data.

[0014] In one possible implementation, the apparatus further includes: an execution module, configured to assemble the numerical consistency rules into a data quality check statement based on a structured query language; the execution module is further configured to execute the data quality check statement on the current data to obtain the abnormal data.

[0015] Thirdly, this application provides an electronic device, including: a processor, and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the method described in any one of the first aspects.

[0016] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which are executed by a processor as described in any one of the first aspects.

[0017] The data inspection method, apparatus, electronic device, and storage medium provided in this application, based on historical data, obtain numerical consistency patterns through a numerical consistency algorithm; use manually verified numerical consistency patterns as numerical consistency rules; and inspect the current data using these numerical consistency rules to identify abnormal data. This approach can comprehensively generate numerical consistency rules and identify abnormal data before quality issues arise in the current data, thereby improving the effectiveness of data inspection. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0019] Figure 1 This is a schematic diagram illustrating an application scenario of a data inspection method provided in an embodiment of this application.

[0020] Figure 2 A flowchart illustrating a data inspection method provided in Embodiment 1 of this application;

[0021] Figure 3 Example of establishing a matrix provided in the embodiments of this application;

[0022] Figure 4 This application provides an example of generating a sparse matrix.

[0023] Figure 5 Data inspection examples provided for embodiments of this application;

[0024] Figure 6 This is a structural example diagram of a data inspection device provided in Embodiment 2 of this application;

[0025] Figure 7 This is a block diagram of a data inspection device provided in Embodiment 3 of this application;

[0026] Figure 8 This is a schematic diagram of the structure of an electronic device provided in Embodiment 4 of this application.

[0027] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0028] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0029] First, let's explain the terms involved:

[0030] Numerical consistency law: linear relationship between continuous data.

[0031] Figure 1 This illustration shows an application scenario of a data inspection method provided in this application. Using the illustrated scenario as an example: A numerical consistency algorithm is used to calculate historical data to obtain numerical consistency patterns. These patterns reflect the relationships between data within the same data type as the historical data. The manually verified numerical consistency patterns are then used as numerical consistency rules. Current data of the same data type as the historical data is then checked against these numerical consistency rules. Data in the current data that does not conform to the numerical consistency rules is identified as abnormal data.

[0032] The technical solutions of this application will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. In the description of this application, unless otherwise expressly specified and limited, the terms should be broadly understood within the art. The embodiments of this application will now be described with reference to the accompanying drawings.

[0033] Example 1

[0034] Figure 2 This is a flowchart illustrating a data inspection method provided in Embodiment 1 of this application. The method includes the following steps:

[0035] S201. Based on historical data, numerical consistency patterns are obtained through a numerical consistency algorithm;

[0036] S202. The numerical consistency patterns that have passed manual verification shall be used as numerical consistency rules.

[0037] S203. Check the current data using the numerical consistency rules to obtain abnormal data.

[0038] As an example, the execution subject of this embodiment can be a data inspection device, which can be implemented in various ways. For example, it can be program software, or a medium storing relevant computer programs, such as a USB flash drive; or, the device can also be a physical device that integrates or installs relevant computer programs, such as a chip, a smart terminal, a computer, a server, etc.

[0039] In one example, S201 specifically includes: establishing a matrix based on the quantity of historical data and the quantity of attributes of the historical data; performing linear decomposition on the matrix to obtain numerical consistency rules; wherein, the numerical consistency rules characterize the numerical consistency rules between different attributes, and the numerical consistency rules include numerical consistency vectors.

[0040] In terms of scenario examples, the attributes of historical data include, but are not limited to, unit price, quantity, total price, cost, and points. The consistency of values ​​refers to a linear relationship, such as unit price * quantity = total price.

[0041] As an feasible approach, such as Figure 3 As shown, Figure 3 Here's an example of building a matrix. For A data points, each data point has B attributes, resulting in an A*B matrix. The coefficients representing the numerical consistency rule are represented by a vector n. For instance, if attribute b1*2 + attribute b2 = attribute 4 for each data point, then we get vector n1 = [2,1,0,-1,0,0,0]. If attribute b2 - attribute b5*3 = attribute b6 for each data point, then we get vector n2 = [0,1,-3,0,0,-1,0]. If any data point * vector n = 0, it means that the sum of the coefficients at the corresponding positions in vector n for each attribute of that data point equals 0, indicating that the data point conforms to the numerical consistency rule corresponding to vector n.

[0042] As an example, change the sign of a vector so that the number of positive coefficients in the vector is no less than the number of negative coefficients.

[0043] For example, for a vector n2 = [0,1,-3,0,0,-1,0], use -n2 = [0,-1,3,0,0,1,0] as the numerical consistency rule.

[0044] It is understandable that the number of positive numbers in the coefficients of a vector is no less than the number of negative numbers, which can improve readability and facilitate manual verification.

[0045] Based on the above implementation methods, numerical consistency algorithms can be used to comprehensively generate numerical consistency patterns from historical data, thereby improving the effectiveness of data inspection.

[0046] In one example, before building a matrix based on the amount of historical data and the number of attributes of the historical data, the method further includes: removing historical outlier data from the historical data.

[0047] In a scenario example, given A historical data points, if A-1 data points conform to the numerical consistency rule, and one data point does not, then that one data point is an outlier. If the numerical consistency rule is generated directly from A historical data points, the outlier will interfere with the rule, leading to inaccuracies in the numerical consistency rule.

[0048] As an feasible approach, historical outliers are removed from historical data to obtain historical normal data. A matrix is ​​then constructed based on the number of historical normal data points and the number of attributes associated with them. Linear decomposition is performed on this matrix to reveal patterns in numerical consistency.

[0049] Based on the above implementation methods, by removing historical abnormal data, the interference of historical abnormal data on the generation of numerical consistency patterns can be avoided, thereby improving the accuracy of numerical consistency patterns.

[0050] Furthermore, in one example, before removing historical outlier data from the historical data, the method further includes: establishing a matrix based on the quantity of historical data and the quantity of attributes of the historical data; performing iterative linear fitting on the matrix to obtain a sparse matrix including the historical outlier data.

[0051] As an feasible approach, such as Figure 4 As shown, Figure 4 This example demonstrates how to generate a sparse matrix. A linear fit is performed on a matrix built from historical data to obtain the numerical consistency rule and the sparse matrix. The sparse matrix includes historical outliers that do not conform to the numerical consistency rule, which can affect the accuracy of the rule. The degree of influence of historical outliers on the numerical consistency rule is determined by judging whether the rule converges to a linear relationship. If the rule does not converge to a linear relationship, it indicates that historical outliers significantly affect the accuracy of the rule. In this case, the sparse matrix is ​​removed from the original matrix, and the linear fit is iteratively performed until the numerical consistency rule converges to a linear relationship. The sparse matrix generated in the current iteration is then output.

[0052] As an example, if the amount of historical outlier data in the sparse matrix generated by the iteration is less than a preset threshold, the iteration stops and the sparse matrix is ​​output.

[0053] Based on the above implementation methods, iterative linear fitting can improve the accuracy of generating sparse matrices through multiple iterations, thereby enhancing the accuracy of numerical consistency patterns.

[0054] In one example, S203 specifically includes: assembling the numerical consistency rules into a data quality check statement based on a structured query language; executing the data quality check statement on the current data to obtain the abnormal data.

[0055] To illustrate with a scenario example, such as Figure 5 As shown, Figure 5 This is an example of data inspection. It reads numerical consistency rules and assembles them into a data quality check statement using Structured Query Language (SQL), for example: `SELECT * FROM order_table WHERE order_table.total_price<> order_table.unit_price* order_table.quantity`. Here, the current data comes from the order table, the numerical consistency rule is unit_price*quantity = total_price, and the data quality check statement indicates that it extracts abnormal data from the order table that does not conform to the rule `unit_price*quantity = total_price`.

[0056] It should be noted that this application does not limit the specific form of the data quality check statement.

[0057] Based on the above implementation methods, by using numerical consistency rules, a comprehensive check can be performed before quality problems occur in the current data, thereby improving the effectiveness of data inspection.

[0058] In one example, the data inspection method also includes: removing numerical consistency patterns that failed manual verification.

[0059] In the context of a scenario, patterns that pass manual verification have practical significance, while numerical consistency patterns that conform to linear patterns but have no practical significance are deleted.

[0060] Based on the above implementation methods, manual verification can improve the accuracy of numerical consistency patterns.

[0061] In the data inspection method provided in this embodiment, numerical consistency patterns are obtained based on historical data using a numerical consistency algorithm. These manually verified patterns are then used as numerical consistency rules. The current data is then inspected using these numerical consistency rules to identify anomalous data. This approach can comprehensively generate numerical consistency rules and identify anomalous data before quality issues arise in the current data, thereby improving the effectiveness of data inspection.

[0062] Example 2

[0063] Figure 6 This is a schematic diagram of the structure of a data inspection device provided in Embodiment 2 of this application, as shown below. Figure 6 As shown, the data inspection device includes:

[0064] Algorithm module 61 is used to obtain numerical consistency patterns based on historical data through a numerical consistency algorithm;

[0065] Verification module 62 is used to take the consistency patterns of numerical values ​​that have passed manual verification as numerical consistency rules;

[0066] The inspection module 63 is used to check the current data according to the numerical consistency rules and obtain abnormal data.

[0067] In one example, algorithm module 61 is specifically used to establish a matrix based on the quantity of historical data and the quantity of attributes of the historical data; algorithm module 61 is also specifically used to perform linear decomposition on the matrix to obtain numerical consistency rules; wherein, the numerical consistency rules characterize the numerical consistency rules between different attributes, and the numerical consistency rules include numerical consistency vectors.

[0068] In terms of scenario examples, the attributes of historical data include, but are not limited to, unit price, quantity, total price, cost, and points. The consistency of values ​​refers to a linear relationship, such as unit price * quantity = total price.

[0069] As an feasible approach, such as Figure 3 As shown, Figure 3 Here's an example of building a matrix. For A data points, each data point has B attributes, resulting in an A*B matrix. The coefficients representing the numerical consistency rule are represented by a vector n. For instance, if attribute b1*2 + attribute b2 = attribute 4 for each data point, then we get vector n1 = [2,1,0,-1,0,0,0]. If attribute b2 - attribute b5*3 = attribute b6 for each data point, then we get vector n2 = [0,1,-3,0,0,-1,0]. If any data point * vector n = 0, it means that the sum of the coefficients at the corresponding positions in vector n for each attribute of that data point equals 0, indicating that the data point conforms to the numerical consistency rule corresponding to vector n.

[0070] As an example, algorithm module 61 is used to change the sign of a vector so that the number of positive coefficients in the vector is not less than the number of negative coefficients.

[0071] For example, for a vector n2 = [0,1,-3,0,0,-1,0], use -n2 = [0,-1,3,0,0,1,0] as the numerical consistency rule.

[0072] It is understandable that the number of positive numbers in the coefficients of a vector is no less than the number of negative numbers, which can improve readability and facilitate manual verification.

[0073] Based on the above implementation methods, numerical consistency algorithms can be used to comprehensively generate numerical consistency patterns from historical data, thereby improving the effectiveness of data inspection.

[0074] In one example, the data inspection device further includes a filtering module 64 for removing historical abnormal data from the historical data.

[0075] In a scenario example, given A historical data points, if A-1 data points conform to the numerical consistency rule, and one data point does not, then that one data point is an outlier. If the numerical consistency rule is generated directly from A historical data points, the outlier will interfere with the rule, leading to inaccuracies in the numerical consistency rule.

[0076] As an feasible approach, the filtering module 64 removes historical outlier data from the historical data to obtain historical normal data. A matrix is ​​then constructed based on the number of historical normal data points and the number of attributes associated with them. The matrix is ​​then linearly decomposed to obtain the numerical consistency patterns.

[0077] Based on the above implementation methods, by removing historical abnormal data, the interference of historical abnormal data on the generation of numerical consistency patterns can be avoided, thereby improving the accuracy of numerical consistency patterns.

[0078] Furthermore, in one example, the data inspection device further includes: an iteration module 65, configured to establish a matrix based on the quantity of historical data and the quantity of attributes of the historical data; the iteration module 65 is also configured to perform iterative linear fitting on the matrix to obtain a sparse matrix including the historical abnormal data.

[0079] As an feasible approach, such as Figure 4 As shown, Figure 4 This example demonstrates the generation of a sparse matrix. Iteration module 65 performs linear fitting on the matrix built from historical data to obtain the numerical consistency rule and the sparse matrix. The sparse matrix includes historical outliers that do not conform to the numerical consistency rule, which can affect the accuracy of the rule. By determining whether the numerical consistency rule converges to a linear relationship, the degree of influence of historical outliers on the rule is determined. If the numerical consistency rule does not converge to a linear relationship, it indicates that historical outliers severely affect the accuracy of the rule. In this case, the sparse matrix is ​​removed from the matrix, and iterative linear fitting is performed until the numerical consistency rule converges to a linear relationship. The sparse matrix generated in the current iteration is then output.

[0080] As an example, the iteration module 65 is used to stop the iteration and output the sparse matrix if the amount of historical outlier data in the sparse matrix generated by the iteration is less than a preset threshold.

[0081] Based on the above implementation methods, iterative linear fitting can improve the accuracy of generating sparse matrices through multiple iterations, thereby enhancing the accuracy of numerical consistency patterns.

[0082] In one example, the data inspection device further includes: an execution module 66, used to assemble the numerical consistency rules into a data quality inspection statement based on a structured query language; the execution module 66 is also used to execute the data quality inspection statement on the current data to obtain the abnormal data.

[0083] To illustrate with a scenario example, such as Figure 5 As shown, Figure 5 This is an example of data inspection. Module 66 reads the numerical consistency rules and assembles them into a data quality check statement in Structured Query Language, for example: `SELECT * FROM order_table WHERE order_table.total_price<> order_table.unit_price* order_table.quantity`. Here, the current data comes from the order table, the numerical consistency rule is unit_price*quantity = total_price, and the data quality check statement indicates that abnormal data that does not conform to the rule `unit_price*quantity = total_price` is extracted from the order table.

[0084] It should be noted that this application does not limit the specific form of the data quality check statement.

[0085] Based on the above implementation methods, by using numerical consistency rules, a comprehensive check can be performed before quality problems occur in the current data, thereby improving the effectiveness of data inspection.

[0086] In one example, the verification module 62 is also used to remove numerical consistency patterns that failed manual verification.

[0087] In the context of a scenario, patterns that pass manual verification have practical significance, while numerical consistency patterns that conform to linear patterns but have no practical significance are deleted.

[0088] Based on the above implementation methods, manual verification can improve the accuracy of numerical consistency patterns.

[0089] In the data inspection device provided in this embodiment, the algorithm module is used to obtain numerical consistency patterns based on historical data using a numerical consistency algorithm; the verification module is used to take the numerical consistency patterns that have passed manual verification as numerical consistency rules; and the inspection module is used to inspect the current data using the numerical consistency rules to identify abnormal data. This solution can comprehensively generate numerical consistency rules and identify abnormal data before quality problems arise in the current data, thereby improving the effectiveness of data inspection.

[0090] Example 3

[0091] Figure 7This is a block diagram of a data inspection apparatus according to an exemplary embodiment. The apparatus may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0092] The device 800 may include one or more of the following components: a processing component 802, a memory 804, a power supply component 806, a multimedia component 808, an audio component 810, an input / output interface 812, a sensor component 814, and a communication component 816.

[0093] Processing component 802 typically controls the overall operation of device 800, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 802 may include one or more processors 820 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 802 may include one or more modules to facilitate interaction between processing component 802 and other components. For example, processing component 802 may include a multimedia module to facilitate interaction between multimedia component 808 and processing component 802.

[0094] Memory 804 is configured to store various types of data to support the operation of device 800. Examples of such data include instructions for any application or method operating on device 800, contact data, phonebook data, messages, pictures, videos, etc. Memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random-Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0095] Power supply component 806 provides power to various components of device 800. Power supply component 806 may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to device 800.

[0096] Multimedia component 808 includes a screen that provides an output interface between the device 800 and the user. In some embodiments, the screen may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a Touch Panel, the screen may be implemented as a touchscreen to receive input signals from the user. The Touch Panel includes one or more touch sensors to sense touches, swipes, and gestures on the Touch Panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 808 includes a front-facing camera and / or a rear-facing camera. When the device 800 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0097] Audio component 810 is configured to output and / or input audio signals. For example, audio component 810 includes a microphone (MIC) configured to receive external audio signals when device 800 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 804 or transmitted via communication component 816. In some embodiments, audio component 810 also includes a speaker for outputting audio signals.

[0098] Input / output interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0099] Sensor assembly 814 includes one or more sensors for providing status assessments of various aspects of device 800. For example, sensor assembly 814 may detect the on / off state of device 800, the relative positioning of components such as the display and keypad of device 800, changes in the position of device 800 or a component of device 800, the presence or absence of user contact with device 800, the orientation or acceleration / deceleration of device 800, and temperature changes of device 800. Sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 814 may also include an optical sensor, such as a complementary metal-oxide-semiconductor (CMOS) image sensor or a charge-coupled device (CCD), for use in imaging applications. In some embodiments, sensor assembly 814 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.

[0100] Communication component 816 is configured to facilitate wired or wireless communication between device 800 and other devices. Device 800 can access wireless networks based on communication standards, such as WiFi, 2G, 3G, 4G, or 5G, or combinations thereof. In one exemplary embodiment, communication component 816 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 816 further includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, NFC modules can be implemented based on Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra-Wide Band (UWB), Bluetooth, and other technologies.

[0101] In an exemplary embodiment, the apparatus 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0102] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, which can be executed by a processor 820 of the device 800 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0103] Example 4

[0104] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application, such as... Figure 8 As shown, the electronic device includes:

[0105] The electronic device includes a processor 291 and a memory 292; it may also include a communication interface 293 and a bus 294. The processor 291, memory 292, and communication interface 293 can communicate with each other via the bus 294. The communication interface 293 can be used for information transmission. The processor 291 can invoke logical instructions stored in the memory 292 to execute the methods of the above embodiments.

[0106] Furthermore, the logic instructions in the aforementioned memory 292 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium.

[0107] The memory 292, as a computer-readable storage medium, can be used to store software programs and computer-executable programs, such as program instructions / modules corresponding to the methods in the embodiments of this application. The processor 291 executes functional applications and data processing by running the software programs, instructions, and modules stored in the memory 292, thereby implementing the methods in the above-described method embodiments.

[0108] The memory 292 may include a program storage area and a data storage area. The program storage area may store the operating system and application programs required for at least one function; the data storage area may store data created based on the use of the terminal device. Furthermore, the memory 292 may include high-speed random access memory and may also include non-volatile memory.

[0109] This application provides a non-transitory computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the methods described in the foregoing embodiments.

[0110] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0111] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A data inspection method, characterized in that, include: A matrix is ​​constructed based on the amount of historical data and the number of attributes of the historical data. The matrix is ​​linearly decomposed to obtain the numerical consistency rule; wherein, the numerical consistency rule represents the numerical consistency rule between different attributes, and the numerical consistency rule includes a numerical consistency vector; the number of positive numbers in the coefficients of the numerical consistency vector is not less than the number of negative numbers; the numerical consistency vector is the coefficient vector of each attribute of the corresponding data, and if the sum of the product of each attribute value of a single data point and the corresponding coefficient of the vector is 0, then the data point conforms to the corresponding numerical consistency rule; The numerical consistency patterns that have passed manual verification will be used as numerical consistency rules. Remove patterns of numerical consistency that failed manual verification; The current data is checked using the numerical consistency rules to identify abnormal data. The step of checking the current data through the numerical consistency rules to obtain abnormal data includes: Based on the structured query language, the numerical consistency rules are assembled into data quality check statements; The data quality check statement is executed on the current data to obtain the abnormal data.

2. The method according to claim 1, characterized in that, Before establishing the matrix based on the quantity of historical data and the quantity of attributes of the historical data, the method further includes: Remove historical outlier data from the historical data.

3. The method according to claim 2, characterized in that, Before removing historical outlier data from the historical data, the process also includes: Establish a matrix based on the quantity of historical data and the number of attributes of the historical data; The matrix is ​​iteratively linearly fitted to obtain a sparse matrix that includes the historical anomaly data.

4. A data inspection device, characterized in that, include: The algorithm module is used to construct a matrix based on the quantity of historical data and the quantity of attributes of the historical data; perform linear decomposition on the matrix to obtain numerical consistency rules; wherein, the numerical consistency rules represent the numerical consistency rules between different attributes, and the numerical consistency rules include numerical consistency vectors; the number of positive numbers in the coefficients of the numerical consistency vectors is not less than the number of negative numbers; the numerical consistency vector is the coefficient vector of each attribute of the corresponding data, and if the sum of the product of each attribute value of a single data point and the corresponding coefficient of the vector is 0, then the data point conforms to the corresponding numerical consistency rule; the verification module is used to take the numerical consistency rules that have passed manual verification as numerical consistency rules; and delete the numerical consistency rules that have failed manual verification. The inspection module is used to check the current data according to the numerical consistency rules and obtain abnormal data; The execution module is used to assemble the numerical consistency rules into data quality check statements based on the structured query language. The execution module is further configured to execute the data quality check statement on the current data to obtain the abnormal data.

5. The apparatus according to claim 4, characterized in that, The device further includes: The filtering module is used to remove historical abnormal data from the historical data.

6. The apparatus according to claim 5, characterized in that, The device further includes: The iteration module is used to build a matrix based on the quantity of historical data and the quantity of attributes of the historical data; The iterative module is further configured to perform iterative linear fitting on the matrix to obtain a sparse matrix including the historical anomaly data.

7. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1-3.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-3.

Images