Data quality detection method and related device

Abstract

The invention discloses a data quality detection method and a related device, and relates to the field of data quality detection, and the method comprises the steps: obtaining the data demand information of a data demand side and the data condition information of a data provider; adopting a pre-trained artificial intelligence large model to generate a data quality detection rule according to the data demand information and the data condition information, and determining an evaluation index corresponding to the data quality detection rule from an index system conforming to a standard specification; performing quality detection on the to-be-evaluated data according to the data quality detection rule to obtain a quality detection result; and according to the quality detection result and an evaluation index corresponding to the data quality detection rule, generating a quality detection report corresponding to the to-be-evaluated data. According to the invention, quality detection can be automatically carried out and the quality detection report can be generated based on the information provided by the data supply and demand parties and the artificial intelligence large model, the whole process does not need artificial participation, and the labor cost is saved.

Description

Technical Field

[0001] The present application relates to the field of data quality detection, and in particular to a data quality detection method and related devices. Background Art

[0002] With the advent of the data element era, data has become an important asset for enterprises and institutions. Data quality, as the core indicator of data, plays an important role in data transactions and data asset registration in the direction of data assets.

[0003] At present, the data quality testing method in the data asset direction is: the testing personnel formulate data quality testing rules and evaluation indicators based on personal experience, and then perform quality testing on the data to be evaluated based on the data quality testing rules and evaluation indicators, and finally write a quality testing report based on the quality testing results. Since the entire data quality testing process is completed manually, the labor cost is extremely high, and the data quality testing process also has the problem of non-standard testing process. Summary of the invention

[0004] In view of the above problems, the present application provides a data quality detection method and related devices to solve the problem of high labor costs in the prior art. The specific solution is as follows:

[0005] The first aspect of the present application provides a data quality detection method, comprising:

[0006] Obtain data demand information from data demanders and data status information from data providers;

[0007] Using a pre-trained artificial intelligence big model to generate data quality detection rules according to the data demand information and the data situation information, and determining the evaluation indicators corresponding to the data quality detection rules from an indicator system that complies with standard specifications;

[0008] Performing quality inspection on the data to be evaluated according to the data quality inspection rules to obtain quality inspection results;

[0009] A quality inspection report corresponding to the data to be evaluated is generated according to the quality inspection result and the evaluation index corresponding to the data quality inspection rule.

[0010] In a possible implementation, performing quality detection on the data to be evaluated according to the data quality detection rule includes:

[0011] Obtaining backup data of the data to be evaluated;

[0012] Performing quality inspection on the backup data according to the data quality inspection rule.

[0013] In a possible implementation, the step of obtaining backup data of the data to be evaluated includes:

[0014] At least one storage object to be reviewed where the data to be reviewed is located is connected through a preset data set connector to obtain backup data of the at least one data to be reviewed from the storage object to be reviewed.

[0015] In a possible implementation, the pre-trained artificial intelligence big model generates data quality detection rules according to the data demand information and the data situation information, and determines the evaluation indicators corresponding to the data quality detection rules from an indicator system that conforms to the standard specification, including:

[0016] Preprocessing the data demand information and the data situation information according to a preset standard format to obtain preprocessed demand situation summary information;

[0017] Inputting the demand situation summary information into the artificial intelligence big model to obtain a data quality detection rule output by the artificial intelligence big model that is semantically consistent with the demand situation summary information;

[0018] The demand situation summary information and the data quality detection rules are input into the artificial intelligence big model, so that the artificial intelligence big model can filter out the evaluation indicators corresponding to the data quality detection rules from the indicator system that meets the standard specifications.

[0019] In a possible implementation, the pre-trained artificial intelligence big model generates data quality detection rules according to the data demand information and the data situation information, and determines the evaluation indicators corresponding to the data quality detection rules from an indicator system that conforms to the standard specification, including:

[0020] Determining whether the correlation between the data situation information and the data demand information is greater than a preset correlation threshold;

[0021] If so, the artificial intelligence big model is used to generate data quality detection rules according to the data demand information and the data situation information, and the evaluation indicators corresponding to the data quality detection rules are determined from the indicator system that meets the standard specifications.

[0022] In a possible implementation, the data demand information includes one or more of the following information: usage demand information, industry demand information, and policy demand information;

[0023] And / or, the data situation information includes one or more of the following information: data source, data category, data level, data generation method, data update frequency, data storage method, data business scenario, data scale, data dictionary and metadata information.

[0024] A second aspect of the present application provides a data quality detection device, comprising:

[0025] A data information collection module is used to obtain data demand information from data demanders and data status information from data providers;

[0026] A rule indicator determination module, used to generate data quality detection rules according to the data demand information and the data situation information using a pre-trained artificial intelligence big model, and determine the evaluation indicators corresponding to the data quality detection rules from an indicator system that conforms to standard specifications;

[0027] A data quality detection module is used to perform quality detection on the data to be evaluated according to the data quality detection rules to obtain quality detection results;

[0028] The detection report generation module is used to generate a quality detection report corresponding to the data to be evaluated based on the quality detection results and the evaluation indicators corresponding to the data quality detection rules.

[0029] A third aspect of the present application provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the data quality detection method of the first aspect or any implementation of the first aspect.

[0030] A fourth aspect of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0031] The memory is used to store computer programs;

[0032] The processor is used to execute the computer program so that the electronic device can implement the data quality detection method of the first aspect or any implementation manner of the first aspect.

[0033] In a fifth aspect, the present application provides a computer storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the data quality detection method of the first aspect or any implementation method of the first aspect.

[0034] By means of the above technical scheme, the data quality detection method provided by the present application obtains the data demand information of the data demander and the data situation information of the data provider, uses the pre-trained artificial intelligence big model to generate data quality detection rules according to the data demand information and data situation information, and determines the evaluation index corresponding to the data quality detection rule from the index system that meets the standard specification, performs quality detection on the data to be evaluated according to the data quality detection rule, obtains the quality detection result, and generates the quality detection report corresponding to the data to be evaluated according to the quality detection result and the evaluation index corresponding to the data quality detection rule. It can be seen that the present application can automatically generate data quality detection rules based on the information provided by both the data supply and demand parties, using the artificial intelligence big model, so that the quality detection report can be recognized by both the data supply and demand parties, and improves the credibility of the quality detection report. The present application can determine the evaluation index corresponding to the data quality detection rule from the index system that meets the standard specification, so that the quality detection report is generated based on the evaluation index, and the credibility of the report as a whole is further improved. The entire detection and report generation process does not require human participation, saving labor costs. In addition, the data quality detection process of the present application is a fully automated process, does not require human participation, and the data detection process is more standardized. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0036] Figure 1 A schematic diagram of a system architecture provided for this application;

[0037] Figure 2 A schematic diagram of an optional hardware structure of the terminal 100 provided in this application;

[0038] Figure 3 A schematic diagram of the structure of a server 200 provided in this application;

[0039] Figure 4 A flowchart of a data quality detection method provided in this application;

[0040] Figure 5 A flowchart of another data quality detection method provided in this application;

[0041] Figure 6 A schematic diagram of the structure of a data quality detection device provided in this application;

[0042] Figure 7 A schematic diagram of the structure of an electronic device provided in this application. DETAILED DESCRIPTION

[0043] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0044] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0045] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0046] See also Figure 1 , Figure 1 A schematic diagram of a system architecture is shown. The system may include a terminal 100 and a server 200. The server 200 may include one or more servers ( Figure 1 In the example, a server is included, and the server 200 can provide the method provided in the embodiment of the present application for one or more terminals.

[0047] Among them, an application can be installed on the terminal 100, and the above application and web page can provide an interface. The terminal 100 can receive relevant parameters entered by the user on the interface and send the above parameters to the server 200. The server 200 can obtain processing results based on the received parameters and return the processing results to the terminal 100.

[0048] It should be understood that in some optional implementations, the terminal 100 can also complete the action of obtaining the processing result based on the received parameters by itself without the cooperation of the server, and the embodiments of the present application are not limited to this.

[0049] Next describe Figure 1 The product form of the mid-terminal 100;

[0050] The terminal 100 in the embodiment of the present application can be a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc., and the embodiment of the present application does not impose any restrictions on this.

[0051] Figure 2 An optional hardware structure diagram of the terminal 100 is shown.

[0052] refer to Figure 2 As shown, the terminal 100 may include a radio frequency unit 110, a memory 120, an input unit 130, a display unit 140, a camera 150 (optional), an audio circuit 160 (optional), a speaker 161 (optional), a microphone 162 (optional), an earphone jack 163 (optional), a processor 170, an external interface 180, a power supply 190 and other components. Those skilled in the art will appreciate that Figure 2 These are merely examples of terminals or multi-function devices and do not constitute limitations on the terminals or multi-function devices, which may include more or fewer components than those shown in the figures, or combinations of certain components, or different components.

[0053] The input unit 130 can be used to receive input digital or character information, and generate key signal input related to the user settings and function control of the portable multifunctional device. Specifically, the input unit 130 may include a touch screen 131 (optional) and / or other input devices 132. The touch screen 131 can collect the user's touch operations on or near it (such as the user's operation on or near the touch screen using any suitable object such as fingers, joints, stylus, etc.), and drive the corresponding connection device according to a pre-set program. The touch screen can detect the user's touch action on the touch screen, convert the touch action into a touch signal and send it to the processor 170, and can receive and execute the command sent by the processor 170; the touch signal at least includes the touch point coordinate information. The touch screen 131 can provide an input interface and an output interface between the terminal 100 and the user. In addition, the touch screen can be implemented using multiple types such as resistive, capacitive, infrared and surface acoustic wave. In addition to the touch screen 131, the input unit 130 can also include other input devices. Specifically, other input devices 132 may include, but are not limited to, one or more of a physical keyboard, function keys (such as a volume control key, a switch key, etc.), a trackball, a mouse, a joystick, and the like.

[0054] Among them, the input device 132 can receive input data and the like.

[0055] The display unit 140 may be used to display information input by a user or provided to a user, various menus of the terminal 100, an interactive interface, file display, and / or playback of any multimedia file.

[0056] The memory 120 can be used to store instructions and data. The memory 120 can mainly include an instruction storage area and a data storage area. The data storage area can store various data, such as multimedia files, texts, etc.; the instruction storage area can store software units such as operating systems, applications, instructions required for at least one function, or their subsets and extensions. It can also include a non-volatile random access memory; provide the processor 170 with hardware, software and data resources including management of computing and processing equipment, and support control software and applications. It is also used for the storage of multimedia files, and the storage of running programs and applications.

[0057] The processor 170 is the control center of the terminal 100. It uses various interfaces and lines to connect various parts of the entire terminal 100. By running or executing instructions stored in the memory 120 and calling data stored in the memory 120, it executes various functions of the terminal 100 and processes data, thereby controlling the terminal device as a whole. Optionally, the processor 170 may include one or more processing units; preferably, the processor 170 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application program, and the modem processor mainly processes wireless communication. It is understandable that the above-mentioned modem processor may not be integrated into the processor 170. In some embodiments, the processor and the memory may be implemented on a single chip, and in some embodiments, they may also be implemented separately on separate chips. The processor 170 may also be used to generate corresponding operation control signals, send them to corresponding components of the computing and processing device, read and process data in the software, especially read and process data and programs in the memory 120, so that each functional module therein performs corresponding functions, thereby controlling the corresponding components to act according to the requirements of the instructions.

[0058] Among them, the memory 120 can be used to store software codes related to the data quality detection method, the processor 170 can execute the steps of the data quality detection method, and can also schedule other units (such as the above-mentioned input unit 130 and display unit 140) to achieve corresponding functions.

[0059] The radio frequency unit 110 (optional) can be used for receiving and sending information or receiving and sending signals during a call, for example, after receiving the downlink information of the base station, it is sent to the processor 170 for processing; in addition, the designed uplink data is sent to the base station. Generally, the RF circuit includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (Low Noise Amplifier, LNA), a duplexer, etc. In addition, the radio frequency unit 110 can also communicate with network devices and other devices through wireless communication. The wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobile Communication (Global System of Mobile Communication, GSM), General Packet Radio Service (General Packet Radio Service, GPRS), Code Division Multiple Access (Code Division Multiple Access, CDMA), Wideband Code Division Multiple Access (Wideband Code Division Multiple Access, WCDMA), Long Term Evolution (Long Term Evolution, LTE), email, Short Messaging Service (SMS), etc.

[0060] In this embodiment of the present application, the RF unit 110 can send data to the server 200 and receive processing results sent by the server 200.

[0061] It should be understood that the radio frequency unit 110 is optional and can be replaced by other communication interfaces, such as a network port.

[0062] The terminal 100 also includes a power supply 190 (such as a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the processor 170 through a power management system, so that the power management system can manage functions such as charging, discharging, and power consumption.

[0063] The terminal 100 further includes an external interface 180 , which may be a standard Micro USB interface or a multi-pin connector, and may be used to connect the terminal 100 to communicate with other devices, or to connect a charger to charge the terminal 100 .

[0064] Although not shown, the terminal 100 may also include a flashlight, a wireless fidelity (WiFi) module, a Bluetooth module, sensors with different functions, etc., which are not described in detail here. Some or all of the methods described below may be applied in the following embodiments. Figure 2 In the terminal 100 shown.

[0065] Next describe Figure 1 The product form of the server 200;

[0066] Figure 3 A schematic diagram of the structure of a server 200 is provided, such as Figure 3 As shown, the server 200 includes a bus 201, a processor 202, a communication interface 203, and a memory 204. The processor 202, the memory 204, and the communication interface 203 communicate with each other via the bus 201.

[0067] The bus 201 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0068] The processor 202 may be any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0069] The memory 204 may include a volatile memory, such as a random access memory (RAM). The memory 204 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0070] The memory 204 may be used to store software codes related to the data quality detection method, and the processor 202 may execute the steps of the chip data quality detection method, and may also schedule other units to implement corresponding functions.

[0071] It should be understood that the above-mentioned terminal 100 and server 200 can be centralized or distributed devices, and the processors in the above-mentioned terminal 100 and server 200 (such as processor 170 and processor 202) can be hardware circuits (such as application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), general-purpose processors, DSPs, microprocessors or microcontrollers, etc.), or a combination of these hardware circuits. For example, the processor can be a hardware system with an instruction execution function, such as a CPU, DSP, etc., or a hardware system without an instruction execution function, such as an ASIC, FPGA, etc., or a combination of the above-mentioned hardware systems without an instruction execution function and hardware systems with an instruction execution function.

[0072] The present application provides a data quality detection method, which can be applied to a detection system. In order to enable those skilled in the art to better understand the present application, the data quality detection method of the present application embodiment is described in detail below in conjunction with the accompanying drawings.

[0073] Reference Figure 4 , Figure 4 A flow chart of a data quality detection method provided in an embodiment of the present application, the method may include:

[0074] Step S401: Obtain data demand information of a data demander and data status information of a data provider.

[0075] Here, the data demander refers to the party that needs to use the data; the data provider refers to the party that produces the data.

[0076] When a data demander wants to use the data of a data provider, the detection system can collect the data demand information of the data demander, that is, collect the data demander's demand or requirement information for the data to be used; at the same time, the detection system can also collect the data situation information of the data provider, that is, collect information describing the details of the data.

[0077] Optionally, the testing system may initiate a data evaluation task through an online method such as a meeting. The data demander may provide its own data demand information online, and the data provider may also provide its own data status information online. Thus, the testing system may obtain the data demand information and data status information.

[0078] It should be noted that the above-mentioned online method is only an example. In addition to this, other methods can also be used. For example, the testing personnel collect data demand information and data status information through offline methods such as visits, and then enter the information into the testing system. In this way, the testing system can obtain the data demand information and data status information.

[0079] It should also be noted that the embodiments of the present application do not limit the specific forms of expression of the data demand information and the data status information. In actual applications, the data demand information and the data status information may be in the form of voice, image, text, etc.

[0080] Optionally, the data demand information may be one or more of the following information: usage demand information, industry demand information, and policy demand information. Here, usage demand information refers to the demand information generated in the process of using data, for example, in the scenario of verifying dishonest persons, the ID number of each person to be verified needs to be complete; industry demand information refers to the demand information of the industry to which the data demander belongs for the data used, for example, the banking industry requires the ID number to be complete; policy demand information refers to the requirement information of data specified in official policies, for example, official policies stipulate that the ID numbers collected by banks must be complete and accurate.

[0081] Optionally, the data status information includes one or more of the following information: data source, data category, data level, data generation method, data update frequency, data storage method, data business scenario, data scale, data dictionary and metadata information.

[0082] Optionally, data sources may include self-development, agreement acquisition, public collection, and other sources.

[0083] Self-Developed: Data is independently developed or generated using resources and technology within the organization, including data collected through internal research, experimentation, or development of applications and services.

[0084] Acquisition by agreement: Data is obtained through data sharing agreements, purchase agreements, or public data authorization operations signed with third parties. This source requires a clear agreement, as well as data ownership and restrictions.

[0085] Publicly collected: Data is collected from publicly available sources, such as public datasets released by a party recognized by industry professionals (such as an official), published research reports, or public information on the Internet.

[0086] Others: The data source does not belong to any of the above situations.

[0087] Optionally, data categories may include public data, enterprise data, and personal data.

[0088] Public data: refers to data held or generated by public institutions, which can be accessed and used by the public. For example, public data includes public service data such as water, electricity, gas, etc.

[0089] Enterprise data: refers to data generated or collected by an enterprise in its business activities. This enterprise data is usually directly related to the operation and management of the enterprise.

[0090] Personal data: refers to data that can identify a specific individual, including but not limited to name, ID number, contact information and other personal information.

[0091] Optionally, the data levels may include general data, important data, and core data.

[0092] General data: other data other than core data and important data.

[0093] Important data: data in specific fields, specific groups, specific regions, or data that reaches a certain accuracy and scale, which once leaked, tampered with, or destroyed, may directly endanger official security, economic operations, social stability, public health and safety.

[0094] Core data: important data that has high coverage of fields, groups, or regions, or has high accuracy, large scale, and a certain depth. Once it is illegally used or shared, it may directly affect official security.

[0095] Optionally, the generation methods may include: manually collected data, data generated by information systems, data generated by sensing devices, original data, secondary processed data and others.

[0096] Manually collected data: Data is collected manually, such as through questionnaires, interviews or observations.

[0097] Information system generated data: Data is automatically generated by information systems or software, such as transaction records, log files, etc.

[0098] Perception devices generate data: Data is collected by perception devices such as sensors and cameras.

[0099] Raw data: Data is unprocessed data obtained directly from the source, maintaining its original form.

[0100] Secondary processed data: data that has undergone some processing or analysis, for example, the original data has been summarized, calculated or converted.

[0101] Optionally, the update frequency may include: real-time update, update by second, update by hour, update by day, update by month and no update.

[0102] Optionally, storage methods may include: relational database, key-value database and column database.

[0103] Business scenarios refer to business scenarios in which data is used, such as financial transaction business use, community group purchasing business use, water company business use, and so on.

[0104] Data scale refers to the size of the data, such as 1MB, 1G, 1TB, etc.

[0105] A data dictionary refers to a specification for certain standard data, such as gender data can only be male and female, province data can only be one of the 34 provinces, and so on.

[0106] Metadata information refers to information that can describe basic attributes of data. Optionally, metadata information may include: database table creation statements, Excel column information, etc.

[0107] It should be noted that the above-mentioned data demand information and data status information may also be other, and this application does not specifically limit them.

[0108] Step S402: Use a pre-trained artificial intelligence big model to generate data quality detection rules based on data demand information and data situation information, and determine evaluation indicators corresponding to the data quality detection rules from an indicator system that meets standard specifications.

[0109] Specifically, this embodiment can pre-train an artificial intelligence big model (i.e., an AI big model) that can fully understand the semantic meaning of natural language information, so that the pre-trained artificial intelligence big model can automatically generate data quality detection rules for quality detection of the data to be evaluated based on data demand information and data situation information. Here, the data quality detection rules refer to the requirements for the data to be evaluated, such as various data requirements such as non-repetitive, non-zero, and must be unique.

[0110] For example, the data to be evaluated is a bus data set, and the data demander requires that the bus routes in the bus data set are mandatory items, then the bus route field in the data quality detection rule cannot be empty; for another example, the data demander requires that the departure time of this bus must be within 2024, then the bus departure time in the data quality detection rule must be greater than or equal to 2024-01-01 00:00:00; for another example, the data provider stipulates that there must be specific departure information in the bus data set, then the departure field in the data quality detection rule cannot be empty.

[0111] In this embodiment, the artificial intelligence big model can also obtain an indicator system that meets the standard specifications, which includes multiple indicators that meet the standard specifications and specifies the scope of rules for the application of the indicators.

[0112] Optionally, the above-mentioned indicator system that complies with standard specifications can be an officially released indicator system, such as "GBT36344-2018 Information Technology Data Quality Evaluation Indicators", or it can be an indicator system recognized by the industry for both data providers and data demanders.

[0113] Furthermore, the artificial intelligence big model can filter out evaluation indicators corresponding to data quality detection rules from the acquired indicator system based on data demand information, data situation information and data quality detection rules.

[0114] For example, if the data demand information or data status information stipulates that information a must be unique, then the evaluation indicators screened out by the artificial intelligence big model may include uniqueness indicators; for another example, for the banking industry, the ID card number needs to be accurate, then the evaluation indicators screened out by the artificial intelligence big model may include accuracy indicators.

[0115] Optionally, the evaluation indicators determined in this embodiment may include various hierarchical indicators such as primary indicators and secondary indicators. In order to facilitate the subsequent generation of reports, this embodiment may also pre-assign weights to indicators of each level through the artificial intelligence big model. For example, after learning the importance of indicators of each level in similar scenarios, the artificial intelligence big model assigns weights to indicators of each level determined in the current scenario based on the learned information.

[0116] Of course, the process of assigning weights to evaluation indicators may not be implemented through a large artificial intelligence model, but may be implemented by manually inputting the indicator weights into the detection system, or there may be other implementation methods, which are not limited in this application.

[0117] Step S403: Perform quality inspection on the data to be evaluated according to data quality inspection rules to obtain quality inspection results.

[0118] Optionally, the quality inspection result includes: the total amount of data to be evaluated and the amount of problems in the data to be evaluated.

[0119] Of course, the quality inspection results may also include other information, which is not specifically limited in this application.

[0120] Step S404: Generate a quality inspection report corresponding to the data to be evaluated based on the quality inspection results and the evaluation indicators corresponding to the data quality inspection rules.

[0121] As described above, in this embodiment, weights can be assigned to the evaluation indicators in advance. Then, in this step, weighted processing can be performed according to the weights of the evaluation indicators corresponding to the quality inspection results and the data quality inspection rules, and a quality inspection report corresponding to the data to be evaluated can be obtained based on the weighted processing results (including but not limited to weighted summation, weighted average, etc.).

[0122] Optionally, the quality inspection report may include the following contents: overview, problem data information, indicator selection information, indicator score information, overall score information and problem analysis information. Among them, overview refers to a summary description of the relevant information of the data to be evaluated, problem data information refers to the data and quantity with problems in the data to be evaluated, indicator selection information refers to the evaluation indicators determined by the artificial intelligence large model in step S402, indicator score information refers to the weighted score value of each evaluation indicator and the corresponding quality inspection result, overall score information refers to the sum of the indicator scores of all evaluation indicators, and problem analysis information refers to the relevant statistical analysis information of the quality inspection results, indicator scores and overall scores, for example, what kind of problem the data with problems in the data to be evaluated is, the proportion of problem data, etc.

[0123] The data quality detection method provided by the present application obtains the data demand information of the data demander and the data situation information of the data provider, uses a pre-trained artificial intelligence big model to generate data quality detection rules according to the data demand information and data situation information, and determines the evaluation index corresponding to the data quality detection rule from the index system that meets the standard specifications, performs quality detection on the data to be evaluated according to the data quality detection rule, obtains the quality detection result, and generates the quality detection report corresponding to the data to be evaluated according to the quality detection result and the evaluation index corresponding to the data quality detection rule. It can be seen that the present application can automatically generate data quality detection rules based on the information provided by both the data supply and demand parties, using the artificial intelligence big model, so that the quality detection report can be recognized by both the data supply and demand parties, and improves the credibility of the quality detection report. The present application can determine the evaluation index corresponding to the data quality detection rule from the index system that meets the standard specifications, so that the quality detection report is generated based on the evaluation index, and the credibility of the report as a whole is further improved. The entire detection and report generation process does not require human participation, saving labor costs.

[0124] Based on the above, this application also provides another data quality detection method, such as Figure 5 , which is a flow chart of another data quality detection method provided in an embodiment of the present application.

[0125] like Figure 5 As shown, the data quality detection method provided in the embodiment of the present application may include:

[0126] Step S501: Obtain data demand information of a data demander and data status information of a data provider.

[0127] Step S502: determine whether the correlation between the data situation information and the data demand information is greater than a preset correlation threshold.

[0128] In a possible implementation, considering that the data demand information contains the requirements of the data demander for the data to be evaluated, and the data situation information contains the specific situation of the data corresponding to the requirements, if the data to be evaluated provided by the data provider meets the requirements of the data demander, then the data demand information and the data situation information should have a certain correlation.

[0129] Based on this, this embodiment can calculate the correlation between the data situation information and the data demand information, and determine whether the correlation meets the preset correlation threshold. If so, it means that the data situation information is basically consistent with the data demand information, that is, the data to be evaluated may meet the requirements of the data demander, then the following step S503 can be executed.

[0130] On the contrary, if the correlation calculated by this embodiment does not meet the preset correlation threshold, it means that the data situation information does not match the data demand information, that is, the data to be evaluated does not meet the requirements of the data demander, then step S402 and subsequent steps will no longer be executed, that is, data quality detection will no longer be performed, so as to avoid unnecessary losses to the data provider and the data demander.

[0131] In this embodiment, the process of calculating the correlation between data situation information and data demand information may be implemented in a variety of ways, and the following two ways are provided here.

[0132] The first implementation method is to pre-set a correlation calculation rule, and specify in the rule the scenarios in which the correlation between the data situation information and the data demand information can be increased. For example, if the data demand information requires a certain field (such as the train number field) to be not empty, and the field in the data situation information is empty, the correlation is reduced by 1; if the data demand information requires that the gender field can only be male or female, and the gender field in the data situation information is only male or female, the correlation is increased by 1, etc. Therefore, this embodiment can calculate the correlation between the data situation information and the data demand information based on the preset correlation calculation rule.

[0133] The second implementation method is to determine the semantic embedding vectors of the data situation information and the data requirement information through a pre-trained large language model such as the BERT (Bidirectional Encoder Representations from Transformers) model, and then calculate the similarity of the semantic embedding vectors of the data situation information and the data requirement information as the correlation between the data situation information and the data requirement information.

[0134] Of course, the calculation process of the correlation between data situation information and data demand information may also be implemented in other ways, which are not specifically limited in this application.

[0135] The above preset correlation threshold can be determined according to the actual scenario and is not specifically limited in this application.

[0136] Step S503: pre-process the data demand information and data situation information according to a preset standard format to obtain pre-processed demand situation summary information.

[0137] As mentioned above, data demand information and data situation information can be obtained through online or offline methods. In this case, the data demand information and data situation information may not be standardized enough. If the data demand information and data situation information are directly input into the artificial intelligence big model, the artificial intelligence big model may obtain incorrect data quality detection rules and evaluation indicators due to its lack of standardization.

[0138] In order to avoid the above-mentioned problems, the embodiment of the present application may first pre-process the data demand information and data situation information according to a preset standard format to obtain the pre-processed demand situation summary information.

[0139] Step S504: input the demand situation summary information into the artificial intelligence big model to obtain data quality detection rules output by the artificial intelligence big model that are semantically consistent with the demand situation summary information.

[0140] Specifically, the artificial intelligence big model can perform semantic understanding and computational analysis on the demand situation summary information, and obtain data quality detection rules that are consistent with the semantics of the demand situation summary information based on the results of the semantic understanding and computational analysis.

[0141] Step S505: Input the demand summary information and data quality detection rules into the artificial intelligence big model, so that the artificial intelligence big model can filter out evaluation indicators corresponding to the data quality detection rules from the indicator system that meets the standard specifications.

[0142] Step S506: Acquire backup data of the data to be evaluated.

[0143] Step S507: Perform quality inspection on the backup data according to data quality inspection rules.

[0144] Considering that the data to be tested in some scenarios may change in real time due to the particularity of the scenarios, for example, users can make transactions at any time, so their transaction records in the bank change in real time.

[0145] In order to avoid inconsistent quality inspection results caused by real-time changes in the data to be tested, this embodiment can back up the data to be tested. For example, if the data demand information is the user's transaction records for the whole year of 2024, then the user's transaction records for 2024 can be backed up.

[0146] Therefore, this embodiment can obtain the backup data of the data to be evaluated, and then perform quality inspection on the backup data according to the data quality inspection rule.

[0147] Since backup data is static data and will not change, the quality inspection of the backup data will not change no matter how many times it is inspected, thus avoiding a trust crisis among the data provider, data demander and data inspector.

[0148] In a possible implementation, considering that the data to be evaluated may be stored in multiple storage objects, for ease of introduction, the storage object where the data to be evaluated is located is defined as a storage object to be evaluated.

[0149] In order to quickly obtain the backup data of the data to be evaluated, this embodiment provides a data set connector, so that this embodiment can connect to at least one storage object where the data to be evaluated is located through a preset data set connector at one time to obtain the backup data of the data to be evaluated from the at least one storage object.

[0150] Optionally, the storage object to be evaluated may be an entity such as a database to be evaluated, a storage device to be evaluated, etc., which is not specifically limited in this application.

[0151] This embodiment uses a data set connector to connect multiple storage objects to be evaluated at one time, so that backup data of the data to be evaluated can be obtained from multiple storage objects to be evaluated at the same time, which improves the data acquisition speed and further improves the data detection speed.

[0152] It should also be noted that the above process of obtaining backup data of the data to be evaluated and performing quality inspection is only an example. In addition to this, there may be other implementation methods, which are not specifically limited in this application.

[0153] Step S508: Generate a quality inspection report corresponding to the data to be evaluated based on the quality inspection results and the evaluation indicators corresponding to the data quality inspection rules.

[0154] This embodiment is a solution further expanded on the basis of the previous embodiment. Parts not described in detail, such as step S501 and step S508, can be referred to the previous description and will not be repeated here.

[0155] It should also be noted that this embodiment is a more preferred embodiment for implementing data quality detection in this application, and some steps therein can also be removed, such as step S502 can be removed without affecting the overall implementation; some steps can be replaced with steps in the previous embodiments, such as steps S503~step S505 can be replaced back to step S402, steps S506~step S507 can be replaced back to step S403, and so on.

[0156] A data quality detection method provided in an embodiment of the present application is introduced above, and a device for executing the above data quality detection method will be introduced below.

[0157] See also Figure 6 , Figure 6 This is a schematic diagram of the structure of a data quality detection device provided in an embodiment of the present application. Figure 6 As shown, the device may include:

[0158] The data information collection module 601 is used to obtain the data demand information of the data demander and the data situation information of the data provider;

[0159] A rule indicator determination module 602 is used to generate data quality detection rules according to data demand information and data situation information using a pre-trained artificial intelligence big model, and determine evaluation indicators corresponding to the data quality detection rules from an indicator system that complies with standard specifications;

[0160] The data quality detection module 603 is used to perform quality detection on the data to be evaluated according to the data quality detection rules to obtain the quality detection result;

[0161] The test report generation module 604 is used to generate a quality test report corresponding to the data to be evaluated according to the quality test results and the evaluation indicators corresponding to the data quality test rules.

[0162] The data quality detection device provided by the present application, the data information collection module obtains the data demand information of the data demander and the data situation information of the data provider, the rule indicator determination module uses the pre-trained artificial intelligence big model to generate the data quality detection rule according to the data demand information and the data situation information, and determines the evaluation index corresponding to the data quality detection rule from the index system that meets the standard specification, the data quality detection module performs quality detection on the data to be evaluated according to the data quality detection rule, and obtains the quality detection result, and the detection report generation module generates the quality detection report corresponding to the data to be evaluated according to the quality detection result and the evaluation index corresponding to the data quality detection rule. It can be seen from this that the present application can automatically generate data quality detection rules based on the information provided by both the data supply and demand parties, using the artificial intelligence big model, so that the quality detection report can be recognized by both the data supply and demand parties, and the credibility of the quality detection report is improved. The present application can determine the evaluation index corresponding to the data quality detection rule from the index system that meets the standard specification, so that the quality detection report is generated based on the evaluation index, and the credibility of the report as a whole is further improved. The entire detection and report generation process does not require human participation, saving labor costs.

[0163] In a possible implementation, the data quality detection module may include: a data backup unit and a backup data detection unit.

[0164] The data backup unit is used to obtain backup data of the data to be evaluated.

[0165] The backup data detection unit is used to perform quality detection on the backup data according to the data quality detection rules.

[0166] In a possible implementation, the data backup unit may be specifically configured to: connect to at least one storage object to be reviewed where the data to be reviewed is located through a preset data set connector, so as to obtain backup data of the data to be reviewed from the at least one storage object to be reviewed.

[0167] In a possible implementation, the rule indicator determination module may include: a preprocessing unit, a rule generating unit and an indicator screening unit.

[0168] The preprocessing unit is used to preprocess the data demand information and data situation information according to a preset standard format to obtain the preprocessed demand situation summary information.

[0169] A rule generation unit is used to input the demand situation summary information into the artificial intelligence big model to obtain data quality detection rules output by the artificial intelligence big model that are semantically consistent with the demand situation summary information.

[0170] The indicator screening unit is used to input the demand summary information and data quality detection rules into the artificial intelligence big model, so that the artificial intelligence big model can screen out the evaluation indicators corresponding to the data quality detection rules from the indicator system that meets the standard specifications.

[0171] In a possible implementation, the rule indicator determination module uses a pre-trained artificial intelligence model to generate data quality detection rules according to data demand information and data situation information, and determines the evaluation indicators corresponding to the data quality detection rules from an indicator system that meets the standard specifications. The process can be specifically used for:

[0172] Determine whether the correlation between the data situation information and the data demand information is greater than a preset correlation threshold;

[0173] If so, an artificial intelligence big model is used to generate data quality detection rules based on data demand information and data situation information, and the evaluation indicators corresponding to the data quality detection rules are determined from an indicator system that meets standard specifications.

[0174] In a possible implementation, the above data demand information includes one or more of the following information: usage demand information, industry demand information and policy demand information.

[0175] In one possible implementation, the above-mentioned data situation information includes one or more of the following information: data source, data category, data level, data generation method, data update frequency, data storage method, data business scenario, data scale, data dictionary and metadata information.

[0176] The data quality detection device provided in the embodiment of the present application corresponds to the data quality detection method provided above. For details, please refer to the above introduction and will not be repeated here.

[0177] The present application also provides an electronic device in an embodiment. Figure 7 As shown, it shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiment of the present application. The electronic device in the embodiment of the present application may include but is not limited to fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 7 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0178] like Figure 7 As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 to a random access memory (RAM) 703. When the electronic device is powered on, various programs and data required for the operation of the electronic device are also stored in the RAM 703. The processing device 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0179] Typically, the following devices may be connected to the I / O interface 705: an input device 706 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 707 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 708 including, for example, a memory card, a hard disk, etc.; and a communication device 709. The communication device 709 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Figure 7 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0180] An embodiment of the present application also provides a computer program product including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any one of the data quality detection methods provided in the embodiments of the present application.

[0181] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any data quality detection method provided in the embodiment of the present application.

[0182] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0183] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0184] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0185] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

Claims

1. A data quality detection method, characterized in that: include: Obtain data demand information from data demanders and data status information from data providers; Using a pre-trained artificial intelligence big model to generate data quality detection rules according to the data demand information and the data situation information, and determining the evaluation indicators corresponding to the data quality detection rules from an indicator system that complies with standard specifications; Performing quality inspection on the data to be evaluated according to the data quality inspection rules to obtain quality inspection results; A quality inspection report corresponding to the data to be evaluated is generated according to the quality inspection result and the evaluation index corresponding to the data quality inspection rule.

2. The data quality detection method according to claim 1, characterized in that: The performing of quality inspection on the data to be evaluated according to the data quality inspection rules includes: Obtaining backup data of the data to be evaluated; Performing quality inspection on the backup data according to the data quality inspection rule.

3. The data quality detection method according to claim 2, characterized in that: The step of obtaining backup data of the data to be evaluated includes: At least one storage object to be reviewed where the data to be reviewed is located is connected through a preset data set connector to obtain backup data of the data to be reviewed from the at least one storage object to be reviewed.

4. The data quality detection method according to claim 1, characterized in that: The pre-trained artificial intelligence big model generates data quality detection rules according to the data demand information and the data situation information, and determines the evaluation indicators corresponding to the data quality detection rules from the indicator system that conforms to the standard specification, including: Preprocessing the data demand information and the data situation information according to a preset standard format to obtain preprocessed demand situation summary information; Inputting the demand situation summary information into the artificial intelligence big model to obtain a data quality detection rule output by the artificial intelligence big model that is semantically consistent with the demand situation summary information; The demand situation summary information and the data quality detection rules are input into the artificial intelligence big model, so that the artificial intelligence big model can filter out the evaluation indicators corresponding to the data quality detection rules from the indicator system that meets the standard specifications.

5. The data quality detection method according to claim 1, characterized in that: The pre-trained artificial intelligence big model generates data quality detection rules according to the data demand information and the data situation information, and determines the evaluation indicators corresponding to the data quality detection rules from the indicator system that conforms to the standard specification, including: Determining whether the correlation between the data situation information and the data demand information is greater than a preset correlation threshold; If so, the artificial intelligence big model is used to generate data quality detection rules according to the data demand information and the data situation information, and the evaluation indicators corresponding to the data quality detection rules are determined from the indicator system that meets the standard specifications.

6. The data quality detection method according to claim 1, characterized in that: The data demand information includes one or more of the following information: usage demand information, industry demand information and policy demand information; And / or, the data situation information includes one or more of the following information: data source, data category, data level, data generation method, data update frequency, data storage method, data business scenario, data scale, data dictionary and metadata information.

7. A data quality detection device, characterized in that: include: A data information collection module is used to obtain data demand information from data demanders and data status information from data providers; A rule indicator determination module, used to generate data quality detection rules according to the data demand information and the data situation information using a pre-trained artificial intelligence big model, and determine the evaluation indicators corresponding to the data quality detection rules from an indicator system that conforms to standard specifications; A data quality detection module is used to perform quality detection on the data to be evaluated according to the data quality detection rules to obtain quality detection results; The detection report generation module is used to generate a quality detection report corresponding to the data to be evaluated based on the quality detection results and the evaluation indicators corresponding to the data quality detection rules.

8. A computer program product, characterized in that It comprises computer-readable instructions, and when the computer-readable instructions are executed on an electronic device, the electronic device implements the data quality detection method as claimed in any one of claims 1 to 6.

9. An electronic device, characterized in that: The method comprises at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the electronic device can implement the data quality detection method as described in any one of claims 1 to 6.

10. A computer storage medium, characterized in that: The storage medium carries one or more computer programs, and when the one or more computer programs are executed by an electronic device, the electronic device can implement the data quality detection method as described in any one of claims 1 to 6.

Images