Data quality offline detection method and related device

Abstract

The invention discloses a data quality offline detection method and a related device, and relates to the technical field of data processing, and the method comprises the following steps: obtaining a field type to which a to-be-detected field in a to-be-detected storage object belongs; obtaining a quality detection rule corresponding to the field type; sending the field type, a quality detection rule corresponding to the field type and a pre-configured offline quality detection tool to a data party, so that the data party obtains a quality detection result of the field value of the field to be detected locally and / or in an intranet environment based on the received field type, quality detection rule and offline quality detection tool; based on the quality detection result and a rule index weight pre-bound for a quality detection rule corresponding to the field type, determining a score value related to the to-be-detected field; and generating a quality detection report based on the score value and the quality detection result. The quality detection process of the field value is carried out by the data party in the local and / or intranet environment, so that the field value is prevented from being exposed to the outside, and the data security is improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method for offline detection of data quality and a related device. Background Art

[0002] With the advent of the big data era, data has become an extremely important asset for major companies and institutions (i.e. data providers). Being able to effectively test and evaluate the quality of their own data is becoming increasingly important for data providers, and it plays an important role in facilitating data asset inventory, quality improvement, value analysis and discovery, and solving key issues such as data assetization and data factorization.

[0003] The existing data quality detection method requires first connecting to the data party's storage object through the external network, and then performing quality detection on the data stored on it. However, the method of connecting to the storage object through the external network requires exposing the data party's data to the outside world, which is prone to data leakage and confidential data being stolen by illegal software systems, causing data security risks. Summary of the invention

[0004] In view of the above problems, the present application provides a data quality offline detection method and related devices to solve the problem of data security risks caused by using external network connection to store objects in the prior art. The specific solution is as follows:

[0005] The first aspect of the present application provides a method for offline detection of data quality, comprising:

[0006] Get the field type of the field to be tested in the storage object to be tested;

[0007] Obtaining quality detection rules corresponding to the field type;

[0008] Sending the field type, the quality detection rule corresponding to the field type and the preconfigured offline quality detection tool to the data party, so that the data party can obtain the quality detection result of the field value of the field to be tested in a local and / or intranet environment based on the received field type, quality detection rule and offline quality detection tool;

[0009] Determine a score value related to the field to be tested based on the quality detection result and a rule indicator weight pre-bound to the quality detection rule corresponding to the field type;

[0010] A quality inspection report of the storage object to be tested is generated based on the score value and the quality inspection result.

[0011] In a possible implementation, the step of obtaining the field type to which the field to be tested in the storage object to be tested belongs includes:

[0012] Acquire a storage structure of the storage object to be tested, wherein the storage structure represents the data organization and storage method of the storage object to be tested;

[0013] The field type to which the field to be tested belongs is determined based on the storage structure.

[0014] In a possible implementation, the obtaining the quality detection rule corresponding to the field type includes:

[0015] Obtain data requirements corresponding to the field type;

[0016] Determine a data attribute to which the field type belongs as a target data attribute, wherein the target data attribute is one of a numerical attribute, a string attribute, and a time and date attribute;

[0017] Acquire a data logic rule item corresponding to the target data attribute, wherein the data logic rule item is a rule framework containing data logic that is customized for the field type;

[0018] Generate a first quality detection rule based on the data requirement corresponding to the field type and the data logic rule item corresponding to the target data attribute;

[0019] The quality detection rule corresponding to the field type is composed of the first quality detection rule and / or the pre-stored second quality detection rule.

[0020] In a possible implementation, the process of obtaining the pre-stored second quality detection rule includes:

[0021] Acquire a quality detection rule set corresponding to the target data attribute from pre-stored quality detection rule sets corresponding to the respective data attributes;

[0022] The second quality detection rule is obtained from a set of quality detection rules corresponding to the target data attribute.

[0023] In a possible implementation, sending the field type, the quality detection rule corresponding to the field type, and a preconfigured offline quality detection tool to the data party includes:

[0024] Generate a rule configuration file based on the field type and the quality detection rule corresponding to the field type;

[0025] Compiling the rule configuration file and the preconfigured offline quality detection tool into a binary executable tool;

[0026] The binary executable tool is sent to the data party.

[0027] In a possible implementation, compiling the rule configuration file and the preconfigured offline quality detection tool into a binary executable tool includes:

[0028] Acquiring the system environment of the data party;

[0029] The rule configuration file and the preconfigured offline quality detection tool are compiled into a binary executable tool that supports the system environment.

[0030] In a possible implementation, the preconfigured offline quality detection tool is obtained based on a program configuration file and a pregenerated offline quality detection running program, wherein the program configuration file includes connection configuration items of the storage object to be tested and control configuration items of the offline quality detection process, and the connection configuration items include the correspondence between the field to be tested, the field type and the identifier of the rule configuration file.

[0031] In a possible implementation, the method further includes:

[0032] Get error sample notification information;

[0033] If the error sample notification information indicates that the error sample is allowed to be displayed in the quality detection report, the error sample configuration item in the connection configuration item is modified to be allowed, so that the data party obtains the error sample while obtaining the quality detection result;

[0034] The step of generating a quality inspection report of the storage object to be tested based on the score value and the quality inspection result includes:

[0035] The quality inspection report is generated based on the score value, the quality inspection result and the error sample.

[0036] A second aspect of the present application provides a data quality offline detection device, comprising:

[0037] A field type acquisition module is used to obtain the field type to which the field to be tested belongs in the storage object to be tested;

[0038] A detection rule generation module, used to obtain quality detection rules corresponding to the field type;

[0039] A detection information sending module, used to send the field type, the quality detection rule corresponding to the field type and the pre-configured offline quality detection tool to the data party, so that the data party can obtain the quality detection result of the field value of the field to be tested in a local and / or intranet environment based on the received field type, quality detection rule and offline quality detection tool;

[0040] A test data scoring module, used to determine a score value related to the test field based on the quality detection result and the rule indicator weight pre-bound to the quality detection rule corresponding to the field type;

[0041] A test report generating module is used to generate a quality test report of the storage object to be tested based on the score value and the quality test result.

[0042] A third aspect of the present application provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the data quality offline detection method of the first aspect or any implementation of the first aspect.

[0043] A fourth aspect of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0044] The memory is used to store computer programs;

[0045] The processor is used to execute the computer program so that the electronic device can implement the data quality offline detection method of the first aspect or any implementation manner of the first aspect.

[0046] In a fifth aspect, the present application provides a computer storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the data quality offline detection method in the first aspect or any implementation method of the first aspect.

[0047] By means of the above technical scheme, the data quality offline detection method provided by the present application obtains the field type to which the field to be tested in the storage object to be tested belongs, obtains the quality detection rule corresponding to the field type, and sends the field type, the quality detection rule corresponding to the field type, and the pre-configured offline quality detection tool to the data party, so that the data party can obtain the quality detection result of the field value of the field to be tested based on the received field type, quality detection rule and offline quality detection tool in the local and / or intranet environment, and determine the score value related to the data in the storage object to be tested based on the quality detection result and the rule indicator weight pre-bound to the quality detection rule corresponding to the field type, and generate a quality detection report of the storage object to be tested based on the score value and the quality detection result. It can be seen from this that the present application can only be processed based on the field type to which the field to be tested in the storage object to be tested belongs, and there is no need to obtain the specific field value. The quality detection process of the field value is performed by the data party itself in the local and / or intranet environment, thereby avoiding the external exposure of the field value in the storage object to be tested and improving data security. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0049] Figure 1 A schematic diagram of a system architecture provided for this application;

[0050] Figure 2 A schematic diagram of an optional hardware structure of the terminal 100 provided in this application;

[0051] Figure 3 A schematic diagram of the structure of a server 200 provided in this application;

[0052] Figure 4 A flowchart of a data quality offline detection method provided in this application;

[0053] Figure 5 A schematic diagram of the structure of a data quality offline detection device provided in this application;

[0054] Figure 6 A schematic diagram of the structure of an electronic device provided in this application. DETAILED DESCRIPTION

[0055] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0056] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0057] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0058] See also Figure 1 , Figure 1A schematic diagram of a system architecture is shown. The system may include a terminal 100 and a server 200. The server 200 may include one or more servers ( Figure 1 In the example, a server is included, and the server 200 can provide the method provided in the embodiment of the present application for one or more terminals.

[0059] Among them, an application can be installed on the terminal 100, and the above application and web page can provide an interface. The terminal 100 can receive relevant parameters entered by the user on the interface and send the above parameters to the server 200. The server 200 can obtain processing results based on the received parameters and return the processing results to the terminal 100.

[0060] It should be understood that in some optional implementations, the terminal 100 can also complete the action of obtaining the processing result based on the received parameters by itself without the cooperation of the server, and the embodiments of the present application are not limited to this.

[0061] Next describe Figure 1 The product form of the mid-terminal 100;

[0062] The terminal 100 in the embodiment of the present application can be a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc., and the embodiment of the present application does not impose any restrictions on this.

[0063] Figure 2 An optional hardware structure diagram of the terminal 100 is shown.

[0064] refer to Figure 2 As shown, the terminal 100 may include a radio frequency unit 110, a memory 120, an input unit 130, a display unit 140, a camera 150 (optional), an audio circuit 160 (optional), a speaker 161 (optional), a microphone 162 (optional), an earphone jack 163 (optional), a processor 170, an external interface 180, a power supply 190 and other components. Those skilled in the art will appreciate that Figure 2 These are merely examples of terminals or multi-function devices and do not constitute limitations on the terminals or multi-function devices, which may include more or fewer components than those shown in the figures, or combinations of certain components, or different components.

[0065] The input unit 130 can be used to receive input digital or character information, and generate key signal input related to the user settings and function control of the portable multifunctional device. Specifically, the input unit 130 may include a touch screen 131 (optional) and / or other input devices 132. The touch screen 131 can collect the user's touch operations on or near it (such as the user's operation on or near the touch screen using any suitable object such as fingers, joints, stylus, etc.), and drive the corresponding connection device according to a pre-set program. The touch screen can detect the user's touch action on the touch screen, convert the touch action into a touch signal and send it to the processor 170, and can receive and execute the command sent by the processor 170; the touch signal at least includes the touch point coordinate information. The touch screen 131 can provide an input interface and an output interface between the terminal 100 and the user. In addition, the touch screen can be implemented using multiple types such as resistive, capacitive, infrared and surface acoustic wave. In addition to the touch screen 131, the input unit 130 can also include other input devices. Specifically, other input devices 132 may include, but are not limited to, one or more of a physical keyboard, function keys (such as a volume control key, a switch key, etc.), a trackball, a mouse, a joystick, and the like.

[0066] Among them, the input device 132 can receive input data and the like.

[0067] The display unit 140 may be used to display information input by a user or provided to a user, various menus of the terminal 100, an interactive interface, file display, and / or playback of any multimedia file.

[0068] The memory 120 can be used to store instructions and data. The memory 120 can mainly include an instruction storage area and a data storage area. The data storage area can store various data, such as multimedia files, texts, etc.; the instruction storage area can store software units such as operating systems, applications, instructions required for at least one function, or their subsets and extensions. It can also include a non-volatile random access memory; provide the processor 170 with hardware, software and data resources including management of computing and processing equipment, and support control software and applications. It is also used for the storage of multimedia files, and the storage of running programs and applications.

[0069] The processor 170 is the control center of the terminal 100. It uses various interfaces and lines to connect various parts of the entire terminal 100. By running or executing instructions stored in the memory 120 and calling data stored in the memory 120, it executes various functions of the terminal 100 and processes data, thereby controlling the terminal device as a whole. Optionally, the processor 170 may include one or more processing units; preferably, the processor 170 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application program, and the modem processor mainly processes wireless communication. It is understandable that the above-mentioned modem processor may not be integrated into the processor 170. In some embodiments, the processor and the memory may be implemented on a single chip, and in some embodiments, they may also be implemented separately on separate chips. The processor 170 may also be used to generate corresponding operation control signals, send them to corresponding components of the computing and processing device, read and process data in the software, especially read and process data and programs in the memory 120, so that each functional module therein performs corresponding functions, thereby controlling the corresponding components to act according to the requirements of the instructions.

[0070] Among them, the memory 120 can be used to store software codes related to the offline data quality detection method, the processor 170 can execute the steps of the offline data quality detection method, and can also schedule other units (such as the above-mentioned input unit 130 and display unit 140) to achieve corresponding functions.

[0071] The radio frequency unit 110 (optional) can be used for receiving and sending information or receiving and sending signals during a call, for example, after receiving the downlink information of the base station, it is sent to the processor 170 for processing; in addition, the designed uplink data is sent to the base station. Generally, the RF circuit includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (Low Noise Amplifier, LNA), a duplexer, etc. In addition, the radio frequency unit 110 can also communicate with network devices and other devices through wireless communication. The wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobile Communication (Global System of Mobile Communication, GSM), General Packet Radio Service (General Packet Radio Service, GPRS), Code Division Multiple Access (Code Division Multiple Access, CDMA), Wideband Code Division Multiple Access (Wideband Code Division Multiple Access, WCDMA), Long Term Evolution (Long Term Evolution, LTE), email, Short Messaging Service (SMS), etc.

[0072] In this embodiment of the present application, the RF unit 110 can send data to the server 200 and receive processing results sent by the server 200.

[0073] It should be understood that the radio frequency unit 110 is optional and can be replaced by other communication interfaces, such as a network port.

[0074] The terminal 100 also includes a power supply 190 (such as a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the processor 170 through a power management system, so that the power management system can manage functions such as charging, discharging, and power consumption.

[0075] The terminal 100 further includes an external interface 180 , which may be a standard Micro USB interface or a multi-pin connector, and may be used to connect the terminal 100 to communicate with other devices, or to connect a charger to charge the terminal 100 .

[0076] Although not shown, the terminal 100 may also include a flashlight, a wireless fidelity (WiFi) module, a Bluetooth module, sensors with different functions, etc., which will not be described in detail here. Some or all of the methods described below may be applied in the following embodiments. Figure 2 In the terminal 100 shown.

[0077] Next describe Figure 1 The product form of the server 200;

[0078] Figure 3 A structural diagram of a server 200 is provided, such as Figure 3 As shown, the server 200 includes a bus 201, a processor 202, a communication interface 203, and a memory 204. The processor 202, the memory 204, and the communication interface 203 communicate with each other via the bus 201.

[0079] The bus 201 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0080] The processor 202 may be any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0081] The memory 204 may include a volatile memory, such as a random access memory (RAM). The memory 204 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0082] The memory 204 may be used to store software codes related to the offline data quality detection method, and the processor 202 may execute the steps of the offline data quality detection method of the chip, and may also schedule other units to implement corresponding functions.

[0083] It should be understood that the above-mentioned terminal 100 and server 200 can be centralized or distributed devices, and the processors in the above-mentioned terminal 100 and server 200 (such as processor 170 and processor 202) can be hardware circuits (such as application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), general-purpose processors, DSPs, microprocessors or microcontrollers, etc.), or a combination of these hardware circuits. For example, the processor can be a hardware system with an instruction execution function, such as a CPU, DSP, etc., or a hardware system without an instruction execution function, such as an ASIC, FPGA, etc., or a combination of the above-mentioned hardware systems without an instruction execution function and hardware systems with an instruction execution function.

[0084] The present application provides a method for offline detection of data quality.

[0085] Optionally, the offline data quality detection method provided in the present application can be applied to a scenario where the data party requests the detection party to perform quality detection on the data party's data. That is, when the data party requests a data quality detection on the detection party, the detection party can perform data quality detection through the offline data quality detection method provided in the present application, obtain a quality detection report and feedback it to the data party.

[0086] Of course, the above scenarios are only examples and are not intended to limit the present application.

[0087] In order to enable those skilled in the art to better understand the present application, the data quality offline detection method of the embodiment of the present application is described in detail below with reference to the accompanying drawings.

[0088] Reference Figure 4 , Figure 4 A flowchart of a method for offline data quality detection provided in an embodiment of the present application may include:

[0089] Step S401: Obtain the field type to which the field to be tested in the storage object to be tested belongs.

[0090] Here, the storage object to be tested refers to the storage object where the field to be tested that needs to be quality tested is located. Optionally, the storage object to be tested may be a database or a storage device.

[0091] In this embodiment, the storage object to be tested includes at least one data table, and the fields to be tested may be all or part of the fields in the at least one data table.

[0092] The above field type can be, for example, a date type, an integer type, a decimal type, etc., which is not limited in this application.

[0093] In specific application scenarios, the quality inspection requirements for different field types may be different. For example, the date type contains three types of data: year, month, and day. The data range of the month is 1 to 12, the data range of the day is 1 to 31 or 1 to 30, the number of digits in the year is 4, and so on. Therefore, this step is used to first obtain the field type of the field to be tested in the storage object to be tested, and then different quality inspections are performed based on different field types.

[0094] Step S402: Obtain quality detection rules corresponding to the field type.

[0095] That is, as mentioned above, the fields to be tested can be all or part of the fields in at least one data table, and different fields may belong to different field types. Therefore, the previous step can obtain one or more field types. Then, this step of obtaining the quality detection rules corresponding to the field type specifically refers to obtaining the quality detection rules corresponding to each field type.

[0096] Optionally, the quality detection rules may be determined based on common knowledge information and / or user data requirements, where the common knowledge information may be, for example: the data range for the month is 1 to 12, the data range for the day is 1 to 31 or 1 to 30, the number of digits for the year is 4, and so on.

[0097] The above data requirements are the requirements that the field types proposed by users (i.e., data demanders) need to meet under predefined standard specifications (i.e., standards and specifications recognized by industry insiders in this field) combined with actual scenarios. In different scenarios, a field type may have only one data requirement or multiple data requirements. Therefore, the quality detection rules obtained for each field type may include one or more rules.

[0098] The above quality detection rules are used to detect whether the field value of the field to be tested in the storage object to be tested meets the above predefined standard specifications.

[0099] Step S403: Send the field type, the quality detection rules corresponding to the field type, and the preconfigured offline quality detection tool to the data party, so that the data party can obtain the quality detection result of the field value of the field to be tested in the local and / or intranet environment based on the received field type, quality detection rules, and offline quality detection tool.

[0100] Specifically, this embodiment pre-configures an offline quality detection tool, which refers to a tool that can perform offline data quality detection based on field types and corresponding quality detection rules. Offline detection refers to deploying the offline quality detection tool to the data party (i.e., the party where the storage object to be tested is located) and performing data quality detection locally on the data party and / or in the intranet environment where the data party is located.

[0101] In this embodiment, the offline detection method prevents the detection party from directly contacting the data (ie, field values) of the data party, thereby preventing data leakage of the data party and improving the data security of the data party.

[0102] Optionally, the quality inspection results in this embodiment include: the total number of field values ​​of the fields to be tested, the number of problematic data (i.e., field values ​​that do not meet standard specifications) in each data type or each field to be tested, and the total number of problematic data in each data table.

[0103] Of course, the above quality inspection results may also be other, and this application does not specifically limit them.

[0104] In this embodiment, the quality inspection results obtained by the data party can be sent to the inspection party, so that the inspection party performs the following scoring and aggregation processing to generate a quality inspection report.

[0105] Step S404: Determine a score value related to the field to be tested based on the quality detection result and the rule indicator weight pre-bound to the quality detection rule corresponding to the field type.

[0106] In this embodiment, at least one rule indicator weight can be bound to each rule contained in the quality detection rules corresponding to each field type. Here, the rule indicator weight refers to the weight of a predefined rule indicator, which is an indicator for evaluating the quality of the field value in the storage object to be tested.

[0107] Optionally, the above rule indicators can be indicators suitable for actual scenarios generated based on standard indicators in "2018-GB / T 36344-2018 Information Technology Data Quality Evaluation Indicators".

[0108] For example, the above-mentioned standard indicators may include: normative indicators, completeness indicators, accuracy indicators, consistency indicators and timeliness indicators; based on this, in a possible scenario, the above-mentioned standard indicators may include one or more of the following indicators: data standard normative indicators, metadata normative indicators, authoritative reference data normative indicators, data element integrity indicators, data record integrity indicators, data content correctness indicators, data format correctness indicators, data non-repetition rate indicators, data uniqueness indicators, precision and accuracy indicators, associated data consistency indicators, time period-based correctness indicators, timing indicators and accessibility indicators.

[0109] Optionally, the “score value related to the field to be tested” includes: the overall quality score of the storage object to be tested, the quality score of each data table in the storage object to be tested, and the quality score of each field to be tested.

[0110] Of course, the “score value associated with the field to be tested” may also be other values, for example, the quality score of each field type, etc., which is not specifically limited in this application.

[0111] Step S405: Generate a quality inspection report of the storage object to be tested based on the score value and the quality inspection result.

[0112] Specifically, a quality inspection report including the above-mentioned scoring value and quality inspection results can be generated for the storage object to be tested, so that the data party can understand the data quality in the storage object to be tested through the quality inspection report, and modify the problematic data, or perform other possible activities based on the quality inspection report, etc.

[0113] Optionally, the quality inspection report may include the evaluation object and evaluation method in addition to the above-mentioned scoring value and quality inspection results.

[0114] For example, a quality inspection report includes a report overview and specific details of each item in the report overview, as well as a directory. The content items in the report overview include: evaluation object, evaluation method, quality inspection result and score value.

[0115] For example, a possible report overview might include the following:

[0116] Evaluation object: The data quality evaluation object is all the incremental data of the laboratory MySQL. The data assets include 3 tables, with a total of 43 fields, a total of 60 records, and a total of 961 data elements.

[0117] Evaluation method: First, establish a data quality evaluation rule base based on relevant national standards, data dictionaries, semantics, etc. for data quality evaluation; second, build a data quality evaluation model; then, automatically generate a data quality inspection script; finally, perform data quality inspection and generate a quality inspection report.

[0118] Quality inspection results: A total of 961 data element records were checked, 961 data element records were actually detected in the evaluation task, the total amount of problem data was 235, the total problem data rate was 0.24, and the data quality was relatively high.

[0119] Score: The overall score of data quality is 53.55 points. The scores of each table are: evaluation score is 59.76, knowledge score is 47.20, and demand score is 53.70.

[0120] It should be noted that the above report overview is only an example and is not intended to limit this application.

[0121] The data quality offline detection method provided by the present application obtains the field type to which the field to be tested in the storage object to be tested belongs, obtains the quality detection rule corresponding to the field type, and sends the field type, the quality detection rule corresponding to the field type, and the pre-configured offline quality detection tool to the data party, so that the data party can obtain the quality detection result of the field value of the field to be tested based on the received field type, quality detection rule and offline quality detection tool in the local and / or intranet environment, and determine the score value related to the data in the storage object to be tested based on the quality detection result and the rule indicator weight pre-bound to the quality detection rule corresponding to the field type, and generate a quality detection report for the storage object to be tested based on the score value and the quality detection result. It can be seen from this that the present application can only be processed based on the field type to which the field to be tested in the storage object to be tested belongs, and there is no need to obtain the specific field value. The quality detection process of the field value is performed by the data party itself in the local and / or intranet environment, thereby avoiding the external exposure of the field value in the storage object to be tested and improving data security.

[0122] In some embodiments of the present application, the process of “step S401, obtaining the field type to which the field to be tested in the storage object to be tested belongs” described above is introduced.

[0123] In a possible implementation, the data party may analyze and obtain the field type to which the field to be tested in the storage object to be tested belongs. This embodiment then obtains the field type to which the field to be tested in the storage object to be tested belongs from the data party.

[0124] In another possible implementation, this embodiment may obtain a storage structure of the storage object to be tested, where the storage structure represents the data organization and storage method of the storage object to be tested, and then determine the field type to which the field to be tested belongs based on the storage structure.

[0125] That is, the present embodiment can pull the storage structure of the storage object to be tested, which includes but is not limited to the following contents: which fields are included in the data table in the storage object to be tested, and what are the field types of these fields. To this end, by analyzing the acquired storage structure, the field type to which the field to be tested in the storage object to be tested belongs can be obtained.

[0126] Of course, the process of “obtaining the field type to which the field to be tested in the storage object to be tested belongs” may also be implemented in other ways, which are not specifically limited in this application.

[0127] As described above, after obtaining the field type, the quality detection rule corresponding to the field type can be obtained through step S402.

[0128] Optionally, the data demander can submit its data requirements for the field to be tested to the tester, so that the present embodiment can obtain the data requirements corresponding to the field type to which the field to be tested belongs. In addition, for the field type to which the field to be tested belongs, the tester can also customize at least one rule framework containing data logic for the data attribute to which the field type belongs. For the convenience of subsequent description, the rule framework is defined as a data logic rule item.

[0129] For example, a custom data logic rule item is: the data range of type xx is xx~xx; the data requirement is: the month of the date type must be between 1 and 12, and no more than 6 months from the current test date. Assuming that the current test date is December 1, a quality detection rule corresponding to the date type generated in this embodiment is: the data range of the date type is July 1 to December 1.

[0130] Then, the process of step S402 may include: obtaining data requirements corresponding to the field type to which the field to be tested belongs; determining the data attributes to which the field type belongs, and for the convenience of subsequent description, taking the determined data attributes as target data attributes; obtaining data logic rule items corresponding to the target data attributes, and generating a first quality detection rule based on the data requirements corresponding to the field type to which the field to be tested belongs and the data logic rule items corresponding to the target data attributes. The quality detection rule corresponding to the field type to which the field to be tested belongs is composed of the first quality detection rule and / or the pre-stored second quality detection rule.

[0131] Here, the process of "generating a first quality detection rule based on data requirements and data logic rule items corresponding to the field type to which the field to be tested belongs" may include: performing semantic analysis on the data requirements to extract data conditions with practical meanings in the data requirements; combining the extracted data conditions with the data logic rule items to obtain a combined first quality detection rule.

[0132] For example, if the data requirement is: the month of the date type must be between 1 and 12 and no more than 6 months from the current detection date, then the data conditions are "1 to 12" and "no more than 6 months from the current detection date".

[0133] The following introduces the process of obtaining the pre-stored second quality detection rule.

[0134] Considering that some quality detection rules are commonly used, in order to improve the quality detection efficiency, this embodiment can pre-store the more commonly used quality detection rule sets for use when needed. Taking the storage in database a as an example, in order to quickly find the required second quality detection rule from database a, this embodiment can classify and store the pre-stored quality detection rule sets according to the data attributes to which the field type belongs, that is, the quality detection rule sets corresponding to each data attribute are built-in and stored in database a.

[0135] Optionally, the data attributes in this embodiment include: numerical attributes, character string attributes, and time and date attributes.

[0136] For example, the quality detection rule set corresponding to the numerical attribute is stored in the first partition of database a, the quality detection rule set corresponding to the string attribute is stored in the second partition of database a, and the quality detection rule set corresponding to the time and date attribute is stored in the third partition of database a.

[0137] Of course, the above storage method is only an example and is not intended to limit the present application.

[0138] In order to obtain the second quality detection rule, the present embodiment may first determine the target data attribute. As described above, the target data attribute is one of a numerical attribute, a string attribute and a time and date attribute, for example, the target data attribute is a numerical attribute.

[0139] Then, the quality detection rule set corresponding to the target data attribute is obtained from the pre-stored quality detection rule sets corresponding to the data attributes, and then the second quality detection rule is obtained from the quality detection rule set corresponding to the target data attribute.

[0140] As described above, this embodiment can directly determine the above-mentioned first quality detection rule or the second quality detection rule as the quality detection rule obtained in step S402, and can also determine both the first quality detection rule and the second quality detection rule as the quality detection rule obtained in step S402.

[0141] Optionally, whether to determine the first quality detection rule and the second quality detection rule as the quality detection rule in step S402 may be pre-configured.

[0142] As described above, in step S403, the field type, the quality detection rules corresponding to the field type and the pre-configured offline quality detection tool can be sent to the data party, so that the data party can obtain the quality detection result of the field value of the field to be tested based on the received field type, quality detection rules and offline quality detection tool in the local and / or intranet environment.

[0143] Optionally, the process of "sending the field type, the quality detection rules corresponding to the field type, and the preconfigured offline quality detection tool to the data party" may include: generating a rule configuration file based on the field type and the quality detection rules corresponding to the field type, compiling the rule configuration file and the preconfigured offline quality detection tool into a binary executable tool, and sending the binary executable tool to the data party.

[0144] Optionally, the preconfigured offline quality detection tool is obtained based on a program configuration file and a pre-generated offline quality detection running program. Here, the program configuration file includes connection configuration items of the storage object to be tested and control configuration items of the offline quality detection process, and the connection configuration items include the corresponding relationship between the identification of the field to be tested, the field type and the rule configuration file.

[0145] In this embodiment, the values ​​of each configuration item in the program configuration file can be substituted into a pre-generated offline quality detection running program to obtain an offline quality detection tool that can pull data (i.e., field values) from the storage object to be tested in the local and / or intranet environment of the data party. The offline quality detection tool can detect the pulled data, but the specific detection method still needs to be determined by the rule configuration file, that is, the offline quality detection tool can match the pulled data with the corresponding quality detection rules in the rule configuration file to obtain the quality detection result of the pulled data, that is, to obtain the quality detection result of the field value of the field to be tested.

[0146] In order to be able to perform quality inspection on the field values ​​of the fields to be tested more quickly in the local and / or intranet environment of the data party, the present embodiment can compile the rule configuration file and the pre-configured offline quality inspection tool into a binary executable tool. Optionally, the present embodiment can pre-acquire the system environment of the data party, and then compile the rule configuration file and the pre-configured offline quality inspection tool into a binary executable tool that supports the system environment of the data party, so that after the binary executable tool is sent to the data party, the data party can successfully run the binary executable tool.

[0147] The system environment of the above-mentioned data party can be, for example, Windows, Linux, Mac and other environments. Of course, the system environment of the data party can also be other environments, and this application does not make specific limitations.

[0148] Considering that in some data quality detection scenarios, the data party not only wants to view content items such as the evaluation object, evaluation method, quality detection results and scoring values ​​mentioned above, but also wants to view error samples. Based on this, this embodiment can also obtain error sample notification information. If the error sample notification information indicates that error samples are allowed to be displayed in the quality detection report, the error sample configuration item in the above connection configuration item is modified to allow, so that the data party can obtain error samples at the same time as the quality detection results.

[0149] Based on this, the process of "generating a quality inspection report of the storage object to be tested based on the score value and the quality inspection result" in the above step S405 may include: generating a quality inspection report based on the score value, the quality inspection result and the error sample.

[0150] That is, this embodiment can display some error samples in the quality detection report if the data source allows it, for example, randomly select 10 error patterns from all error samples and display them in the quality detection report.

[0151] It should be noted that the number of 10 error samples mentioned above is only an example and is not intended to limit the present application.

[0152] In summary, the data quality offline detection method provided in this embodiment has the following advantages:

[0153] (1) This embodiment enables the data party to perform data quality detection on the data in the storage object to be tested by running a binary executable tool in a secure environment such as a local and / or intranet without connecting the storage object to the external network. This embodiment does not require an external network connection, avoids the risk of data in the storage object to be tested being exposed to the outside world, and ensures data security.

[0154] (2) This embodiment can automatically pull the storage structure of the storage object to be tested, and obtain the field type to which the data in the storage object to be tested belongs based on it, so that quality detection can be performed more flexibly and efficiently.

[0155] (3) As time goes by, this embodiment can have commonly used data requirements and data logic rule items built into its own system, so that this embodiment can select corresponding built-in data requirements and built-in data logic rule items according to different field types, and can customize data requirements and data logic rule items based on the current scenario during the quality detection process, thereby improving the accuracy of data quality detection.

[0156] (4) For built-in commonly used data requirements and data logic rule items, this embodiment can classify and store them based on data attributes, thereby improving the efficiency of data requirements and data logic rule items required for subsequent queries.

[0157] (5) This embodiment can dynamically calculate the score value related to the data in the storage object to be tested according to the pre-set rule indicator weights. The rule indicator weights can be modified and maintained and updated in a targeted manner according to needs and relevant policies, thereby ensuring the accuracy and timeliness of quality detection.

[0158] (6) This embodiment can automatically generate a quality evaluation report without the need for manual writing. The report content can be automatically modified according to the template format, saving time and effort.

[0159] In order to enable those skilled in the art to better understand the present application, the following introduces the process of offline quality detection of data in the MySQL database, taking the storage object to be tested as a MySQL database and the fields to be tested as all fields in the MySQL database as an example.

[0160] Step 1: Pull the storage structure in the MySQL database, which includes the table structure information of the data table in the MySQL database, including: all field names, field descriptions, field types, etc. in the data table. Determine the field type to which each field in the data table belongs based on the storage structure, and then integrate and classify the fields in the data table according to the field type, and classify them into three data attributes: value, string, and time and date. The data attribute to which the classified field type belongs is recorded as the target data attribute.

[0161] Step 2: Filter the quality detection rule set corresponding to the target data attribute from the quality detection rule sets corresponding to the pre-stored built-in data attributes, and then obtain the second quality detection rule corresponding to the field type to which each field in the data table belongs from the filtered quality detection rule set.

[0162] The specific implementation process of this step can be referred to the previous introduction and will not be repeated here.

[0163] Optionally, the second quality detection rule generated in this step is divided into two parts: basic information and rule information. The basic information includes the rule name and rule description; the rule information is further divided into public rules and special rules.

[0164] Optionally, the above-mentioned common rules can be specifically divided into general rules and attribute rules. General rules refer to rules that can be configured for all data types under data attributes, and attribute rules refer to rules specific to each data type under data attributes.

[0165] Optionally, the above-mentioned special rules can be specifically divided into custom SQL rules and data comparison rules. Custom SQL rules refer to custom rules that are not included in the common rules but are required for the current data quality detection scenario. Data comparison rules refer to rules that require comparison of two fields of the same type.

[0166] It should be noted that the second quality detection rule may be divided in other ways, which is not limited in this application.

[0167] Step three: Obtain the data requirements corresponding to the above-mentioned field type and the data logic rule items corresponding to the target data attributes, wherein the data logic rule items are rule frameworks customized for the field type and containing data logic; generate a first quality detection rule based on the data requirements corresponding to the field type and the data logic rule items corresponding to the target data attributes; and form a quality detection rule corresponding to the field type composed of the first quality detection rule and / or the pre-stored second quality detection rule.

[0168] Step 4: Get the rule indicator weights bound to each rule contained in the quality detection rules in step 3, and export the rule indicator weights, quality detection rules, and corresponding field types as a rule configuration file. The rule configuration file includes: fields and field types in the data table, quality detection rule details corresponding to the fields (i.e., text descriptions of quality detection rules), and rule detection SQL (Structured Query Language) statements to be executed (i.e., machine language of quality detection rules).

[0169] Optionally, the SQL statement includes a problem quantity statistics statement and an error sample statistics statement.

[0170] Step 5: Compile the rule configuration file and the pre-configured offline quality detection tool into a binary executable tool. The binary executable tool can be packaged and compiled according to the different systems of the data party to support running under different systems such as Windows, Linux, Mac, etc.

[0171] Among them, the preconfigured offline quality detection tool is obtained based on a program configuration file and a pre-generated offline quality detection running program. The program configuration file contains connection configuration items of the storage object to be tested and control configuration items of the offline quality detection process. The connection configuration items contain the correspondence between the identification of the field to be tested, the field type and the rule configuration file.

[0172] Optionally, the connection configuration item also includes a correspondence between the data table where the field to be tested is located and the identifier of the rule configuration file.

[0173] Step 6: Send the binary executable tool to the data party, so that the data party can obtain the quality inspection results of the field values ​​in the data table in the MySQL database based on the received binary executable tool in the local and / or intranet environment.

[0174] When the binary executable tool is run in the local and / or intranet environment of the data party, the binary executable tool will connect to the MySQL database of the data party. The above-mentioned offline quality detection running program will read the rule detection SQL statements of each field in the rule configuration file corresponding to the data table in the order of the actual data table according to the mapping relationship between the data table and the rule configuration file in the program configuration file, and execute them in the MySQL database to count the amount of data that does not comply with the rules.

[0175] If the data provider allows, you can also randomly query 10 error samples.

[0176] Optionally, after all data tables in the program configuration file are executed, the binary executable tool will generate an offline evaluation report for each data table based on the execution status of the rule detection SQL statement in each data table. The report content includes all field information in the data table, details of the executed rules, rule detection SQL statements corresponding to the rules, the number of elements that do not comply with the rules under the field, and 10 random sample information of elements that do not comply with the rules (i.e., error samples) if allowed by the data party.

[0177] Step 7: Integrate and analyze the offline evaluation reports generated for each data table, count the percentage of each field that does not comply with the rules, and combine the rule indicator weights to obtain the final quality inspection report.

[0178] It should be noted that the above steps are merely examples of steps in a scenario that can be implemented in this application and are not intended to limit this application.

[0179] A method for offline data quality detection provided in an embodiment of the present application is introduced above. A device for executing the method for offline data quality detection will be introduced below.

[0180] See also Figure 5 , Figure 5 This is a schematic diagram of the structure of a data quality offline detection device provided in an embodiment of the present application. Figure 5 As shown, the device may include:

[0181] The field type acquisition module 501 is used to acquire the field type to which the field to be tested in the storage object to be tested belongs;

[0182] A detection rule acquisition module 502 is used to acquire quality detection rules corresponding to the field type;

[0183] The detection information sending module 503 is used to send the field type, the quality detection rule corresponding to the field type and the pre-configured offline quality detection tool to the data party, so that the data party can obtain the quality detection result of the field value of the field to be tested in the local and / or intranet environment based on the received field type, quality detection rule and offline quality detection tool;

[0184] The test data scoring module 504 is used to determine the score value related to the test field based on the quality detection result and the rule indicator weight pre-bound to the quality detection rule corresponding to the field type;

[0185] The test report generation module 505 is used to generate a quality test report of the storage object to be tested based on the score value and the quality test result.

[0186] In a possible implementation, the field type acquisition module may include: a storage structure acquisition unit and a field type generation unit.

[0187] A storage structure acquisition unit, used to acquire the storage structure of the storage object to be tested, where the storage structure represents the data organization and storage method of the storage object to be tested;

[0188] The field type generating unit is used to determine the field type to which the field to be tested belongs based on the storage structure.

[0189] In a possible implementation, the detection rule acquisition module may include: a data information acquisition unit, a data attribute determination module, a rule item acquisition module, a detection rule generation unit and a detection rule combination unit.

[0190] A data information acquisition unit, used to acquire data requirements corresponding to a field type;

[0191] A data attribute determination module, used to determine the data attribute to which the field type belongs as a target data attribute, wherein the target data attribute is one of a numerical attribute, a string attribute, and a time and date attribute;

[0192] A rule item acquisition module is used to acquire data logic rule items corresponding to target data attributes, wherein the data logic rule items are rule frameworks containing data logic that are customized for field types;

[0193] A detection rule generating unit, configured to generate a first quality detection rule based on a data requirement corresponding to a field type and a data logic rule item corresponding to a target data attribute;

[0194] The detection rule combination unit is used to compose a quality detection rule corresponding to the field type from the first quality detection rule and / or the pre-stored second quality detection rule.

[0195] In a possible implementation, the process of the detection rule combination unit acquiring the pre-stored second quality detection rule may include:

[0196] Obtaining a quality detection rule set corresponding to the target data attribute from the pre-stored quality detection rule sets corresponding to the respective data attributes;

[0197] A second quality detection rule is obtained from a set of quality detection rules corresponding to the target data attribute.

[0198] In a possible implementation, the detection information sending module may include: a file generating unit, a tool compiling unit and a tool sending unit.

[0199] A file generating unit, used for generating a rule configuration file based on the field type and the quality detection rule corresponding to the field type;

[0200] A tool compilation unit, used for compiling a rule configuration file and a pre-configured offline quality detection tool into a binary executable tool;

[0201] The tool sending unit is used to send the binary executable tool to the data party.

[0202] In a possible implementation, the tool compilation unit may include: an environment acquisition subunit and a tool compilation subunit.

[0203] The environment acquisition subunit is used to obtain the system environment of the data party;

[0204] The tool compilation subunit is used to compile the rule configuration file and the pre-configured offline quality detection tool into a binary executable tool that supports the system environment.

[0205] In a possible implementation, the above-mentioned preconfigured offline quality detection tool is obtained based on a program configuration file and a pre-generated offline quality detection running program. The program configuration file includes connection configuration items of the storage object to be tested and control configuration items of the offline quality detection process. The connection configuration items include the correspondence between the identification of the field to be tested, the field type and the rule configuration file.

[0206] In a possible implementation, the data quality offline detection device provided by the present application may also include: an error sample notification module and a configuration file modification module.

[0207] Error sample notification module, used to obtain error sample notification information;

[0208] The configuration file modification module is used to modify the error sample configuration item in the connection configuration item to allow if the error sample notification information indicates that the error sample is allowed to be displayed in the quality detection report, so that the data party can obtain the error sample at the same time as the quality detection result.

[0209] On this basis, the above-mentioned test report generation module can be specifically used to generate a quality test report based on the score value, quality test results and error samples.

[0210] The data quality offline detection device provided in this application corresponds to the data quality offline detection method provided in the previous text. For details, please refer to the previous text introduction and will not be repeated here.

[0211] The present application also provides an electronic device in an embodiment. Figure 6 As shown, it shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiment of the present application. The electronic device in the embodiment of the present application may include but is not limited to fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 6 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0212] like Figure 6 As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 to a random access memory (RAM) 603. When the electronic device is powered on, various programs and data required for the operation of the electronic device are also stored in the RAM 603. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0213] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a memory card, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Figure 6 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0214] An embodiment of the present application also provides a computer program product including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any one of the data quality offline detection methods provided in the embodiments of the present application.

[0215] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any one of the data quality offline detection methods provided in the embodiment of the present application.

[0216] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0217] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0218] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0219] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

Claims

1. A data quality offline detection method, characterized in that: include: Get the field type of the field to be tested in the storage object to be tested; Obtaining quality detection rules corresponding to the field type; Sending the field type, the quality detection rule corresponding to the field type and the preconfigured offline quality detection tool to the data party, so that the data party can obtain the quality detection result of the field value of the field to be tested in a local and / or intranet environment based on the received field type, quality detection rule and offline quality detection tool; Determine a score value related to the field to be tested based on the quality detection result and a rule indicator weight pre-bound to the quality detection rule corresponding to the field type; A quality inspection report of the storage object to be tested is generated based on the score value and the quality inspection result.

2. The data quality offline detection method according to claim 1, characterized in that: The step of obtaining the field type to which the field to be tested in the storage object to be tested belongs includes: Acquire a storage structure of the storage object to be tested, wherein the storage structure represents the data organization and storage method of the storage object to be tested; The field type to which the field to be tested belongs is determined based on the storage structure.

3. The data quality offline detection method according to claim 1, characterized in that: The obtaining of the quality detection rule corresponding to the field type includes: Obtain data requirements corresponding to the field type; Determine a data attribute to which the field type belongs as a target data attribute, wherein the target data attribute is one of a numerical attribute, a string attribute, and a time and date attribute; Acquire a data logic rule item corresponding to the target data attribute, wherein the data logic rule item is a rule framework containing data logic that is customized for the field type; Generate a first quality detection rule based on the data requirement corresponding to the field type and the data logic rule item corresponding to the target data attribute; The quality detection rule corresponding to the field type is composed of the first quality detection rule and / or the pre-stored second quality detection rule.

4. The method for offline data quality detection according to claim 3, characterized in that: The process of obtaining the pre-stored second quality detection rule includes: Acquire a quality detection rule set corresponding to the target data attribute from pre-stored quality detection rule sets corresponding to the respective data attributes; The second quality detection rule is obtained from a set of quality detection rules corresponding to the target data attribute.

5. The method for offline data quality detection according to claim 1, characterized in that: The sending the field type, the quality detection rule corresponding to the field type and the preconfigured offline quality detection tool to the data party includes: Generate a rule configuration file based on the field type and the quality detection rule corresponding to the field type; Compiling the rule configuration file and the preconfigured offline quality detection tool into a binary executable tool; The binary executable tool is sent to the data party.

6. The method for offline data quality detection according to claim 5, characterized in that: The step of compiling the rule configuration file and the preconfigured offline quality detection tool into a binary executable tool includes: Acquiring the system environment of the data party; The rule configuration file and the preconfigured offline quality detection tool are compiled into a binary executable tool that supports the system environment.

7. The method for offline data quality detection according to claim 5, characterized in that: The preconfigured offline quality detection tool is obtained based on a program configuration file and a pregenerated offline quality detection running program, wherein the program configuration file includes connection configuration items of the storage object to be tested and control configuration items of the offline quality detection process, and the connection configuration items include the correspondence between the field to be tested, the field type and the identifier of the rule configuration file.

8. The method for offline data quality detection according to claim 7, characterized in that: Also includes: Get error sample notification information; If the error sample notification information indicates that the error sample is allowed to be displayed in the quality detection report, the error sample configuration item in the connection configuration item is modified to be allowed, so that the data party obtains the error sample while obtaining the quality detection result; The step of generating a quality inspection report of the storage object to be tested based on the score value and the quality inspection result includes: The quality inspection report is generated based on the score value, the quality inspection result and the error sample.

9. A data quality offline detection device, characterized in that: include: A field type acquisition module is used to obtain the field type to which the field to be tested belongs in the storage object to be tested; A detection rule acquisition module, used to obtain the quality detection rule corresponding to the field type; A detection information sending module, used to send the field type, the quality detection rule corresponding to the field type and the pre-configured offline quality detection tool to the data party, so that the data party can obtain the quality detection result of the field value of the field to be tested in a local and / or intranet environment based on the received field type, quality detection rule and offline quality detection tool; A test data scoring module, used to determine a score value related to the test field based on the quality detection result and the rule indicator weight pre-bound to the quality detection rule corresponding to the field type; A test report generating module is used to generate a quality test report of the storage object to be tested based on the score value and the quality test result.

10. An electronic device, characterized in that: The method comprises at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the electronic device can implement the data quality offline detection method as described in any one of claims 1 to 8.

Images