Data quality detection method and device, equipment, medium and program product

Abstract

The invention provides a data quality detection method and device, equipment, a medium and a program product, and relates to the field of big data. Metadata of database tables of a plurality of application systems are obtained, the metadata of the database tables of the application systems are analyzed to obtain an analysis result, one or more data quality detection rules are generated according to the analysis result, and each data quality detection rule corresponds to one or more application systems. Converting each data quality detection rule into a corresponding data query statement, sending the data query statement to one or more application systems corresponding to the data quality detection rule, and generating a quality detection result according to a data query result returned by the one or more application systems; according to the method, unified management and efficient detection of data quality are achieved, and the problems that data quality problems are scattered and centralized management is difficult are effectively solved.

Description

Technical Field

[0001] This application relates to the field of big data, and in particular, to a data quality detection method, apparatus, device, medium, and program product. Background Art

[0002] With the acceleration of the digital transformation pace, the deployment mode of application systems is changing from the traditional centralized mode to the distributed and cloud-based directions. This change has led to a sharp increase in both the number of servers deployed in the data center and the data storage scale, thereby making the data quality problem become more prominent and important. The quality of data is not only directly related to the smooth operation of business operations but also may potentially trigger major risks. Therefore, implementing data quality detection has become an essential key to ensuring the stable operation of the business.

[0003] In the prior art, the data quality detection platform adopts an independent deployment method, which means that each application system sets quality detection rules on its respective database for data quality detection.

[0004] However, in this decentralized deployment mode, the quality detection rules set by each application system are independent and different from each other, making it difficult to centrally manage and uniformly solve data quality problems. Summary of the Invention

[0005] This application provides a data quality detection method, apparatus, device, medium, and program product, which is used to solve the technical problem that it is difficult to centrally manage and uniformly solve data quality problems.

[0006] In a first aspect, this application provides a data quality detection method, including:

[0007] Obtaining metadata of database tables of multiple application systems respectively;

[0008] Parsing the metadata of the database tables of each application system to obtain a parsing result, where the parsing result includes the database table fields of each application system and their corresponding business meanings, and the database table fields of different application systems with the same business meaning;

[0009] Generating one or more data quality detection rules according to the parsing result, and each data quality detection rule corresponds to one or more application systems;

[0010] Converting each data quality detection rule into a corresponding data query statement, sending the data query statement to the one or more application systems corresponding to the data quality detection rule, and generating a quality detection result according to the data query results returned by the one or more application systems.

[0011] In a possible implementation, obtaining the metadata of the database tables of each of the multiple application systems includes:

[0012] Access the databases of each of the multiple application systems through an application interface or direct database connection to obtain the metadata of the database tables of each of the multiple application systems.

[0013] In a possible implementation, parsing the metadata of the database tables of each application system to obtain a parsing result includes:

[0014] Input the metadata of the database tables of each application system into a pre-trained large model, and through the large model, retrieve knowledge related to the metadata from the expert knowledge base corresponding to each of the preset application systems, and generate the parsing result according to the metadata and the knowledge related to the metadata.

[0015] In a possible implementation, generating a quality detection result according to the data query result returned by one or more application systems includes:

[0016] According to the data query result returned by the application system, determine whether the data query result matches the abnormal determination condition corresponding to the quality detection rule. If it matches, determine that the quality detection result is abnormal; if it does not match, determine that the quality detection result is normal.

[0017] In a possible implementation, it further includes:

[0018] Generate an abnormal problem list according to the abnormal quality detection results in the quality detection results of one or more application systems corresponding to each data quality detection rule. The abnormal problem list includes one or more abnormal problems and the processing status corresponding to the abnormal problems;

[0019] In response to a processing operation for any abnormal problem in the abnormal problem list, update the processing status of the any abnormal problem in the abnormal problem list.

[0020] In a possible implementation, converting each data quality detection rule into a corresponding data query statement includes:

[0021] Generate an audit request for the data quality detection rule;

[0022] In response to the audit request being passed, convert each data quality detection rule into a corresponding data query statement.

[0023] In a possible implementation, after parsing the metadata of the database tables of each application system to obtain a parsing result, the method further includes:

[0024] Create a data directory corresponding to the metadata of the database tables of each application system according to the parsing result, where the data directory includes the correspondence between the application system, the database table fields, and the business meanings.

[0025] In a second aspect, the present application provides a data quality detection device, including:

[0026] An acquisition module 201, configured to acquire the metadata of the database tables of multiple application systems respectively;

[0027] A generation module 202, configured to parse the metadata of the database tables of each application system to obtain a parsing result, where the parsing result includes the database table fields of each application system and the corresponding business meanings, and the database table fields of the same business meaning in different application systems;

[0028] A generation module 203, configured to generate one or more data quality detection rules according to the parsing result, and each data quality detection rule corresponds to one or more application systems;

[0029] The processing module 202 is further configured to convert each data quality detection rule into a corresponding data query statement;

[0030] A sending module 204, configured to send the data query statement to the one or more application systems corresponding to the data quality detection rule;

[0031] The generation module 203 is further configured to generate a quality detection result according to the data query results returned by the one or more application systems.

[0032] In a possible implementation manner, it includes:

[0033] The acquisition module 201 is further configured to access the databases of the multiple application systems respectively through an application interface or a direct database connection to acquire the metadata of the database tables of the multiple application systems respectively.

[0034] In a possible implementation manner, it includes:

[0035] The processing module 202 is further configured to input the metadata of the database tables of each application system into a pre-trained large model, and retrieve knowledge related to the metadata from a preset expert knowledge base corresponding to each application system through the large model;

[0036] The generation module 203 is further configured to generate the parsing result according to the metadata and the knowledge related to the metadata.

[0037] In a possible implementation, the device further includes: a determination module 205;

[0038] The determination module 205 is configured to determine, according to the data query result returned by the application system, whether the data query result matches the abnormal determination condition corresponding to the quality detection rule. If it matches, it determines that the quality detection result is abnormal; if it does not match, it determines that the quality detection result is normal.

[0039] In a possible implementation, it further includes:

[0040] The generation module 203 is further configured to generate an abnormal problem list according to the abnormal quality detection results among the quality detection results of one or more application systems corresponding to each data quality detection rule. The abnormal problem list includes one or more abnormal problems and the processing status corresponding to the abnormal problems;

[0041] The processing module 202 is further configured to update the processing status of any abnormal problem in the abnormal problem list in response to a processing operation for any abnormal problem in the abnormal problem list.

[0042] In a possible implementation, it includes:

[0043] The generation module 203 is further configured to generate an audit request for the data quality detection rule;

[0044] The processing module 202 is further configured to convert each data quality detection rule into a corresponding data query statement in response to the passing of the audit request.

[0045] In a possible implementation, the method further includes:

[0046] The processing module 202 is further configured to establish a data directory corresponding to the metadata of the database tables of each application system according to the parsing result. The data directory includes the correspondence between the application system, the database table fields, and the business meaning.

[0047] In a third aspect, an embodiment of the present application provides a data quality detection device, including: a memory, a processor;

[0048] The memory stores computer execution instructions;

[0049] The processor executes the computer execution instructions stored in the memory, so that the processor executes the above first aspect and / or various possible implementations of the first aspect.

[0050] Fourthly, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored, and when the computer-executable instructions are executed by a processor, they are used to implement the first aspect and / or various possible implementation manners of the first aspect as described above.

[0051] Fifthly, an embodiment of the present application provides a computer program product, including a computer program, which when executed by a processor, implements the first aspect and / or various possible implementation manners of the first aspect as described above.

[0052] The data quality detection method provided by the present application obtains the metadata of the database tables of multiple application systems respectively, parses the metadata of the database tables of each application system to obtain a parsing result, the parsing result includes the database table fields of each application system and the corresponding business meanings, and the database table fields of the same business meaning in different application systems, generates one or more data quality detection rules according to the parsing result, each data quality detection rule corresponds to one or more application systems, converts each data quality detection rule into a corresponding data query statement, sends the data query statement to one or more application systems corresponding to the data quality detection rule, and generates a quality detection result according to the data query results returned by the one or more application systems; this method realizes the unified management and efficient detection of data quality, and effectively solves the problems of scattered data quality problems and difficult centralized management. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The drawings here are incorporated into the specification and constitute a part of this specification, showing embodiments consistent with the present application, and are used together with the specification to explain the principles of the present application.

[0054] Figure 1 It is a schematic diagram of a data quality detection scenario provided by the present application;

[0055] Figure 2 It is a schematic flowchart of the data quality detection method provided by the present application;

[0056] Figure 3 It is a schematic structural diagram of the data quality detection device provided by the present application;

[0057] Figure 4 It is a schematic structural diagram of the data quality detection device provided by the present application.

[0058] Through the above-mentioned drawings, the clear embodiments of the present application have been shown, and there will be more detailed descriptions hereinafter. These drawings and textual descriptions are not intended to limit the scope of the concept of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0059] Exemplary embodiments will be described in detail herein, and examples thereof are shown in the accompanying drawings. When the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0060] It should be noted that the data quality detection methods, devices, equipment, media, and program products provided by the present application can be used in the field of big data and can also be used in any field other than the field of big data. The present application does not limit the application fields of the data quality detection methods, devices, equipment, media, and program products.

[0061] With the continuous acceleration of digital transformation, the deployment mode of application systems is undergoing a transformation from traditional centralized to distributed and cloud-based directions. This transformation has directly led to a sharp increase in the number of servers in the data center and a rapid expansion of the data storage scale, making data quality issues increasingly prominent and becoming an important issue that cannot be ignored. The quality of data is not only directly related to the smoothness of business operations but also potentially involves the prevention of major risks. Therefore, implementing efficient data quality detection has become the key to ensuring the stable operation of the business.

[0062] In the prior art, data quality detection platforms often operate in an independent deployment manner, that is, specific data quality detection rules are set on the databases of each application system to monitor data quality.

[0063] However, in this decentralized deployment mode, the quality detection rules set for each application system are independent and different from each other, making it difficult to centrally manage and uniformly solve data quality problems.

[0064] In response to the above problems, the data quality detection method provided by the present application collects and analyzes the metadata of database tables of multiple application systems, clarifies the business meanings of data fields in each application system and their cross-system corresponding relationships, and then formulates data quality detection rules based on the analysis results. Subsequently, the data quality detection rules are converted into executable data query statements, and these data query statements are sent to the corresponding application systems for execution. Finally, based on the data query results returned by each application system, the method realizes the unified management and efficient detection of data quality, effectively solving the problem that data quality problems are scattered and difficult to centrally manage.

[0065] Figure 1Schematic diagram of the data quality detection scenario provided for this application. The schematic diagram of the data quality detection scenario includes: a data quality detection platform and multiple application systems (including: Application System 1, Application System 2, ……, Application System n). The data quality detection platform includes: a data link module, a metadata processing module, a data asset module, a quality detection module, a quality rule module, and a quality problem module.

[0066] The data link module links to the databases of multiple application systems respectively through configuring application interfaces or direct database connections, extracts metadata from the database tables of multiple application systems respectively, sends the extracted metadata to the metadata processing module to parse the metadata, and sends the parsing result to the data asset module for storage and query. The data asset module sends the parsing result to the quality rule module to generate one or more data quality detection rules. The quality rule module sends the data quality detection rules to the quality detection module. The quality detection module converts each data quality detection rule into a corresponding data query statement, and sends the data query statement to one or more application systems corresponding to the data quality detection rule through the data link module. The application system performs data quality detection according to the data query statement, and returns the data quality detection result to the data link module. The data link module returns the quality detection result to the quality detection module. The quality detection module sends the quality detection result to the quality problem module.

[0067] The following uses specific embodiments to elaborate in detail on the technical solution of this application and how the technical solution of this application solves the above technical problems. These several specific embodiments below can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of this application will be described below in conjunction with the accompanying drawings.

[0068] Figure 2 Schematic diagram of the process of the data quality detection method provided for this application. The execution subject of this embodiment is, for example, a data quality detection system. As Figure 2 shown, this method includes:

[0069] S101: Obtain the metadata of the database tables of multiple application systems respectively.

[0070] Among them, metadata usually includes the name of the table, field name, field type, field length, whether it allows null values, primary key, foreign key, and index.

[0071] Metadata is the basis for data governance, data quality management, data security, and privacy protection. In data integration and migration projects, accurate metadata is the key to ensuring data consistency and integrity in data quality detection.

[0072] S102: Parse the metadata of the database tables of each application system to obtain the parsing result, where the parsing result includes the database table fields of each application system and their corresponding business meanings, as well as the database table fields of the same business meaning in different application systems.

[0073] First, the data quality detection system uses a metadata parsing tool or writes a custom script to analyze the collected metadata, extract the field names, data types, and annotation or description information. Then, combined with business knowledge and industry standards, the extracted information is manually reviewed and confirmed to ensure that the business meaning of each field is accurately understood. During this process, special attention also needs to be paid to the cases where fields with the same business meaning have different names in different application systems, and they are marked and recorded. Finally, the parsing result is organized into a format that is easy to query and use, such as a database view, a data dictionary, or metadata.

[0074] S103: Generate one or more data quality detection rules according to the parsing result, where each data quality detection rule corresponds to one or more application systems.

[0075] First, based on the parsing result, determine the data range and fields to be detected. Second, according to business requirements and industry standards, the detection rules include data format verification, value range check, uniqueness constraint, association relationship verification, etc. Ensure that the detection rules can accurately reflect the data requirements of the same business meaning in different application systems. Then, use a data quality management tool or write a custom script to convert the detection rules into executable programs or query statements. Finally, deploy the detection rules to the corresponding data quality detection system, configure the detection frequency and reporting mechanism to ensure the timely discovery and reporting of data quality problems.

[0076] For example, taking the sales management system and inventory management system of a certain company as an example, according to the parsing result of the metadata of the database tables of these two systems, a data quality detection rule is generated: "Ensure that the value of the 'Order Quantity' field in the sales management system is consistent with the value of the 'Outbound Quantity' field of the corresponding order in the inventory management system." This rule aims to verify the data consistency of the order quantity between the sales management system and the inventory management system, ensure the accuracy and integrity of the data, and it applies to both the sales management system and the inventory management system.

[0077] S104: Convert each data quality detection rule into a corresponding data query statement, send the data query statement to one or more application systems corresponding to the data quality detection rule, and generate a quality detection result according to the data query results returned by the one or more application systems.

[0078] First, data quality detection rules, such as which fields are required and what value ranges are reasonable; second, write corresponding data query statements according to the data quality detection rules to ensure that these statements can accurately extract the data to be verified; then, send these query statements to the specified application systems, and these application systems execute the queries and return the results; finally, based on the returned results, generate detailed quality detection results by writing scripts or using existing data quality tools. The quality detection results should include detailed descriptions of data quality problems, the quantity of problematic data, possible solutions, and other information.

[0079] The data quality detection method provided in this embodiment obtains the metadata of the database tables of multiple application systems respectively, parses the metadata of the database tables of each application system to obtain the parsing results. The parsing results include the database table fields and corresponding business meanings of each application system, as well as the database table fields of the same business meaning in different application systems. One or more data quality detection rules are generated according to the parsing results, and each data quality detection rule corresponds to one or more application systems. Each data quality detection rule is converted into a corresponding data query statement, and the data query statement is sent to one or more application systems corresponding to the data quality detection rule, and a quality detection result is generated according to the data query results returned by the one or more application systems. This method realizes the unified management and efficient detection of data quality, and effectively solves the problems of scattered data quality problems and difficult centralized management.

[0080] In a possible implementation manner, the specific implementation manner of obtaining the metadata of the database tables of multiple application systems respectively includes:

[0081] Access the databases of multiple application systems respectively through application interfaces or direct database connections to obtain the metadata of the database tables of multiple application systems respectively.

[0082] The data quality detection system can either configure application interfaces to achieve data interaction with application systems or directly connect to the database tables of each application system to directly obtain metadata. During this process, the data quality detection system follows the preset access permissions to ensure the legality and security of data access. At the same time, by regulating the access frequency, the data quality detection system can query data at a reasonable rhythm and cycle, avoiding unnecessary burdens or interferences on the original application system. The entire data quality detection process is completely in read-only mode, and the data quality detection system does not have the permission to modify data, thus fundamentally ensuring the security and stability of the original application system database and ensuring that its normal operation is not affected in any way.

[0083] In a possible implementation manner, the specific implementation manner of parsing the metadata of the database tables of each application system to obtain the parsing results includes:

[0084] Input the metadata of the database tables of each application system into a pre-trained large model. The large model retrieves knowledge related to the metadata from the expert knowledge bases corresponding to the preset application systems, and generates a parsing result based on the metadata and the knowledge related to the metadata.

[0085] Among them, the expert knowledge base is a comprehensive database integrating data quality detection industry, enterprise standards and professional knowledge. It gathers the wisdom of experts in various fields, aiming to provide users with accurate, authoritative and comprehensive industry guidance, specification reference and professional knowledge support, and promote the efficient utilization and innovative development of knowledge.

[0086] Select one or more pre-trained large models, which should have the ability to process natural language text and structured data and be able to interact with the expert knowledge base; then, input the metadata into the large model. The model uses its own deep learning algorithms and semantic understanding capabilities to retrieve knowledge related to the metadata from the expert knowledge base; then, combining the metadata and the retrieved knowledge, the large model generates a parsing result through pattern matching. The parsing result may include the business meaning of fields, cross-system field mapping, data quality assessment, etc.; finally, verify and optimize the parsing result to ensure its accuracy and practicality.

[0087] With the improvement of enterprise informatization level, the amount of data generated by each application system has increased sharply, and these data often follow different formats and standards, resulting in extremely complex data integration and analysis. By using a combination of pre-trained large models and expert knowledge bases, the efficiency and accuracy of data parsing can be improved. The large model can process large-scale data, quickly identify key information in the metadata, and ensure that the parsing result not only conforms to business logic but also meets industry and enterprise standards.

[0088] In a possible implementation manner, generate a quality detection result according to the data query result returned by one or more application systems. The specific implementation manner includes:

[0089] According to the data query result returned by the application system, determine whether the data query result matches the abnormal determination condition corresponding to the quality detection rule. If it matches, determine that the quality detection result is abnormal; if it does not match, determine that the quality detection result is normal.

[0090] It can be understood that obtain the data query result from the application system; then, according to the preset quality detection rule, extract the abnormal determination conditions therein; then, compare the data query result with these abnormal determination conditions one by one; finally, according to the comparison result, determine that the quality status of the data query result is abnormal or normal. If it is determined to be abnormal, a corresponding warning mechanism or processing process can be further triggered.

[0091] In a possible implementation, after determining that the quality inspection result is abnormal, the specific implementation method includes:

[0092] Generate an exception problem list based on the abnormal quality inspection results in the quality inspection results of one or more application systems corresponding to each data quality inspection rule. The exception problem list includes one or more exception problems and the processing status corresponding to the exception problems;

[0093] In response to a processing operation for any exception problem in the exception problem list, update the processing status of any exception problem in the exception problem list.

[0094] The data quality inspection system will detect the application system according to the preset data quality inspection rules and automatically record the abnormal results. Then, the data quality inspection system will generate an exception problem list based on these abnormal results. When processing any exception problem in the list, corresponding operations need to be performed in the data quality inspection system to update the processing status of the problem.

[0095] The practice of generating an exception problem list and updating the processing status aims to improve the efficiency and transparency of data quality management. By centrally managing exception problems, all current quality problems and the processing situation of each problem can be quickly understood. This not only helps to discover and solve problems in a timely manner but also effectively avoids the omission and delay of problems.

[0096] In a possible implementation, when converting each data quality inspection rule into a corresponding data query statement, the specific implementation method includes:

[0097] Generate an audit request for the data quality inspection rule;

[0098] In response to the audit request being passed, convert each data quality inspection rule into a corresponding data query statement.

[0099] The data quality inspection system will generate an audit request and send it to the auditor's terminal. The auditor will review and confirm the data quality inspection rule. Once the auditor determines that the audit is passed on the terminal, the data quality inspection system will automatically convert the data quality inspection rule into a specific data query statement for execution in the database to verify whether the quality of the data meets the established standards.

[0100] In a possible implementation, after parsing the metadata of the database tables of each application system to obtain a parsing result, the method further includes:

[0101] Establish a data directory corresponding to the metadata of the database tables of each application system according to the parsing result. The data directory includes the corresponding relationships among the application system, the database table fields, and the business meanings.

[0102] It is understandable that to establish a data catalog, it is first necessary to extract the metadata of database tables from each application system, which includes information such as table structures, field names, and data types. Then, according to business requirements, the metadata is parsed to clarify the business meaning represented by each field. Next, these parsing results are integrated into a unified data catalog to establish the correspondence between application systems, database table fields, and business meanings.

[0103] Figure 3 This is a schematic structural diagram of the data quality detection device provided by this application. As Figure 3 shown, the data quality detection device 200 provided in this embodiment includes:

[0104] An acquisition module 201, configured to acquire the metadata of database tables of multiple application systems respectively;

[0105] A generation module 202, configured to parse the metadata of database tables of each application system to obtain a parsing result, where the parsing result includes the database table fields of each application system and the corresponding business meanings, and the database table fields of the same business meaning in different application systems;

[0106] A generation module 203, configured to generate one or more data quality detection rules according to the parsing result, and each data quality detection rule corresponds to one or more application systems;

[0107] A processing module 202 is further configured to convert each data quality detection rule into a corresponding data query statement;

[0108] A sending module 204, configured to send the data query statement to one or more application systems corresponding to the data quality detection rule;

[0109] The generation module 203 is further configured to generate a quality detection result according to the data query results returned by one or more application systems.

[0110] In a possible implementation manner, it includes:

[0111] The acquisition module 201 is further configured to access the databases of multiple application systems respectively through application interfaces or direct database connections to acquire the metadata of database tables of multiple application systems respectively.

[0112] In a possible implementation manner, it includes:

[0113] The processing module 202 is further configured to input the metadata of database tables of each application system into a pre-trained large model, and retrieve knowledge related to the metadata from the expert knowledge bases corresponding to the preset application systems through the large model;

[0114] The generating module 203 is further configured to generate a parsing result according to the metadata and the knowledge related to the metadata.

[0115] In a possible implementation manner, the apparatus further includes: a determining module 205;

[0116] The determining module 205 is configured to determine, according to the data query result returned by the application system, whether the data query result matches the exception determination condition corresponding to the quality detection rule. If it matches, it is determined that the quality detection result is abnormal. If it does not match, it is determined that the quality detection result is normal.

[0117] In a possible implementation manner, it further includes:

[0118] The generating module 203 is further configured to generate a list of abnormal problems according to the abnormal quality detection results in the quality detection results of one or more application systems corresponding to each data quality detection rule. The list of abnormal problems includes one or more abnormal problems and the processing status corresponding to the abnormal problems.

[0119] The processing module 202 is further configured to update the processing status of any abnormal problem in the list of abnormal problems in response to a processing operation for any abnormal problem in the list of abnormal problems.

[0120] In a possible implementation manner, it includes:

[0121] The generating module 203 is further configured to generate an audit request for the data quality detection rule;

[0122] The processing module 202 is further configured to convert each data quality detection rule into a corresponding data query statement in response to the passing of the audit request.

[0123] In a possible implementation manner, the method further includes:

[0124] The processing module 202 is further configured to establish a data directory corresponding to the metadata of the database tables of each application system according to the parsing result. The data directory includes the corresponding relationship between the application system, the database table fields, and the business meaning.

[0125] The data quality detection apparatus provided in this embodiment can execute the method provided in the above method embodiment, and its implementation principle and technical effect are similar. Details are not described herein again.

[0126] Figure 4 It is a schematic structural diagram of the data quality detection device provided in this application. As Figure 4As shown in the figure, the electronic device 300 provided in this embodiment includes: at least one processor 301 and a memory 302. Optionally, the device 300 further includes a communication component 303. Among them, the processor 301, the memory 302, and the communication component 303 are connected through a bus 304.

[0127] In the specific implementation process, at least one processor 301 executes the computer-executable instructions stored in the memory 302, so that at least one processor 301 executes the above-mentioned method.

[0128] For the specific implementation process of the processor 301, reference can be made to the above method embodiment. The implementation principle and technical effects are similar, and will not be elaborated here in this embodiment.

[0129] In the above embodiment, it should be understood that the processor may be a central processing unit (English: Central Processing Unit, abbreviated: CPU), or other general-purpose processors, digital signal processors (English: Digital Signal Processor, abbreviated: DSP), application-specific integrated circuits (English: Application Specific Integrated Circuit, abbreviated: ASIC), etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in combination with the invention can be directly embodied as being executed by a hardware processor, or executed by a combination of hardware and software modules in the processor.

[0130] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (Non-volatile Memory, NVM), such as at least one disk memory.

[0131] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, the bus in the drawings of this application is not limited to only one bus or one type of bus.

[0132] This application also provides a computer program product, including a computer program, which implements the above-mentioned method when executed by a processor.

[0133] The present application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above method.

[0134] The above-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium accessible by a general-purpose or special-purpose computer.

[0135] An exemplary readable storage medium is coupled to the processor so that the processor can read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the readable storage medium can also exist as discrete components in a device.

[0136] The division of units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection between each other can be through some interfaces, and the indirect coupling or communication connection of devices or units can be in electrical, mechanical or other forms.

[0137] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0138] In addition, in each embodiment of the present invention, the functional units can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit.

[0139] If a function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present invention. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0140] Those of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When this program is executed, it executes the steps including the above method embodiments; and the aforementioned storage medium includes: various media such as ROM, RAM, magnetic disks, or optical discs that can store program codes.

[0141] It should be noted that for the above method embodiments, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should know that this application is not limited by the described action sequence, because according to this application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0142] Furthermore, it should be noted that although the steps in the flowchart are shown in sequence according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps has no strict order limit, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential either, but can be executed alternately or in turn with at least a part of other steps or sub-steps or stages of other steps.

[0143] It should be understood that the above device embodiments are merely illustrative, and the devices of the present application can also be implemented in other ways. For example, the division of units / modules in the above embodiments is only a logical function division, and there can be other division methods in actual implementation. For example, multiple units, modules or components can be combined, or can be integrated into another system, or some features can be ignored or not executed.

[0144] In addition, without special instructions, in each embodiment of the present application, each functional unit / module can be integrated in one unit / module, or each unit / module can exist physically alone, or two or more units / modules can be integrated together. The above integrated unit / module can be implemented in the form of hardware or in the form of a software program module.

[0145] When the integrated unit / module is implemented in the form of hardware, the hardware can be a digital circuit, an analog circuit, etc. The physical implementation of the hardware structure includes but is not limited to transistors, memristors, etc. Without special instructions, the processor can be any suitable hardware processor, such as CPU, GPU, FPGA, DSP, and ASIC, etc. Without special instructions, the storage unit can be any suitable magnetic storage medium or magneto-optical storage medium, such as resistive random access memory RRAM (Resistive Random Access Memory), dynamic random access memory DRAM (Dynamic Random Access Memory), static random access memory SRAM (Static Random-Access Memory), enhanced dynamic random access memory EDRAM (Enhanced Dynamic Random Access Memory), high-bandwidth memory HBM (High-Bandwidth Memory), hybrid memory cube HMC (Hybrid Memory Cube), etc.

[0146] When the integrated unit / module is implemented in the form of a software program module and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present application. The aforementioned memory includes: various media that can store program codes, such as USB flash drives, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), mobile hard disks, magnetic disks, or optical discs.

[0147] In the above embodiments, the descriptions of the various embodiments each have their own emphases. For the parts not detailed in a certain embodiment, reference can be made to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as falling within the scope described in this specification.

[0148] Those skilled in the art will readily think of other implementation manners of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any variations, uses, or adaptations of the present application, which follow the general principles of the present application and include the common general knowledge or conventional technical means in the technical field not disclosed in the present application. The specification and the embodiments are only regarded as exemplary, and the true scope and spirit of the present application are pointed out by the following claims.

[0149] It should be understood that the present application is not limited to the exact structure already described and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present application is only limited by the appended claims.

Claims

1. A data quality detection method, characterized in that: include: Get metadata of database tables of multiple application systems; Parsing the metadata of the database table of each application system to obtain a parsing result, wherein the parsing result includes the database table fields of each application system and the corresponding business meanings, and the database table fields with the same business meaning in different application systems; Generate one or more data quality detection rules according to the analysis results, each of the data quality detection rules corresponding to one or more application systems; Each of the data quality detection rules is converted into a corresponding data query statement, the data query statement is sent to the one or more application systems corresponding to the data quality detection rule, and a quality detection result is generated according to the data query results returned by the one or more application systems.

2. The method according to claim 1, characterized in that The step of obtaining metadata of database tables of the plurality of application systems includes: The respective databases of the multiple application systems are accessed through an application interface or a database direct connection to obtain metadata of the respective database tables of the multiple application systems.

3. The method according to claim 1, characterized in that The analysis results obtained by analyzing the metadata of the database tables of each application system include: The metadata of the database tables of the application systems are input into a pre-trained big model, and the knowledge related to the metadata is retrieved from a preset expert knowledge base corresponding to the application systems through the big model, and the analysis result is generated based on the metadata and the knowledge related to the metadata.

4. The method according to any one of claims 1 to 3, characterized in that: Generating the quality inspection result according to the data query result returned by the one or more application systems includes: According to the data query result returned by the application system, determine whether the data query result matches the abnormality judgment condition corresponding to the quality detection rule. If it matches, determine that the quality detection result is abnormal; if it does not match, determine that the quality detection result is normal.

5. The method according to claim 4, characterized in that Also includes: Generate an abnormal problem list according to abnormal quality detection results among the quality detection results of one or more application systems corresponding to each of the data quality detection rules, wherein the abnormal problem list includes one or more abnormal problems and processing statuses corresponding to the abnormal problems; In response to a processing operation on any abnormal problem in the abnormal problem list, a processing status of any abnormal problem in the abnormal problem list is updated.

6. The method according to any one of claims 1 to 3, characterized in that: The step of converting each of the data quality detection rules into a corresponding data query statement includes: Generating a review request for the data quality detection rule; In response to the review request being passed, each of the data quality detection rules is converted into a corresponding data query statement.

7. The method according to any one of claims 1 to 3, characterized in that: After parsing the metadata of the database tables of each application system to obtain the parsing results, the method further includes: A data directory corresponding to the metadata of the database table of each application system is established according to the analysis result, wherein the data directory includes the correspondence between the application system, the database table field and the business meaning.

8. A data quality detection device, characterized in that: include: An acquisition module is used to acquire metadata of database tables of multiple application systems; A processing module, used to parse the metadata of the database table of each application system to obtain a parsing result, wherein the parsing result includes the database table fields of each application system and the corresponding business meanings, and the database table fields with the same business meaning in different application systems; A generating module, used to generate one or more data quality detection rules according to the analysis result, each of which corresponds to one or more application systems; The processing module is further used to convert each of the data quality detection rules into a corresponding data query statement; A sending module, used for sending the data query statement to the one or more application systems corresponding to the data quality detection rule; The generating module is further used to generate quality detection results according to the data query results returned by the one or more application systems.

9. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor.

11. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 7 when being executed by a processor.

Images