Metadata-driven Data Patrol Inspection and Version Management Method, Device, and Electronic Equipment

Abstract

An embodiment of the present invention discloses a metadata-driven data inspection and version management method, device, and electronic device, including: establishing communication connections with respective target data sources based on the target data sources, establishing an initial metadata set according to the respective target data sources, and establishing a metadata version number for each metadata in the initial metadata set; establishing a storage directory according to the external metadata in the initial metadata set, and establishing a directory version number for each directory entry in the storage directory; inspecting the external metadata associated with each directory entry based on a first inspection instruction and the target data source, and updating the external metadata, the metadata version number of the external metadata, and the directory version number of the directory entry corresponding to the updated external metadata based on the inspection result; inspecting the remaining metadata based on a second inspection instruction and the target data source, and updating the remaining metadata and the metadata version number of the remaining metadata, achieving the effect of improving the inspection speed and pertinence.

Description

Technical Field

[0001] The embodiments of the present invention relate to the technical field of data patrol inspection, and in particular, to a metadata-driven data patrol inspection and version management method, apparatus, and electronic device. Background Art

[0002] Data has become an important asset in various industries. The quality of data is directly related to the accuracy of information and also affects the normal operation of the industry and the integrity of data. Therefore, it is necessary to ensure the accuracy and uniformity of each data.

[0003] Currently, the way to ensure the accuracy and uniformity of data is to conduct a full-scale patrol inspection of data regularly. However, due to the large amount of data stored in the system, conducting a full-scale patrol inspection will lead to problems such as slow speed and lack of pertinence, and it is difficult to determine the changes in data after the data patrol inspection.

[0004] There is a large amount of data such as backup data, intermediate data, and temporary data in medical data. Conducting a full-scale patrol inspection of these data is a waste of resources and has no practical business significance. Therefore, through the full-scale patrol inspection method, it is impossible to patrol and update meaningful data in a targeted manner. Moreover, there is no unified and convenient access method for the data sources of different medical data, and the versions of the data obtained after access cannot be standardized managed and displayed. Summary of the Invention

[0005] The embodiments of the present invention provide a metadata-driven data patrol inspection and version management method, apparatus, and electronic device to achieve the technical effects of improving the data patrol inspection speed and pertinence and recording the differences in data patrol inspection.

[0006] In a first aspect, the embodiments of the present invention provide a metadata-driven data patrol inspection and version management method, and the method includes:

[0007] Based on at least one target data source, communication connections with each target data source are respectively established, and an initial metadata set is established according to each target data source, and a metadata version number is established for each metadata in the initial metadata set; wherein, the initial metadata set includes external metadata and remaining metadata;

[0008] According to the external metadata in the initial metadata set, a storage directory is established, and a directory version number is established for each directory entry in the storage directory;

[0009] When receiving the first inspection instruction, based on the first inspection instruction and the target data source, inspect the external metadata associated with each directory entry in the storage directory, and update the external metadata, the metadata version number of the external metadata, and the directory version number of the directory entry corresponding to the updated external metadata based on the inspection result;

[0010] When receiving the second inspection instruction, based on the second inspection instruction and the target data source, inspect the remaining metadata in the initial metadata set, and update the remaining metadata and the metadata version number of the remaining metadata based on the inspection result.

[0011] In a second aspect, an embodiment of the present invention further provides a metadata-driven data inspection and version management device, which includes:

[0012] An initial metadata set establishment module, configured to respectively establish communication connections with each target data source based on at least one target data source, establish an initial metadata set according to each target data source, and establish a metadata version number for each metadata in the initial metadata set; wherein, the initial metadata set includes external metadata and remaining metadata;

[0013] A storage directory establishment module, configured to establish a storage directory according to the external metadata in the initial metadata set, and establish a directory version number for each directory entry in the storage directory;

[0014] A first inspection module, configured to when receiving the first inspection instruction, based on the first inspection instruction and the target data source, inspect the external metadata associated with each directory entry in the storage directory, and update the external metadata, the metadata version number of the external metadata, and the directory version number of the directory entry corresponding to the updated external metadata based on the inspection result;

[0015] A second inspection module, configured to when receiving the second inspection instruction, based on the second inspection instruction and the target data source, inspect the remaining metadata in the initial metadata set, and update the remaining metadata and the metadata version number of the remaining metadata based on the inspection result.

[0016] In a third aspect, an embodiment of the present invention further provides an electronic device, and the electronic device includes:

[0017] One or more processors;

[0018] A storage device, configured to store one or more programs,

[0019] When the one or more programs are executed by the one or more processors, the one or more processors implement the metadata-driven data inspection and version management method according to any one of the embodiments of the present invention.

[0020] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, it implements the metadata-driven data inspection and version management method according to any one of the embodiments of the present invention.

[0021] The technical solution of the embodiment of the present invention is to respectively establish communication connections with each target data source based on at least one target data source, establish an initial metadata set according to each target data source, establish a metadata version number for each metadata in the initial metadata set, so as to collect and obtain the metadata in each target data source and assign a metadata version number. Furthermore, according to the external metadata in the initial metadata set, a storage directory is established, and a directory version number is established for each directory entry in the storage directory, so as to establish a storage directory and assign a directory version number. When a first inspection instruction is received, based on the first inspection instruction and the target data source, the external metadata associated with each directory entry in the storage directory is inspected, and based on the inspection result, the external metadata, the metadata version number of the external metadata, and the directory version number of the directory entry corresponding to the updated external metadata are updated; when a second inspection instruction is received, based on the second inspection instruction and the target data source, the remaining metadata in the initial metadata set is inspected, and based on the inspection result, the remaining metadata and the metadata version number of the remaining metadata are updated, which solves the problems of large amount of data inspection, long time consumption, and difficulty in determining the data differences before and after data inspection due to a large number of target data sources, realizes the technical effect of dividing data elements through directory entries, improving the data inspection speed and pertinence, and recording the differences in data inspection. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the introduced drawings are only the drawings of a part of the embodiments to be described by the present invention, rather than all the drawings. For those of ordinary skill in the art, other drawings can be obtained according to these drawings without creative efforts.

[0023] Figure 1 It is a schematic flowchart of a metadata-driven data inspection and version management method provided by Embodiment 1 of the present invention;

[0024] Figure 2 It is a schematic flowchart of a metadata-driven data inspection and version management method provided by Embodiment 2 of the present invention;

[0025] Figure 3 The structural schematic diagram of a metadata-driven data inspection and version management system provided in the second embodiment of the present invention;

[0026] Figure 4 The process schematic diagram of reviewing the processing operations for inspection results provided in the second embodiment of the present invention;

[0027] Figure 5 The structural schematic diagram of a metadata-driven data inspection and version management device provided in the third embodiment of the present invention;

[0028] Figure 6 The structural schematic diagram of an electronic device provided in the fourth embodiment of the present invention. Detailed implementation manners

[0029] The present invention will be further described in detail below with reference to the drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the present invention, rather than limiting the present invention. Additionally, it should be noted that for the convenience of description, only the parts related to the present invention rather than all the structures are shown in the drawings.

[0030] Embodiment 1

[0031] Figure 1 The process schematic diagram of a metadata-driven data inspection and version management method provided in the first embodiment of the present invention. This embodiment is applicable to the situation of performing data inspection and version management on the metadata in the accessed data source. This method can be executed by a metadata-driven data inspection and version management device, which can be implemented in the form of software and / or hardware. The hardware can be an electronic device. Optionally, the electronic device can be a mobile terminal, a PC, a server, etc.

[0032] As Figure 1 described, the method of this embodiment specifically includes the following steps:

[0033] S110. Based on at least one target data source, establish communication connections with each target data source respectively, and establish an initial metadata set according to each target data source, and establish a metadata version number for each metadata in the initial metadata set. Among them, the target data source can be various data sources corresponding to data integration, which are devices or original media that provide the required data, such as databases, data servers, etc. The initial metadata set can be a data set composed of metadata in each target data source. Metadata can be data about data, such as metadata information of databases, tables, columns, indexes, stored procedures, views, etc. Through metadata, information support can be provided for business functions such as data quality management. The metadata version number can be an independent version number established for different metadata. The initial metadata set includes external metadata and remaining metadata. The external metadata can be metadata that each external platform needs to obtain. The external platform can be understood as a third-party business platform, etc., such as a data quality platform, a Business Intelligence (BI) platform, etc. The remaining metadata can be the metadata of the remaining part after removing the external metadata from the metadata in the initial metadata set, such as backup data, intermediate data, temporary data, etc.

[0034] Specifically, determine at least one target data source that needs to establish a connection according to actual needs, and establish communication connections with these target data sources to obtain all the metadata in the target data sources. Furthermore, integrate all the obtained metadata to obtain an initial metadata set, and establish a metadata version number for each metadata in the initial metadata set. At this time, the metadata version number can be the initial metadata version number, such as v1.0, etc.

[0035] Exemplarily, if the target data source is the in-hospital anesthesia database A, the data acquisition and adaptation module can be used to connect to database A and obtain metadata to generate an initial metadata set, and assign an initial metadata version number, such as v1.0, to each metadata in the initial metadata set.

[0036] S120. According to the external metadata in the initial metadata set, establish a storage directory, and establish a directory version number for each directory entry in the storage directory.

[0037] Among them, the storage directory can be a metadata directory constructed according to different hierarchical structures of metadata. The hierarchical structure includes but is not limited to data domain, data source, data type, etc. The directory entry can be the content of each level of directory in the storage directory. The directory version number can be an independent version number established for different directory entries.

[0038] Specifically, the external metadata is determined from the initial metadata set according to the external platform, and the external metadata is used to construct a storage directory according to a certain hierarchical structure. Furthermore, directory version numbers can be established for different directory entries in the storage directory. At this time, the directory version number can be the initial directory version number, such as v1.0, etc.

[0039] Exemplarily, the data corresponding to each directory entry in the storage directory is all referenced from the data set obtained by the acquisition adaptation module. The data in the data set can be divided according to the directory entries. For example, it can be divided according to the processing flow to the data lake, data center, data domain, and data supermarket. This flow can be dynamically added or deleted according to medical needs. The directory entries determined according to the flow can be further logically divided into the following directory entries. For example, the directory entries corresponding to the three-level hierarchy of library-table-field. It can be understood that the data element belongs to the field level. The attributes of the data element include but are not limited to the logical layer it belongs to, data type, type length, allowed values, etc. Each data element has a version number. Similarly, there are also attributes and version numbers for the library and table. For example, the library type, library connection string, master-slave library, etc. of the library, and the table type, table resource identifier, table description, etc. of the table. When subsequent data inspection detects that any of the above attributes is modified, a new version number and version history record will be generated accordingly.

[0040] Exemplarily, for the data allowed to be introduced into the storage directory in the initial metadata set, the version number of the data referenced in the storage directory can directly inherit the version number obtained by acquisition adaptation (the data not allowed does not affect the version number). If the data in the storage directory changes (including but not limited to manual changes, inspection and repair changes), the version number of the changed data element in the acquisition adaptation module will change according to the number of modifications. For example, it is modified to v1.1, v1.2, etc. The version number of the data corresponding to the storage directory can directly reference the modified version number.

[0041] S130. When the first inspection instruction is received, based on the first inspection instruction and the target data source, inspect the external metadata associated with each directory entry in the storage directory, and update the external metadata, the metadata version number of the external metadata, and the directory version number of the directory entry corresponding to the updated external metadata based on the inspection result.

[0042] Among them, the first inspection instruction can be a code or command for data inspection of each external metadata associated with the storage directory. The inspection result can be the result of determining whether there are differences in the data after data inspection. For example, the inspection result can include differences exist and no differences exist, etc.

[0043] Specifically, when the first inspection instruction is received, a data inspection operation based on the storage directory can be triggered based on the first inspection instruction. Data inspection is performed on the external metadata corresponding to each directory entry in the storage directory according to each target data source to obtain an inspection result. If the inspection result shows a difference, the external metadata with differences is updated, and the metadata version number corresponding to the external metadata is updated. For example, it is updated from v1.0 to v2.0, etc. Further, the update of the external metadata will cause the associated content of the directory entry corresponding to the updated external metadata to change. Therefore, the directory version number of the directory entry corresponding to the updated external metadata is updated.

[0044] Exemplarily, the inspection results of data inspection for a library can include two types: consistent and inconsistent. Among them, consistent means that all table contents under the library are the same; inconsistent means that there is at least one different table under the library. The inspection results of data inspection for a table or a field can include four types: consistent, inconsistent, new, and missing.

[0045] S140. When the second inspection instruction is received, based on the second inspection instruction and the target data source, the remaining metadata in the initial metadata set is inspected, and the remaining metadata and the metadata version number of the remaining metadata are updated based on the inspection result.

[0046] Among them, the second inspection instruction can be code or a command for data inspection of each remaining metadata outside the storage directory, etc.

[0047] Specifically, when the second inspection instruction is received, a data inspection operation for the remaining metadata can be triggered based on the second inspection instruction. Data inspection is performed on each remaining metadata according to each target data source to obtain an inspection result. If the inspection result shows a difference, the remaining metadata with differences is updated, and the metadata version number corresponding to the remaining metadata is updated. For example, it is updated from v1.0 to v2.0, etc.

[0048] In a medical scenario, through the technical solution of this embodiment, the problem of long full-scale inspection time caused by the large amount of data in the original business databases such as HIS (Hospital Information System) / LIS (Laboratory Information Management System) can be solved. By classifying the data into different directory items, such as dividing according to the business logic layer, and there are corresponding business databases under each business logic layer, such as RDR (Research Data Repository) / CDR (Clinical Data Repository), etc. Furthermore, determine the data corresponding to the storage directory according to the classification result. During use, hospital information department personnel or department doctors can select the required storage directory according to actual business needs for targeted data verification or other operations.

[0049] The technical solution of the embodiment of the present invention is to respectively establish communication connections with each target data source based on at least one target data source, and establish an initial metadata set according to each target data source, and establish a metadata version number for each metadata in the initial metadata set to collect and obtain the metadata in each target data source and assign a metadata version number. Furthermore, establish a storage directory according to the external metadata in the initial metadata set, and establish a directory version number for each directory item in the storage directory to establish a storage directory and assign a directory version number. When receiving a first inspection instruction, based on the first inspection instruction and the target data source, inspect the external metadata associated with each directory item in the storage directory, and update the external metadata, the metadata version number of the external metadata, and the directory version number of the directory item corresponding to the updated external metadata based on the inspection result; when receiving a second inspection instruction, based on the second inspection instruction and the target data source, inspect the remaining metadata in the initial metadata set, and update the remaining metadata and the metadata version number of the remaining metadata based on the inspection result, solving the problems of large amount of data and long time-consuming for data inspection, and difficulty in determining the data differences before and after data inspection, achieving the technical effects of improving the data inspection speed and pertinence, and recording the differences in data inspection.

[0050] Embodiment 2

[0051] Figure 2 It is a schematic flowchart of a metadata-driven data inspection and version management method provided by the second embodiment of the present invention. Based on the above embodiments, for the establishment method of the storage directory and the specific method of data inspection, refer to the technical solution of this embodiment. Among them, the explanations of the same or corresponding terms as the above embodiments will not be repeated here.

[0052] As Figure 2 described above, the method of this embodiment specifically includes the following steps:

[0053] S210. Based on at least one target data source, establish communication connections with each target data source respectively, and establish an initial metadata set according to each target data source, and establish a metadata version number for each metadata in the initial metadata set.

[0054] Specifically, establish a communication connection with at least one target data source according to actual requirements to obtain all the metadata in the target data source. Furthermore, integrate all the obtained metadata to obtain an initial metadata set, and establish a metadata version number for each metadata in the initial metadata set.

[0055] Optionally, the target data source can be determined and a communication connection can be established with the target data source in the following manner:

[0056] Determine at least one target data source according to the data source information in the data source connection information; establish a communication connection with at least one target data source according to the configuration information in the data source connection information.

[0057] Among them, the data source connection information can be information indicating the connection of the data source. The data source information can be various basic information identifying the data source, such as: data source name, data source identifier, storage location, etc. The configuration information can include the account, password, IP (Internet Protocol) address, etc. required to connect to the data source.

[0058] Specifically, according to the data source name, data source identifier, storage location, etc. of the target data source mentioned in the data source connection information, the target data source can be located. Furthermore, match and log in with the located target data source according to the configuration information in the target data source connection information to establish a communication connection, so as to facilitate subsequent acquisition of the data in the target data source.

[0059] Optionally, the initial metadata set can be established and the metadata version number can be assigned through the following steps:

[0060] Step 1. Collect the metadata in each target data source, perform data inspection on the metadata according to the preset inspection items and each target data source, and establish an initial metadata set according to the metadata after data inspection.

[0061] Among them, the inspection configuration can be used to inspect the collected metadata according to the data in the target data source. The preset inspection items include at least one of data type, data length, data element definition, and metadata model. The inspection content can include: database connection exception, database new table, database delete table, data table new field, data table delete field, field attribute change, etc.

[0062] Specifically, collect each metadata from each target data source, and perform a full-scale data inspection on the collected metadata to ensure the integrity and accuracy of the loaded metadata. Furthermore, perform a one-by-one inspection on the loaded metadata according to the preset inspection items. After the metadata passes the data inspection, integrate the data-inspected metadata to establish an initial metadata set.

[0063] Step 2: For each metadata in the initial metadata set, establish an initial metadata version number corresponding to the metadata.

[0064] Among them, the initial metadata version number can be a pre-set version number used to identify the original metadata, for example: v1.0, etc.

[0065] Specifically, establish an initial metadata version number corresponding to each metadata in the initial metadata set to identify that these metadata are initially collected.

[0066] If the metadata is updated subsequently, the metadata version number can be updated, from v1.0 to v2.0, from v1.1 to v1.2, etc.

[0067] S220: Determine the external metadata corresponding to the external platform from the initial metadata set.

[0068] Among them, the external platform can be a platform that uses the data in the initial metadata set, for example: a business platform, etc.

[0069] Specifically, according to the data requirements of each external platform, determine the metadata that matches the data requirements from the initial metadata set as the external metadata.

[0070] S230: Divide each external metadata according to the data domain corresponding to each external metadata.

[0071] Among them, the data domain can be each stage in the data flow direction. For example: Data Lake (DL), Data Center (DC), Data Domain, Data Market, etc. Exemplarily, DL can be used to store complete external metadata. DC can be used to initially integrate the external metadata in DL. In the data domain, it can be used to integrate the same data domain in DC together after processing the data according to different data domains and based on preset criteria. The data market can be used to store the derivative data of the external metadata.

[0072] Specifically, according to the requirements of the external platform and the data flow direction, the external metadata can be divided according to different data domains, so as to subsequently match each directory entry in the storage directory.

[0073] It should be noted that dividing each external metadata according to the data domain is a way of metadata division in this embodiment, and it can also be divided according to different division methods, such as project requirements, platform requirements, etc.

[0074] S240. Determine each directory entry of the storage directory according to the hierarchical structure of the data domain, establish the corresponding relationship between each external metadata and each directory entry according to the division result of each external metadata, and establish a directory version number for each directory entry in the storage directory.

[0075] Specifically, construct the directory entries of each level according to the hierarchical structure of the data domain to construct the structure of the storage directory. Furthermore, according to the division result of the external metadata, match the external metadata with each directory entry to establish the corresponding relationship between the external metadata and the directory entry. And an initial directory version number can be assigned to each directory entry in the storage directory.

[0076] S250. When receiving the first inspection instruction, based on the first inspection instruction and the target data source, inspect the external metadata associated with each directory entry in the storage directory, and update the external metadata, the metadata version number of the external metadata, and the directory version number of the directory entry corresponding to the updated external metadata based on the inspection result.

[0077] Specifically, when receiving the first inspection instruction, perform data inspection on the external metadata corresponding to each directory entry in the storage directory according to each target data source to obtain the inspection result. If the inspection result shows a difference, update the external metadata with the difference, update the metadata version number corresponding to the external metadata, and update the directory version number of the directory entry corresponding to the updated external metadata.

[0078] Optionally, the content of the inspection can include: abnormal database connection, new database tables, deleted database tables, new fields in data tables, deleted fields in data tables, changed field attributes, etc.

[0079] Optionally, during subsequent data inspections, for the inspection results, partial or full updates can be made to the external metadata. The specific data selected for update can be chosen according to requirements and is not specifically limited in this embodiment.

[0080] Optionally, the following steps can be used to perform data inspection based on the first inspection instruction with the storage directory as the reference point:

[0081] Step 1: Parse the first inspection instruction to determine the inspection directory item corresponding to the first inspection instruction.

[0082] Among them, the inspection directory item can be the directory item to be inspected carried in the first inspection instruction and can be at least one directory item.

[0083] Specifically, parse the first inspection instruction and determine the inspection directory item carried in the first inspection instruction from the parsing result to perform data inspection on the external metadata according to the inspection directory item.

[0084] Step 2: Determine the data source to be inspected from the target data source according to the data lineage of the external metadata corresponding to the inspection directory item.

[0085] Among them, the data lineage can be used to describe the source and destination of data, etc., and mainly can include: the source of data, the data processing method, the mapping relationship, and the data export, etc. The inspection data source can be the target data source that needs to be matched when inspecting the inspection directory item.

[0086] Exemplarily, the data lineage is used to trace back the data source of a data object upwards. For example: In the first case, there is a field named "length of hospitalization" under the DOMAIN directory item, and the source of this field is the "admission time" and "discharge time" of the upper-level DC. In the second case, downstream data directly references upstream data, such as the patient identifier in the in-hospital original business database being referenced by downstream data tables.

[0087] Specifically, after determining the inspection directory item, the external metadata corresponding to the inspection directory item can be determined. Furthermore, according to the data lineage of the determined external metadata, trace back to determine the data source of these external metadata, that is, the inspection data source.

[0088] Step 3: Perform data inspection on the external metadata corresponding to the inspection directory item according to the data source to be inspected.

[0089] Specifically, metadata corresponding to the external metadata corresponding to the inspection catalog item is obtained from the data source to be inspected. Data inspection is performed on the external metadata corresponding to the inspection catalog item according to the metadata obtained from the data source to be inspected to obtain an inspection result.

[0090] Step 4: If the external metadata corresponding to the inspection catalog item is updated according to the inspection result, the metadata version number of the external metadata to be updated is updated, an external metadata update record is generated based on the external metadata before the update and the external metadata after the update, and the catalog version number of the inspection catalog item corresponding to the external metadata to be updated is updated.

[0091] Among them, the external metadata update record can be a record that records the external metadata before and after the update.

[0092] Specifically, if the inspection result shows a difference and the external metadata corresponding to the inspection catalog item is updated, it indicates that the external metadata has changed, and the metadata version number of the external metadata to be updated can be updated. In order to record the changes in the external metadata, the external metadata before the update and the external metadata after the update can be integrated to obtain an external metadata update record. For example: combining the external metadata in JSON format before the update and the external metadata in JSON format after the update as the external metadata update record for this update. Since the data inspection is performed based on the inspection catalog item as the reference point, when the external metadata changes, the catalog version number of the inspection catalog item needs to be updated to indicate that the external metadata corresponding to the inspection catalog item has changed.

[0093] Exemplarily, updating the metadata version number of the external metadata can be understood as: taking the initial metadata version number of the external metadata as the current metadata version number, performing data inspection on the external metadata. If the external metadata is updated according to the inspection result, the current metadata version number is updated, and the updated current metadata version number is used as the new current metadata version number, and the operation of performing data inspection on the external metadata is returned for execution. It should be noted that a similar method can also be used for the update operation of the catalog version number of the catalog item and the metadata version number of the remaining metadata mentioned later, which will not be elaborated here.

[0094] S260: When a second inspection instruction is received, based on the second inspection instruction and the target data source, inspect the remaining metadata in the initial metadata set, and update the remaining metadata and the metadata version number of the remaining metadata based on the inspection result.

[0095] Specifically, when the second patrol inspection instruction is received, the data patrol inspection operation of the remaining metadata can be triggered based on the second patrol inspection instruction, and the data patrol inspection of each remaining metadata is performed according to each target data source to obtain a patrol inspection result. If the patrol inspection result shows a difference, the remaining metadata with differences is updated, and the metadata version number corresponding to the remaining metadata is updated.

[0096] Optionally, the following steps can be used to perform data patrol inspection based on the remaining metadata with the second patrol inspection instruction as the reference point:

[0097] Step 1: Determine the data sources to be patrolled from the target data sources according to the remaining metadata in the initial metadata set.

[0098] Among them, the patrolled data source can be the target data source that needs to be matched when patrolling the remaining metadata.

[0099] Specifically, for each remaining metadata in the initial metadata set, determine the data sources to be patrolled corresponding to the remaining metadata. For example: Determine the data sources to be patrolled from the target data sources through the data lineage of the remaining metadata.

[0100] Step 2: Perform data patrol inspection on the remaining metadata according to the data sources to be patrolled.

[0101] Specifically, collect the data information corresponding to the remaining metadata from the data sources to be patrolled, and compare the determined data information with the remaining metadata to perform data patrol inspection.

[0102] Step 3: If the remaining metadata is updated according to the patrol inspection result, update the metadata version number of the remaining metadata that is updated, and generate a remaining metadata update record based on the remaining metadata before the update and the remaining metadata after the update.

[0103] Among them, the remaining metadata update record can be a record that records the remaining metadata before and after the update.

[0104] Specifically, if the patrol inspection result shows a difference and the remaining metadata is updated, it indicates that the remaining metadata has changed, and the metadata version number of the remaining metadata that is updated can be updated. In order to record the changes in the remaining metadata, the remaining metadata before the update and the remaining metadata after the update can be integrated to obtain a remaining metadata update record. For example: Combine the remaining metadata in JSON format before the update and the remaining metadata in JSON format after the update as the remaining metadata update record for this update.

[0105] Based on the above embodiments, since there are differences in the inspection results, metadata will be updated. There may be frequent updates, and metadata that does not need to be updated will also be updated, wasting time and resources. Therefore, an audit operation can be added after data inspection and before metadata update. Specifically, it can be:

[0106] Step 1: If the inspection result shows differences, generate a request for audit instruction to obtain the audit result.

[0107] Among them, the audit instruction can be an instruction to request an audit of the metadata differences to determine whether to perform synchronous update. The audit instruction can carry the metadata identifier and differences with differences. The audit result can be the result of whether to update the metadata obtained by auditing the metadata differences. For example, the audit result can include passing the audit and failing the audit.

[0108] Specifically, if the inspection result shows differences, generate an audit instruction based on the metadata with differences and send the audit instruction to the terminal device of the auditor to obtain the audit result.

[0109] It should be noted that the audit instruction can indicate the location information of the metadata with differences. The auditor can retrieve the metadata according to the location information for manual audit to determine whether synchronous update is required. The audit instruction can also directly carry the metadata with differences. For example, name and nameCn. The auditor can directly judge whether synchronous update is required according to the information carried in the audit instruction.

[0110] Step 2: If the audit result is failing the audit, keep the external metadata and the remaining metadata unchanged.

[0111] Among them, failing the audit means ignoring the differences, that is, the audit result that does not require updating the metadata with differences.

[0112] Specifically, if the audit result is failing the audit, it indicates that the differences in the metadata detected by data inspection are not necessary to be updated and can be ignored. Therefore, the external metadata and the remaining metadata can be kept unchanged for continued use.

[0113] It should be noted that each piece of metadata with differences can be audited separately. Then, the audit results of different metadata with differences may be different, and subsequent operations can be performed separately according to different audit results.

[0114] Step 3: If the audit result is passing the audit, update the external metadata and perform the operation of updating the metadata version number of the external metadata and the directory version numbers of each directory item based on the inspection result and / or the operation of updating the remaining metadata and the metadata version number of the remaining metadata based on the inspection result.

[0115] Among them, passing the review indicates synchronizing differences, that is, the review result for data synchronization and update of metadata with differences.

[0116] Specifically, if the review result is passing the review, it is necessary to perform synchronous update for the metadata corresponding to the review result. If the metadata to be synchronously updated is external metadata, after updating the external metadata, perform the operation of updating the metadata version number of the external metadata and the directory version numbers of each directory entry based on the inspection result. If the metadata to be synchronously updated is the remaining connected metadata, after updating the remaining metadata, perform the operation of updating the remaining metadata and the metadata version number of the remaining metadata based on the inspection result.

[0117] Exemplarily, for the review method, it can be divided into the review of libraries, tables, and fields, and can implement: specifying to ignore, specifying to synchronize, and one-key batch ignoring and synchronizing, as well as version release - the function of releasing all events at this level. For the inspection results with differences, the control of the approval process can be carried out, such as: the one-key ignore operation needs to be approved by the approver to take effect.

[0118] Based on the above embodiments, in order to select different data inspection benchmark points for data inspection according to business requirements, the first inspection instruction and / or the second inspection instruction can be generated in the following manner:

[0119] If the current time reaches the preset inspection cycle, generate the first inspection instruction and / or the second inspection instruction.

[0120] Among them, the preset inspection cycle can be a cycle for data inspection set in advance, for example: 1 day, etc. The preset inspection cycle can be artificially set and modified according to data inspection requirements, and is not specifically limited in this embodiment.

[0121] Specifically, when the current time reaches the preset data inspection cycle, it indicates that data inspection of the stored metadata is required. At this time, the first inspection instruction and / or the second inspection instruction can be generated to perform data inspection on the external metadata and / or the remaining metadata.

[0122] It should be noted that the preset inspection cycle can include the first inspection cycle and the second inspection cycle. If the current time reaches the first inspection cycle, generate the first inspection instruction; if the current time reaches the second inspection cycle, generate the second inspection instruction.

[0123] If an inspection trigger behavior for at least one directory entry is detected, generate the second inspection instruction according to the at least one directory entry.

[0124] Among them, the inspection trigger behavior can be an operation behavior for data inspection on the external metadata corresponding to the directory entry. For example, the inspection trigger behavior can be an operation such as clicking the inspection control of the directory entry or executing the code for inspecting the directory entry.

[0125] Specifically, when an inspection trigger behavior for at least one directory entry is detected, at least one directory entry corresponding to the inspection trigger behavior is determined. Furthermore, a second inspection instruction is generated according to the at least one directory entry to perform data inspection on the external metadata corresponding to these directory entries.

[0126] As an optional implementation solution of the above embodiments, Figure 3 FIG. is a schematic structural diagram of a metadata-driven data inspection and version management system provided in Embodiment II of the present invention. Among them, the explanations of the same or corresponding terms as those in the above embodiments are not repeated here.

[0127] As Figure 3 shown, the metadata-driven data inspection and version management system includes a data acquisition adaptation module and a storage directory module.

[0128] The technical solution of the embodiment of the present invention is executed based on an execution entity, that is, executed based on the metadata-driven data inspection and version management system. The data acquisition adaptation module in the metadata-driven data inspection and version management system can establish a communication connection between the system and each data source (target data source) to obtain data from each data source.

[0129] The data acquisition adaptation module can complete the acquisition of various types of database information and specific table and field contents in the data source through basic information such as IP, account, and password, and realize the association of the corresponding acquisition library and application, providing a data basis and blood relationship parsing and positioning for the later storage directory.

[0130] Specifically, the metadata-driven data inspection and version management system loads the data in the specified external data source into the acquisition adaptation module. During the first data loading process, a full-scale data inspection is performed simultaneously. The inspection items (preset inspection items) include but are not limited to: data type, data type length, data element definition, meta-model, etc., and the initial version (initial metadata version number) of this data is recorded simultaneously. When the data is first incorporated into the data acquisition adaptation module, an initial version, such as a version number of v1.0, will be assigned. Subsequently, whether it is manually modified or automatically modified during the inspection, it will be updated based on this initial version, and the modification history of the data version will be recorded.

[0131] After the data is loaded through the acquisition adaptation module, a reference to the acquired data is established through the storage directory module to obtain the storage directory.

[0132] Among them, the data source corresponding to the storage directory is the acquisition adaptation module, but it is not necessarily all the data collected in the acquisition adaptation module. What the storage directory corresponds to is all the data used by each platform (external platform), such as library, table, field information, lineage information, meta-model, etc.

[0133] It should be noted that version updates do not require manual settings. For sensitive operations such as editing and deletion, there will be operation approvals (audit instructions). After the approval is passed, the version will be automatically recorded and the difference records before and after the modification will be generated (external metadata update records and / or remaining metadata update records).

[0134] During the data inspection process, data inspection can be carried out in the following two ways:

[0135] Method 1: If data inspection is carried out on the metadata (remaining metadata) that is loaded into the data acquisition adaptation module but not referenced by the storage directory, the data in the data source is used as the reference point to inspect and verify the data.

[0136] Method 2: If data inspection is carried out on the metadata (external metadata) that is loaded into the data acquisition adaptation module, referenced by the storage directory and associated with relevant applications, the data in the storage directory is used as the reference point to inspect and verify the data source.

[0137] For the inspection results, only the inconsistent inspection results (with differences) are recorded. The version recording methods of the two methods are the same: record the differences before and after the data update.

[0138] Regarding the process of operation approval, the following example can be referred to. Exemplarily, if the metadata targeted by the data inspection is: Table B in Database A - field name name, and it is found after the data inspection that name in the field name name of Table B in Database A in the data source has been changed to nameCn. At this time, the auditor has two options: 1. Ignore the difference (audit not passed), 2. Synchronize the difference (audit passed). If the option to ignore the difference is selected, no further attention will be paid to any differences and the metadata will remain unchanged; if the option to synchronize the difference is selected, the metadata in the data source will be synchronized to the system, that is, name will be changed to nameCn, and two version records of name and nameCn will be created during the synchronization process. Since a unique identifier is assigned to each data in the system, all version records of this data can be queried through this unique identifier.

[0139] Optionally, the processing operations for the inspection results can be reviewed. The specific process is as Figure 4As shown in the figure, specifically, the processing operations for the discrepant inspection results include two types: ignore operation and synchronization operation. Submit the inspection results and the selected processing operations for the inspection results for review. If the review is passed, the system will perform the ignore operation or the synchronization operation according to the selected processing operation; if the review fails, the review process will end, and it can return to perform the selection of processing operations for the discrepant inspection results.

[0140] Optionally, a scheduled task can be set in the system to perform a full-library data scan at regular intervals. The scheduled period can be preset to be daily, and manual modification of the scheduled period is also supported.

[0141] Through the metadata-driven data inspection and version management system, the data differences at different reference points can be visually seen from the page, which is convenient for discussing, solving, data differences and version determination from different reference point perspectives. Moreover, when performing data inspection, two methods are provided: scheduled inspection and manual inspection. The scheduled inspection has a large amount of data and a comprehensive inspection, while the manual inspection has a small inspection scope and is targeted. Inspecting according to different needs brings great convenience to data management.

[0142] The technical solution of the embodiment of the present invention is as follows: Based on at least one target data source, communication connections with each target data source are respectively established, and an initial metadata set is established according to each target data source. A metadata version number is established for each metadata in the initial metadata set to collect and obtain the metadata in each target data source and assign a metadata version number. Furthermore, the external metadata corresponding to the external platform is determined from the initial metadata set, the external metadata is divided according to the data domain corresponding to each external metadata, each directory entry of the storage directory is determined according to the hierarchical structure of the data domain, and the corresponding relationship between each external metadata and each directory entry is established according to the division result of each external metadata. A directory version number is established for each directory entry in the storage directory to establish the storage directory and assign a directory version number. When receiving a first inspection instruction, based on the first inspection instruction and the target data source, the external metadata associated with each directory entry in the storage directory is inspected, and the external metadata, the metadata version number of the external metadata, and the directory version number of the directory entry corresponding to the updated external metadata are updated based on the inspection results; when receiving a second inspection instruction, based on the second inspection instruction and the target data source, the remaining metadata in the initial metadata set is inspected, and the remaining metadata and the metadata version number of the remaining metadata are updated based on the inspection results. This solves the problems of large amount of data and long time consumption in data inspection, and it is difficult to determine the data differences before and after data inspection, achieving the technical effects of improving the data inspection speed and pertinence, and recording the differences in data inspection.

[0143] Embodiment Three

[0144] Figure 5Schematic diagram of a metadata-driven data inspection and version management device provided in Embodiment 3 of the present invention. The device includes: an initial metadata set establishment module 310, a storage directory establishment module 320, a first inspection module 330, and a second inspection module 340.

[0145] Among them, the initial metadata set establishment module 310 is used to establish communication connections with each target data source based on at least one target data source, establish an initial metadata set according to each target data source, and establish a metadata version number for each metadata in the initial metadata set; among them, the initial metadata set includes external metadata and remaining metadata; the storage directory establishment module 320 is used to establish a storage directory according to the external metadata in the initial metadata set, and establish a directory version number for each directory entry in the storage directory; the first inspection module 330 is used to, when receiving a first inspection instruction, inspect the external metadata associated with each directory entry in the storage directory based on the first inspection instruction and the target data source, and update the external metadata, the metadata version number of the external metadata, and the directory version number of the directory entry corresponding to the updated external metadata based on the inspection result; the second inspection module 340 is used to, when receiving a second inspection instruction, inspect the remaining metadata in the initial metadata set based on the second inspection instruction and the target data source, and update the remaining metadata and the metadata version number of the remaining metadata based on the inspection result.

[0146] Optionally, the initial metadata set establishment module 310 is further used to determine at least one target data source according to the data source information in the data source connection information; establish a communication connection with the at least one target data source according to the configuration information in the data source connection information.

[0147] Optionally, the initial metadata set establishment module 310 is further used to collect the metadata in each target data source, perform data inspection on the metadata according to preset inspection items and each target data source, and establish an initial metadata set according to the metadata after data inspection; among them, the preset inspection items include at least one of data type, data length, data element definition, and metadata model; for each metadata in the initial metadata set, establish an initial metadata version number corresponding to the metadata.

[0148] Optionally, the storage directory establishment module 320 is further used to determine the external metadata corresponding to the external platform from the initial metadata set; divide each external metadata according to the data domain corresponding to each external metadata; determine each directory entry of the storage directory according to the hierarchical structure of the data domain, and establish the corresponding relationship between each external metadata and each directory entry according to the division result of each external metadata.

[0149] Optionally, the first inspection module 330 is further configured to parse the first inspection instruction to determine an inspection directory entry corresponding to the first inspection instruction; determine a data source to be inspected from the target data source according to the data lineage of the external metadata corresponding to the inspection directory entry; perform data inspection on the external metadata corresponding to the inspection directory entry according to the data source to be inspected; if the external metadata corresponding to the inspection directory entry is updated according to the inspection result, update the metadata version number of the updated external metadata, generate an external metadata update record based on the external metadata before the update and the external metadata after the update, and update the directory version number of the inspection directory entry corresponding to the updated external metadata.

[0150] Optionally, the second inspection module 340 is further configured to determine a data source to be inspected from the target data source according to the remaining metadata in the initial metadata set; perform data inspection on the remaining metadata according to the data source to be inspected; if the remaining metadata is updated according to the inspection result, update the metadata version number of the updated remaining metadata, and generate a remaining metadata update record based on the remaining metadata before the update and the remaining metadata after the update.

[0151] Optionally, the device further includes an audit module, configured to generate a request for audit instruction to obtain an audit result if the inspection result shows a difference; keep the external metadata and the remaining metadata unchanged if the audit result is not passed; and update the external metadata and perform the operation of updating the metadata version number of the external metadata and the directory version numbers of each directory entry based on the inspection result and / or the operation of updating the remaining metadata and the metadata version number of the remaining metadata based on the inspection result if the audit result is passed.

[0152] Optionally, the device further includes an inspection instruction generation module, configured to generate the first inspection instruction and / or the second inspection instruction if the current time reaches a preset inspection period; or generate the second inspection instruction according to the at least one directory entry if an inspection trigger behavior for at least one directory entry is detected.

[0153] In the technical solution of the embodiment of the present invention, communication connections are respectively established with at least one target data source based on the at least one target data source, and an initial metadata set is established according to each target data source. A metadata version number is established for each metadata in the initial metadata set to collect and obtain the metadata in each target data source and assign a metadata version number. Furthermore, a storage directory is established according to the external metadata in the initial metadata set, and a directory version number is established for each directory entry in the storage directory to establish the storage directory and assign a directory version number. When a first inspection instruction is received, based on the first inspection instruction and the target data source, the external metadata associated with each directory entry in the storage directory is inspected, and the external metadata, the metadata version number of the external metadata, and the directory version number of the directory entry corresponding to the updated external metadata are updated based on the inspection result; when a second inspection instruction is received, based on the second inspection instruction and the target data source, the remaining metadata in the initial metadata set is inspected, and the remaining metadata and the metadata version number of the remaining metadata are updated based on the inspection result, solving the problems of large amount of data and long time consumption for data inspection, and difficulty in determining the data differences before and after data inspection, achieving the technical effects of improving the data inspection speed and pertinence and recording the differences in data inspection.

[0154] The metadata-driven data inspection and version management device provided by the embodiment of the present invention can execute the metadata-driven data inspection and version management method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing the method.

[0155] It should be noted that the various units and modules included in the above device are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be realized; in addition, the specific names of the functional units are only for the convenience of mutual distinction and do not limit the protection scope of the embodiment of the present invention.

[0156] Embodiment 4

[0157] Figure 6 It is a schematic structural diagram of an electronic device provided by Embodiment 4 of the present invention. Figure 6 It shows a block diagram of an exemplary electronic device 40 suitable for implementing the implementation manner of the embodiment of the present invention. Figure 6 The shown electronic device 40 is only an example and should not bring any limitation to the functions and usage scope of the embodiment of the present invention.

[0158] As Figure 6 shown, the electronic device 40 is presented in the form of a general-purpose computing device. The components of the electronic device 40 may include, but are not limited to: one or more processors or processing units 401, a system memory 402, and a bus 403 connecting different system components (including the system memory 402 and the processing unit 401).

[0159] The bus 403 represents one or more of several types of bus architectures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or a local bus using any of the multiple bus architectures. By way of example, these architectures include, but are not limited to, Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MAC) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus.

[0160] The electronic device 40 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the electronic device 40, including volatile and nonvolatile media, removable and non-removable media.

[0161] The system memory 402 can include computer system readable media in the form of volatile memory, such as random access memory (RAM) 404 and / or cache 405. The electronic device 40 can further include other removable / non-removable, volatile / nonvolatile computer system storage media. By way of example only, the storage system 406 can be used for reading and writing non-removable, nonvolatile magnetic media ( Figure 6 not shown, and typically referred to as a "hard disk drive"). Although Figure 6 not shown in the figure, a disk drive for reading and writing removable nonvolatile disks (such as a "floppy disk"), and an optical disk drive for reading and writing removable nonvolatile optical disks (such as CD-ROM, DVD-ROM, or other optical media) can be provided. In these cases, each drive can be connected to the bus 403 through one or more data media interfaces. The system memory 402 can include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the embodiments of the present invention.

[0162] A program / utility 408 having a set (at least one) of program modules 407 can be stored, for example, in the system memory 402. Such program modules 407 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data, and an implementation of a network environment may be included in each or some combination of these examples. The program modules 407 generally perform the functions and / or methods described in the embodiments of the present invention.

[0163] The electronic device 40 can also communicate with one or more external devices 409 (such as a keyboard, a pointing device, a display 410, etc.), and can also communicate with one or more devices that enable a user to interact with the electronic device 40, and / or communicate with any device that enables the electronic device 40 to communicate with one or more other computing devices (such as a network card, a modem, etc.). Such communication can be carried out through the input / output (I / O) interface 411. Moreover, the electronic device 40 can also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through the network adapter 412. As shown in the figure, the network adapter 412 communicates with other modules of the electronic device 40 through the bus 403. It should be understood that although Figure 6 not shown in the figure, other hardware and / or software modules can be used in combination with the electronic device 40, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0164] The processing unit 401 executes various functional applications and data processing by running programs stored in the system memory 402, such as implementing the metadata-driven data inspection and version management method provided by the embodiments of the present invention.

[0165] Embodiment 5

[0166] Embodiment 5 of the present invention also provides a storage medium containing computer-executable instructions, and the computer-executable instructions are used to execute a metadata-driven data inspection and version management method when executed by a computer processor. The method includes:

[0167] Based on at least one target data source, communication connections with each target data source are respectively established, and an initial metadata set is established according to each of the target data sources, and a metadata version number is established for each metadata in the initial metadata set; wherein, the initial metadata set includes external metadata and remaining metadata;

[0168] According to the external metadata in the initial metadata set, a storage directory is established, and a directory version number is established for each directory entry in the storage directory;

[0169] When a first inspection instruction is received, based on the first inspection instruction and the target data source, the external metadata associated with each directory entry in the storage directory is inspected, and the external metadata, the metadata version number of the external metadata, and the directory version number of the directory entry corresponding to the updated external metadata are updated based on the inspection result;

[0170] When a second patrol inspection instruction is received, based on the second patrol inspection instruction and the target data source, conduct a patrol inspection on the remaining metadata in the initial metadata set, and update the remaining metadata and the metadata version number of the remaining metadata based on the patrol inspection result.

[0171] The computer storage medium of the embodiments of the present invention may adopt any combination of one or more computer-readable media. The computer-readable media may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples (a non-exhaustive list) of the computer-readable storage medium include: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, the computer-readable storage medium may be any tangible medium that contains or stores a program, and this program can be used by or in combination with an instruction execution system, apparatus, or device.

[0172] The computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which computer-readable program code is carried. Such a propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium may also be any computer-readable medium other than the computer-readable storage medium, and this computer-readable medium can send, propagate, or transmit a program for use by or in combination with an instruction execution system, apparatus, or device.

[0173] The program code contained on the computer-readable medium can be transmitted by any appropriate medium, including - but not limited to - wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0174] Computer program code for performing the operations of the embodiments of the present invention may be written in one or more programming languages or combinations thereof. The programming languages include object-oriented programming languages such as Java, Smalltalk, C++, and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, executed as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., by connecting through the Internet using an Internet service provider).

[0175] Note that the above is only the preferred embodiment of the present invention and the technical principles applied. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein. Various obvious changes, re-adjustments, and substitutions can be made by those skilled in the art without departing from the protection scope of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments. Without departing from the concept of the present invention, more other equivalent embodiments may be included, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A metadata-driven data inspection and version management method, characterized in that, Including: Based on at least one target data source, establish communication connections with each target data source respectively, and establish an initial metadata set according to each of the target data sources, and establish a metadata version number for each metadata in the initial metadata set; wherein, the initial metadata set includes external metadata and remaining metadata; the external metadata refers to the metadata that each external platform needs to obtain, and the remaining metadata refers to the metadata that is the remainder of the metadata in the initial metadata set after removing the external metadata; Establish a storage directory according to the external metadata in the initial metadata set, and establish a directory version number for each directory entry in the storage directory; When receiving a first inspection instruction, based on the first inspection instruction and the target data source, inspect the external metadata associated with each directory entry in the storage directory, and update the external metadata, the metadata version number of the external metadata, and the directory version number of the directory entry corresponding to the updated external metadata based on the inspection result; When receiving a second inspection instruction, based on the second inspection instruction and the target data source, inspect the remaining metadata in the initial metadata set, and update the remaining metadata and the metadata version number of the remaining metadata based on the inspection result.

2. The method according to claim 1, wherein The establishing communication connections with each target data source respectively based on at least one target data source includes: Determine at least one target data source according to the data source information in the data source connection information; Establish a communication connection with the at least one target data source according to the configuration information in the data source connection information.

3. The method according to claim 1, wherein The establishing an initial metadata set according to each of the target data sources and establishing a metadata version number for each metadata in the initial metadata set includes: Collect the metadata in each of the target data sources, perform data inspection on the metadata according to preset inspection items and each of the target data sources, and establish an initial metadata set according to the metadata after data inspection; wherein, the preset inspection items include at least one of data type, data length, data element definition, and metadata model; For each metadata in the initial metadata set, establish an initial metadata version number corresponding to the metadata.

4. The method according to claim 1, wherein The establishing a storage directory according to the external metadata in the initial metadata set includes: Determine the external metadata corresponding to the external platform from the initial metadata set; Divide each of the external metadata according to the data domain corresponding to each of the external metadata; Determine each directory entry of the storage directory according to the hierarchical structure of the data domain, and establish the corresponding relationship between each of the external metadata and each of the directory entries according to the division result of each of the external metadata.

5. The method according to claim 1, wherein The inspecting the external metadata associated with each directory entry in the storage directory based on the first inspection instruction and the target data source, and updating the external metadata, the metadata version number of the external metadata, and the directory version number of the directory entry corresponding to the updated external metadata based on the inspection result includes: Parse the first patrol inspection instruction to determine the patrol inspection directory entry corresponding to the first patrol inspection instruction; Determine the data source to be patrolled from the target data source according to the data lineage of the external metadata corresponding to the patrol inspection directory entry; Perform data patrol inspection on the external metadata corresponding to the patrol inspection directory entry according to the data source to be patrolled; If the external metadata corresponding to the patrol inspection directory entry is updated according to the patrol inspection result, update the metadata version number of the updated external metadata, generate an external metadata update record based on the external metadata before update and the external metadata after update, and update the directory version number of the patrol inspection directory entry corresponding to the updated external metadata.

6. The method according to claim 1, wherein The patrol inspection of the remaining metadata in the initial metadata set based on the second patrol inspection instruction and the target data source, and updating the remaining metadata and the metadata version number of the remaining metadata based on the patrol inspection result includes: Determine the data source to be patrolled from the target data source according to the remaining metadata in the initial metadata set; Perform data patrol inspection on the remaining metadata according to the data source to be patrolled; If the remaining metadata is updated according to the patrol inspection result, update the metadata version number of the updated remaining metadata, and generate a remaining metadata update record based on the remaining metadata before update and the remaining metadata after update.

7. The method according to claim 1, wherein It further includes: If the patrol inspection result is that there are differences, generate a request for review instruction to obtain the review result; If the review result is that the review fails, keep the external metadata and the remaining metadata unchanged; If the review result is that the review passes, update the external metadata and perform the operation of updating the metadata version number of the external metadata and the directory version number of each directory entry based on the patrol inspection result and / or the operation of updating the remaining metadata and the metadata version number of the remaining metadata based on the patrol inspection result.

8. The method according to claim 1, characterized in that It further includes: If the current time reaches the preset patrol inspection period, generate the first patrol inspection instruction and / or the second patrol inspection instruction; Or, If a patrol inspection trigger behavior for at least one directory entry is detected, generate the second patrol inspection instruction according to the at least one directory entry.

9. A metadata-driven data inspection and version management device, characterized in that, It includes: An initial metadata set establishment module, configured to respectively establish communication connections with each target data source based on at least one target data source, establish an initial metadata set according to each target data source, and establish a metadata version number for each metadata in the initial metadata set; wherein, the initial metadata set includes external metadata and remaining metadata; the external metadata refers to the metadata that each external platform needs to obtain, and the remaining metadata refers to the remaining part of the metadata in the initial metadata set after removing the external metadata; A storage directory establishment module, configured to establish a storage directory according to the external metadata in the initial metadata set, and establish a directory version number for each directory entry in the storage directory; The first patrol inspection module is used to, when receiving a first patrol inspection instruction, perform a patrol inspection on the external metadata associated with each directory entry in the storage directory based on the first patrol inspection instruction and the target data source, and update the external metadata, the metadata version number of the external metadata, and the directory version number of the directory entry corresponding to the updated external metadata based on the patrol inspection result; The second patrol inspection module is used to, when receiving a second patrol inspection instruction, perform a patrol inspection on the remaining metadata in the initial metadata set based on the second patrol inspection instruction and the target data source, and update the remaining metadata and the metadata version number of the remaining metadata based on the patrol inspection result.

10. An electronic device, characterized in that, The electronic device includes: One or more processors; A storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the metadata-driven data patrol inspection and version management method as described in any one of claims 1-8.

Images