Data standard governance method and device, computer equipment and storage medium

Abstract

The invention belongs to the technical field of artificial intelligence and big data, and relates to a data standard governance method and device, computer equipment and a storage medium, and the method comprises the steps: obtaining field metadata contained in a table structure of a target information system; obtaining standard field data in an enterprise data standard library, and comparing the field metadata with the standard field data to generate a difference benchmarking report; generating a field correction script set according to each difference field contained in the difference benchmarking report; and integrating the field correction script set into a batch transaction, and executing the batch transaction to perform data standard correction on the table structure of the target information system. In addition, the invention further relates to a block chain technology, and data such as field metadata and difference benchmarking reports can be stored in a block chain. According to the invention, the efficiency and accuracy of data standard governance can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence and big data technology, and in particular to a data standard management method, device, computer equipment and storage medium. Background Art

[0002] In current data management practices, data standardization refers to effectively integrating the unified data standards within the enterprise into the data dictionaries of various information systems to ensure data consistency and standardized implementation. However, existing technical means mainly detect whether business data has changed by scanning the target business system, and then collect the metadata of the changed business data. In this process, it is necessary to manually compare and analyze the collected metadata attribute information with the standard data elements in the standard data pool. This model relies on manual subsequent maintenance of the metadata of the business system through the server, showing an obvious passive feature, that is, the corresponding metadata maintenance operation can only be triggered when the business system changes. This passive metadata management method is not only inefficient, but also unable to trace the implementation of data standards for metadata in historical business systems, limiting the efficiency and accuracy of data standard governance.

[0003] Therefore, there is an urgent need to develop new data standard governance technologies to improve the efficiency and accuracy of data standard governance. Summary of the invention

[0004] The purpose of the embodiments of the present application is to propose a data standard management method, apparatus, computer equipment and storage medium, which can improve the efficiency and accuracy of data standard management.

[0005] In order to solve the above technical problems, the present application embodiment provides a data standard governance method, which adopts the following technical solutions:

[0006] Obtain the field metadata contained in the table structure of the target information system;

[0007] Obtain standard field data in the enterprise data standard library, compare the field metadata with the standard field data, and generate a difference benchmarking report;

[0008] Generate a field correction script set according to each difference field included in the difference benchmarking report;

[0009] The field modification script set is integrated into a batch transaction, and the batch transaction is executed to perform data standard modification on the table structure of the target information system.

[0010] In order to solve the above technical problems, the embodiment of the present application also provides a data standard management device, which adopts the following technical solution:

[0011] A data acquisition module is used to obtain the field metadata contained in the table structure of the target information system;

[0012] A difference comparison module is used to obtain standard field data in the enterprise data standard library, compare the field metadata with the standard field data, and generate a difference comparison report;

[0013] A script generation module, used to generate a field correction script set according to each difference field included in the difference benchmarking report;

[0014] The transaction processing module is used to integrate the field modification script set into a batch transaction, and execute the batch transaction to perform data standard modification on the table structure of the target information system.

[0015] In order to solve the above technical problems, an embodiment of the present application also provides a computer device, which adopts the technical solution described below: the computer device includes a memory and a processor, the memory stores computer-readable instructions, and the processor implements the steps of the data standard governance method as described in any one of the above items when executing the computer-readable instructions.

[0016] In order to solve the above technical problems, an embodiment of the present application also provides a computer-readable storage medium, which adopts the following technical solution: the computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps of the data standard governance method as described in any one of the above items are implemented.

[0017] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0018] The embodiment of the present application obtains the field metadata contained in the table structure of the target information system, and compares the field metadata with the standard field data in the enterprise data standard library. This can avoid the tediousness and inefficiency of manual data extraction through automated means, and quickly generate a difference benchmarking report to clearly display the fields that do not meet the enterprise data standards; by generating a set of field correction scripts based on the various difference fields contained in the difference benchmarking report, specific correction measures can be designed for each difference field, ensuring the accuracy and pertinence of the correction; by integrating the generated field correction scripts into batch transactions and executing them once to correct the table structure of the target information system, the inefficiency and error-proneness of correcting each field one by one is avoided, thereby significantly improving the efficiency and accuracy of data standard governance. Based on this application plan, by deeply integrating metadata management with data standard implementation, text analysis technology is used to automatically realize the rapid association and benchmarking difference analysis between field metadata and existing data standards, effectively avoiding the inefficiency and omission problems of one-by-one verification, and greatly improving the accuracy of data standardization processing; at the same time, field correction scripts are automatically generated based on the difference benchmarking report and directly applied to the table structure correction of the target information system, which significantly reduces the time consumption and error risk of script writing and accelerates the implementation of data standards. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0020] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0021] Figure 2 It is a flowchart of an embodiment of the data standard governance method of the present application;

[0022] Figure 3 It is a structural diagram of an embodiment of the data standard management device of the present application;

[0023] Figure 4 It is a structural diagram of an embodiment of a computer device of the present application. DETAILED DESCRIPTION

[0024] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0025] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0026] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0027] like Figure 1 As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The terminal device 101 may be a laptop 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables.

[0028] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0029] The terminal device 101 can be any electronic device with a display screen and supporting web browsing. In addition to a laptop computer 1011, a tablet computer 1012 or a mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV), a laptop computer, a desktop computer, etc.

[0030] The server 103 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal device 101 .

[0031] It should be noted that the data standard management method provided in the embodiment of the present application is generally executed by a server / terminal device, and accordingly, the data standard management device is generally set in the server / terminal device.

[0032] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0033] Continue to refer Figure 2 , showing a flow chart of an embodiment of the question-answering method according to the present application. The data standard governance method comprises the following steps:

[0034] Step S201, obtaining field metadata contained in the table structure of the target information system;

[0035] In this embodiment, the electronic device (eg, Figure 1 The server / terminal device shown in the figure can obtain the field metadata contained in the table structure of the target information system through a wired connection or a wireless connection. It should be noted that the above wireless connection method may include but is not limited to 2G / 3G / 4G / 5G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0036] Specifically, the database type (such as MySQL, Oracle, etc.) of the target information system is determined in advance, and the connection information of the database is collected, wherein the connection information may include the host address, port number, database name, user name and password, etc. The database connection with the target information system is established based on the connection information of the database by using an API interface or writing an SQL script. The field metadata contained in the table structure of the target information system is extracted through database query (such as using SQL statements). Field metadata is data that describes the characteristics and attributes of each field (also called column) in the database table, which may include but is not limited to the field name (such as the Chinese name of the field and the English name of the field), data type, field length, value range, field usage description, whether it is allowed to be empty (NULL), etc.

[0037] Step S202, obtaining standard field data in the enterprise data standard library, comparing the field metadata with the standard field data, and generating a difference comparison report;

[0038] In this embodiment, standard field data is obtained from the enterprise data standard library, wherein the enterprise data standard library is a database or data set for storing enterprise data standards, which contains standard information such as data definition, data format, structure and rules that each information system in the enterprise needs to follow; standard field data may include standard field name, standard data type, standard field length, standard value range, etc. The obtained standard field data and field metadata are compared. During the comparison process, the difference between the field metadata and the standard field data is analyzed according to predefined rules (such as whether the data type is consistent, whether the length meets the requirements, whether the value range matches, etc.), and a difference benchmarking report is generated based on the analyzed differences. The difference benchmarking report contains information about the field metadata and standard field data with differences, such as the association pair of field name-standard field name, and the difference type (such as inconsistent field name, mismatched data type, insufficient field length, mismatched value range, etc.).

[0039] In some optional implementations, before comparing the field metadata with the standard field data and generating a difference comparison report, the following steps may also be included:

[0040] The standard field data and the field metadata are segmented respectively, and a standard segmentation result and a segmentation result to be checked are obtained accordingly;

[0041] Specifically, the standard field data extracted from the enterprise data standard library and the field metadata in the target information system table structure are segmented, wherein both the standard field data and the field metadata exist in the form of continuous text, and the continuous text is cut into independent words or phrases to achieve the segmentation of the standard field data to obtain the corresponding standard segmentation results; the field metadata is segmented to obtain the corresponding segmentation results to be checked.

[0042] Using a word vector model, the standard word segmentation results and the to-be-checked word segmentation results are respectively mapped to vector spaces to obtain a standard field vector space set and a to-be-checked field vector;

[0043] Specifically, a preset word vector model (such as a Word2Vec model) is obtained. The word vector model uses a word embedding method to represent the input words as high-dimensional vectors. The word vector model is used to map the standard word segmentation results to the vector space to obtain a standard field vector space set, which contains the vector representation of all standard field data. The word segmentation results to be checked are converted into the field vectors to be checked. For example, the word vector model is used to map the word segmentation results to be checked to the vector space to obtain the field vectors to be checked.

[0044] Calculating a similarity value between the field vector to be checked and each standard field vector in the standard field vector space set;

[0045] Specifically, a preset similarity calculation formula is obtained, and the similarity calculation formula is used to calculate the similarity value between the field vector to be checked and each standard field vector in the standard field vector space set. For example, the similarity calculation formula may be a cosine similarity formula:

[0046] Cosine similarity = (A·B) / (||A||*||B||);

[0047] Among them, A is the vector of the field to be checked, B is the standard field vector, ||A|| and ||B|| are the moduli (i.e., the length of the vector) of A and B respectively. The value range of cosine similarity is [-1, 1]. The larger the value, the more similar the two vectors are.

[0048] If the similarity value is equal to 1, the field vector to be checked and the standard field vector are marked as benchmarking standards;

[0049] Specifically, when the similarity value between the field vector to be checked and the standard field vector is equal to 1, it indicates that the field vector to be checked and the standard field vector are completely matched, and at this time, the field vector to be checked and the standard field vector are marked as the benchmarking standard. When the field vector to be checked is marked as the benchmarking standard, it indicates that the field vector to be checked has completed the data standard.

[0050] If the similarity value is not equal to 1, a preset threshold is obtained, the threshold is compared with the similarity value, and a target similarity value having a similarity value greater than the threshold is selected;

[0051] Specifically, when the similarity value between the field vector to be checked and the standard field vector is not equal to 1, it indicates that the field vector to be checked partially matches or does not match the standard field vector at all. At this time, a preset threshold is obtained, and the threshold is used to screen the standard field vector with higher similarity. The calculated similarity value between the field vector to be checked and the standard field vector is compared with the preset threshold, and the target similarity value whose similarity value is greater than the threshold is selected.

[0052] Optionally, when there is no similarity value greater than the threshold in the standard field vector set, it means that there is no suitable standard field for the field vector to be checked, and the field vector to be checked is not marked.

[0053] The to-be-checked field vector corresponding to the target similarity value is associated with the standard field vector to obtain an associated field vector.

[0054] Specifically, for the selected target similarity value, the to-be-checked field vector and the standard field vector corresponding to the target similarity value are determined, and the to-be-checked field vector and the standard field vector are associated to obtain an associated field vector, which can be marked as a partial match.

[0055] The embodiment of the present application can obtain finer-grained text units by segmenting the standard field data and field metadata, which helps to more accurately identify the differences between the two. By converting the results after word segmentation into vector representations and using word vector models for comparison, the semantic similarities between words can be captured, thereby improving the accuracy of the comparison. By setting a threshold to screen standard field vectors and associating the field vectors to be checked with the standard field vectors, standard field vectors with high similarity can be accurately identified and associated, thereby improving the accuracy of data standard labeling.

[0056] In some optional implementations, the above-mentioned comparing the field metadata with the standard field data to generate a difference comparison report may include the following steps:

[0057] Extracting the to-be-checked field name of the to-be-checked field vector from the associated field vector, and extracting the standard field name of the standard field vector from the associated field vector;

[0058] Compare the name of the field to be checked with the name of the standard field;

[0059] If the name of the field to be checked is inconsistent with the standard field name, a difference comparison report is generated according to the name of the field to be checked and the standard field name.

[0060] Specifically, the associated field vector is obtained by associating the field vector to be checked with the standard field vector, extracting the field name to be checked corresponding to the field vector to be checked from the associated field vector, and extracting the standard field name corresponding to the standard field vector from the associated field vector. The extracted field names to be checked are compared with the standard field names one by one, for example, to check whether the spelling of the field names is wrong, whether the abbreviations are different, whether the naming is inconsistent, etc., to determine whether the field names to be checked are consistent with the standard field names. If the field names to be checked are inconsistent with the standard field names, a difference benchmarking report is generated based on the field names to be checked and the standard field names, and the difference benchmarking report contains detailed information on the differences in the field names, such as spelling errors in the field names to be checked, abnormal abbreviations in the field names to be checked, or inconsistent naming of the field names to be checked and the standard field names.

[0061] In some optional implementations, the above-mentioned comparing the field metadata with the standard field data to generate a difference comparison report may include the following steps:

[0062] Extracting the to-be-checked data type of the to-be-checked field vector from the associated field vector, and extracting the standard data type of the standard field vector from the associated field vector;

[0063] Comparing the to-be-checked data type with the standard data type;

[0064] If the data type to be checked is inconsistent with the standard data type, a difference comparison report is generated according to the data type to be checked and the standard data type.

[0065] Specifically, the data type to be checked corresponding to the field vector to be checked is extracted from the associated field vector, and the standard data type corresponding to the standard field vector is extracted from the associated field vector. Among them, the data type may include integers, floating point numbers, strings, dates, etc. The extracted data types to be checked are compared with the standard data types one by one to determine whether the data types to be checked are consistent with the standard data types. If the data type to be checked is inconsistent with the standard data type, a difference benchmarking report is generated based on the data type to be checked and the standard data type, and the difference benchmarking report contains detailed information on the differences in data types, such as the data type to be checked is A, the standard data type is B, etc.

[0066] In some optional implementations, the above-mentioned comparing the field metadata with the standard field data to generate a difference comparison report may include the following steps:

[0067] Extracting the to-be-checked value range of the to-be-checked field vector from the associated field vector, and extracting the standard value range of the standard field vector from the associated field vector;

[0068] Comparing the to-be-checked value range with the standard value range;

[0069] If the value range to be checked exceeds the standard value range, a difference comparison report is generated according to the value range to be checked and the standard value range.

[0070] Specifically, the value range to be checked corresponding to the field vector to be checked is extracted from the associated field vector, and the standard value range corresponding to the standard field vector is extracted from the associated field vector. The value range refers to the possible value range of the field value, which can be a specific numerical range (such as 1-100) or a category range (such as male or female). The value range to be checked is compared with the standard value range one by one, for example, the upper limit, lower limit, whether it contains a specific value or category, etc., to determine whether the value range to be checked is consistent with the standard value range. If the value range to be checked is inconsistent with the standard value range, a difference benchmarking report is generated based on the value range to be checked and the standard value range, and the difference benchmarking report contains detailed information on the difference in the value range to be checked, such as inconsistent upper limits of the value range, inconsistent lower limits of the value range, and the value range to be checked does not contain a specific value or category.

[0071] Step S203, generating a field correction script set according to each difference field included in the difference benchmarking report;

[0072] In this embodiment, based on the generated difference benchmarking report, the difference type (such as field name difference, data type difference, value range difference, etc.) of each difference field contained in the difference benchmarking report is determined, and a field correction script (such as field renaming script, field type conversion script, value range constraint script, etc.) is generated according to the difference type of each difference field, and the generated field correction scripts are combined into a field correction script set.

[0073] In some optional implementations, the generating of the field correction script set according to each difference field included in the difference benchmarking report may include the following steps:

[0074] Identify the difference type of each difference field included in the difference benchmarking report;

[0075] Specifically, according to the generated difference benchmarking report, the difference type (such as field name difference, data type difference, value range difference, etc.) of each difference field included in the difference benchmarking report is identified.

[0076] If the difference type is a field name difference, then obtaining the to-be-checked field name and the standard field name included in the difference field, and generating a field renaming script for the to-be-checked field name according to the standard field name;

[0077] Specifically, if the difference type of the difference field is a field name difference, the name of the field to be checked and the standard field name contained in the difference field are obtained, and for the obtained field name to be checked, a field renaming instruction is generated according to the standard field name corresponding to the field name to be checked. The field renaming instruction is used to instruct to change the field name to be checked to the standard field name. All field renaming instructions are combined into a field renaming script, and the field renaming script will perform the corresponding standard field name change for all field names to be checked.

[0078] If the difference type is a data type difference, then obtaining the to-be-checked data type and the standard data type included in the difference field, and generating a field type conversion script for the to-be-checked data type according to the standard data type;

[0079] Specifically, if the difference type of the difference field is a data type difference, the to-be-checked data type and the standard data type contained in the difference field are obtained, and for the obtained to-be-checked data type, a field type conversion instruction is generated according to the standard data type corresponding to the to-be-checked data type, and the field type conversion instruction is used to instruct to convert the to-be-checked data type into the standard data type. All field type conversion instructions are combined into a field type conversion script, and the field type conversion script will perform corresponding standard data type conversion for all to-be-checked data types.

[0080] If the difference type is a value range difference, the value range to be checked and the standard value range included in the difference field are obtained, and a value range constraint script for the value range to be checked is generated according to the standard value range.

[0081] Specifically, if the difference type of the difference field is a value range difference, the value range to be checked and the standard value range contained in the difference field are obtained, and for the obtained value range to be checked, a value range constraint instruction is generated according to the standard value range corresponding to the value range to be checked, and the value range constraint instruction is used to instruct to adjust the value range to be checked to the standard value range. All value range constraint instructions are combined into a value range constraint script, and the value range constraint script will perform corresponding standard value range adjustments for all value ranges to be checked.

[0082] The embodiment of the present application generates a field renaming script to change the name of the field to be checked to the standard field name, thereby ensuring the uniformity and standardization of the field name in the data system, and helping to eliminate data confusion and misunderstanding caused by inconsistent field names. By generating a field type conversion script, the data type to be checked is made consistent with the standard data type, which helps to ensure the correct processing and storage of data in the system and avoids data errors or losses caused by data type mismatches. By generating a value domain constraint script, the value domain of the field to be checked is limited to the standard value domain, which helps to ensure the accuracy and validity of the data and prevents data values ​​beyond the expected range from causing adverse effects on the system. Based on this embodiment, by correcting the field name, data type and value domain range, data quality problems caused by data inconsistency or errors are reduced, and the reliability and availability of data standard governance are improved.

[0083] Step S204: Integrate the field modification script set into a batch transaction, and execute the batch transaction to perform data standard modification on the table structure of the target information system.

[0084] In this embodiment, the field correction scripts in the field correction script set are connected in series according to a preset order, and the connected field correction scripts are integrated into a batch transaction to ensure that the correction operations of all field correction scripts are executed in one transaction. The batch transaction is started, and the connected field correction scripts are executed in sequence, such as renaming the field name, converting the field type, correcting the field length, and constraining the value range according to the script content. After the batch transaction is executed, the data standard correction of the table structure of the target information system is realized.

[0085] In some optional implementations, the above-mentioned integrating the field modification script set into a batch transaction and executing the batch transaction to perform data standard management on the table structure of the target information system may include the following steps:

[0086] Integrate one or more of the field renaming script, the field type conversion script, and the value range constraint script into a batch transaction;

[0087] Specifically, all field correction scripts generated according to the difference benchmarking report are collected, and the field correction scripts include field renaming scripts, field type conversion scripts, and value range constraint scripts, etc. The collected one or more field correction scripts are sorted in a preset order (such as renaming first, then converting the type, and finally constraining the value range), and the sorted field correction scripts are integrated into batch transactions.

[0088] Executing the batch transaction and monitoring the execution process of the batch transaction, and when all scripts in the batch transaction are successfully executed, correcting the data standard of the table structure of the target information system;

[0089] Specifically, a batch transaction is started and each field correction script in the batch transaction is executed in sequence according to the order in which the scripts are sorted. For example, each field correction script is used to rename the fields to be checked, convert data types, and constrain the value range. At the same time, during the execution of the batch transaction, the execution status of each field correction script is monitored in real time. When each field correction script in the batch transaction is successfully executed, it indicates that the data standard of the table structure of the target information system has been corrected according to the data standard. At this time, the batch transaction can be submitted to make all data standard corrections take effect.

[0090] When a script execution fails in the batch transaction, the execution process of the batch transaction is terminated and the batch transaction is rolled back; the step of generating a field correction script set according to each difference field included in the difference benchmarking report is returned to execution; until all scripts in the batch transaction are successfully executed, the data standard of the table structure of the target information system is corrected.

[0091] Specifically, when a field correction script fails to execute in a batch transaction, the execution process of the current batch transaction is terminated and the batch transaction is rolled back. The rollback operation will undo all executed script changes to restore the table structure of the target information system to the state before the batch transaction started. Return to the above steps of generating a set of field correction scripts based on the difference fields contained in the difference benchmarking report, and integrate the regenerated set of field correction scripts into a batch transaction and execute it until all scripts in the batch transaction are successfully executed, indicating that the data standard of the table structure of the target information system has been corrected in accordance with the data standard. At this time, the batch transaction can be submitted to make all data standard corrections take effect.

[0092] The embodiment of the present application can ensure data consistency by integrating multiple field correction scripts into one batch transaction and ensuring that all scripts are executed in the same transaction. In addition, by integrating multiple scripts into one batch transaction execution, the overhead of opening and closing transactions multiple times can be reduced, thereby improving execution efficiency. When any script in a batch transaction fails to execute, the rollback operation of the batch transaction is automatically triggered, all executed script changes are undone, and the state before the transaction starts is quickly restored, thereby avoiding inconsistencies caused by partial updates of data standards.

[0093] For example, a company's financial system (i.e., the target information system) is being connected to the company's master data management platform (i.e., the company's data standard library). The data dictionary of the financial system (i.e., the field metadata of the table structure) has not yet fully complied with the company's unified data standards (such as field naming specifications, data types, and value ranges). By adopting the following steps S1-S4, the data of the financial system is standardized and integrated with the company's data governance system:

[0094] Step S1: Data dictionary extraction

[0095] Extract the field metadata contained in the table structure of the financial system through SQL statements. The field metadata may include the Chinese and English names of the field, data types (such as varchar, int), field length (such as 50-character limit) and value range (such as the value of the status field is ["ACTIVE", "INACTIVE"]). Convert the extracted field metadata into a structured data dictionary format.

[0096] For example:

[0097] The extracted field metadata includes: the English name of the field is CUSTOMER_ID (the data type is varchar(20)), and the corresponding Chinese name of the field is customer number.

[0098] The extracted field metadata is converted into a structured data dictionary format and stored in the original data dictionary module of the metadata management system to form the following table:

[0099]

[0100] Get the standard field data related to the enterprise data standard library as follows:

[0101]

[0102] Step S2: Data benchmarking

[0103] Compare the field metadata with the enterprise data standard library, and the differences are analyzed as follows:

[0104] Field association: Use text analysis technology to match the Chinese field names "customer number" and "account status" in the field metadata with the standard field data in the enterprise data standard library, and finally associate them with the information items CUS010001 and FIN010001;

[0105] Field name comparison: The English name of the information item CUS010001 in the enterprise data standard library is CustomerNumber, and the English name of the field in the field metadata is CUSTOMER_ID. It is determined that the field names are inconsistent.

[0106] Data type verification: The data type of CUS010001 in the enterprise data standard library is varchar(30), and the data type of CUSTOMER_ID in the field metadata is varchar(20), so the data types are judged to be inconsistent.

[0107] Range check: The standard range of FIN010001 is ["ENABLED", "DISABLED"], and the range of ACC_STA in the field metadata is ["ACTIVE", "INACTIVE"]. It is determined that the ranges are inconsistent.

[0108] Generate a difference benchmarking report based on the difference fields after the above comparison:

[0109]

[0110] Step S3: Data labeling

[0111] Based on the difference benchmarking report, a field correction script is automatically generated to update the fields in the financial system. The specific execution is as follows:

[0112] Field rename script: ALTER TABLE customer RENAME COLUMN CUSTOMER_ID TOCUST_NO;

[0113] Field type conversion script: ALTER TABLE customer MODIFY COLUMN CUST_NO varchar(30);

[0114] Range constraint script:

[0115] Add a range constraint: ALTER TABLE customer ADD CONSTRAINT CHECK(ACC_STA IN('ENABLED','DISABLED'));

[0116] Modify the existing value range: UPDATE customer SET ACC_STA = CASE;

[0117] WHEN ACC_STA='ACTIVE'THEN'ENABLED';

[0118] WHEN ACC_STA='INACTIVE'THEN'DISABLED';

[0119] Integrate the above field correction scripts into a batch transaction and execute it:

[0120] BEGIN TRANSACTION;

[0121] -- Field rename

[0122] ALTER TABLE customer RENAME COLUMN CUSTOMER_ID TO CUST_NO;

[0123] --Data type conversion

[0124] ALTER TABLE customer MODIFY COLUMN CUST_NO varchar(30);

[0125] --Add value range constraints

[0126] ALTER TABLE customer ADD CONSTRAINT CHECK(ACC_STA IN('ENABLED','DISABLED'));

[0127] --Value range correction

[0128] UPDATE customer SET ACC_STA=CASE;

[0129] WHEN ACC_STA='ACTIVE'THEN'ENABLED';

[0130] WHEN ACC_STA='INACTIVE'THEN'DISABLED';

[0131] After the table structure of the financial system (i.e. the target information system) is updated, the data dictionary of the target information system is re-extracted and a secondary benchmarking verification is performed with the enterprise data standard library to ensure that the update results meet expectations.

[0132] Step S4: Closed-loop management

[0133] Archive the updated data dictionary as a new version and record the change log:

[0134] Field name change record: CUSTOMER_ID → CUST_NO;

[0135] Data type change record: varchar(20) → varchar(30);

[0136] Value range change record: ["ACTIVE","INACTIVE"]→["ENABLED","DISABLED"];

[0137] The real-time monitoring module continues to detect whether newly added fields in the financial system meet the standards. For example, if it is found that a newly added field has an undefined value range, the system will issue a warning and generate update suggestions to support the continuous optimization of data standards.

[0138] The embodiment of the present application adopts the above scheme, specifically by obtaining the field metadata contained in the table structure of the target information system, and comparing the field metadata with the standard field data in the enterprise data standard library. It can avoid the tediousness and inefficiency of manual data extraction through automated means, and quickly generate a difference benchmarking report to clearly display the fields that do not meet the enterprise data standards; by generating a field correction script set according to each difference field contained in the difference benchmarking report, it can design specific correction measures for each difference field, ensuring the accuracy and pertinence of the correction; by integrating the generated field correction scripts into batch transactions and executing them once to correct the table structure of the target information system, it avoids the inefficiency and error-proneness of correcting each field one by one, and significantly improves the efficiency and accuracy of data standard governance. Based on this application plan, by deeply integrating metadata management with data standard implementation, text analysis technology is used to automatically realize the rapid association and benchmarking difference analysis between field metadata and existing data standards, effectively avoiding the inefficiency and omission problems of one-by-one verification, and greatly improving the accuracy of data standardization processing; at the same time, field correction scripts are automatically generated based on the difference benchmarking report and directly applied to the table structure correction of the target information system, which significantly reduces the time consumption and error risk of script writing and accelerates the implementation of data standards.

[0139] It should be emphasized that in order to further ensure the privacy and security of the above-mentioned field metadata, difference benchmarking report and other information as well as other relevant data, the above-mentioned field metadata, difference benchmarking report and other information as well as other relevant data can also be stored in a blockchain node.

[0140] The blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0141] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The above-mentioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0142] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0143] Further references Figure 3 , as a response to the above Figure 2 In order to realize the method shown in the figure, the present application provides an embodiment of a data standard management device, and the device embodiment is Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0144] like Figure 3 As shown, the data standard management device 400 described in this embodiment includes: a data acquisition module 401, a difference comparison module 402, a script generation module 403 and a transaction processing module 404. Among them:

[0145] The data acquisition module 401 is used to acquire the field metadata contained in the table structure of the target information system;

[0146] The difference comparison module 402 is used to obtain standard field data in the enterprise data standard library, compare the field metadata with the standard field data, and generate a difference comparison report;

[0147] The script generation module 403 is used to generate a field correction script set according to each difference field included in the difference benchmarking report;

[0148] The transaction processing module 404 is used to integrate the field modification script set into a batch transaction, and execute the batch transaction to perform data standard modification on the table structure of the target information system.

[0149] In this embodiment, the data acquisition module 401 is used to predetermine the database type (such as MySQL, Oracle, etc.) of the target information system and collect the connection information of the database, wherein the connection information may include the host address, port number, database name, user name and password, etc. An API interface or SQL script is used to establish a database connection with the target information system based on the connection information of the database. The field metadata contained in the table structure of the target information system is extracted through database query (such as using SQL statements). Field metadata is data that describes the characteristics and attributes of each field (also called column) in the database table, and may include but is not limited to the field name (such as the Chinese name of the field and the English name of the field), data type, field length, value range, field usage description, whether it is allowed to be empty (NULL), etc.

[0150] Specifically, the difference comparison module 402 is used to obtain standard field data from the enterprise data standard library, wherein the enterprise data standard library is a database or data set for storing enterprise data standards, which contains standard information such as data definition, data format, structure and rules that each information system in the enterprise needs to follow; standard field data may include standard field name, standard data type, standard field length, standard value range, etc. The obtained standard field data and field metadata are compared. During the comparison process, the difference between the field metadata and the standard field data is analyzed according to predefined rules (such as whether the data type is consistent, whether the length meets the requirements, whether the value range matches, etc.), and a difference benchmarking report is generated according to the analyzed difference. The difference benchmarking report contains information about the field metadata and standard field data with differences, such as the association pair of field name-standard field name, and the difference type (such as inconsistent field name, mismatched data type, insufficient field length, mismatched value range, etc.).

[0151] Specifically, the script generation module 403 is used to determine the difference type (such as field name difference, data type difference, value range difference, etc.) of each difference field contained in the difference benchmarking report based on the generated difference benchmarking report, and generate a field correction script (such as field renaming script, field type conversion script, value range constraint script, etc.) according to the difference type of each difference field, and combine the generated field correction scripts into a field correction script set.

[0152] Specifically, the transaction processing module 404 is used to concatenate the field correction scripts in the field correction script set in a preset order, and integrate the concatenated field correction scripts into a batch transaction to ensure that the correction operations of all field correction scripts are executed in one transaction. The batch transaction is started, and the concatenated field correction scripts are executed in sequence, such as renaming the field name, converting the field type, correcting the field length, and constraining the value range according to the script content, and after the batch transaction is executed, the data standard correction of the table structure of the target information system is realized.

[0153] The data standard management device 400 of the embodiment of the present application obtains the field metadata contained in the table structure of the target information system, compares the field metadata with the standard field data in the enterprise data standard library, and can avoid the tediousness and inefficiency of manual data extraction through automated means, and quickly generate a difference benchmarking report to clearly display the fields that do not meet the enterprise data standards; by generating a set of field correction scripts according to the various difference fields contained in the difference benchmarking report, specific correction measures can be designed for each difference field, ensuring the accuracy and pertinence of the correction; by integrating the generated field correction scripts into batch transactions and executing them once to correct the table structure of the target information system, the inefficiency and error-proneness of correcting each field is avoided, thereby significantly improving the efficiency and accuracy of data standard management. Based on this application plan, by deeply integrating metadata management with data standard implementation, text analysis technology is used to automatically realize the rapid association and benchmarking difference analysis between field metadata and existing data standards, effectively avoiding the inefficiency and omission problems of one-by-one verification, and greatly improving the accuracy of data standardization processing; at the same time, field correction scripts are automatically generated based on the difference benchmarking report and directly applied to the table structure correction of the target information system, which significantly reduces the time consumption and error risk of script writing and accelerates the implementation of data standards.

[0154] In some optional implementations of this embodiment, the data standard management device 400 may also include a data segmentation module, a vector mapping module, a similarity calculation module, a standard marking module, a threshold comparison module and a vector association module. Among them:

[0155] A data segmentation module, used to segment the standard field data and the field metadata respectively, and obtain a standard segmentation result and a segmentation result to be checked;

[0156] A vector mapping module, used to use a word vector model to map the standard word segmentation results and the to-be-checked word segmentation results to vector spaces respectively, to obtain a standard field vector space set and a to-be-checked field vector;

[0157] A similarity calculation module, used to calculate the similarity value between the field vector to be checked and each standard field vector in the standard field vector space set;

[0158] A standard marking module, configured to mark the field vector to be checked and the standard field vector as a benchmark standard if the similarity value is equal to 1;

[0159] A threshold comparison module, configured to obtain a preset threshold if the similarity value is not equal to 1, compare the threshold with the similarity value, and select a target similarity value whose similarity value is greater than the threshold;

[0160] The vector association module is used to associate the to-be-checked field vector corresponding to the target similarity value with the standard field vector to obtain an associated field vector.

[0161] Specifically, the data segmentation module is used to segment the standard field data extracted from the enterprise data standard library and the field metadata in the target information system table structure, wherein the standard field data and the field metadata both exist in the form of continuous text, and the continuous text is cut into independent words or phrases to achieve segmentation of the standard field data to obtain the corresponding standard segmentation results; segmentation of the field metadata to obtain the corresponding segmentation results to be checked.

[0162] Specifically, the vector mapping module is used to obtain a preset word vector model (such as a Word2Vec model). The word vector model uses a word embedding method to represent the input words as high-dimensional vectors. The word vector model is used to map the standard word segmentation results to the vector space to obtain a standard field vector space set, which contains the vector representation of all standard field data. The word segmentation results to be checked are converted into the field vectors to be checked. For example, the word vector model is used to map the word segmentation results to be checked to the vector space to obtain the field vectors to be checked.

[0163] Specifically, the similarity calculation module is used to obtain a preset similarity calculation formula, and use the similarity calculation formula to calculate the similarity value between the field vector to be checked and each standard field vector in the standard field vector space set. For example, the similarity calculation formula can be a cosine similarity formula:

[0164] Cosine similarity = (A·B) / (||A||*||B||);

[0165] Among them, A is the vector of the field to be checked, B is the standard field vector, ||A|| and ||B|| are the moduli (i.e., the length of the vector) of A and B respectively. The value range of cosine similarity is [-1, 1]. The larger the value, the more similar the two vectors are.

[0166] Specifically, the standard marking module is used to mark the field vector to be checked and the standard field vector as the benchmarking standard when the similarity value between the field vector to be checked and the standard field vector is equal to 1, indicating that the field vector to be checked and the standard field vector are completely matched. When the field vector to be checked is marked as the benchmarking standard, it indicates that the field vector to be checked has completed the data standard mark.

[0167] Specifically, the threshold comparison module is used to obtain a preset threshold value when the similarity value between the field vector to be checked and the standard field vector is not equal to 1, indicating that the field vector to be checked partially matches or does not match the standard field vector at all. The threshold value is used to screen the standard field vector with a higher similarity. The calculated similarity value between the field vector to be checked and the standard field vector is compared with the preset threshold value, and the target similarity value with a similarity value greater than the threshold value is selected.

[0168] Optionally, when there is no similarity value greater than the threshold in the standard field vector set, it means that there is no suitable standard field for the field vector to be checked, and the field vector to be checked is not marked.

[0169] Specifically, the vector association module is used to determine the to-be-checked field vector and the standard field vector corresponding to the selected target similarity value, associate the to-be-checked field vector with the standard field vector to obtain an associated field vector, which can be marked as a partial match.

[0170] The data standard management device 400 of the embodiment of the present application can obtain finer-grained text units by segmenting the standard field data and field metadata, which helps to more accurately identify the differences between the two. By converting the results after word segmentation into vector representations and using word vector models for comparison, the semantic similarity between words can be captured, thereby improving the accuracy of the comparison. By setting a threshold to screen standard field vectors and associating the field vectors to be checked with the standard field vectors, standard field vectors with high similarity can be accurately identified and associated, thereby improving the accuracy of data standard labeling.

[0171] In some optional implementations of this embodiment, the difference comparison module 402 may include a first extraction submodule, a first comparison submodule and a first generation submodule.

[0172] A first extraction submodule is used to extract the to-be-checked field name of the to-be-checked field vector from the associated field vector, and to extract the standard field name of the standard field vector from the associated field vector;

[0173] A first comparison submodule, used for comparing the name of the field to be checked with the name of the standard field;

[0174] The first generating submodule is used to generate a difference comparison report according to the name of the field to be checked and the name of the standard field if the name of the field to be checked is inconsistent with the name of the standard field.

[0175] Specifically, the first extraction submodule is used to associate the field vector to be checked with the standard field vector, extract the field name to be checked corresponding to the field vector to be checked from the association field vector, and extract the standard field name corresponding to the standard field vector from the association field vector. The first comparison submodule is used to compare the extracted field names to be checked with the standard field names one by one, for example, to check whether the spelling of the field names is wrong, whether the abbreviations are different, whether the naming is inconsistent, etc., to determine whether the field names to be checked are consistent with the standard field names. The first generation submodule is used to generate a difference comparison report based on the field names to be checked and the standard field names if the field names to be checked are inconsistent with the standard field names. The difference comparison report contains detailed information on the differences in field names, such as spelling errors in the field names to be checked, abnormal abbreviations in the field names to be checked, or inconsistent naming of the field names to be checked and the standard field names.

[0176] In some optional implementations of this embodiment, the difference comparison module 402 may include a second extraction submodule, a second comparison submodule, and a second generation submodule.

[0177] A second extraction submodule is used to extract the to-be-checked data type of the to-be-checked field vector from the associated field vector, and to extract the standard data type of the standard field vector from the associated field vector;

[0178] A second comparison submodule, used for comparing the data type to be checked with the standard data type;

[0179] The second generating submodule is used to generate a difference comparison report according to the data type to be checked and the standard data type if the data type to be checked is inconsistent with the standard data type.

[0180] Specifically, the second extraction submodule is used to extract the data type to be checked corresponding to the field vector to be checked from the associated field vector, and to extract the standard data type corresponding to the standard field vector from the associated field vector. Among them, the data type may include integers, floating point numbers, strings, dates, etc. The second comparison submodule is used to compare the extracted data type to be checked with the standard data type one by one to determine whether the data type to be checked is consistent with the standard data type. The second generation submodule is used to generate a difference benchmarking report based on the data type to be checked and the standard data type if the data type to be checked is inconsistent with the standard data type. The difference benchmarking report contains detailed information on the differences in data types, such as the data type to be checked is A, the standard data type is B, etc.

[0181] In some optional implementations of this embodiment, the difference comparison module 402 may include a third extraction submodule, a third comparison submodule and a third generation submodule.

[0182] A third extraction submodule is used to extract the to-be-checked value range of the to-be-checked field vector from the associated field vector, and to extract the standard value range of the standard field vector from the associated field vector;

[0183] A third comparison submodule, used for comparing the to-be-checked value range with the standard value range;

[0184] The third generating submodule is used to generate a difference comparison report according to the value range to be checked and the standard value range if the value range to be checked exceeds the standard value range.

[0185] Specifically, the third extraction submodule is used to extract the value range to be checked corresponding to the field vector to be checked from the associated field vector, and to extract the standard value range corresponding to the standard field vector from the associated field vector. Wherein, the value range refers to the possible value range of the field value, which can be a specific numerical range (such as 1-100) or a category range (such as male or female). The third comparison submodule is used to compare the value range to be checked with the standard value range one by one, for example, to compare the upper limit, lower limit, whether it contains a specific value or category, etc. of the value range, so as to determine whether the value range to be checked is consistent with the standard value range. The third generation submodule is used to generate a difference benchmarking report based on the value range to be checked and the standard value range if the value range to be checked is inconsistent with the standard value range. The difference benchmarking report contains detailed information on the difference in the value range to be checked, such as inconsistent upper limits of the value range, inconsistent lower limits of the value range, and the value range to be checked does not contain a specific value or category.

[0186] In some optional implementations of this embodiment, the script generation module 403 may include a difference type identification submodule, a first script generation submodule, a second script generation submodule, and a third script generation submodule, wherein:

[0187] A difference type identification submodule, used to identify the difference type of each difference field included in the difference benchmarking report;

[0188] A first script generation submodule is used for obtaining the to-be-checked field name and the standard field name included in the difference field if the difference type is a field name difference, and generating a field renaming script for the to-be-checked field name according to the standard field name;

[0189] A second script generation submodule is used for obtaining the data type to be checked and the standard data type included in the difference field if the difference type is a data type difference, and generating a field type conversion script for the data type to be checked according to the standard data type;

[0190] The third script generation submodule is used to obtain the to-be-checked value range and the standard value range included in the difference field if the difference type is a value range difference, and generate a value range constraint script for the to-be-checked value range according to the standard value range.

[0191] Specifically, the difference type identification submodule is used to identify the difference type (such as field name difference, data type difference, value range difference, etc.) of each difference field contained in the difference benchmarking report based on the generated difference benchmarking report.

[0192] Specifically, the first script generation submodule is used to obtain the to-be-checked field name and the standard field name contained in the difference field if the difference type of the difference field is a field name difference, and for the obtained to-be-checked field name, generate a field renaming instruction according to the standard field name corresponding to the to-be-checked field name, and the field renaming instruction is used to instruct to change the to-be-checked field name to the standard field name. All field renaming instructions are combined into a field renaming script, and the field renaming script will perform the corresponding standard field name change for all to-be-checked field names.

[0193] Specifically, the second script generation submodule is used to obtain the to-be-checked data type and the standard data type contained in the difference field if the difference type of the difference field is a data type difference, and for the obtained to-be-checked data type, generate a field type conversion instruction according to the standard data type corresponding to the to-be-checked data type, the field type conversion instruction is used to instruct to convert the to-be-checked data type to the standard data type. All field type conversion instructions are combined into a field type conversion script, and the field type conversion script will perform corresponding standard data type conversion for all to-be-checked data types.

[0194] Specifically, the third script generation submodule is used to obtain the to-be-checked value range and the standard value range contained in the difference field if the difference type of the difference field is a value range difference, and for the obtained to-be-checked value range, generate a value range constraint instruction according to the standard value range corresponding to the to-be-checked value range, and the value range constraint instruction is used to instruct to adjust the to-be-checked value range to the standard value range. All value range constraint instructions are combined into a value range constraint script, and the value range constraint script will perform corresponding standard value range adjustments for all value ranges to be checked.

[0195] The script generation module 403 of the embodiment of the present application generates a field renaming script to change the name of the field to be checked to the standard field name, thereby ensuring the uniformity and standardization of the field name in the data system, and helping to eliminate data confusion and misunderstanding caused by inconsistent field names. By generating a field type conversion script, the data type to be checked is made consistent with the standard data type, which helps to ensure the correct processing and storage of data in the system and avoids data errors or losses caused by data type mismatches. By generating a value domain constraint script, the value domain of the field to be checked is limited to the standard value domain range, which helps to ensure the accuracy and validity of the data and prevents data values ​​beyond the expected range from causing adverse effects on the system. Based on this embodiment, by correcting the field name, data type and value domain range, data quality problems caused by data inconsistency or errors are reduced, and the reliability and availability of data standard governance are improved.

[0196] In some optional implementations of this embodiment, the transaction processing module 404 may include a transaction integration submodule, a first execution submodule, and a second execution submodule.

[0197] A transaction integration submodule, used to integrate one or more of the field renaming script, the field type conversion script and the value range constraint script into a batch transaction;

[0198] A first execution submodule, configured to execute the batch transaction and monitor the execution process of the batch transaction, and to modify the data standard of the table structure of the target information system when all scripts in the batch transaction are successfully executed;

[0199] The second execution submodule is used to terminate the execution process of the batch transaction and roll back the batch transaction when a script execution fails in the batch transaction; return to the step of generating a field correction script set according to each difference field included in the difference benchmarking report; until all scripts in the batch transaction are successfully executed, the data standard of the table structure of the target information system is corrected.

[0200] Specifically, the transaction integration submodule is used to collect all field correction scripts generated according to the difference benchmarking report. The field correction scripts include field renaming scripts, field type conversion scripts, and value range constraint scripts, etc. The collected one or more field correction scripts are sorted in a preset order (such as renaming first, then converting the type, and finally constraining the value range), and the sorted field correction scripts are integrated into batch transactions.

[0201] Specifically, the first execution submodule is used to start a batch transaction and execute each field correction script in the batch transaction in sequence according to the order in which the scripts are sorted, such as renaming the fields to be checked, converting data types, and constraining the value range according to each field correction script. At the same time, during the execution of the batch transaction, the execution status of each field correction script is monitored in real time. When each field correction script in the batch transaction is successfully executed, it indicates that the data standard of the table structure of the target information system has been corrected according to the data standard. At this time, the batch transaction can be submitted to make all data standard corrections take effect.

[0202] Specifically, the second execution submodule is used to terminate the execution process of the current batch transaction and roll back the batch transaction when there is a field correction script execution failure in the batch transaction. The rollback operation will undo all the executed script changes to restore the table structure of the target information system to the state before the batch transaction started. Return to the above step of generating a set of field correction scripts based on the various difference fields contained in the difference benchmarking report, and integrate the regenerated set of field correction scripts into a batch transaction and execute it until all scripts in the batch transaction are successfully executed, indicating that the data standard of the table structure of the target information system has been corrected according to the data standard. At this time, the batch transaction can be submitted to make all data standard corrections take effect.

[0203] The transaction processing module 404 of the embodiment of the present application can ensure data consistency by integrating multiple field correction scripts into a batch transaction and ensuring that all scripts are executed in the same transaction. In addition, by integrating multiple scripts into a batch transaction execution, the overhead of opening and closing transactions multiple times can be reduced, thereby improving execution efficiency. When any script in a batch transaction fails to execute, the rollback operation of the batch transaction is automatically triggered, all script changes that have been executed are undone, and the state before the transaction starts is quickly restored, thereby avoiding inconsistencies caused by partial updates of data standards.

[0204] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0205] The computer device 6 includes a memory 61, a processor 62, and a network interface 63 that are interconnected and communicated through a system bus. It should be noted that the figure only shows a computer device 6 with a memory 61, a processor 62, and a network interface 63, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0206] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0207] The memory 61 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 61 can be an internal storage unit of the computer device 6, such as a hard disk or memory of the computer device 6. In other embodiments, the memory 61 can also be an external storage device of the computer device 6, such as a plug-in hard disk equipped on the computer device 6, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. Of course, the memory 61 can also include both the internal storage unit of the computer device 6 and its external storage device. In this embodiment, the memory 61 is generally used to store the operating system and various application software installed on the computer device 6, such as computer-readable instructions of the data standard management method, etc. In addition, the memory 61 can also be used to temporarily store various types of data that have been output or are to be output.

[0208] The processor 62 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor or other data processing chip in some embodiments. The processor 62 is generally used to control the overall operation of the computer device 6. In this embodiment, the processor 62 is used to run the computer-readable instructions stored in the memory 61 or process data, such as running the computer-readable instructions of the data standard governance method.

[0209] The network interface 63 may include a wireless network interface or a wired network interface. The network interface 63 is generally used to establish a communication connection between the computer device 6 and other electronic devices.

[0210] The present application also provides another embodiment, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the data standard governance method as described above.

[0211] The computer device, computer-readable storage medium and computer-readable instructions provided in the embodiments of the present application can obtain the field metadata contained in the table structure of the target information system through the processor execution, and compare the field metadata with the standard field data in the enterprise data standard library. It can avoid the tediousness and inefficiency of manual data extraction through automated means, and quickly generate a difference benchmarking report to clearly display the fields that do not meet the enterprise data standards; by generating a set of field correction scripts according to the various difference fields contained in the difference benchmarking report, specific correction measures can be designed for each difference field, ensuring the accuracy and pertinence of the correction; by integrating the generated field correction scripts into batch transactions and executing them once to correct the table structure of the target information system, the inefficiency and error-proneness of correcting each field is avoided, and the efficiency and accuracy of data standard governance are significantly improved. Based on this application plan, by deeply integrating metadata management with data standard implementation, text analysis technology is used to automatically realize the rapid association and benchmarking difference analysis between field metadata and existing data standards, effectively avoiding the inefficiency and omission problems of one-by-one verification, and greatly improving the accuracy of data standardization processing; at the same time, field correction scripts are automatically generated based on the difference benchmarking report and directly applied to the table structure correction of the target information system, which significantly reduces the time consumption and error risk of script writing and accelerates the implementation of data standards.

[0212] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0213] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to replace some of the technical features therein with equivalents. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

[0214] The non-Company software tools or components appearing in the embodiments of this application are merely examples and do not represent actual use.

Claims

1. A data standard governance method, characterized in that: The steps include: Obtain the field metadata contained in the table structure of the target information system; Obtain standard field data in the enterprise data standard library, compare the field metadata with the standard field data, and generate a difference benchmarking report; Generate a field correction script set according to each difference field included in the difference benchmarking report; The field modification script set is integrated into a batch transaction, and the batch transaction is executed to perform data standard modification on the table structure of the target information system.

2. The data standard management method according to claim 1, characterized in that: Before the step of comparing the field metadata with the standard field data to generate a difference comparison report, the step further includes: The standard field data and the field metadata are segmented respectively, and a standard segmentation result and a segmentation result to be checked are obtained correspondingly; Using a word vector model, the standard word segmentation results and the to-be-checked word segmentation results are respectively mapped to vector spaces to obtain a standard field vector space set and a to-be-checked field vector; Calculating a similarity value between the field vector to be checked and each standard field vector in the standard field vector space set; If the similarity value is equal to 1, the field vector to be checked and the standard field vector are marked as benchmarking standards; If the similarity value is not equal to 1, a preset threshold is obtained, the threshold is compared with the similarity value, and a target similarity value having a similarity value greater than the threshold is selected; The to-be-checked field vector corresponding to the target similarity value is associated with the standard field vector to obtain an associated field vector.

3. The data standard management method according to claim 2, characterized in that: The step of comparing the field metadata with the standard field data to generate a difference comparison report includes: Extracting the to-be-checked field name of the to-be-checked field vector from the associated field vector, and extracting the standard field name of the standard field vector from the associated field vector; Compare the name of the field to be checked with the name of the standard field; If the name of the field to be checked is inconsistent with the standard field name, a difference comparison report is generated according to the name of the field to be checked and the standard field name.

4. The data standard management method according to claim 2, characterized in that: The step of comparing the field metadata with the standard field data to generate a difference comparison report includes: Extracting the to-be-checked data type of the to-be-checked field vector from the associated field vector, and extracting the standard data type of the standard field vector from the associated field vector; Comparing the to-be-checked data type with the standard data type; If the data type to be checked is inconsistent with the standard data type, a difference comparison report is generated according to the data type to be checked and the standard data type.

5. The data standard management method according to claim 2, characterized in that: The step of comparing the field metadata with the standard field data to generate a difference comparison report includes: Extracting the to-be-checked value range of the to-be-checked field vector from the associated field vector, and extracting the standard value range of the standard field vector from the associated field vector; Comparing the to-be-checked value range with the standard value range; If the value range to be checked exceeds the standard value range, a difference comparison report is generated according to the value range to be checked and the standard value range.

6. The data standard management method according to claim 2, characterized in that: The step of generating a field correction script set according to each difference field included in the difference benchmarking report includes: Identify the difference type of each difference field included in the difference benchmarking report; If the difference type is a field name difference, then obtaining the to-be-checked field name and the standard field name included in the difference field, and generating a field renaming script for the to-be-checked field name according to the standard field name; If the difference type is a data type difference, then obtaining the to-be-checked data type and the standard data type included in the difference field, and generating a field type conversion script for the to-be-checked data type according to the standard data type; If the difference type is a value range difference, the value range to be checked and the standard value range included in the difference field are obtained, and a value range constraint script for the value range to be checked is generated according to the standard value range.

7. The data standard management method according to claim 6, characterized in that: The step of integrating the field modification script set into a batch transaction and executing the batch transaction to perform data standard management on the table structure of the target information system includes: Integrate one or more of the field renaming script, the field type conversion script, and the value range constraint script into a batch transaction; Executing the batch transaction and monitoring the execution process of the batch transaction, and when all scripts in the batch transaction are successfully executed, correcting the data standard of the table structure of the target information system; When a script execution fails in the batch transaction, the execution process of the batch transaction is terminated and the batch transaction is rolled back; the step of generating a field correction script set according to each difference field included in the difference benchmarking report is returned to execution; until all scripts in the batch transaction are successfully executed, the data standard of the table structure of the target information system is corrected.

8. A data standard management device, characterized in that: The device comprises: A data acquisition module is used to obtain the field metadata contained in the table structure of the target information system; A difference comparison module is used to obtain standard field data in the enterprise data standard library, compare the field metadata with the standard field data, and generate a difference comparison report; A script generation module, used to generate a field correction script set according to each difference field included in the difference benchmarking report; The transaction processing module is used to integrate the field modification script set into a batch transaction, and execute the batch transaction to perform data standard modification on the table structure of the target information system.

9. A computer device, characterized in that: It comprises a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the data standard governance method as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the data standard governance method according to any one of claims 1 to 7 are implemented.

Images