Metadata benchmarking method and device, electronic equipment and storage medium

Abstract

The invention discloses a metadata benchmarking method and device, electronic equipment and a storage medium. The method comprises the steps that in response to a target import operation, a to-be-benchmarked file is determined, and the to-be-benchmarked file comprises a plurality of to-be-benchmarked data items; in response to the data benchmarking operation, benchmarking the to-be-benchmarked data item through a data benchmarking model to obtain a benchmarked to-be-audited data item, and displaying the to-be-audited data item based on a target format, the data benchmarking model being obtained by training a contrast learning sentence vector model based on a training sample; determining a target data item corresponding to each to-be-audited data item in response to a target auditing operation for the displayed to-be-audited data items; and in response to the target exporting operation, exporting target benchmarking files corresponding to the multiple target data items. According to the technical scheme, the effect of performing automatic benchmarking on the metadata through the learning model in a one-key mode can be achieved, and the efficiency and quality of benchmarking of the metadata are improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a metadata alignment method, device, electronic device and storage medium. Background Art

[0002] Metadata refers to the fields that describe a data set. Data analysis of metadata can obtain certain characteristics described by the metadata, such as business characteristics, technical characteristics, or management characteristics. Before data analysis, benchmarking metadata to obtain standardized metadata can effectively improve the efficiency and quality of data analysis.

[0003] At present, metadata is usually standardized by manually performing metadata benchmarking. However, manual benchmarking is a single repetitive operation. When the amount of data is large, the benchmarking cycle is long and benchmarking errors are prone to occur. Therefore, it is difficult to ensure the efficiency and quality of data benchmarking. Summary of the invention

[0004] The present invention provides a metadata benchmarking method, device, electronic device and storage medium to solve the technical problem that the current method of performing data benchmarking on metadata is difficult to ensure the efficiency and quality of data benchmarking.

[0005] According to one aspect of the present invention, a metadata alignment method is provided, the method comprising:

[0006] In response to the target import operation, determining a file to be benchmarked, wherein the file to be benchmarked includes a plurality of data items to be benchmarked, and the data items to be benchmarked include metadata;

[0007] In response to the data benchmarking operation, the data item to be benchmarked is benchmarked by a data benchmarking model to obtain the benchmarked data item to be reviewed, and the data item to be reviewed is displayed based on the target format, wherein the data benchmarking model is obtained by training a sentence vector model for contrast learning based on training samples;

[0008] In response to the target review operation on the displayed plurality of data items to be reviewed, determining a target data item corresponding to each of the data items to be reviewed;

[0009] In response to the target export operation, target benchmarking files corresponding to the plurality of target data items are exported.

[0010] According to another aspect of the present invention, a metadata alignment device is provided, the device comprising:

[0011] A data import module, configured to determine a file to be benchmarked in response to a target import operation, wherein the file to be benchmarked includes a plurality of data items to be benchmarked, and the data items to be benchmarked include metadata;

[0012] A data benchmarking module, used for, in response to the data benchmarking operation, benchmarking the data items to be benchmarked through a data benchmarking model to obtain the benchmarked data items to be reviewed, and displaying the data items to be reviewed based on a target format, wherein the data benchmarking model is obtained by training a sentence vector model for contrast learning based on training samples;

[0013] A benchmarking review module, configured to determine a target data item corresponding to each of the data items to be reviewed in response to a target review operation on the displayed plurality of data items to be reviewed;

[0014] The file export module is used to export target benchmarking files corresponding to multiple target data items in response to a target export operation.

[0015] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0016] at least one processor; and a memory in communication with the at least one processor; wherein,

[0017] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the metadata alignment method described in any embodiment of the present invention.

[0018] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the metadata alignment method described in any embodiment of the present invention when executed.

[0019] The technical solution of the embodiment of the present invention determines a file to be benchmarked in response to a target import operation, wherein the file to be benchmarked includes multiple data items to be benchmarked, and the data items to be benchmarked include metadata; in response to a data benchmarking operation, the data items to be benchmarked are benchmarked through a data benchmarking model to obtain the benchmarked data items to be reviewed, and the data items to be reviewed are displayed based on a target format, wherein the data benchmarking model is obtained by training a contrastive learning sentence vector model based on training samples, thereby achieving an effect of one-click metadata automatic benchmarking, and metadata benchmarking can be improved by performing metadata benchmarking based on the data benchmarking model trained based on the contrastive learning sentence vector model; in response to a target review operation for the displayed multiple data items to be reviewed, a target data item corresponding to each of the data items to be reviewed is determined, thereby achieving an effect of reviewing the metadata benchmarking results (i.e., the data items to be reviewed) based on the personalized needs of the user; in response to a target export operation, a target benchmarking file corresponding to the multiple target data items is exported, thereby achieving a one-click file export for the metadata benchmarking results (i.e., the target data items) that meet the personalized needs of the user.

[0020] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 is a flowchart of a metadata alignment method provided according to Embodiment 1 of the present invention;

[0023] Figure 2 is an example diagram of a data item to be aligned provided according to an embodiment of the present invention;

[0024] Figure 3 is an example diagram of a data item to be reviewed provided according to an embodiment of the present invention;

[0025] Figure 4 is a flowchart of a metadata alignment method provided according to Embodiment 2 of the present invention;

[0026] Figure 5 is an overall flow chart of a metadata alignment method provided according to an embodiment of the present invention;

[0027] Figure 6 It is a structural diagram of a metadata alignment device provided according to Embodiment 3 of the present invention. DETAILED DESCRIPTION

[0028] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0029] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0030] Embodiment 1

[0031] Figure 1 A flowchart of a metadata alignment method is provided for the first embodiment of the present invention. This embodiment is applicable to the case of automatic alignment of metadata. The method can be executed by a metadata alignment device. The metadata alignment device can be implemented in the form of hardware and / or software. The metadata alignment device can be configured in a computer. Figure 1 As shown, the method includes:

[0032] S110 . In response to the target import operation, determine the file to be compared.

[0033] In the embodiment of the present invention, the file to be benchmarked includes a plurality of data items to be benchmarked.

[0034] The target import operation may be an operation of importing a file to be aligned that contains data items to be aligned, or an operation of importing a data table that contains metadata.

[0035] The file to be aligned can be understood as a file to be aligned with metadata. In the embodiment of the present invention, the file to be aligned can be one or more files.

[0036] The data item to be aligned can be understood as the data item to be aligned. The data item to be aligned can include metadata. Optionally, the data item to be aligned can include a data sentence corresponding to the metadata or a data word corresponding to the metadata. Figure 2 As shown, Figure 2 is an example diagram of a data item to be aligned according to an embodiment of the present invention. Figure 2 For example, the data items to be benchmarked may be measure number, organizer's opinion, review opinion, CSBH, and CSRYJ, etc.

[0037] Optionally, in response to the target import operation, determining the file to be benchmarked includes:

[0038] In response to the target import operation, multiple data tables to be imported are determined, and data table related information corresponding to the data tables is determined, wherein the data table related information includes at least one of a total storage capacity of the table, a field type in the table, a fill rate of the table, a connection relationship between the tables, and table source information;

[0039] Determine the files to be compared corresponding to the multiple data tables according to the relevant information of the data tables.

[0040] The data table may be a data table in a database, and the data table may include metadata.

[0041] The data table related information can be understood as information related to the data table.

[0042] Specifically, metadata in a plurality of data tables are written into an initialized table file according to the relevant information of the data tables to obtain the file to be aligned.

[0043] S120 . In response to the data benchmarking operation, the data items to be benchmarked are benchmarked through a data benchmarking model to obtain the benchmarked data items to be reviewed, and the data items to be reviewed are displayed based on a target format.

[0044] In an embodiment of the present invention, the data benchmarking model is obtained by training a sentence vector model based on training samples for contrast learning.

[0045] The data matching operation may be understood as an operation of starting metadata matching. Optionally, the data matching operation may be a trigger operation for a data matching control. The data matching control may be a data matching button displayed on the screen. Exemplarily, the data matching operation may be clicking the data matching button.

[0046] Based on the solution of the embodiment of the present invention, a one-click automatic metadata alignment effect can be achieved.

[0047] The data benchmarking model can be understood as a machine model with metadata benchmarking function. The training samples can be used to train the data benchmarking model. The Simple Contrastive Learning of Sentence Embeddings (SimCSE) model can perform sentence embedding representation based on contrastive learning.

[0048] The data items to be reviewed can be understood as the benchmarking results of the data items to be manually reviewed. Figure 3 As shown, Figure 3 It is an example diagram of a data item to be reviewed provided according to an embodiment of the present invention. Figure 3 Can be Figure 2 A corresponding metadata benchmark result. Figure 2 Corresponding to the data items to be benchmarked given as examples, the data items to be reviewed may be measure number, undertaker_opinion, review_opinion, CSBH and CSR_YJ, etc.

[0049] The target format may be a format for displaying the data item to be reviewed. Optionally, the target format may be a data storage format corresponding to the file to be compared. Figure 3 , exemplary, the target format may be in a tabular format.

[0050] S130: In response to a target review operation on the displayed data item to be reviewed, determine a target data item corresponding to the data item to be reviewed.

[0051] The target review operation may be an operation of manually reviewing the metadata matching result. Optionally, the target review operation may include a review pass operation and / or a review fail operation.

[0052] The target data item can be understood as the benchmarking result of the data item that has passed the manual review, that is, the data item to be reviewed that has passed the manual review.

[0053] Optionally, the target review operation includes a review pass operation; and the determining of the target data item corresponding to each of the to-be-reviewed data items includes:

[0054] In the case where the target audit operation is the audit pass operation, the data item to be audited is used as the target data item.

[0055] Specifically, when the target audit operation is the audit pass operation, for each of the data items to be audited, the data item to be audited is directly used as the target data item.

[0056] S140. In response to the target export operation, export target benchmarking files corresponding to the plurality of target data items.

[0057] The target export operation may be an operation of exporting the target data item.

[0058] Specifically, in response to the target export operation, the target data item is exported to obtain a plurality of target benchmarking files corresponding to the target data items.

[0059] The target alignment file may be understood as a file containing a plurality of target data items, or may be understood as a file that completes metadata alignment.

[0060] The technical solution of the embodiment of the present invention determines a file to be benchmarked in response to a target import operation, wherein the file to be benchmarked includes multiple data items to be benchmarked, and the data items to be benchmarked include metadata; in response to a data benchmarking operation, the data items to be benchmarked are benchmarked through a data benchmarking model to obtain the benchmarked data items to be reviewed, and the data items to be reviewed are displayed based on a target format, wherein the data benchmarking model is obtained by training a contrastive learning sentence vector model based on training samples, thereby achieving an effect of one-click metadata automatic benchmarking, and metadata benchmarking can be improved by performing metadata benchmarking based on the data benchmarking model trained based on the contrastive learning sentence vector model; in response to a target review operation for the displayed multiple data items to be reviewed, a target data item corresponding to each of the data items to be reviewed is determined, thereby achieving an effect of reviewing the metadata benchmarking results (i.e., the data items to be reviewed) based on the personalized needs of the user; in response to a target export operation, a target benchmarking file corresponding to the multiple target data items is exported, thereby achieving a one-click file export for the metadata benchmarking results (i.e., the target data items) that meet the personalized needs of the user.

[0061] Embodiment 2

[0062] Figure 4 This is a flowchart of a metadata benchmarking method provided in the second embodiment of the present invention. This embodiment is to benchmark the data items to be benchmarked through the data benchmarking model, and refine the data items to be reviewed after the benchmarking. Figure 4 As shown, the method includes:

[0063] S210: In response to the target import operation, determine the file to be compared.

[0064] S220. For each of the data items to be benchmarked, perform keyword extraction on the data item to be benchmarked through the data benchmarking model of the first benchmarking parameter to obtain at least one first keyword, and determine a first other word in the data item to be benchmarked except the first keyword, and determine the data item to be reviewed based on the first keyword and the first other word.

[0065] The first benchmarking parameter can be understood as a model benchmarking parameter preset for the data benchmarking model. Optionally, the model benchmarking parameter can be used to provide a benchmarking prompt for the data benchmarking model, so that the data benchmarking model performs metadata benchmarking based on the model benchmarking parameter. Exemplarily, the model benchmarking parameter can be parameter a, parameter b, or parameter c, etc.

[0066] The first keyword can be understood as a keyword in the metadata to be benchmarked. In an embodiment of the present invention, the first keyword corresponding to each metadata to be benchmarked can be one or more. In addition to the above situation, keywords may not be extracted from the metadata to be benchmarked. In layman's terms, the first keyword can be what technicians in the field commonly call standard metadata, that is, national standard metadata.

[0067] The first other words can be understood as other words in the metadata to be benchmarked except the first keyword. In an embodiment of the present invention, the first other words corresponding to each metadata to be benchmarked can be one or more. In addition to the above situation, other words may not be extracted from the metadata to be benchmarked. In layman's terms, the first other words can be what technicians in the field commonly call qualifiers.

[0068] Exemplarily, the data item to be matched is: organizer_opinion, for example, the first keyword may be opinion, and the first other word may be organizer.

[0069] Optionally, extracting keywords from the data item to be benchmarked by the data benchmarking model using the first benchmarking parameter to obtain at least one first keyword includes:

[0070] Determine a first word set corresponding to the first benchmarking parameter by using the data benchmarking model, wherein the first word set includes a plurality of preset keywords;

[0071] Keywords are extracted from the data item to be aligned based on the first word set to obtain at least one first keyword.

[0072] The first keyword may be a target word existing in the to-be-matched data item and matching the keyword in the first word set.

[0073] The first word set includes a plurality of preset keywords. In layman's terms, the first word set is a word set including a plurality of standard metadata, that is, a national standard word set.

[0074] Specifically, target words existing in the to-be-matched data item and matching the keywords in the first word set are determined, and at least one of the target words is used as the first keyword.

[0075] Optionally, determining the data item to be reviewed based on the first keyword and the first other words includes:

[0076] Determine a first delimiter corresponding to the first alignment parameter;

[0077] A data item to be reviewed is determined based on the first keyword, the first other word, and the first separator.

[0078] Among them, the first separator can be understood as a preset separator. In an embodiment of the present invention, the separator corresponding to each of the benchmarking parameters can be preset according to the scene requirements, and is not specifically limited here. The separators corresponding to different benchmarking parameters can be the same or different. Exemplarily, the separator can be "-", ", " or "_", etc.

[0079] Optionally, determining the data item to be reviewed based on the first keyword, the first other word, and the first separator includes:

[0080] The first keyword, the first other word and the first separator are combined based on a preset combination method to obtain a data item to be reviewed; wherein the preset combination method may be to separate the keyword and other words based on the separator.

[0081] Optionally, the determining the data item to be reviewed based on the first keyword, the first other word and the first separator further includes:

[0082] The first replacement word corresponding to each of the first keywords is determined through a keyword replacement library, wherein the keyword replacement library includes multiple keywords, multiple replacement words and a word mapping relationship, and the word mapping relationship is a mapping relationship between keywords and replacement words.

[0083] In the embodiment of the present invention, the keywords, mapping words and mapping relationships in the keyword replacement library can be preset according to scene requirements and are not specifically limited here.

[0084] For example, Figure 2 and Figure 3 For example, the keyword may be the start time, and the replacement word corresponding to the start time may be the start time. The keyword may be the audit result, and the replacement word corresponding to the audit result may be the audit result code.

[0085] S230: Display the data items to be reviewed based on the target format.

[0086] S240: In response to a target review operation on the displayed plurality of data items to be reviewed, determine a target data item corresponding to each of the data items to be reviewed.

[0087] Optionally, the target review operation includes a review failure operation; and the determining of the target data item corresponding to each of the to-be-reviewed data items includes:

[0088] When the target audit operation is the audit failure operation, switching the first benchmarking parameter corresponding to the data benchmarking model to obtain a second benchmarking parameter;

[0089] For the data benchmarking model of the second benchmarking parameter, returning to execute the operation of benchmarking the data item to be benchmarked by using the data benchmarking model to obtain the updated data item to be reviewed;

[0090] In response to the target review operation on the displayed multiple updated data items to be reviewed, a target data item corresponding to each of the data items to be reviewed is determined.

[0091] The second benchmarking parameter may be understood as a model benchmarking parameter different from the first benchmarking parameter.

[0092] In an embodiment of the present invention, optionally, in response to the audit failure of the updated data item to be audited, the second benchmarking parameter corresponding to the data benchmarking model can be switched to obtain a third benchmarking parameter to obtain the updated data item to be audited under the third benchmarking parameter; until the audit is passed and the target data item is obtained.

[0093] Based on the above-mentioned embodiment scheme, the metadata benchmarking result (i.e., the data item to be reviewed) of the data benchmarking model can be updated based on the user's review operation by switching the benchmarking parameters corresponding to the data benchmarking model until a benchmarking result that meets the user's personalized needs is obtained.

[0094] S250. In response to the target export operation, export target benchmarking files corresponding to the plurality of target data items.

[0095] The technical solution of the embodiment of the present invention is to extract keywords from the data item to be benchmarked through the data benchmarking model of the first benchmarking parameter for each of the data items to be benchmarked, obtain at least one first keyword, and determine the first other words in the data item to be benchmarked except the first keyword, and determine the data item to be reviewed based on the first keyword and the first other words, wherein the first benchmarking parameter is a model benchmarking parameter preset for the data benchmarking model. The embodiment of the present invention benchmarks the data items by keyword extraction to obtain standardized data items, thereby avoiding the loss of important information represented by the original data items during the data benchmarking process, and effectively retains the original information represented by the original data items on the basis of data benchmarking to obtain more standardized data items.

[0096] Figure 5 is an overall flow chart of a metadata alignment method provided according to an embodiment of the present invention. Figure 5 The overall process of the metadata benchmarking method is further explained:

[0097] First, obtain the data table to be benchmarked. The data table contains the basic information of the data items to be benchmarked, which may include connection information, source information, and storage information; then use the corresponding exploration tools to explore the total storage volume, field length, type, fill rate, sample data, and outliers of the data table, and output the exploration report; then generate the file to be benchmarked based on the exploration report; prepare the dependent environment before automatic benchmarking (for example, standard data elements, qualifiers, and dictionary items, etc.), debug the benchmarking parameters of the SimCSE model of the PyTorch framework and start it normally, execute the main program that calls the automatic benchmarking script, and obtain the benchmarking results; review the benchmarking results by humans; the entire benchmarking process ends when the review is passed.

[0098] Furthermore, the data benchmarking model is further elaborated as follows:

[0099] For the SimCSE model. The data benchmarking model is trained based on the SimCSE model, which can represent sentence vectors based on contrastive learning. The core goal of contrastive learning is to learn an encoder so that similar samples (i.e. positive samples) are closer in the encoded vector space, while dissimilar samples (i.e. negative samples) are farther away in the space. In this way, the model can better learn the representation of the data.

[0100] For contrastive learning in the SimCSE model. In SimCSE, similar sentences are considered positive samples, and dissimilar sentences are considered negative samples. The model learns the representation of sentences by shortening the distance of positive samples and increasing the distance of negative samples. Specifically, cosine similarity is used as the similarity metric function of the SimCSE model to calculate the cosine similarity between two sentence vectors. Then, the model parameters are adjusted by optimizing the objective function so that the cosine similarity of similar sentences is increased and the cosine similarity of dissimilar sentences is reduced.

[0101] Data enhancement strategy for the SimCSE model: The following data enhancement strategy for the SimCSE model can be used to construct positive and negative samples.

[0102] Positive sample construction: For unsupervised SimCSE models, dropout technology can be used to construct positive samples. Specifically, for the same sentence, two different dropout masks are used in training, and the sentence pairs after the two dropouts are regarded as a set of positive samples of each other. This can avoid complex text enhancement operations and effectively learn the semantic representation of sentences. For supervised SimCSE models, positive samples can be constructed with the help of textual entailment (natural language inference) data items. The entailment-pair is used as a positive example for subsequent contrastive learning training.

[0103] Negative sample construction: In the unsupervised SimCSE model, negative samples are usually constructed by sampling the rest of the sentences in the same batch. In the supervised SimCSE model, the contradiction-pair can be regarded as a hard negative example to train the model to better distinguish dissimilar sentences.

[0104] Optimization goal for the SimCSE model. The optimization goal of the SimCSE model is to maximize the similarity between positive samples and minimize the similarity between negative samples. This can be achieved through a contrast loss function, such as the InfoNCE loss. During the training process, by continuously adjusting the model parameters to optimize this loss function, a more effective representation of the sentence can be learned, achieving better matching results in text tasks.

[0105] The technical solution of the present invention can greatly improve the efficiency of metadata benchmarking; reduce manual operations and significantly reduce labor costs; and achieve the effect of high-quality data application after data benchmarking.

[0106] Embodiment 3

[0107] Figure 6 This is a schematic diagram of the structure of a metadata alignment device provided in Embodiment 3 of the present invention. Figure 6 As shown, the device includes: a data import module 310, a data benchmarking module 320, a benchmarking review module 330 and a file export module 340.

[0108] Among them, the data import module 310 is used to determine the file to be benchmarked in response to the target import operation, wherein the file to be benchmarked includes multiple data items to be benchmarked, and the data items to be benchmarked include metadata; the data benchmarking module 320 is used to respond to the data benchmarking operation, benchmark the data items to be benchmarked through the data benchmarking model, obtain the benchmarked data items to be reviewed, and display the data items to be reviewed based on the target format, wherein the data benchmarking model is trained based on the training sample pair contrast learning sentence vector model; the benchmarking review module 330 is used to respond to the target review operation for the displayed multiple data items to be reviewed, and determine the target data item corresponding to each of the data items to be reviewed; the file export module 340 is used to export the target benchmarking files corresponding to the multiple target data items in response to the target export operation.

[0109] The technical solution of the embodiment of the present invention determines a file to be benchmarked in response to a target import operation, wherein the file to be benchmarked includes multiple data items to be benchmarked; in response to a data benchmarking operation, the data items to be benchmarked are benchmarked through a data benchmarking model to obtain the benchmarked data items to be reviewed, and the data items to be reviewed are displayed based on a target format, wherein the data benchmarking model is obtained by training a contrastive learning sentence vector model based on training samples, thereby achieving an effect of one-click metadata automatic benchmarking, and metadata benchmarking can be improved by performing metadata benchmarking based on the data benchmarking model trained based on the contrastive learning sentence vector model; in response to a target review operation for the displayed multiple data items to be reviewed, a target data item corresponding to each of the data items to be reviewed is determined, thereby achieving an effect of reviewing the metadata benchmarking results (i.e., the data items to be reviewed) based on the personalized needs of the user; in response to a target export operation, a target benchmarking file corresponding to the multiple target data items is exported, thereby achieving a one-click file export for the metadata benchmarking results (i.e., the target data items) that meet the personalized needs of the user.

[0110] Optionally, the data benchmarking module 320 includes:

[0111] A data benchmarking unit is used to extract keywords from each of the data items to be benchmarked through the data benchmarking model of the first benchmarking parameter to obtain at least one first keyword, and determine first other words in the data item to be benchmarked except the first keyword, and determine the data item to be reviewed based on the first keyword and the first other words, wherein the first benchmarking parameter is a model benchmarking parameter preset for the data benchmarking model.

[0112] Optionally, the data benchmarking unit includes: a word set determination subunit and a keyword extraction subunit;

[0113] The word set determination subunit is used to determine a first word set corresponding to the first benchmarking parameter through the data benchmarking model, wherein the first word set includes a plurality of preset keywords;

[0114] The keyword extraction subunit is used to extract keywords from the to-be-matched data item based on the first word set to obtain at least one first keyword.

[0115] Optionally, the data matching unit includes: a separator determination subunit and a data combination subunit;

[0116] The delimiter determination subunit is used to determine a first delimiter corresponding to the first alignment parameter;

[0117] The data combination subunit is used to determine the data item to be reviewed based on the first keyword, the first other word and the first separator.

[0118] Optionally, the target audit operation includes an audit failure operation; the benchmarking audit module 330 is specifically used to:

[0119] When the target audit operation is the audit failure operation, switching the first benchmarking parameter corresponding to the data benchmarking model to obtain a second benchmarking parameter;

[0120] For the data benchmarking model of the second benchmarking parameter, returning to execute the operation of benchmarking the data item to be benchmarked by using the data benchmarking model to obtain the updated data item to be reviewed;

[0121] In response to the target review operation on the displayed multiple updated data items to be reviewed, a target data item corresponding to each of the data items to be reviewed is determined.

[0122] Optionally, the benchmarking review module 330 is specifically used to:

[0123] In the case where the target audit operation is the audit pass operation, for each of the to-be-audited data items, the to-be-audited data item is used as a target data item.

[0124] Optionally, the data import module 310 is specifically used for:

[0125] In response to the target import operation, multiple data tables to be imported are determined, and data table related information corresponding to the data tables is determined, wherein the data table related information includes at least one of a total storage capacity of the table, a field type in the table, a fill rate of the table, a connection relationship between the tables, and table source information;

[0126] Determine the files to be compared corresponding to the multiple data tables according to the relevant information of the data tables.

[0127] The metadata alignment device provided in the embodiment of the present invention can execute the metadata alignment method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

Claims

1. A metadata benchmarking method, characterized in that: include: In response to the target import operation, determining a file to be benchmarked, wherein the file to be benchmarked includes a plurality of data items to be benchmarked, and the data items to be benchmarked include metadata; In response to the data benchmarking operation, the data item to be benchmarked is benchmarked by a data benchmarking model to obtain the benchmarked data item to be reviewed, and the data item to be reviewed is displayed based on the target format, wherein the data benchmarking model is obtained by training a sentence vector model for contrast learning based on training samples; In response to the target review operation on the displayed plurality of data items to be reviewed, determining a target data item corresponding to each of the data items to be reviewed; In response to the target export operation, target benchmarking files corresponding to the plurality of target data items are exported.

2. The method according to claim 1, characterized in that: The step of benchmarking the data items to be benchmarked by using a data benchmarking model to obtain the benchmarked data items to be reviewed includes: For each of the data items to be benchmarked, keywords are extracted for the data items to be benchmarked through the data benchmarking model of the first benchmarking parameter to obtain at least one first keyword, and first other words other than the first keyword in the data item to be benchmarked are determined, and the data item to be reviewed is determined based on the first keyword and the first other words, wherein the first benchmarking parameter is a model benchmarking parameter preset for the data benchmarking model.

3. The method according to claim 2, characterized in that The step of extracting keywords from the data item to be benchmarked by the data benchmarking model using the first benchmarking parameter to obtain at least one first keyword includes: Determine a first word set corresponding to the first benchmarking parameter by using the data benchmarking model, wherein the first word set includes a plurality of preset keywords; Keywords are extracted from the data item to be aligned based on the first word set to obtain at least one first keyword.

4. The method according to claim 2, characterized in that: The determining of the data item to be reviewed based on the first keyword and the first other words includes: Determine a first delimiter corresponding to the first alignment parameter; A data item to be reviewed is determined based on the first keyword, the first other word, and the first separator.

5. The method according to claim 2, characterized in that: The target audit operation includes an audit failure operation; and the determining of the target data item corresponding to each of the to-be-audited data items includes: When the target audit operation is the audit failure operation, switching the first benchmarking parameter corresponding to the data benchmarking model to obtain a second benchmarking parameter; For the data benchmarking model of the second benchmarking parameter, returning to execute the operation of benchmarking the data item to be benchmarked by using the data benchmarking model to obtain the updated data item to be reviewed; In response to the target review operation on the displayed multiple updated data items to be reviewed, a target data item corresponding to each of the data items to be reviewed is determined.

6. The method according to claim 1, characterized in that The target review operation includes a review pass operation; the determining of the target data item corresponding to each of the to-be-reviewed data items includes: In the case where the target audit operation is the audit pass operation, for each of the to-be-audited data items, the to-be-audited data item is used as a target data item.

7. The method according to claim 1, characterized in that The step of determining the file to be compared in response to the target import operation includes: In response to the target import operation, multiple data tables to be imported are determined, and data table related information corresponding to the data tables is determined, wherein the data table related information includes at least one of a total storage capacity of the table, a field type in the table, a fill rate of the table, a connection relationship between the tables, and table source information; Determine the files to be compared corresponding to the multiple data tables according to the relevant information of the data tables.

8. A metadata benchmarking device, characterized in that: include: A data import module, configured to determine a file to be benchmarked in response to a target import operation, wherein the file to be benchmarked includes a plurality of data items to be benchmarked, and the data items to be benchmarked include metadata; A data benchmarking module, used for, in response to the data benchmarking operation, benchmarking the data items to be benchmarked through a data benchmarking model to obtain the benchmarked data items to be reviewed, and displaying the data items to be reviewed based on a target format, wherein the data benchmarking model is obtained by training a sentence vector model for contrast learning based on training samples; A benchmarking review module, configured to determine a target data item corresponding to each of the data items to be reviewed in response to a target review operation on the displayed plurality of data items to be reviewed; The file export module is used to export target benchmarking files corresponding to multiple target data items in response to a target export operation.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory in communication with the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the metadata alignment method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the metadata alignment method according to any one of claims 1 to 7 when executed.

Images