A method for determining a data standard and related apparatus

Abstract

The application discloses a data standard determination method and related device. Based on the data to be benchmarked, a standard judgment model is used to output a standard judgment result corresponding to the data to be benchmarked; based on the data to be benchmarked, a standard benchmarking model is used to output a data standard name corresponding to the data to be benchmarked; based on the matching result of the standard judgment result and the data standard name, a target data standard name corresponding to the data to be benchmarked is determined; based on the mapping relationship between the data standard and the data standard name and the target data standard name, a target data standard corresponding to the data to be benchmarked is determined from a plurality of data standards. It can be seen that the data to be benchmarked is automatically benchmarked without human intervention, which can reduce the labor cost and the work burden, and the output results of the two models are mutually corrected, which can improve the accuracy of the target data standard name determined, and further improve the accuracy of the data benchmarking result (i.e., the target data standard determined).

Description

Technical Field

[0001] This application relates to the field of data governance, and in particular to a method and apparatus for determining data standards. Background Technology

[0002] Data governance is the process of processing, formatting, and standardizing data. It is a crucial means to improve the quality of data within enterprises and other organizations, promote widespread data sharing, strengthen data security, and unlock the value of data resources. Data standards are the normative constraints that ensure the consistency and accuracy of data use and exchange both internally and externally. They are the core activity and primary task of data governance for enterprises, playing a key role in reducing governance complexity, improving data quality, breaking down data silos, accelerating data exchange and sharing, and releasing data value. Therefore, determining the corresponding data standards (also known as data benchmarking) is a particularly important process in data governance.

[0003] In related technologies, data benchmarking is usually done manually. However, internal data of enterprises and other organizations often contains a large amount of existing data as well as incremental data, which requires a lot of manpower and time, increases labor costs and workload, and the final data benchmarking results are also prone to errors.

[0004] Therefore, how to improve the accuracy of data benchmarking results while reducing labor costs and workload has become an urgent problem to be solved. Summary of the Invention

[0005] In view of this, embodiments of this application provide a method and related apparatus for determining data standards, aiming to reduce labor costs and workload while improving the accuracy of data benchmarking results.

[0006] In a first aspect, embodiments of this application provide a method for determining data standards, the method comprising:

[0007] Based on the data to be benchmarked, a standard judgment model is used to output the standard judgment result corresponding to the data to be benchmarked; the standard judgment result is the judgment result of whether the data to be benchmarked exists; the standard judgment model is trained based on a first data sample and the standard judgment label corresponding to the first data sample; the standard judgment label includes a label indicating the existence of a data standard and a label indicating the non-existence of a data standard; the data to be benchmarked includes business data related to the target industry and computer system data related to the target industry.

[0008] Based on the data to be benchmarked, the data standard name corresponding to the data to be benchmarked is output through the standard benchmarking model; the standard benchmarking model is trained based on the second data sample and the data standard name label corresponding to the second data sample; the data standard name label includes name labels representing data standards and name labels representing no data standards; the data standard corresponding to the name label representing data standards includes data standard content corresponding to at least one data standard field.

[0009] Based on the matching results of the standard judgment and the data standard name, the target data standard name corresponding to the data to be benchmarked is determined;

[0010] Based on the mapping relationship between data standards and data standard names, as well as the target data standard name, the target data standard corresponding to the data to be benchmarked is determined from multiple data standards.

[0011] Optionally, determining the target data standard name corresponding to the data to be benchmarked based on the matching result of the standard judgment result and the data standard name specifically includes:

[0012] If the matching result is a match between the standard judgment result and the data standard name, then the data standard name is determined as the target data standard name corresponding to the data to be benchmarked.

[0013] If the matching result is that the standard judgment result and the data standard name do not match, then the target data standard name corresponding to the data to be benchmarked is determined based on the first probability corresponding to the standard judgment result and the second probability corresponding to the data standard name; the first probability is the probability corresponding to the result output by the standard judgment model; the second probability is the probability corresponding to the result output by the standard benchmarking model.

[0014] Optionally, determining the target data standard name corresponding to the data to be benchmarked based on the first probability corresponding to the standard judgment result and the second probability corresponding to the data standard name specifically includes:

[0015] If the first probability corresponding to the standard judgment result is less than the second probability corresponding to the data standard name, then the data standard name is determined as the target data standard name corresponding to the data to be benchmarked.

[0016] If the first probability corresponding to the standard judgment result is greater than or equal to the second probability corresponding to the data standard name, and the standard judgment result indicates that the data to be benchmarked has a data standard, then the data standard name corresponding to the second largest probability in the output result of the standard benchmarking model is determined as the target data standard name corresponding to the data to be benchmarked.

[0017] If the first probability corresponding to the standard judgment result is greater than or equal to the second probability corresponding to the data standard name, and the standard judgment result is that the data to be benchmarked does not have a data standard, then the data standard name characterized as having no data standard name is determined as the target data standard name corresponding to the data to be benchmarked.

[0018] Optionally, the standard judgment model includes a vector transformation layer, a convolutional layer, a pooling layer, and a fully connected layer; the step of outputting the standard judgment result corresponding to the data to be benchmarked through the standard judgment model specifically includes:

[0019] The vector transformation layer performs vector transformation on the data to be benchmarked to obtain the vector representation of the data to be benchmarked.

[0020] The vector representation of the data to be benchmarked is convolved by the convolutional layer to obtain the first vector feature matrix of the data to be benchmarked.

[0021] The pooling layer is used to perform pooling processing on the first vector feature matrix of the data to be processed to obtain the second vector feature matrix of the data to be benchmarked.

[0022] The second vector feature matrix of the data to be benchmarked is normalized by the fully connected layer to obtain the standard judgment result corresponding to the data to be benchmarked.

[0023] Optionally, the standard judgment model is obtained through the following steps:

[0024] Obtain the first data sample and the standard judgment label corresponding to the first data sample;

[0025] Based on the first data sample, the prediction standard judgment result corresponding to the first data sample is output through the first model to be trained;

[0026] Based on the difference between the standard judgment label corresponding to the first data sample and the prediction standard judgment result corresponding to the first data sample, the parameters of the first model to be trained are adjusted.

[0027] When the training cutoff condition is met, training ends and the standard judgment model is obtained.

[0028] Optionally, the standard benchmarking model is obtained through the following steps:

[0029] Obtain the second data sample and the corresponding data standard name label;

[0030] Based on the second data sample, the predicted data standard name corresponding to the second data sample is output by the second model to be trained;

[0031] Based on the difference between the standard data name label corresponding to the second data sample and the predicted standard data name corresponding to the second data sample, the parameters of the second model to be trained are adjusted.

[0032] When the training cutoff condition is met, the training ends and the standard benchmark model is obtained.

[0033] Optionally, the first data sample or the second data sample is obtained through the following steps:

[0034] Obtain multiple initial data samples; the multiple initial data samples include business field metadata related to the target industry and computer system field metadata related to the target industry;

[0035] From the plurality of initial data samples, a first initial data sample of Chinese words is determined based on the industry thesaurus corresponding to the target industry; the industry thesaurus includes a plurality of Chinese words related to the target industry;

[0036] The remaining initial data samples, excluding the first initial data sample, are segmented into words to obtain a second initial data sample consisting of English words or abbreviations.

[0037] The first initial data sample and the second initial data sample are combined to obtain the target industry thesaurus;

[0038] The target industry thesaurus includes multiple words as the first data sample or the second data sample.

[0039] Secondly, embodiments of this application provide a data standard determination apparatus, the apparatus comprising:

[0040] A standard judgment module is used to output a standard judgment result corresponding to the data to be benchmarked based on the data to be benchmarked, through a standard judgment model. The standard judgment result is the judgment result of whether the data to be benchmarked exists. The standard judgment model is trained based on a first data sample and the standard judgment label corresponding to the first data sample. The standard judgment label includes a label indicating the existence of a data standard and a label indicating the non-existence of a data standard. The data to be benchmarked includes business data related to the target industry and computer system data related to the target industry.

[0041] The standard name output module is used to output the data standard name corresponding to the data to be benchmarked based on the data to be benchmarked, through a standard benchmarking model; the standard benchmarking model is trained based on the second data sample and the data standard name label corresponding to the second data sample; the data standard name label includes name labels representing data standard and name labels representing no data standard; the data standard corresponding to the name label representing data standard includes data standard content corresponding to at least one data standard field.

[0042] The standard name determination module is used to determine the target data standard name corresponding to the data to be benchmarked based on the standard judgment result and the matching result of the data standard name;

[0043] The standard determination module is used to determine the target data standard corresponding to the data to be benchmarked from multiple data standards based on the mapping relationship between data standards and data standard names and the target data standard name.

[0044] Thirdly, embodiments of this application provide a data standard determination device, the device including a memory and a processor:

[0045] The memory is used to store computer programs and transmit the computer programs to the processor;

[0046] The processor is configured to execute the computer program to cause the device to perform the data standard determination method described in the first aspect above.

[0047] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, wherein when the computer program is run, a device running the computer program implements the data standard determination method described in the first aspect above.

[0048] Compared with the prior art, the embodiments of this application have the following beneficial effects:

[0049] This application provides a method and related apparatus for determining data standards. In this method, based on the data to be benchmarked, a standard judgment model outputs the standard judgment result corresponding to the data to be benchmarked; the standard judgment result is the judgment result of whether the data to be benchmarked has a data standard; the standard judgment model is trained based on a first data sample and the standard judgment labels corresponding to the first data sample; the standard judgment labels include labels indicating the existence of a data standard and labels indicating the absence of a data standard; the data to be benchmarked includes business data related to the target industry and computer system data related to the target industry; based on the data to be benchmarked, a standard benchmarking model outputs the data standard name corresponding to the data to be benchmarked; the standard benchmarking model is trained based on a second data sample and the data standard name labels corresponding to the second data sample; the data standard name labels include name labels indicating the existence of a data standard and name labels indicating the absence of a data standard; the data standard corresponding to the name label indicating the existence of a data standard includes data standard content corresponding to at least one data standard field; based on the matching result of the standard judgment result and the data standard name, the target data standard name corresponding to the data to be benchmarked is determined; based on the mapping relationship between data standards and data standard names and the target data standard name, the target data standard corresponding to the data to be benchmarked is determined from multiple data standards.

[0050] As can be seen, the target data standard name corresponding to the data to be benchmarked, determined by this method, is obtained by matching the standard judgment result output by the standard judgment model with the data standard name output by the standard benchmarking model. The standard judgment result and the data standard name can be matched to determine a target data standard that more accurately corresponds to the data to be benchmarked. Furthermore, this application requires no manual intervention; it automatically performs benchmarking processing on the data to be benchmarked (i.e., determines the target data standard corresponding to the data to be benchmarked). Thus, this application reduces labor costs and workload while improving the accuracy of the data benchmarking results (i.e., the determined target data standard). Attached Figure Description

[0051] To more clearly illustrate the technical solutions in this embodiment or the prior art, the drawings used in the description of the embodiment or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0052] Figure 1 A flowchart illustrating a method for determining a data standard, as provided in an embodiment of this application;

[0053] Figure 2 A model structure diagram of a standard judgment model provided in an embodiment of this application;

[0054] Figure 3 A schematic diagram of a target industry thesaurus provided for an embodiment of this application;

[0055] Figure 4 A flowchart illustrating a method for determining a data standard provided in an embodiment of this application;

[0056] Figure 5 This is a schematic diagram of a data standard determination device provided in an embodiment of this application. Detailed Implementation

[0057] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0058] Currently, the data standards corresponding to the internal data of enterprises and other organizations are often determined manually based on multiple pre-defined data standards. However, the internal data of enterprises and other organizations usually contains a large amount of existing and incremental data. It requires a lot of manpower and time to determine the corresponding data standards. Moreover, due to differences in human understanding of data standards, errors may occur in determining the data standards, leading to incorrect data benchmarking results.

[0059] In order to solve the above problems, this application provides a method and related apparatus for determining data standards. In this method, based on the data to be benchmarked, a standard judgment model is used to output the standard judgment result corresponding to the data to be benchmarked. The standard judgment result is the judgment result of whether the data to be benchmarked exists. The standard judgment model is trained based on the first data sample and the standard judgment label corresponding to the first data sample. The standard judgment label includes a label indicating the existence of a data standard and a label indicating the absence of a data standard. The data to be benchmarked includes business data related to the target industry and computer system data related to the target industry. Based on the data to be benchmarked, the standard benchmarking model is used to output the data standard name corresponding to the data to be benchmarked. The standard benchmarking model is trained based on the second data sample and the data standard name label corresponding to the second data sample. The data standard name label includes a name label indicating the existence of a data standard and a name label indicating the absence of a data standard. The data standard corresponding to the name label indicating the existence of a data standard includes the data standard content corresponding to at least one data standard field. Based on the matching result of the standard judgment result and the data standard name, the target data standard name corresponding to the data to be benchmarked is determined. Based on the mapping relationship between the data standard and the data standard name and the target data standard name, the target data standard corresponding to the data to be benchmarked is determined from multiple data standards.

[0060] As can be seen, the target data standard name corresponding to the data to be benchmarked, determined by this method, is obtained by matching the standard judgment result output by the standard judgment model with the data standard name output by the standard benchmarking model. The standard judgment result and the data standard name can be matched to determine a target data standard that more accurately corresponds to the data to be benchmarked. Furthermore, this application requires no manual intervention; it automatically performs the benchmarking process on the data to be benchmarked. Thus, this application reduces labor costs and workload while improving the accuracy of the data benchmarking results (i.e., the determined target data standard).

[0061] It should be noted that this application does not limit the entity performing the data standard determination method. For example, the data standard determination method in the embodiments of this application can be applied to data processing devices such as terminal devices or servers. The terminal device can be an electronic device such as a smartphone, computer, or tablet computer. The server can be a standalone server, a cluster server, or a cloud server. This application does not specifically limit the terminal devices or servers mentioned above.

[0062] The following, in conjunction with the accompanying drawings, describes in detail the specific implementation of the data standard determination method and related apparatus in the embodiments of this application.

[0063] See Figure 1 The figure is a flowchart of a method for determining data standards provided in an embodiment of this application, combined with... Figure 1 As shown, it can specifically include:

[0064] S101: Based on the data to be benchmarked, output the standard judgment result corresponding to the data to be benchmarked through the standard judgment model.

[0065] The benchmarking data includes business data related to the target industry and computer system data related to the target industry. Business data refers to the data used and recorded in the various business processes handled daily in the target industry. Computer system data refers to the metadata in the IT systems used by the target industry to process business.

[0066] Taking the insurance industry as an example, business data could include the policy number of the insured, the insurance type code of the insured, and the employee ID responsible for handling the relevant business of the insured; computer system data could include media access control MAC address binding switches, policy storage addresses, etc. This application does not limit the target industry, the business data related to the target industry, or the computer system data related to the target industry. For example, the target industry could also be other industries such as banking or securities.

[0067] The standard judgment result is the result of judging whether the data to be benchmarked exists. In other words, the standard judgment result of the data to be benchmarked may be that the data to be benchmarked exists, or that the data to be benchmarked does not exist.

[0068] The aforementioned standard judgment model is trained based on a first data sample and the corresponding standard judgment labels. The standard judgment labels include labels indicating the existence of data standards and labels indicating the absence of data standards.

[0069] In one possible implementation of this application, the standard judgment model may include a vector transformation layer, a convolutional layer, a pooling layer, and a fully connected layer. Accordingly, the above S101 can be further subdivided into the following steps:

[0070] A1: The vector transformation layer performs vector transformation on the data to be benchmarked to obtain the vector representation of the data to be benchmarked.

[0071] The vector transformation layer is used to map the target data from the semantic space to the vector space. For example, the vector transformation layer can be an embedding layer, but this application does not limit this.

[0072] A2: The vector representation of the data to be benchmarked is convolved by the convolutional layer to obtain the first vector feature matrix of the data to be benchmarked.

[0073] Specifically, a first vector feature matrix is ​​obtained through convolutional layers, which can be used to represent the data to be compared. This first vector feature matrix has a smaller data volume compared to the vector representation of the data to be compared, thus improving the processing speed of the standard judgment model during subsequent processing of the data to be compared. For example, the convolutional layer can be a one-dimensional convolutional layer, which may include three one-dimensional convolutional sliding windows (also called convolutional kernels). This application does not limit the structural parameters of the convolutional layer or the number of sliding windows included in the convolutional layer.

[0074] It should be noted that the number of feature matrices in the first vector is the same as the number of sliding windows included in the convolutional layer.

[0075] A3: The pooling layer is used to perform pooling processing on the first vector feature matrix of the data to be processed to obtain the second vector feature matrix of the data to be benchmarked.

[0076] The pooling layer is used to perform dimensionality reduction, removal of redundant information, feature compression, and selection of more discriminative features on the first vector feature matrix. In other words, the resulting second vector feature matrix is ​​obtained by performing dimensionality reduction, redundancy removal, feature compression, and feature selection on the first vector feature matrix.

[0077] It should be noted that the number of feature matrices in the second vector is the same as the number of feature matrices in the first vector.

[0078] A4: The second vector feature matrix of the data to be benchmarked is normalized through the fully connected layer to obtain the standard judgment result corresponding to the data to be benchmarked.

[0079] After normalization, the probability of each category (i.e., the categories of data standard existence and data standard non-existence) can be obtained, and the category with the higher probability is output as the standard judgment result corresponding to the data to be benchmarked.

[0080] It should be noted that if there are multiple second vector feature matrices, the multiple second vector feature matrices need to be concatenated first through a fully connected layer, and then the concatenated vector feature matrices need to be normalized.

[0081] As can be seen, the above model structure, through vector transformation layer, convolutional layer and pooling layer, can obtain a second vector feature matrix with more obvious representation and smaller data volume corresponding to the data to be benchmarked, and then obtain a more accurate standard judgment result through fully connected layer.

[0082] As an example, see Figure 2 This figure is a model structure diagram of a standard judgment model provided in an embodiment of this application. Combined with... Figure 2As shown, the data to be benchmarked is "customer ID card". The vector transformation layer is an embedding layer, and the convolutional layer includes three one-dimensional convolutional sliding windows. A 1D CNN represents the one-dimensional convolution. First, the embedding layer obtains the text vector corresponding to "customer ID card" (i.e., the vector representation mentioned above). The three 1D CNN sliding windows obtain three feature matrices (i.e., the first vector feature matrix mentioned above). The pooling layer obtains three feature matrices (i.e., the second vector feature matrix mentioned above). Then, the fully connected layer performs concatenation and normalization processing to obtain a feature matrix, and the result is output (i.e., the standard judgment result corresponding to the data to be benchmarked mentioned above).

[0083] In one possible implementation of this application, the aforementioned standard judgment model can be trained through the following steps: obtaining the first data sample and the standard judgment label corresponding to the first data sample; based on the first data sample, outputting the predicted standard judgment result corresponding to the first data sample through a first model to be trained; adjusting the parameters of the first model to be trained based on the difference between the standard judgment label corresponding to the first data sample and the predicted standard judgment result corresponding to the first data sample; and ending the training when the training cutoff condition is met to obtain the standard judgment model. Thus, a standard judgment model capable of determining whether a data standard exists is obtained.

[0084] For example, the first data sample may include business data samples related to the target industry and computer system data samples related to the target industry; this application does not limit this. Specific examples can be found in the above description and will not be repeated here. The training cutoff condition may be that the loss value of the corruption function of the first model to be trained is less than or equal to a loss value threshold; it may also be that the number of training iterations for the first model to be trained reaches a training iteration threshold, etc.; this application does not limit this.

[0085] S102: Based on the data to be benchmarked, output the data standard name corresponding to the data to be benchmarked through the standard benchmarking model.

[0086] The benchmarking model is trained based on a second data sample and its corresponding data standard name labels. These data standard name labels include labels indicating the presence of a data standard and labels indicating the absence of a data standard. For example, labels indicating the presence of a data standard may include Standard 1, Standard 2, Standard 3, etc., or First Standard, Second Standard, Third Standard, etc. This application does not limit this, as long as it can distinguish different data standards. For example, a label indicating the absence of a data standard could be "No Data Standard," indicating that the data to be benchmarked does not have a specific corresponding data standard.

[0087] The data standard represented by the aforementioned name tag with data standards includes the data standard content corresponding to at least one data standard field. One name tag with data standards corresponds to one data standard. Taking the name tag with data standards "Standard 1" as an example, the data standard corresponding to "Standard 1" can include different data standard fields such as business theme, primary category, secondary category, and Chinese name of data standard. For different data standard fields, "Standard 1" has the specific content of the corresponding data standard.

[0088] Based on the above examples, in one possible implementation of this application, the model structure of the standard benchmarking model may include a vector transformation layer, a convolutional layer, a pooling layer, and a fully connected layer. For specific implementations, please refer to the implementation of the model structure of the standard judgment model in the above examples, which will not be repeated here.

[0089] Furthermore, in one possible implementation of this application, the data to be benchmarked can first undergo target preprocessing. The preprocessed data is then input into a standard judgment model to obtain the standard judgment result corresponding to the data to be benchmarked, and the preprocessed data is also input into a standard benchmarking model to obtain the data standard name corresponding to the data to be benchmarked. For example, target preprocessing may include data preprocessing methods such as Chinese word matching, word segmentation, and quantization, which are not limited in this application.

[0090] In one possible implementation of this application, the standard benchmarking model can be trained through the following steps: obtaining the second data sample and the data standard name label corresponding to the second data sample; based on the second data sample, outputting the predicted data standard name corresponding to the second data sample through the second model to be trained; adjusting the parameters of the second model to be trained based on the difference between the data standard name label corresponding to the second data sample and the predicted data standard name corresponding to the second data sample; and ending the training when the training cutoff condition is met to obtain the standard benchmarking model.

[0091] For example, the second data sample may include business data samples related to the target industry and computer system data samples related to the target industry; this application does not limit this. Specific examples can be found in the above description and will not be repeated here. The training cutoff condition may be that the loss value of the corruption function of the second model to be trained is less than or equal to a loss value threshold; it may also be that the number of training iterations for the second model to be trained reaches a training iteration threshold, etc.; this application does not limit this.

[0092] In one possible implementation of this application, the first data sample or the second data sample is obtained through the following steps:

[0093] B1: Obtain multiple initial data samples.

[0094] The initial data samples include business field metadata and computer system field metadata related to the target industry. For example, the initial data samples might be all the data contained in an insurance policy, which may include policy number, user identification number, employee ID, etc.

[0095] B2: From the plurality of initial data samples, a first initial data sample of Chinese words is determined based on the industry thesaurus corresponding to the target industry.

[0096] The industry terminology database includes multiple Chinese terms related to the target industry. For example, the industry terminology database could be an insurance industry terminology database, which includes multiple Chinese terms related to the insurance industry.

[0097] B3: Perform word segmentation on the remaining initial data samples other than the first initial data sample among the plurality of initial data samples to obtain a second initial data sample which consists of English words or abbreviations.

[0098] After matching Chinese words in the initial data sample, the remaining words are English words or abbreviations. The remaining initial data samples are then segmented. For example, segmentation can be performed using characters such as underscores and spaces. This application does not limit the specific method of segmentation.

[0099] B4: Combine the first initial data sample and the second initial data sample to obtain the target industry thesaurus.

[0100] B5: Use multiple words included in the target industry thesaurus as the first data sample or the second data sample.

[0101] In this way, by preprocessing multiple initial data samples in the manner described above, training samples that are more suitable for the first and second models to be trained are obtained, thereby further improving the training effect of the models.

[0102] Furthermore, in an optional embodiment of this application, the target industry thesaurus may also include the frequency of occurrence of each word. As an example, see [link to example]. Figure 3 This figure is a schematic diagram of a target industry thesaurus provided in an embodiment of this application. Combined with... Figure 3 The chart shows multiple words and their respective frequencies. For example, the word "check" (meaning to inspect) appears 516 times. Other words and their frequencies can be found in [link to chart]. Figure 3As shown, it will not be elaborated further here. Among them, the words "file" means document, "index" means index, "dataowner" means data owner, and "idtype" means ID type.

[0103] Furthermore, in one optional embodiment of this application, multiple words included in the target industry thesaurus can be quantified and converted into numbers that can be directly recognized by a standard judgment model or a standard benchmarking model. This further reduces the difficulty of model training and improves the training speed of the model.

[0104] Furthermore, in one optional embodiment of this application, the industry lexicon can be updated periodically to process newly added initial data samples and update the target industry lexicon. This increases the number of training samples for the model, further improving the model training effect.

[0105] S103: Based on the matching result of the standard judgment and the data standard name, determine the target data standard name corresponding to the data to be benchmarked.

[0106] The standard judgment result refers to whether the data to be benchmarked exists. The data standard name includes a name label indicating that a data standard exists and a name label indicating that no data standard exists. Therefore, the standard judgment result and the data standard name may match or not.

[0107] In one possible implementation of this application, S103 can be further subdivided into the following steps:

[0108] C1: If the matching result is that the standard judgment result matches the data standard name, then the data standard name is determined as the target data standard name corresponding to the data to be benchmarked.

[0109] For example, if the standard judgment result indicates that the data to be benchmarked has a data standard, the data standard name of the data to be benchmarked is a name label representing the existence of a data standard; or if the standard judgment result indicates that the data to be benchmarked does not have a data standard, the data standard name of the data to be benchmarked is a name label representing the absence of a data standard. Both cases indicate a match, meaning the outputs of the standard judgment model and the standard benchmarking model are consistent. In such cases, the data standard name can be directly determined as the target data standard name corresponding to the data to be benchmarked, without any correction.

[0110] C2: If the matching result is that the standard judgment result and the data standard name do not match, then based on the first probability corresponding to the standard judgment result and the second probability corresponding to the data standard name, determine the target data standard name corresponding to the data to be benchmarked; the first probability is the probability corresponding to the result output by the standard judgment model; the second probability is the probability corresponding to the result output by the standard benchmarking model.

[0111] For example, if the standard judgment result indicates that the data to be benchmarked has a data standard, and the name of the data standard for the data to be benchmarked is a name label indicating that there is no data standard; or if the standard judgment result indicates that the data to be benchmarked does not have a data standard, and the name of the data standard for the data to be benchmarked is a name label indicating that there is a data standard, both of these situations indicate a mismatch. This also indicates that the outputs of the standard judgment model and the standard benchmarking model are inconsistent, suggesting that the name of the data standard for the data to be benchmarked may be incorrect and needs to be corrected.

[0112] Understandably, the standard judgment model will include the probabilities of belonging to the category of "existing data standard" and "non-existent data standard" for the data to be compared. The standard judgment model will output the category with the higher probability, i.e., the category corresponding to the first probability. Similarly, the standard benchmarking model will include the probabilities of belonging to the category of "name label" represented by "no data standard" and "multiple name labels" represented by "data standard" for the data to be compared. The standard judgment model will output the category with the higher probability, i.e., the category corresponding to the second probability.

[0113] In one possible implementation of this application, C2 can specifically be: if the first probability corresponding to the standard judgment result is less than the second probability corresponding to the data standard name, then the data standard name is determined as the target data standard name corresponding to the data to be benchmarked; if the first probability corresponding to the standard judgment result is greater than or equal to the second probability corresponding to the data standard name, and the standard judgment result indicates that the data to be benchmarked has a data standard, then the data standard name corresponding to the second largest probability in the output result of the standard benchmarking model is determined as the target data standard name corresponding to the data to be benchmarked; if the first probability corresponding to the standard judgment result is greater than or equal to the second probability corresponding to the data standard name, and the standard judgment result indicates that the data to be benchmarked does not have a data standard, then the data standard name characterized as having no data standard name is determined as the target data standard name corresponding to the data to be benchmarked.

[0114] A higher probability indicates that the data to be compared is more likely to belong to this category. Therefore, we first compare the size of the first probability and the second probability, and then determine a more appropriate target data standard name for the data to be compared based on the comparison result.

[0115] For example, suppose the standard judgment result is that there is no data standard for the data to be benchmarked, and its corresponding first probability is 0.7, the data standard name is "Standard 1", and its corresponding second probability is 0.6, then "no data standard" can be determined as the target data standard name corresponding to the data to be benchmarked, indicating that the data to be benchmarked has no corresponding data standard.

[0116] S104: Based on the mapping relationship between data standards and data standard names, and the target data standard name, determine the target data standard corresponding to the data to be benchmarked from multiple data standards.

[0117] As an example, see Table 1, which is a mapping table between data standards and data standard names provided in the embodiments of this application.

[0118] Table 1. Mapping Relationship Between Data Standards and Data Standard Names

[0119]

[0120] As shown in Table 1, the fields are data standard fields, and the names are data standard names. A name tag indicating the presence of a data standard (i.e., Standard 1-Standard 6) corresponds to a data standard that includes at least one data standard field. A name tag indicating the absence of a data standard (i.e., no data standard) has no corresponding data standard content.

[0121] For ease of understanding, combined with Figure 4 The flowchart illustrating a method for determining data standards is provided to specifically illustrate the method for determining data standards provided in this application embodiment. For example... Figure 4 As shown, assuming the benchmarking data consists of Chinese words, the process begins by matching the Chinese words in the benchmarking data based on an industry thesaurus, followed by quantification to obtain the corresponding numbers. These converted numbers are then input into a standard judgment model, which outputs the standard judgment result corresponding to the benchmarking data. Next, the converted numbers are input into a standard benchmarking model, which outputs the data standard name corresponding to the benchmarking data. The standard judgment result and the data standard name are then mutually corrected (i.e., matched and corrected based on the matching result) to determine the final target data standard name. Finally, based on the mapping relationship between the data standard and the data standard name, the target data standard corresponding to the benchmarking data is determined.

[0122] The above are some specific implementations of the data standard determination method provided in the embodiments of this application. Based on this, this application also provides a corresponding apparatus. The data standard determination apparatus provided in the embodiments of this application will be described below from the perspective of functional modularity.

[0123] See Figure 5 The figure is a schematic diagram of a data standard determination device provided in an embodiment of this application. The data standard determination device 500 may include:

[0124] The standard judgment module 510 is used to output a standard judgment result corresponding to the data to be benchmarked based on the data to be benchmarked through a standard judgment model; the standard judgment result is the judgment result of whether the data to be benchmarked exists; the standard judgment model is trained based on a first data sample and the standard judgment label corresponding to the first data sample; the standard judgment label includes a label of data standard existence and a label of data standard non-existence; the data to be benchmarked includes business data related to the target industry and computer system data related to the target industry;

[0125] The standard name output module 520 is used to output the data standard name corresponding to the data to be benchmarked based on the data to be benchmarked, through a standard benchmarking model; the standard benchmarking model is trained based on the second data sample and the data standard name label corresponding to the second data sample; the data standard name label includes name labels representing data standard and name labels representing no data standard; the data standard corresponding to the name label representing data standard includes data standard content corresponding to at least one data standard field.

[0126] The standard name determination module 530 is used to determine the target data standard name corresponding to the data to be benchmarked based on the standard judgment result and the matching result of the data standard name;

[0127] The standard determination module 540 is used to determine the target data standard corresponding to the data to be benchmarked from multiple data standards based on the mapping relationship between data standards and data standard names and the target data standard name.

[0128] As one implementation method, the standard name determination module 530 specifically includes:

[0129] The first standard name determination unit is used to determine the data standard name as the target data standard name corresponding to the data to be benchmarked if the matching result is that the standard judgment result matches the data standard name;

[0130] The second standard name determination unit is used to determine the target data standard name corresponding to the data to be benchmarked based on the first probability corresponding to the standard judgment result and the second probability corresponding to the data standard name if the matching result is that the standard judgment result and the data standard name do not match; the first probability is the probability corresponding to the result output by the standard judgment model; the second probability is the probability corresponding to the result output by the standard benchmarking model.

[0131] As one implementation method, the second standard name determination unit is specifically used for:

[0132] If the first probability corresponding to the standard judgment result is less than the second probability corresponding to the data standard name, then the data standard name is determined as the target data standard name corresponding to the data to be benchmarked.

[0133] If the first probability corresponding to the standard judgment result is greater than or equal to the second probability corresponding to the data standard name, and the standard judgment result indicates that the data to be benchmarked has a data standard, then the data standard name corresponding to the second largest probability in the output result of the standard benchmarking model is determined as the target data standard name corresponding to the data to be benchmarked.

[0134] If the first probability corresponding to the standard judgment result is greater than or equal to the second probability corresponding to the data standard name, and the standard judgment result is that the data to be benchmarked does not have a data standard, then the data standard name characterized as having no data standard name is determined as the target data standard name corresponding to the data to be benchmarked.

[0135] As one implementation, the standard judgment model in the standard judgment module 510 includes a vector transformation layer, a convolutional layer, a pooling layer, and a fully connected layer; correspondingly, the standard judgment module is specifically used for:

[0136] The vector transformation layer performs vector transformation on the data to be benchmarked to obtain the vector representation of the data to be benchmarked.

[0137] The vector representation of the data to be benchmarked is convolved by the convolutional layer to obtain the first vector feature matrix of the data to be benchmarked.

[0138] The pooling layer is used to perform pooling processing on the first vector feature matrix of the data to be processed to obtain the second vector feature matrix of the data to be benchmarked.

[0139] The second vector feature matrix of the data to be benchmarked is normalized by the fully connected layer to obtain the standard judgment result corresponding to the data to be benchmarked.

[0140] As one implementation method, the standard judgment model in the standard judgment module 510 is obtained through training the following units:

[0141] The first acquisition unit is used to acquire the first data sample and the standard judgment label corresponding to the first data sample;

[0142] The first output unit is used to output the prediction standard judgment result corresponding to the first data sample based on the first data sample and through the first model to be trained.

[0143] The first parameter adjustment unit is used to adjust the parameters of the first model to be trained based on the difference between the standard judgment label corresponding to the first data sample and the prediction standard judgment result corresponding to the first data sample.

[0144] The first model acquisition unit is used to end training and obtain the standard judgment model when the training cutoff condition is met.

[0145] As one implementation method, the standard benchmarking model in the standard name output module 520 is obtained through training the following units:

[0146] The second acquisition unit is used to acquire the second data sample and the data standard name label corresponding to the second data sample;

[0147] The second output unit is used to output the predicted data standard name corresponding to the second data sample through the second training model based on the second data sample.

[0148] The second parameter adjustment unit is used to adjust the parameters of the second model to be trained based on the difference between the data standard name label corresponding to the second data sample and the predicted data standard name corresponding to the second data sample.

[0149] The second model acquisition unit is used to end the training and obtain the standard benchmark model when the training cutoff condition is met.

[0150] As one implementation, the first data sample in the first acquisition unit or the second data sample in the second acquisition unit is obtained through the following unit:

[0151] The first sample acquisition unit is used to acquire multiple initial data samples; the multiple initial data samples include business field metadata related to the target industry and computer system field metadata related to the target industry;

[0152] The sample determination unit is used to determine a first initial data sample that is a Chinese word from the plurality of initial data samples based on the industry thesaurus corresponding to the target industry; the industry thesaurus includes a plurality of Chinese words related to the target industry;

[0153] The word segmentation unit is used to perform word segmentation on the remaining initial data samples other than the first initial data sample among the plurality of initial data samples to obtain a second initial data sample which is English words or abbreviations.

[0154] A sample combination unit is used to combine the first initial data sample and the second initial data sample to obtain a target industry thesaurus;

[0155] The second sample acquisition unit is used to use multiple words included in the target industry thesaurus as the first data sample or the second data sample.

[0156] This application also provides a device for determining the corresponding data standard and a computer-readable storage medium for implementing the solution provided in this application.

[0157] The data standard determination device includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to enable the data standard determination device to perform the data standard determination method described in any embodiment of this application.

[0158] The computer-readable storage medium stores a computer program. When the computer program is run, the data standard determination device running the computer program implements the data standard determination method described in any embodiment of this application.

[0159] In the embodiments of this application, the terms "first" and "second" (if they exist) are used only as name identifiers and do not represent the order of first and second.

[0160] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that all or part of the steps in the methods of the above embodiments can be implemented by means of software plus a general-purpose hardware platform. Based on this understanding, the technical solution of this application can be embodied in the form of a software product. This computer software product can be stored in a readable storage medium, such as a read-only memory (ROM) / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, a server, or a network communication device such as a router) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0161] It should be noted that the various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for the device embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiments. The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separate. The components indicated as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment solution according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0162] The above description is merely one specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for determining data standards, characterized in that, The method includes: Based on the data to be benchmarked, a standard judgment model is used to output the standard judgment result corresponding to the data to be benchmarked; the standard judgment result is the judgment result of whether the data to be benchmarked exists; the standard judgment model is trained based on a first data sample and the standard judgment label corresponding to the first data sample; the standard judgment label includes a label indicating the existence of a data standard and a label indicating the non-existence of a data standard; the data to be benchmarked includes business data related to the target industry and computer system data related to the target industry. Based on the data to be benchmarked, the data standard name corresponding to the data to be benchmarked is output through the standard benchmarking model; the standard benchmarking model is trained based on the second data sample and the data standard name label corresponding to the second data sample; the data standard name label includes name labels representing data standards and name labels representing no data standards; the data standard corresponding to the name label representing data standards includes data standard content corresponding to at least one data standard field. Based on the matching result of whether the standard judgment result and the data standard name are consistent, the target data standard name corresponding to the data to be benchmarked is determined; Based on the mapping relationship between data standards and data standard names, as well as the target data standard name, the target data standard corresponding to the data to be benchmarked is determined from multiple data standards.

2. The method according to claim 1, characterized in that, The determination of the target data standard name corresponding to the data to be benchmarked, based on the matching result of the standard judgment result and the data standard name, specifically includes: If the matching result is a match between the standard judgment result and the data standard name, then the data standard name is determined as the target data standard name corresponding to the data to be benchmarked. If the matching result is that the standard judgment result and the data standard name do not match, then the target data standard name corresponding to the data to be benchmarked is determined based on the first probability corresponding to the standard judgment result and the second probability corresponding to the data standard name; the first probability is the probability corresponding to the result output by the standard judgment model; the second probability is the probability corresponding to the result output by the standard benchmarking model.

3. The method according to claim 2, characterized in that, The determination of the target data standard name corresponding to the data to be benchmarked, based on the first probability corresponding to the standard judgment result and the second probability corresponding to the data standard name, specifically includes: If the first probability corresponding to the standard judgment result is less than the second probability corresponding to the data standard name, then the data standard name is determined as the target data standard name corresponding to the data to be benchmarked. If the first probability corresponding to the standard judgment result is greater than or equal to the second probability corresponding to the data standard name, and the standard judgment result indicates that the data to be benchmarked has a data standard, then the data standard name corresponding to the second largest probability in the output result of the standard benchmarking model is determined as the target data standard name corresponding to the data to be benchmarked. If the first probability corresponding to the standard judgment result is greater than or equal to the second probability corresponding to the data standard name, and the standard judgment result is that the data to be benchmarked does not have a data standard, then the data standard name characterized as having no data standard name is determined as the target data standard name corresponding to the data to be benchmarked.

4. The method according to any one of claims 1-3, characterized in that, The standard judgment model includes a vector transformation layer, a convolutional layer, a pooling layer, and a fully connected layer; the step of outputting the standard judgment result corresponding to the data to be benchmarked through the standard judgment model specifically includes: The vector transformation layer performs vector transformation on the data to be benchmarked to obtain the vector representation of the data to be benchmarked. The vector representation of the data to be benchmarked is convolved by the convolutional layer to obtain the first vector feature matrix of the data to be benchmarked. The pooling layer is used to perform pooling processing on the first vector feature matrix of the data to be benchmarked, so as to obtain the second vector feature matrix of the data to be benchmarked. The second vector feature matrix of the data to be benchmarked is normalized by the fully connected layer to obtain the standard judgment result corresponding to the data to be benchmarked.

5. The method according to claim 1, characterized in that, The standard judgment model is obtained through training the following steps: Obtain the first data sample and the standard judgment label corresponding to the first data sample; Based on the first data sample, the prediction standard judgment result corresponding to the first data sample is output through the first model to be trained; Based on the difference between the standard judgment label corresponding to the first data sample and the prediction standard judgment result corresponding to the first data sample, the parameters of the first model to be trained are adjusted. When the training cutoff condition is met, training ends and the standard judgment model is obtained.

6. The method according to claim 1, characterized in that, The standard benchmarking model is obtained through the following steps: Obtain the second data sample and the corresponding data standard name label; Based on the second data sample, the predicted data standard name corresponding to the second data sample is output by the second model to be trained; Based on the difference between the standard data name label corresponding to the second data sample and the predicted standard data name corresponding to the second data sample, the parameters of the second model to be trained are adjusted. When the training cutoff condition is met, the training ends and the standard benchmark model is obtained.

7. The method according to claim 5 or 6, characterized in that, The first data sample or the second data sample is obtained through the following steps: Obtain multiple initial data samples; the multiple initial data samples include business field metadata related to the target industry and computer system field metadata related to the target industry; From the plurality of initial data samples, a first initial data sample of Chinese words is determined based on the industry thesaurus corresponding to the target industry; the industry thesaurus includes a plurality of Chinese words related to the target industry; The remaining initial data samples, excluding the first initial data sample, are segmented into words to obtain a second initial data sample consisting of English words or abbreviations. The first initial data sample and the second initial data sample are combined to obtain the target industry thesaurus; The target industry thesaurus includes multiple words as the first data sample or the second data sample.

8. A data standard determination device, characterized in that, The device includes: A standard judgment module is used to output a standard judgment result corresponding to the data to be benchmarked based on the data to be benchmarked, through a standard judgment model. The standard judgment result is the judgment result of whether the data to be benchmarked exists. The standard judgment model is trained based on a first data sample and the standard judgment label corresponding to the first data sample. The standard judgment label includes a label indicating the existence of a data standard and a label indicating the non-existence of a data standard. The data to be benchmarked includes business data related to the target industry and computer system data related to the target industry. The standard name output module is used to output the data standard name corresponding to the data to be benchmarked based on the data to be benchmarked, through a standard benchmarking model; the standard benchmarking model is trained based on the second data sample and the data standard name label corresponding to the second data sample; the data standard name label includes name labels representing data standard and name labels representing no data standard; the data standard corresponding to the name label representing data standard includes data standard content corresponding to at least one data standard field. The standard name determination module is used to determine the target data standard name corresponding to the data to be benchmarked based on the matching result of whether the standard judgment result and the data standard name are consistent. The standard determination module is used to determine the target data standard corresponding to the data to be benchmarked from multiple data standards based on the mapping relationship between data standards and data standard names and the target data standard name.

9. A data standard determination device, characterized in that, The device includes a memory and a processor: The memory is used to store computer programs and transmit the computer programs to the processor; The processor is configured to execute the computer program to cause the device to perform the steps of the data standard determination method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, and when the computer program is run, the device running the computer program implements the steps of the method for determining the data standard as described in any one of claims 1-7.

Images