Data model field mapping method and device, storage medium and processor

Abstract

The invention discloses a data model field mapping method and device, a storage medium and a processor. According to the scheme, metadata of a source model and metadata of a target model are obtained; constructing a candidate field mapping set based on the metadata of the source model and the metadata of the target model; traversing each pair of candidate field mappings in the candidate field mapping set, calculating a comprehensive similarity value of each pair of candidate field mappings in the candidate field mapping set, and determining a mapping rule corresponding to each pair of candidate field mappings based on the comprehensive similarity value of each pair of candidate field mappings, and converting each pair of candidate field mapping from the source model to the target model based on the determined mapping rule. Compared with the problems of low efficiency, high error rate and the like of a manual maintenance mode in the prior art for a large number of fields and complex data structures, the method has obvious advantages.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data model field mapping method, apparatus, storage medium and processor. Background Technology

[0002] In the process of cross-disciplinary and cross-domain network data acquisition and processing, data model field mapping is an essential part. It integrates differentiated data from multiple heterogeneous data sources into a unified target model in order to achieve data interoperability and sharing.

[0003] In the process of data model field mapping, it is necessary to address issues such as differences in data structure, format, and semantics between the data source model and the target model. Data structure differences include differences in model definitions between different data sources and differences between the definitions of the data source model and the target model, such as field names, data types, and field granularity. Data format differences include technical format differences, differences in data value levels, differences in numerical formats, and differences in units of measurement. Data semantic differences involve differences in field names and their meanings; for example, the same business concept may use different field names in different models, or different business concepts may use the same field name in different models. Existing data model field mapping methods rely on manual maintenance of field mapping tables, which suffers from low efficiency and high error rates when dealing with a large number of fields and complex data structures.

[0004] While ensuring the accuracy of data model field mapping, how to improve the efficiency of data model field mapping is a technical problem that urgently needs to be solved. Summary of the Invention

[0005] To address the aforementioned issues, this application provides a data model field mapping method, apparatus, storage medium, and processor, with the aim of improving the efficiency of data model field mapping while ensuring its accuracy.

[0006] The embodiments of this application disclose the following technical solutions:

[0007] The first aspect of this application provides a data model field mapping method, the method comprising:

[0008] Obtain the metadata of the source model and the target model;

[0009] Based on the metadata of the source model and the metadata of the target model, a candidate field mapping set is constructed;

[0010] Traverse each pair of candidate field mappings in the candidate field mapping set, calculate the comprehensive similarity value of each pair of candidate field mappings in the candidate field mapping set, and determine the mapping rule corresponding to each pair of candidate field mappings based on the comprehensive similarity value of each pair of candidate field mappings, so that each pair of candidate field mappings can be transformed from the source model to the target model based on the determined mapping rule.

[0011] Optionally, calculating the comprehensive similarity value for each pair of candidate field mappings in the candidate field mapping set includes:

[0012] The formal similarity value of each pair of candidate field mappings in the candidate field mapping set is calculated using the edit distance algorithm;

[0013] Using the BERT model, the semantic similarity value of each pair of candidate field mappings in the candidate field mapping set is calculated;

[0014] The formal similarity value and semantic similarity value of each pair of candidate field mappings in the candidate field mapping set are weighted and summed to obtain the comprehensive similarity value of each pair of candidate field mappings in the candidate field mapping set.

[0015] Optionally, determining the mapping rule corresponding to each pair of candidate field mappings based on the comprehensive similarity value of each pair of candidate field mappings includes:

[0016] Determine whether the comprehensive similarity value of each pair of candidate field mappings meets the preset conditions to obtain the determination result;

[0017] If the judgment result is yes, then the mapping rule corresponding to the current candidate field mapping is generated;

[0018] If the judgment result is negative, the current candidate field mapping is removed from the candidate field mapping set.

[0019] Optionally, constructing a candidate field mapping set based on the metadata of the source model and the metadata of the target model includes:

[0020] Based on the metadata of the source model and the metadata of the target model, extract the field information contained in the source model and the field information contained in the target model;

[0021] A one-to-one mapping relationship is established between each field in the source model and each field in the target model to generate multiple pairs of candidate field mappings, and a candidate field mapping set is formed based on the multiple pairs of candidate mappings.

[0022] Optionally, after obtaining the metadata of the source model and the target model, the method further includes:

[0023] Data cleaning operations are performed on the metadata of the source model and the metadata of the target model; the data cleaning operations include unifying the naming style of English names, unifying the capitalization of letters, removing spaces, standardizing field types, removing stop words, and removing modal particles;

[0024] Based on the metadata of the source model and the target model after data cleaning, a field information string is constructed according to a preset method.

[0025] A second aspect of this application provides a data model field mapping apparatus, the apparatus comprising:

[0026] The data model metadata acquisition module is used to acquire the metadata of the source model and the target model.

[0027] A candidate field mapping set construction module is used to construct a candidate field mapping set based on the metadata of the source model and the metadata of the target model;

[0028] The data model field mapping module is used to traverse each pair of candidate field mappings in the candidate field mapping set, calculate the comprehensive similarity value of each pair of candidate field mappings in the candidate field mapping set, and determine the mapping rule corresponding to each pair of candidate field mappings based on the comprehensive similarity value of each pair of candidate field mappings, so that each pair of candidate field mappings can be converted from the source model to the target model based on the determined mapping rule.

[0029] Optionally, the data model field mapping module is specifically used for:

[0030] The formal similarity value of each pair of candidate field mappings in the candidate field mapping set is calculated using the edit distance algorithm;

[0031] Using the BERT model, the semantic similarity value of each pair of candidate field mappings in the candidate field mapping set is calculated;

[0032] The formal similarity value and semantic similarity value of each pair of candidate field mappings in the candidate field mapping set are weighted and summed to obtain the comprehensive similarity value of each pair of candidate field mappings in the candidate field mapping set.

[0033] Optionally, the data model field mapping module is specifically used for:

[0034] Determine whether the comprehensive similarity value of each pair of candidate field mappings meets the preset conditions to obtain the determination result;

[0035] If the judgment result is yes, then the mapping rule corresponding to the current candidate field mapping is generated;

[0036] If the judgment result is negative, the current candidate field mapping is removed from the candidate field mapping set.

[0037] A third aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements a data model field mapping method as provided in any implementation of the first aspect.

[0038] The fourth aspect of this application provides a processor for running a computer program that, when running, executes a data model field mapping method as provided in any implementation of the first aspect.

[0039] Compared with the prior art, this application has the following beneficial effects:

[0040] This application provides a data model field mapping method. Based on the metadata of the source model and the target model, a candidate field mapping set is constructed. This set comprehensively covers potential field mapping relationships, providing a foundation for subsequent intelligent selection of qualified field mappings and generation of corresponding mapping rules. Each pair of candidate field mappings in the set is traversed, and a comprehensive similarity value is calculated for each pair. Based on this comprehensive similarity value, a mapping rule is determined for each pair. This allows each pair to be converted from the source model to the target model based on the determined mapping rule. By introducing the calculation of the comprehensive similarity value, the mapping relationship between field pairs can be intelligently determined, avoiding errors that may occur in manual mapping and improving the accuracy and consistency of the mapping results. Moreover, this automated generation of mapping rules significantly reduces the need for manual intervention and improves the efficiency of data model field mapping. Especially when thousands of data models and fields are involved, automated data model field mapping can efficiently complete the mapping task, greatly improving data processing efficiency. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 A flowchart illustrating a data model field mapping method provided in this application embodiment;

[0043] Figure 2A flowchart illustrating yet another data model field mapping method provided in this application embodiment;

[0044] Figure 3 This is a schematic diagram of the structure of a data model field mapping device provided in an embodiment of this application. Detailed Implementation

[0045] As described earlier, existing data model field mapping methods rely on manual maintenance of field mapping tables. This manual maintenance method suffers from problems such as low efficiency and high error rate when faced with a large number of fields and complex data structures.

[0046] In view of the above problems, the inventors, after research, proposed a data model field mapping method, device, storage medium, and processor, which obtains the metadata of the source model and the metadata of the target model; constructs a candidate field mapping set based on the metadata of the source model and the metadata of the target model; traverses each pair of candidate field mappings in the candidate field mapping set, calculates the comprehensive similarity value of each pair of candidate field mappings in the candidate field mapping set, and determines the mapping rule corresponding to each pair of candidate field mappings based on the comprehensive similarity value of each pair of candidate field mappings, so that each pair of candidate field mappings can perform the conversion from the source model to the target model based on the determined mapping rule.

[0047] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0048] See Figure 1 This figure is a flowchart of a data model field mapping method provided in an embodiment of this application. Figure 1 As shown, the method includes the following steps:

[0049] S101. Obtain the metadata of the source model and the target model.

[0050] Model metadata refers to data related to the model definition, including the model's Chinese and English names, field names in Chinese and English, and field types.

[0051] S102. Based on the metadata of the source model and the metadata of the target model, construct a candidate field mapping set.

[0052] Based on the metadata of the source model and the target model, each field in the source model is paired with each field in the target model to generate all possible field pairing combinations, forming a candidate field mapping set.

[0053] By constructing a candidate field mapping set, all potential field mapping relationships are fully covered, providing a foundation for subsequent intelligent filtering of field mappings that meet the conditions and generating corresponding mapping rules.

[0054] S103. Traverse each pair of candidate field mappings in the candidate field mapping set, calculate the comprehensive similarity value of each pair of candidate field mappings in the candidate field mapping set, and determine the mapping rule corresponding to each pair of candidate field mappings based on the comprehensive similarity value of each pair of candidate field mappings, so that each pair of candidate field mappings can be converted from the source model to the target model based on the determined mapping rule.

[0055] By introducing a comprehensive similarity calculation, the matching relationship between field pairs can be intelligently determined, avoiding errors that may occur in manual mapping and improving the accuracy and consistency of the mapping results. Moreover, the generation of this automated mapping rule significantly reduces the need for manual intervention and accelerates data processing. Especially when dealing with thousands of data models and fields, automated data model field mapping can efficiently complete the mapping task, greatly improving the overall efficiency of data processing.

[0056] This application provides a data model field mapping method that constructs a candidate field mapping set based on the metadata of the source model and the target model. This set comprehensively covers potential field mapping relationships, providing a foundation for subsequent intelligent selection of qualified field mappings and generation of corresponding mapping rules. Each pair of candidate field mappings in the set is traversed, and a comprehensive similarity value is calculated. Based on this value, a mapping rule is determined for each pair, enabling the conversion from the source model to the target model. By introducing the calculation of the comprehensive similarity value, the mapping relationship between field pairs can be intelligently determined, avoiding errors that may occur in manual mapping and improving the accuracy and consistency of the mapping results. Furthermore, this automated generation of mapping rules significantly reduces the need for manual intervention and improves the efficiency of data model field mapping. Especially when dealing with thousands of data models and fields, automated data model field mapping can efficiently complete the mapping task, greatly improving data processing efficiency.

[0057] Based on the above embodiments, in order to further improve the data model field mapping method, a step of preprocessing the metadata of the source model and the target model is added.

[0058] See Figure 2 This figure is a flowchart of another data model field mapping method provided in an embodiment of this application. Figure 2 As shown, the method includes the following steps:

[0059] S201. Obtain the metadata of the source model and the target model.

[0060] Model metadata refers to data related to the model definition, including the model's Chinese and English names, field names in Chinese and English, and field types.

[0061] S202. Preprocess the metadata of the source model and the metadata of the target model.

[0062] In one feasible implementation:

[0063] Data cleaning operations are performed on the metadata of the source model and the metadata of the target model.

[0064] The data cleaning process includes standardizing the naming style of English names, standardizing letter case, removing spaces, standardizing field types, removing stop words, and removing modal particles.

[0065] Based on the metadata of the source model and the target model after data cleaning, a field information string is constructed according to a preset method.

[0066] The specific data cleaning steps are as follows:

[0067] To standardize the naming style of English names, the English names of models and fields should be converted to underscores. For example, EutranCellFDD should be converted to Eutran_Cell_FDD, and userLabel should be converted to user_Label.

[0068] Unify letter case by converting all English letters in the model definition to lowercase. For example, convert Eutran_Cell_FDD to Eutran_cell_fdd and user_Label to user_label.

[0069] Remove spaces, specifically the spaces before and after the English names of the model and fields. Replace spaces in the middle of the names with underscores.

[0070] Standardize the field types, and uniformly convert the field types in the model definition, such as varchar, String, character, etc., into standard type names, such as string.

[0071] Remove stop words or modal particles. Especially for the Chinese names of models and fields, remove common words that are not important for semantics, such as "de", "shi", etc.

[0072] Exemplarily, the metadata of the source model is shown in Table 1.

[0073] Table 1

[0074]

[0075]

[0076] After performing data cleaning operations on the metadata of the source model, it is shown in Table 2.

[0077] Table 2

[0078]

[0079] Perform data cleaning operations on the metadata of the source model and the target model to ensure the consistency of the input information for similarity calculation, and improve the accuracy and consistency of the field mapping between the source model and the target model.

[0080] The preset method refers to the format of the field information string preset in advance, for example:

[0081] <Model type><Chinese model name><English model name><Chinese field name><English field name><Field type>;

[0082] Process the metadata of the source model and the target model after data cleaning based on the preset method to construct a unified field representation form and include the context information of the model.

[0083] Exemplarily, the constructed field information string is as follows:

[0084] # Source model field information string (sourceFieldStr):

[0085] source 4G cell eutran_cell_fdd user-friendly name user_label string;

[0086] # Target model field information string (targetFieldStr):

[0087] target 4G cell eutran_cell cell name user_label string;

[0088] By constructing a string of field information based on the model definition metadata, conflicts caused by field duplication are avoided in the subsequent similarity value calculation process, and sufficient contextual information is provided for the calculation.

[0089] S203. Based on the preprocessed metadata of the source model and the metadata of the target model, construct a candidate field mapping set.

[0090] In one feasible implementation:

[0091] Based on the metadata of the source model and the metadata of the target model, extract the field information contained in the source model and the field information contained in the target model.

[0092] A one-to-one mapping relationship is established between each field in the source model and each field in the target model to generate multiple pairs of candidate field mappings, and a candidate field mapping set is formed based on the multiple pairs of candidate mappings.

[0093] For example, given the known mapping relationships at the model granularity level, all potentially relevant fields in the source model EutranCellFDD and the target model eutran_cell are identified. These fields typically include cell name, user-friendly name, and eNodeB identifier. The relevant fields in the source and target models are then permuted and combined to generate a candidate field mapping set. Specifically, each field in the source model is paired with each field in the target model to generate all possible field pairings, which serve as the candidate field mapping set. Taking the source model EutranCellFDD and the target model eutran_cell as an example, the constructed candidate field mapping set is as follows:

[0094]

[0095] In the example above, there are four pairs of candidate field mappings, each explicitly indicating a potential correspondence between the source and target fields. Specifically, the first pair indicates that the "user-friendly name" field in the source model may correspond to the "cell name" field in the target model. The second pair indicates that the "user-friendly name" field in the source model can be mapped to the "identifier of the eNodeB" field in the target model. The third pair indicates that the "eNodeB identifier" field in the source model can be mapped to the "cell name" field in the target model. The fourth pair indicates that the "eNodeB identifier" field in the source model can be mapped to the "identifier of the eNodeB" field in the target model.

[0096] By constructing a candidate field mapping set, all potential field mapping relationships are fully covered, providing a foundation for subsequent intelligent filtering of field mappings that meet the conditions and generating corresponding mapping rules.

[0097] S204. Traverse each pair of candidate field mappings in the candidate field mapping set, calculate the comprehensive similarity value of each pair of candidate field mappings in the candidate field mapping set, and determine the mapping rule corresponding to each pair of candidate field mappings based on the comprehensive similarity value of each pair of candidate field mappings, so that each pair of candidate field mappings can be converted from the source model to the target model based on the determined mapping rule.

[0098] In one feasible implementation, calculating the comprehensive similarity value for each pair of candidate field mappings in the candidate field mapping set includes:

[0099] The formal similarity value of each pair of candidate field mappings in the candidate field mapping set is calculated using the edit distance algorithm;

[0100] Using the BERT model, the semantic similarity value of each pair of candidate field mappings in the candidate field mapping set is calculated;

[0101] The formal similarity value and semantic similarity value of each pair of candidate field mappings in the candidate field mapping set are weighted and summed to obtain the comprehensive similarity value of each pair of candidate field mappings in the candidate field mapping set.

[0102] For example, for each pair of candidate mappings, the edit distance algorithm is used to calculate the edit distance between the source model information string `sourceFieldStr` and the target model information string `targetFieldStr`, using the edit distance between the two strings to measure their formal similarity. The specific calculation formula is as follows:

[0103]

[0104] Here, levenshteinDistance(sourceModelStr, targetModelStr) represents the edit distance between the source model information string sourceFieldStr and the target model information string targetFieldStr; len(sourceModelStr) represents the length of the source model information string; len(targetModelStr) represents the length of the target model information string; formalSimilarity represents the similarity value between the forms of the source model field and the target model field. The value ranges from 0 to 1, and the closer it is to 1, the more similar the two field names are in form.

[0105] This paper utilizes the text similarity calculation capability of the BERT model to measure the semantic similarity of candidate field mapping pairs between the source and target models. The specific implementation includes fine-tuning the BERT model and the calculation of semantic similarity values.

[0106] The BERT model fine-tuning process consists of the following seven steps:

[0107] S11, Load the pre-trained model.

[0108] Load a publicly available BERT pre-trained model, such as bert-base-uncased weights. This process allows you to initialize the model's general language representation capabilities for a specific task.

[0109] S12, Prepare for fine-tuning the data.

[0110] Data preparation and preprocessing are fundamental to fine-tuning the BERT model, and mainly include the following:

[0111] Source and target model fields are organized. By organizing the field information of the source and target models, a standardized dataset is constructed, including model name, field name, field type, etc.

[0112] Similarity labeling: In fine-tuning data, each pair of source model fields and target model fields has a similarity label. This label is a continuous value that represents the similarity between the source model fields and the target model fields. The value range is [0,1]. A value of 0 indicates that they are completely different, and a value of 1 indicates that they are completely the same.

[0113] Preprocessing of fine-tuning data includes cleaning and standardizing the field information of the source and target models to ensure that the data input meets the requirements of the BERT model and optimizes the fine-tuning effect.

[0114] S13. Define the BERT model type, using BertForSequenceClassification from the pre-trained model, and set the num_labels parameter to 1 to indicate a regression task.

[0115] S14. Set training parameters. The hyperparameter settings during training are crucial for fine-tuning. Commonly used training parameters include learning rate, batch size, and number of training epochs.

[0116] S15. Fine-tune the BERT model using the prepared fine-tuning data. Adjust the weights of the BERT model by optimizing the loss function.

[0117] S16. Evaluate the model by using the fine-tuned BERT model to predict the input test data and convert the model's output to the range [0,1] using the Sigmoid function to represent the semantic similarity between fields.

[0118] The error between the predicted similarity value and the actual similarity label value is calculated, and the accuracy of the BERT model is evaluated by this error.

[0119] S17. Save the BERT model. After the BERT model has been fine-tuned and passed the evaluation, save the fine-tuned model so that it can be loaded and used in practical applications.

[0120] Calculating the semantic similarity between fields in the source and target models using a fine-tuned BERT model mainly involves four steps: loading the fine-tuned BERT model, encoding the input information, predicting similarity, and normalizing the prediction results.

[0121] S21. Loading the fine-tuned BERT model: Load the fine-tuned BERT model and word segmenter.

[0122] S22. Input information encoding: Transform the information strings of the source model and the target model into a format acceptable to the BERT model. For semantic similarity calculation, the two field information strings (source model fields and target model fields) need to be encoded into their corresponding input formats and processed by a tokenizer.

[0123] S23. Similarity Prediction: After loading the fine-tuned BERT model, the next step is model evaluation, where predicted similarity values ​​are calculated. In this process, the BERT model predicts semantic similarity values ​​based on the input field information.

[0124] S24. Normalization of Prediction Results: The output of the BERT model is usually a real value, which may exceed the range [0,1]. To maintain consistency in similarity label values, the Sigmoid activation function is used to normalize the prediction results to the range [0,1]. This helps to ensure the consistency of the results and also ensures that the output of the BERT model better meets actual business needs.

[0125] For example, the formula for calculating the overall similarity value between the source model fields and the target model fields is as follows:

[0126] comprehensipeSimilarity

[0127] =α.formalSimilarity+(1-α)·semamticSimilarity;

[0128] Here, `formalSimilarity` represents the formal similarity value of the field mapping pairs between the source and target models. `semanticSimilarity` represents the semantic similarity value of the field mapping pairs between the source and target models. `α` is a weight parameter used to control the balance between formal and semantic similarity, with a value range of [0,1]. By adjusting the value of the weight parameter `α`, different model difference scenarios can be used. For example, if the field names of the source and target models are relatively similar, the value of `α` can be increased to make the overall similarity value primarily based on similarity. If the field names of the source and target models are significantly different, the value of `α` can be decreased to make the overall similarity primarily based on semantic similarity. `comprehensiveSimilarity` represents the comprehensive similarity value, used to measure the overall similarity between the source and target fields.

[0129] The model mapping rules are automatically established by measuring the comprehensive similarity between the source model fields and the target model fields.

[0130] In one feasible implementation, based on the comprehensive similarity value of each pair of candidate field mappings, the mapping rule corresponding to each pair of candidate field mappings is determined, including:

[0131] Determine whether the comprehensive similarity value of each pair of candidate field mappings meets the preset conditions to obtain the determination result;

[0132] If the judgment result is yes, then the mapping rule corresponding to the current candidate field mapping is generated;

[0133] If the judgment result is negative, the current candidate field mapping is removed from the candidate field mapping set.

[0134] The preset condition refers to the overall similarity value of the candidate field mapping pair being greater than or equal to the preset threshold.

[0135] If the overall similarity value of the candidate field mapping pairs between the source model and the target model is greater than or equal to a preset threshold, it is considered a valid mapping, and a mapping relationship between the fields of the source model and the target model is established, generating the corresponding preset rules. If the overall similarity is less than the preset threshold, the candidate field mapping is discarded.

[0136] The strictness of the generated mapping rules is determined by setting different preset thresholds. The choice of preset threshold can vary depending on the application scenario. If the business requires high mapping accuracy, a higher preset threshold can be set, such as 0.9 or higher. This ensures that only field pairs with a high overall similarity value are mapped, guaranteeing high-quality mapping relationships, but some potential matches may be missed. To generate a wider range of mapping rules, a lower preset threshold can be set, such as 0.7 or lower. This allows even field pairs with low similarity to be mapped, generating more mapping rules, but may introduce some less accurate matches.

[0137] This flexible threshold adjustment allows for compatibility with different application scenarios, reduces the need for manual intervention, and improves the overall data processing quality.

[0138] The calculation of the comprehensive similarity value between the source and target models combines formal and semantic similarity, providing strong support for establishing accurate field mapping relationships. The introduced weight parameter α can adjust the balance between formal and semantic similarity according to different scenarios, enabling the BERT model to achieve optimal results in scenarios with different types of model differences. By selecting field pairs in this way and generating corresponding mapping rules, the need for manual intervention can be significantly reduced, improving the efficiency of data model field mapping.

[0139] This application provides a data model field mapping method that constructs a candidate field mapping set based on the metadata of the source model and the target model. This set comprehensively covers potential field mapping relationships, providing a foundation for subsequent intelligent selection of qualified field mappings and generation of corresponding mapping rules. Each pair of candidate field mappings in the set is traversed, and a comprehensive similarity value is calculated. Based on this value, a mapping rule is determined for each pair, enabling the conversion from the source model to the target model. By introducing the calculation of the comprehensive similarity value, the mapping relationship between field pairs can be intelligently determined, avoiding errors that may occur in manual mapping and improving the accuracy and consistency of the mapping results. Furthermore, this automated generation of mapping rules significantly reduces the need for manual intervention and improves the efficiency of data model field mapping. Especially when dealing with thousands of data models and fields, automated data model field mapping can efficiently complete the mapping task, greatly improving data processing efficiency.

[0140] Based on the data model field mapping method described in the preceding embodiments, this application also provides a data model field mapping apparatus. Figure 3This is a schematic diagram of the device. Figure 3 As shown, the data model field mapping device includes:

[0141] The data model metadata acquisition module 301 is used to acquire the metadata of the source model and the metadata of the target model.

[0142] The candidate field mapping set construction module 302 is used to construct a candidate field mapping set based on the metadata of the source model and the metadata of the target model.

[0143] The data model field mapping module 303 is used to traverse each pair of candidate field mappings in the candidate field mapping set, calculate the comprehensive similarity value of each pair of candidate field mappings in the candidate field mapping set, and determine the mapping rule corresponding to each pair of candidate field mappings based on the comprehensive similarity value of each pair of candidate field mappings, so that each pair of candidate field mappings can be converted from the source model to the target model based on the determined mapping rule.

[0144] Optionally, the data model field mapping module is specifically used for:

[0145] The formal similarity value of each pair of candidate field mappings in the candidate field mapping set is calculated using the edit distance algorithm;

[0146] Using the BERT model, the semantic similarity value of each pair of candidate field mappings in the candidate field mapping set is calculated;

[0147] The formal similarity value and semantic similarity value of each pair of candidate field mappings in the candidate field mapping set are weighted and summed to obtain the comprehensive similarity value of each pair of candidate field mappings in the candidate field mapping set.

[0148] Optionally, the data model field mapping module is specifically used for:

[0149] Determine whether the comprehensive similarity value of each pair of candidate field mappings meets the preset conditions to obtain the determination result;

[0150] If the judgment result is yes, then the mapping rule corresponding to the current candidate field mapping is generated;

[0151] If the judgment result is negative, the current candidate field mapping is removed from the candidate field mapping set.

[0152] Optionally, the candidate field mapping set construction module is specifically used for:

[0153] Based on the metadata of the source model and the metadata of the target model, extract the field information contained in the source model and the field information contained in the target model;

[0154] A one-to-one mapping relationship is established between each field in the source model and each field in the target model to generate multiple pairs of candidate field mappings, and a candidate field mapping set is formed based on the multiple pairs of candidate mappings.

[0155] Optionally, the apparatus further includes: a preprocessing module;

[0156] The preprocessing module is specifically used to perform data cleaning operations on the metadata of the source model and the metadata of the target model; the data cleaning operations include unifying the naming style of English names, unifying the capitalization of letters, removing spaces, standardizing field types, removing stop words, and removing modal particles.

[0157] Based on the metadata of the source model and the target model after data cleaning, a field information string is constructed according to a preset method.

[0158] Furthermore, embodiments of this application also provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the data model field mapping method as described in any of the method embodiments.

[0159] Furthermore, this application embodiment also provides a processor for running a computer program, which executes the data model field mapping method as described in any of the foregoing method embodiments.

[0160] It should be noted that the various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for the device embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiments. The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separate. The components indicated as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment solution according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0161] The above description is merely one specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data model field mapping method, characterized in that, include: Obtain the metadata of the source model and the target model; Based on the metadata of the source model and the metadata of the target model, a candidate field mapping set is constructed; Traverse each pair of candidate field mappings in the candidate field mapping set, calculate the comprehensive similarity value of each pair of candidate field mappings in the candidate field mapping set, and determine the mapping rule corresponding to each pair of candidate field mappings based on the comprehensive similarity value of each pair of candidate field mappings, so that each pair of candidate field mappings can be transformed from the source model to the target model based on the determined mapping rule.

2. The method according to claim 1, characterized in that, The calculation of the comprehensive similarity value for each pair of candidate field mappings in the candidate field mapping set includes: The formal similarity value of each pair of candidate field mappings in the candidate field mapping set is calculated using the edit distance algorithm; Using the BERT model, the semantic similarity value of each pair of candidate field mappings in the candidate field mapping set is calculated; The formal similarity value and semantic similarity value of each pair of candidate field mappings in the candidate field mapping set are weighted and summed to obtain the comprehensive similarity value of each pair of candidate field mappings in the candidate field mapping set.

3. The method according to claim 1, characterized in that, The step of determining the mapping rule corresponding to each pair of candidate field mappings based on the comprehensive similarity value of each pair of candidate field mappings includes: Determine whether the comprehensive similarity value of each pair of candidate field mappings meets the preset conditions to obtain the determination result; If the judgment result is yes, then the mapping rule corresponding to the current candidate field mapping is generated; If the judgment result is negative, the current candidate field mapping is removed from the candidate field mapping set.

4. The method according to claim 1, characterized in that, The process of constructing a candidate field mapping set based on the metadata of the source model and the metadata of the target model includes: Based on the metadata of the source model and the metadata of the target model, extract the field information contained in the source model and the field information contained in the target model; A one-to-one mapping relationship is established between each field in the source model and each field in the target model to generate multiple pairs of candidate field mappings, and a candidate field mapping set is formed based on the multiple pairs of candidate mappings.

5. The method according to claim 1, characterized in that, After obtaining the metadata of the source model and the target model, the process also includes: Data cleaning operations are performed on the metadata of the source model and the metadata of the target model; the data cleaning operations include unifying the naming style of English names, unifying the capitalization of letters, removing spaces, standardizing field types, removing stop words, and removing modal particles; Based on the metadata of the source model and the target model after data cleaning, a field information string is constructed according to a preset method.

6. A data model field mapping device, characterized in that, include: The data model metadata acquisition module is used to acquire the metadata of the source model and the target model. A candidate field mapping set construction module is used to construct a candidate field mapping set based on the metadata of the source model and the metadata of the target model; The data model field mapping module is used to traverse each pair of candidate field mappings in the candidate field mapping set, calculate the comprehensive similarity value of each pair of candidate field mappings in the candidate field mapping set, and determine the mapping rule corresponding to each pair of candidate field mappings based on the comprehensive similarity value of each pair of candidate field mappings, so that each pair of candidate field mappings can be converted from the source model to the target model based on the determined mapping rule.

7. The apparatus according to claim 6, characterized in that, The data model field mapping module is specifically used for: The formal similarity value of each pair of candidate field mappings in the candidate field mapping set is calculated using the edit distance algorithm; Using the BERT model, the semantic similarity value of each pair of candidate field mappings in the candidate field mapping set is calculated; The formal similarity value and semantic similarity value of each pair of candidate field mappings in the candidate field mapping set are weighted and summed to obtain the comprehensive similarity value of each pair of candidate field mappings in the candidate field mapping set.

8. The apparatus according to claim 6, characterized in that, The data model field mapping module is specifically used for: Determine whether the comprehensive similarity value of each pair of candidate field mappings meets the preset conditions to obtain the determination result; If the judgment result is yes, then the mapping rule corresponding to the current candidate field mapping is generated; If the judgment result is negative, the current candidate field mapping is removed from the candidate field mapping set.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the data model field mapping method as described in any one of claims 1-5.

10. A processor, characterized in that, Used to run a computer program, which, when running, executes the data model field mapping method as described in any one of claims 1-5.

Images