A data mapping method

Abstract

This application provides a data mapping method, which includes acquiring data information of the field to be mapped and a mapping rule base for the field to be mapped; performing a first-level match between the data information and the mapping rule base; the first-level match is an exact match between the data information and the mapping rule base; if the first-level match fails, performing a second-level match between the data information and the mapping rule base; the second-level match includes determining the comprehensive similarity and reference similarity between the field to be mapped and the mapping rule base; if the comprehensive similarity is greater than or equal to the reference similarity, the second-level match is successful; if either the first-level match or the second-level match is successful, the mapping result of the field to be mapped is output. This application provides a novel automatic and intelligent mapping method, reducing the manpower and resources required for manual data mapping of each new field to be mapped, thus improving work efficiency and saving time and manpower costs.

Description

Technical Field

[0001] This application relates to the field of data retrieval technology, specifically to a data mapping method. Background Technology

[0002] Field mapping is a fundamental operation in the field of data integration. Its core task is to match data fields in the source system with corresponding fields in the target system when data is being transferred from one system to another. In real-world business scenarios, the same business concept often has different names in different systems. For example, when aggregating contact information from multiple address books, there may be differences in the data used to refer to contact phone numbers, such as "mobile number," "telephone," "phone," and "tel." In the construction of an enterprise data platform, there may be differences in the data used to refer to source IP addresses, such as "src_ip," "srcaddr," and "source address." To achieve unified analysis of multi-source data, these fields with different names but the same semantics must be accurately associated with their corresponding fields in the standard data model; this process is called data mapping.

[0003] Currently, field mapping in the development process still relies heavily on manual verification. When integrating a new data source, engineers need to manually process hundreds or even thousands of fields, which is inefficient, error-prone, and results in a large amount of repetitive work. Developing an intelligent data mapping method has become a pressing technical problem. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention discloses a data mapping method that improves upon the technical issues in the prior art where engineers must manually process hundreds or thousands of fields when connecting to a new data source, resulting in low efficiency, susceptibility to errors, and a large amount of repetitive work.

[0005] To achieve the above objectives, this application provides a data mapping method, which includes the following steps:

[0006] Obtain the data information of the field to be mapped and the mapping rule library of the field to be mapped;

[0007] The data information is matched against the mapping rule base at the first level; the first level match is an exact match between the data information and the mapping rule base.

[0008] If the first-level matching fails, the data information is matched with the mapping rule base for a second-level matching; the second-level matching includes: determining the comprehensive similarity and reference similarity between the field to be mapped and the mapping rule base; if the comprehensive similarity is greater than or equal to the reference similarity, the second-level matching is successful.

[0009] If the first-level match or the second-level match is successful, the mapping result of the field to be mapped is output.

[0010] In one possible embodiment, obtaining the data information of the field to be mapped and the mapping rule base of the field to be mapped includes:

[0011] Obtain the data information of the field to be mapped; the data information includes the data name; the data information includes the data type and / or comment information;

[0012] Obtain several sample values ​​within the field to be mapped, and combine these sample values ​​to obtain the data features of the field to be mapped; the data features include the proportion of null values, the number of different values, and / or value format features;

[0013] Obtain the field mapping database; the field mapping database includes the mapping rule library of the field to be mapped.

[0014] In one possible embodiment, the step of performing a first-level match between the data information and the mapping rule base; the first-level match being an exact match between the data information and the mapping rule base, includes:

[0015] The data information of the field to be mapped is retrieved from the mapping rule base; if the mapping rule base contains a mapping rule that completely corresponds to the data information, the first-level matching is successful.

[0016] In one possible embodiment, the data information based on the field to be mapped is retrieved from the mapping rule base; if the mapping rule base contains a mapping rule that completely corresponds to the data information, after the first-level matching is successful, the process further includes:

[0017] If the first-level matching fails, determine the highest similarity between the field to be mapped and all rules in the mapping rule base;

[0018] Obtain the first reference similarity of the field to be mapped;

[0019] The highest similarity is compared with the first reference similarity. If the highest similarity is greater than the first reference similarity, the mapping rule corresponding to the highest similarity is recorded as a candidate mapping rule.

[0020] In one possible embodiment, if the first-level matching fails, the data information is matched with the mapping rule base for a second-level matching; the second-level matching includes: determining the comprehensive similarity and reference similarity between the field to be mapped and the mapping rule base; if the comprehensive similarity is less than the reference similarity, the second-level matching is successful, including:

[0021] If the first-level matching fails, the data information is processed to obtain a semantic vector, and vector retrieval is performed based on the semantic vector;

[0022] By combining the vector retrieval results and data information of the field to be mapped, semantic reasoning is performed to obtain the target field;

[0023] The highest similarity score, the result of the vector retrieval, and the result of the semantic reasoning are weighted and calculated to obtain a comprehensive similarity score.

[0024] Obtain the second, third, and fourth reference similarities of the field to be mapped;

[0025] If the overall similarity of the field to be mapped is greater than or equal to the second reference similarity, output the mapping result with the highest similarity of the field to be mapped in the mapping rule base;

[0026] If the overall similarity of the field to be mapped is greater than the third similarity and less than or equal to the second similarity, output the mapping result with the highest similarity of the field to be mapped in the mapping rule base, and provide a similarity prompt.

[0027] If the overall similarity of the field to be mapped is greater than the fourth similarity and less than or equal to the third similarity, output the mapping result with the highest similarity of the field to be mapped in the mapping rule base, and provide similarity prompts and / or list candidate mapping rules;

[0028] If the overall similarity of the field to be mapped is less than or equal to the fourth similarity, a no-match result message will be displayed.

[0029] In one possible embodiment, the step of outputting the mapping result of the field to be mapped if the first-level match or the second-level match is successful includes:

[0030] The mapping result of the field to be mapped is standardized and encapsulated; the standardized encapsulation includes matching source encapsulation; the matching source type includes first-level matching result, second-level matching result, and no matching result;

[0031] The output is a standardized and encapsulated mapping result; the standardized and encapsulated mapping result includes similarity and mapping process; when the matching source type includes a first-level matching result, the mapping process includes a first-level matching process; when the matching source type includes a second-level matching result, the mapping process includes a second-level matching process.

[0032] In one possible embodiment, after outputting the mapping result of the field to be mapped if the first-level match or the second-level match is successful, the method further includes:

[0033] The system receives user feedback and updates the mapping rule base based on the feedback; the feedback includes confirmation, correction, and / or rejection.

[0034] In one possible embodiment, before obtaining the data information of the field to be mapped and the mapping rule base of the field to be mapped, and / or after outputting the mapping result of the field to be mapped if the first-level match is successful or the second-level match is successful, the method further includes:

[0035] Obtain the confirmation rate and correction rate of all mapping rules in the mapping rule base;

[0036] The credibility rating of each rule in the mapping rule base is performed based on the confirmation rate and correction rate of each rule.

[0037] In one possible embodiment, the credibility rating of each rule in the mapping rule base based on the confirmation rate and correction rate of each rule in the mapping rule base includes:

[0038] If the confirmation rate of the current rating mapping rule in the mapping rule base is greater than or equal to the first preset frequency, and the number of times the mapping rule is used is greater than the first preset number of times, the current rating mapping rule is rated as a highly reliable rule.

[0039] If the correction rate of the current rating mapping rule in the mapping rule base is greater than or equal to the second preset frequency, and the number of times the mapping rule is used is greater than or equal to the second preset number of times, the current rating mapping rule is rated as a low-confidence rule.

[0040] If the correction rate of the current rating mapping rule in the mapping rule library is greater than or equal to the third preset frequency, the current rating mapping rule will be removed from the mapping rule library.

[0041] If none of the current rating mapping rules in the mapping rule base are used within a first preset time, the current rating mapping rule is marked as dormant.

[0042] In one possible embodiment, the step of rating the credibility of each rule in the mapping rule base based on the confirmation rate and correction rate of each rule in the mapping rule base further includes:

[0043] After traversing all mapping rules in the mapping rule base, an optimization log for the mapping rule base is generated based on the credibility rating result of each rule; the optimization log includes the number of upgrades and downgrades of all mapping rules in the mapping rule base and / or the accuracy changes in the mapping rule base.

[0044] The technical solution provided in this application has the following technical effects:

[0045] In the embodiments of this application, a first-level matching can be performed on the field to be mapped to automatically query the mapping rule base that completely corresponds to the field to be mapped. If the first-level matching fails, a second-level matching can be performed on the field to be mapped to automatically query the mapping rule base that has a high degree of matching with the field to be mapped. This provides a novel automatic and intelligent mapping method, which reduces the manpower and resources required to manually process the data mapping of each new field to be mapped, improves work efficiency, and saves time and manpower costs. Attached Figure Description

[0046] To more clearly illustrate the technical solutions and advantages in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 This is a flowchart illustrating a data mapping method provided in an embodiment of this application. Figure 1 ;

[0048] Figure 2 This is a flowchart illustrating a data mapping method provided in an embodiment of this application. Figure 2 ;

[0049] Figure 3 This is a flowchart illustrating a data mapping method provided in an embodiment of this application. Figure 3 ;

[0050] Figure 4 This is a flowchart illustrating a data mapping method provided in an embodiment of this application. Figure 4 ;

[0051] Figure 5 This is a flowchart illustrating a data mapping method provided in an embodiment of this application. Figure 5 ;

[0052] Figure 6 This is a flowchart illustrating a data mapping method provided in an embodiment of this application. Figure 6 ;

[0053] Figure 7 This is a flowchart illustrating a data mapping method provided in an embodiment of this application. Figure 7 . Detailed Implementation

[0054] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0055] It should be noted that the term "an embodiment" or "embodiment" in the specification of the embodiments of this application refers to a specific feature, structure, or characteristic that can be included in at least one implementation of this application. It should be understood that in the specification, claims, and accompanying drawings of the embodiments of this application, the terms "upper," "lower," "top," "bottom," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this application. The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined with "first" and "second" may explicitly or implicitly include one or more of that feature. Moreover, the terms "first," "second," etc., are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0056] Please see Figures 1 to 7 This application provides a data mapping method, which includes the following steps:

[0057] S1: Obtain the data information of the field to be mapped and the mapping rule library of the field to be mapped.

[0058] In one possible embodiment, the data information of the field to be mapped may include the name, data type and / or annotation information of the field to be mapped. The name of the field to be mapped may be the field name of the original data of the field to be mapped. The data type of the field to be mapped may be used to characterize which system the field to be mapped originates from. The mapping rule base of the field to be mapped may be a general database that includes the mapping relationship of the field to be mapped. The mapping rule base may be used to store mapping rules including but not limited to historical data mapping records.

[0059] S2: Perform a first-level match between the data information and the mapping rule base; the first-level match is an exact match between the data information and the mapping rule base.

[0060] Optionally, the mapping rule base can be traversed to find mapping rules that match the data information of the field to be mapped. When the mapping rule base contains mapping rules that are completely consistent with the data information of the field to be mapped, it is considered a first-level match.

[0061] S3: If the first-level matching fails, the data information is matched with the mapping rule base for a second-level matching; the second-level matching includes: determining the comprehensive similarity and reference similarity between the field to be mapped and the mapping rule base; if the comprehensive similarity is greater than or equal to the reference similarity, the second-level matching is successful.

[0062] Optionally, during the secondary matching process, the field to be mapped can be processed to obtain the comprehensive similarity between the field to be mapped and the mapping rule library. The reference similarity between the field to be mapped and the mapping rule library can be obtained. The comprehensive similarity and the reference similarity are compared. If the comprehensive similarity is greater than or equal to the reference similarity, it indicates that during the secondary matching process, there exists a mapping rule in the mapping rule library that has passed the secondary matching with the field to be mapped.

[0063] S4: If the first-level match or the second-level match is successful, output the mapping result of the field to be mapped.

[0064] Optionally, if the first-level match or the second-level match is successful, it indicates that there is a mapping rule in the mapping rule library that corresponds to the data to be mapped, and the mapping rule can be output at this time.

[0065] In the embodiments of this application, a first-level matching can be performed on the field to be mapped to query the mapping rule base for mapping rules that completely correspond to the field to be mapped. If the first-level matching fails, a second-level matching can be performed on the field to be mapped to query the mapping rule base for mapping rules that have a high degree of matching with the field to be mapped. This provides a novel automatic and intelligent mapping method, which reduces the manpower and resources required to manually process the data mapping of each new field to be mapped, improves work efficiency, and saves time and manpower costs.

[0066] In one possible embodiment, when performing S1: obtaining the data information of the field to be mapped and the mapping rule base of the field to be mapped, the following steps may be specifically performed:

[0067] S101: Obtain the data information of the field to be mapped; the data information includes the data name; the data information includes the data type and / or comment information.

[0068] Optionally, the data information of the field to be mapped is used to perform subsequent first-level and second-level matching, and the data information includes the data name.

[0069] Optionally, in addition to the data name, the data information may include data type and / or annotation information.

[0070] S102: Obtain several sample values ​​within the field to be mapped, and combine the several sample values ​​to obtain the data features of the field to be mapped; the data features include the proportion of null values, the number of different values, and / or value format features.

[0071] Optionally, the data features of the field to be mapped can be determined by several sample values ​​within the field to be mapped. These data features are then used to perform subsequent secondary matching.

[0072] In addition, data information such as field values, database names, and data types in the fields to be mapped can be obtained, and these data information can be combined with several sample values ​​to obtain data features, thereby further improving the accuracy of data mapping.

[0073] S103: Obtain the field mapping database; the field mapping database includes the mapping rule library of the field to be mapped.

[0074] Optionally, a field mapping database can be obtained, which can be a general database containing a mapping rule library of fields to be mapped.

[0075] In this embodiment, data information or data features of the field to be mapped can be obtained to provide data preparation for subsequent first-level and second-level matching, thereby enabling subsequent data matching of the field to be mapped.

[0076] In one possible embodiment, during S2: performing a first-level match between the data information and the mapping rule base; the first-level match is an exact match between the data information and the mapping rule base, which can be specifically executed as follows:

[0077] S201: Based on the data information of the field to be mapped, a search is performed in the mapping rule base; if the mapping rule base contains a mapping rule that completely corresponds to the data information, the first-level matching is successful.

[0078] Optionally, the mapping rule base can be traversed to query whether there is a mapping rule that is completely consistent with the field to be mapped. When a mapping rule that completely contains the data name of the field to be mapped is detected, it is considered that the first-level match has passed.

[0079] In one possible embodiment, after executing S201: a search is performed in the mapping rule base based on the data information of the field to be mapped; if the mapping rule base contains a mapping rule that completely corresponds to the data information, after the first-level matching is successful, the process further includes:

[0080] S202: If the first-level matching fails, determine the highest similarity between the field to be mapped and all rules in the mapping rule base.

[0081] S203: Obtain the first reference similarity of the field to be mapped.

[0082] Optionally, the first reference similarity of the field to be mapped can be obtained. The first reference similarity can be the minimum similarity that indicates the field to be mapped meets the requirements to be a candidate mapping result.

[0083] S204: Compare the highest similarity with the first reference similarity. If the highest similarity is greater than the first reference similarity, record the mapping rule corresponding to the highest similarity as a candidate mapping rule.

[0084] Optionally, the alternative mapping rule can be a backup result after the first-level match fails. That is, when the first-level match fails, the result can still be fed back to the user by providing the alternative mapping rule, thereby increasing the probability that the result feedback of the field to be mapped is correct.

[0085] In one possible embodiment, when executing S3: if the first-level matching fails, the data information is matched with the mapping rule base for a second-level matching; the second-level matching includes: determining the comprehensive similarity and reference similarity between the field to be mapped and the mapping rule base; if the comprehensive similarity is less than the reference similarity, and the second-level matching succeeds, it includes:

[0086] S301: If the first-level matching fails, the data information is processed to obtain a semantic vector, and vector retrieval is performed based on the semantic vector.

[0087] Optionally, the data information of the field to be mapped can be encoded into a semantic vector, and a similarity search can be performed in the vector index corresponding to the mapping rule base to perform vector indexing. Specifically, a Top-K semantic similarity search can be performed to obtain vector search results, which can be RAG (Retrieval Enhancement Generation) results.

[0088] S302: Combine the vector retrieval results and data information of the field to be mapped to perform semantic reasoning to obtain the target field.

[0089] Optionally, semantic reasoning can be performed by combining the vector retrieval results and data information of the field to be mapped. The target table structure and industry domain knowledge of the field to be mapped can also be obtained. The vector retrieval results, data information, target table structure and industry domain knowledge of the field to be mapped are submitted to the natural language big model for semantic reasoning, and the similarity between the target field and the natural language big model is output. The target field is the mapping result.

[0090] S303: Calculate the weights of the highest similarity score, the result of the vector retrieval, and the result of the semantic reasoning to obtain a comprehensive similarity score.

[0091] Optionally, the calculation weights of the highest similarity, vector retrieval results, and semantic reasoning results can be obtained, and a comprehensive similarity numerical calculation can be performed based on the calculation weights of the highest similarity, vector retrieval results, and semantic reasoning results.

[0092] Specifically, the weight for calculating the highest similarity can be 'a', the weight for calculating the vector retrieval result can be 'b', and the weight for calculating the semantic reasoning result can be 'c', with the sum of a+b+c being 1. Here, a can be 0.5, b can be 0.2, and c can be 0.3.

[0093] S304: Obtain the second reference similarity, third reference similarity, and fourth reference similarity of the field to be mapped.

[0094] Among them, the second reference similarity, the third reference similarity, and the fourth reference similarity are all used to compare with the calculated comprehensive similarity to form a reference. The second reference similarity can be greater than the third reference similarity, and the third reference similarity can be greater than the fourth reference similarity.

[0095] S305: If the overall similarity of the field to be mapped is greater than or equal to the second reference similarity, output the target field.

[0096] If the overall similarity of the field to be mapped is greater than or equal to the second reference similarity, it indicates that the confidence level is high at this time, and the mapping result of the field to be mapped is automatically mapped, i.e., the target field.

[0097] Optionally, if the overall similarity of the field to be mapped is greater than the third similarity and less than or equal to the second similarity, the target field is output and a similarity prompt is given.

[0098] If the overall similarity of the field to be mapped is greater than the third similarity and less than or equal to the second similarity, it indicates that the confidence level is average. The target field will be automatically mapped, and the user is advised to confirm.

[0099] Optionally, if the overall similarity of the field to be mapped is greater than the fourth similarity and less than or equal to the third similarity, the target field of the field to be mapped is output, and similarity prompts and / or candidate mapping rules are listed.

[0100] If the overall similarity of the field to be mapped is greater than the fourth similarity and less than or equal to the third similarity, it indicates that the confidence level is average. The mapping result of the field to be mapped is automatically mapped, and the annotation suggests that the user confirm and / or lists the candidate mapping rules obtained from the S204 operation.

[0101] Optionally, if the overall similarity of the field to be mapped is less than or equal to the fourth similarity, a no-match result message will be displayed.

[0102] If the overall similarity of the field to be mapped is less than or equal to the fourth similarity, a no-match result message will be displayed, and candidate mapping rules can be listed.

[0103] In one possible embodiment, when executing S4: if the first-level match is successful or the second-level match is successful, output the mapping result of the field to be mapped, including:

[0104] S401: Standardize and encapsulate the mapping result of the field to be mapped; the standardization encapsulation includes matching source encapsulation; the matching source type includes first-level matching result, second-level matching result, and no matching result.

[0105] Optionally, the data information and mapping results of the field to be mapped can be standardized and encapsulated according to the MCP protocol, and the source matching encapsulation is used to characterize the source of the mapping result of the field to be mapped.

[0106] S402: Output the standardized and encapsulated mapping result; the standardized and encapsulated mapping result includes similarity and mapping process; when the matching source type includes a first-level matching result, the mapping process includes a first-level matching process; when the matching source type includes a second-level matching result, the mapping process includes a second-level matching process.

[0107] Optionally, the standardized and encapsulated mapping results can be displayed in a list on the system operation interface. The mapping process can include the entire mapping process of the field to be mapped, that is, the specific operation information from operation S1 to operation S3.

[0108] In one possible embodiment, after executing S4: if the first-level match is successful or the second-level match is successful, outputting the mapping result of the field to be mapped, the method further includes:

[0109] S5: Receive user feedback and update the mapping rule base based on the feedback; the feedback includes confirmation, correction and / or rejection.

[0110] Optionally, the mapping rule base can be updated by receiving user feedback and updating the mapping rule base accordingly, thereby enabling real-time / periodic updates and maintenance of the mapping rule base.

[0111] In one possible embodiment, before executing S4: obtaining the data information of the field to be mapped and the mapping rule base of the field to be mapped, and / or after outputting the mapping result of the field to be mapped if the first-level match is successful or the second-level match is successful, the method further includes:

[0112] S501: Obtain the confirmation rate and correction rate of all mapping rules in the mapping rule base.

[0113] Optionally, the user's feedback tendency towards mapping rules in the mapping rule base can be determined by the confirmation rate and correction rate of the mapping rules. When the confirmation rate is high and the correction rate is low, the mapping rule feedback is better.

[0114] S502: Assess the credibility of each rule in the mapping rule base based on its confirmation rate and correction rate.

[0115] Optionally, the credibility rating of the rules in the mapping rule base can be performed based on the confirmation rate and correction rate of each rule in the mapping rule base. When outputting the results of the field to be mapped, the rating results of each rule in the mapping rule base can be output.

[0116] In one possible embodiment, performing S502: rating the credibility of each rule in the mapping rule base based on the confirmation rate and correction rate of each rule in the mapping rule base, may specifically include:

[0117] S503: If the confirmation rate of the current rating mapping rule in the mapping rule base is greater than or equal to the first preset frequency, and the number of times the mapping rule is used is greater than the first preset number of times, the current rating mapping rule is rated as a highly reliable rule.

[0118] Optionally, the first preset frequency can be 90%, and the first preset number of times can be 20.

[0119] S504: If the correction rate of the current rating mapping rule in the mapping rule base is greater than or equal to the second preset frequency, and the number of times the mapping rule is used is greater than or equal to the second preset number of times, the current rating mapping rule is rated as a low-confidence rule.

[0120] Optionally, the second preset frequency can be 30%, and the second preset number of times can be 10.

[0121] S505: If the correction rate of the current rating mapping rule in the mapping rule library is greater than or equal to the third preset frequency, the current rating mapping rule is removed from the mapping rule library.

[0122] Optionally, the third preset frequency can be 50%.

[0123] S506: If none of the current rating mapping rules in the mapping rule base are used within a first preset time, the current rating mapping rule is marked as dormant.

[0124] Optionally, the first preset time can be 6 months.

[0125] S507: Traverse the user's historical correction records. If the number of times the user has corrected the same mapping rule is greater than or equal to the third preset number, the corrected mapping rule will be included as a new mapping rule in the mapping rule library.

[0126] Optionally, the third preset number of times can be 5.

[0127] In one possible embodiment, after performing a credibility rating on each rule in the mapping rule base based on the confirmation rate and correction rate of each rule in the mapping rule base, step S502 further includes:

[0128] S508: After traversing all mapping rules in the mapping rule base, an optimization log for the mapping rule base is generated based on the credibility rating result of each rule; the optimization log includes the number of upgrades and downgrades of all mapping rules in the mapping rule base and / or the accuracy changes in the mapping rule base.

[0129] Optionally, by optimizing the logging of changes in the rank, accuracy, and other parameters of each mapping rule in the mapping rule base, the changes in each mapping rule in the mapping rule base can be recorded for easy reference later.

[0130] The following describes a mapping method of this application in conjunction with the first embodiment.

[0131] Step 1: Obtain the data information of the field to be mapped and the mapping rule library of the field to be mapped.

[0132] Specifically, it includes:

[0133] Get the name, data type, and comment information of the field to be mapped.

[0134] Randomly select 10-20 sample values ​​from the actual data of this field, and obtain data characteristics based on the sample values: the proportion of empty values, the number of different values, and the value format characteristics.

[0135] Obtain the constructed field mapping database, i.e. the mapping rule base, which stores historical mapping records and their usage statistics.

[0136] For example, the field to be mapped is srcaddr, the type is VARCHAR(50), the sample values ​​are 192.168.1.100, 10.0.0.50, etc., the proportion of empty values ​​is 2%, the number of different values ​​is about 15,000, and the value format characteristics are IPv4 addresses.

[0137] Step 2: Perform a first-level match between the data information and the mapping rule base.

[0138] The first-level matching is an exact match: it searches the rule base to see if there is a completely identical field name. If so, the match is successful, and the process proceeds directly to step 6 to output the mapping result.

[0139] In this example, there is no exact match for the field name "srcaddr" in the rule base, so the first-level match fails.

[0140] Step 3: After a first-level match fails, determine the highest similarity between the field to be mapped and all rules in the rule base, and compare it with the first reference similarity.

[0141] Calculate the semantic similarity of the field names and obtain the highest similarity score (e.g., 0.88). Compare this score with a preset first reference similarity (e.g., 0.9). Since 0.88 < 0.9, the first reference similarity is not met, so this rule is not considered as a candidate mapping rule and proceeds to the second-level matching.

[0142] Step 4: Perform secondary matching.

[0143] Second-level matching includes the following sub-steps:

[0144] Vector retrieval: The data information (field name, type, data features, etc.) of the field to be mapped is encoded into a semantic vector. Top-K semantic similarity retrieval is performed in the vector index of the rule base to obtain the K most similar historical records and their vector retrieval scores (e.g., 0.92).

[0145] Semantic reasoning: The data information of the field to be mapped, the target table structure, vector retrieval results, and industry domain knowledge are submitted to the large language model through structured prompts. The model performs semantic reasoning and returns the recommended target field and LLM inference score (e.g., 0.95). The large language model can include commonly available large language models.

[0146] Weighted calculation of comprehensive similarity: The highest similarity score (0.88), vector retrieval score (0.92), and LLM inference score (0.95) obtained in step 3 are weighted and calculated to obtain the comprehensive similarity. Let the weights be α=0.5, β=0.2, and γ=0.3, then the comprehensive similarity = 0.5×0.88 + 0.2×0.92 + 0.3×0.95 = 0.44 + 0.184 + 0.285 = 0.909.

[0147] Compare with reference similarities and output the results: Preset second reference similarity = 0.9, third reference similarity = 0.7, fourth reference similarity = 0.5. The overall similarity is 0.909 ≥ 0.9, therefore the second-level match is successful, and the recommended target field (source_ip) is output and marked as automatic mapping.

[0148] If the overall similarity falls between 0.7 and 0.9, it will be automatically mapped and marked "recommended for confirmation"; if it is between 0.5 and 0.7, multiple candidates will be listed for manual selection; if it is below 0.5, it will be marked as "cannot be judged".

[0149] Step 5: Output the mapping results.

[0150] The mapping results are standardized and encapsulated (MCP protocol can be used). The encapsulated content includes source field information, recommended target field, comprehensive similarity, matching source type (secondary matching result in this example), and a summary of the reasoning process. The encapsulated results are displayed in a list format on the interface, along with similarity and reasoning basis.

[0151] Step 6: Receive user feedback and update the rule base.

[0152] Users can confirm, correct, or reject the mapping results. For example, if a user confirms that srcaddr→source_ip is correct, the confirmation count for that mapping in the rule base is incremented by 1; if the user corrects it to a different target field, the correction count for the original mapping is incremented by 1, and the new mapping relationship is written to the rule base.

[0153] Step 7: Iterative optimization of the rule base.

[0154] The system periodically or under triggered conditions optimizes the rule base:

[0155] The confirmation rate and correction rate of each rule are calculated.

[0156] If a rule has a confirmation rate of ≥90% and is used ≥20 times, it is upgraded to a highly reliable rule (the same field will be used directly next time without AI inference).

[0157] If the correction rate is ≥30% and the number of uses is ≥10, it will be downgraded to a low-trust rule; if the correction rate is ≥50%, it will be removed directly.

[0158] Analyze patterns in user correction records and automatically discover new rules (such as "the _ts suffix usually corresponds to the _time field").

[0159] Rules that have not been used for more than 6 months are marked as dormant.

[0160] Generate optimization logs to record rule upgrades and downgrades, as well as changes in the overall accuracy of the rule base.

[0161] The following description, in conjunction with the second embodiment, explains steps four and five of the first embodiment:

[0162] The complete prompt to send to the AI ​​===

[0163] [System Message]

[0164] You are a professional data field mapping analyst. Your job is to determine the correspondences between data fields.

[0165] You only need to determine the field correspondence; you don't do anything else.

[0166] If you are unsure, be honest and don't guess.

[0167] If the overall similarity is below 0.6 (i.e. below the fourth reference similarity), it must be marked "requires manual confirmation".

[0168] The results must be returned in JSON format.

[0169] [User Messages]

[0170] ## Data information of the field to be mapped

[0171] Field name: evt_ts

[0172] Field type: BIGINT

[0173] Data samples: 1705312456000, 1705312789000, 1705313012000

[0174] Data source: User behavior logs

[0175] Other fields in the same table: user_id, event_type, page_url, device_type

[0176] Null value percentage: 0%

[0177] Number of distinct values: close to the total number of records (approximately 500,000)

[0178] Value format characteristics: 13-digit pure numeric (millisecond timestamp)

[0179] ## Target Table Structure

[0180] Table name: dwd_user_behavior

[0181] Has the field name type meaning been mapped?

[0182] event_time DATETIME: Event time is not mapped.

[0183] `create_time DATETIME` indicates that the creation time is not mapped.

[0184] update_time DATETIME update time is not mapped

[0185] ## Reference Records for Mapping Rule Base (Vector Retrieval Results)

[0186] - Source field evt_ts → Target field event_time (used 67 times, historical accuracy 97%)

[0187] - Source field event_ts → Target field event_time (used 45 times, historical accuracy 100%)

[0188] ## Industry-Specific Knowledge

[0189] Area: User behavior analysis

[0190] Common naming conventions: Timestamp fields often use the suffixes _ts or _time.

[0191] ## Analysis Requirements

[0192] Please analyze in 5 steps: Field Name Analysis → Data Analysis → Context Analysis → Candidate Filtering → Final Conclusion

[0193] ## Reference Example

[0194] ## Output Format

[0195] Please strictly adhere to the following JSON format for output:

[0196] {

[0197] "mapping_result": {

[0198] "source_field": "field name",

[0199] "target_field": "target field name",

[0200] "confidence": 0.xx, / / Overall similarity

[0201] "match_type": "Match source type" / / For example, "Level 1 match (exact match)" or "Level 2 match (semantic match)"

[0202] },

[0203] "reasoning": {

[0204] "step1_name_analysis": "...",

[0205] "step2_data_analysis": "...",

[0206] "step3_context_analysis": "...",

[0207] "step4_candidates": "...",

[0208] "step5_conclusion": "..."

[0209] },

[0210] "alternatives": [], / / List of alternative mapping rules (provided when the overall similarity is between the fourth and third reference similarities)

[0211] "needs_human_review": true or false

[0212] }

[0213] The following MCP message example encapsulates the above information in a unified way, achieving decoupling from the specific model.

[0214] json

[0215] {

[0216] "version": "1.0",

[0217] "message_type": "Field mapping request",

[0218] "context": {

[0219] "source_field": {

[0220] "name": "evt_ts",

[0221] "data_type": "BIGINT",

[0222] "comment": null,

[0223] "samples": ["1705312456000", "1705312789000", "1705313012000"],

[0224] "null_ratio": 0.0,

[0225] "distinct_count": 500000,

[0226] "format_pattern": "13-digit millisecond timestamp"

[0227] },

[0228] "source_metadata": {

[0229] "system_name": "User Behavior Log System",

[0230] "table_name": "raw_user_events",

[0231] "vendor": null,

[0232] "data_format": "log"

[0233] },

[0234] "co_occurring_fields": [

[0235] {"name": "user_id", "type": "BIGINT", "comment": "user ID"},

[0236] {"name": "event_type", "type": "VARCHAR", "comment": "event type"},

[0237] {"name": "page_url", "type": "VARCHAR", "comment": "page URL"},

[0238] {"name": "device_type", "type": "VARCHAR", "comment": "device type"}

[0239] ],

[0240] "target_table": {

[0241] "name": "dwd_user_behavior",

[0242] "description": "User Behavior Details"

[0243] "fields": [

[0244] {"name": "event_time", "type": "DATETIME", "comment": "event time", "mapped": false},

[0245] {"name": "create_time", "type": "DATETIME", "comment": "creation time", "mapped": false},

[0246] {"name": "update_time", "type": "DATETIME", "comment": "update time", "mapped": false} ]

[0248] },

[0249] "vector_retrieval_results": [

[0250] {

[0251] "source_field": "evt_ts",

[0252] "target_field": "event_time",

[0253] "similarity_score": 0.95,

[0254] "usage_count": 67,

[0255] "accuracy": 0.97

[0256] },

[0257] {

[0258] "source_field": "event_ts",

[0259] "target_field": "event_time",

[0260] "similarity_score": 0.92,

[0261] "usage_count": 45,

[0262] "accuracy": 1.0

[0263] }

[0264] ],

[0265] "domain_knowledge": {

[0266] "industry": "user behavior analysis",

[0267] "field_naming_conventions": ["_ts usually represents a timestamp", "_time usually represents a time field"]

[0268] }

[0269] },

[0270] "task": {

[0271] "type": "field_mapping",

[0272] "requirements": ["Type compatibility", "Semantic matching", "Give overall similarity", "Explanation of reasoning process"]

[0273] },

[0274] "constraints": {

[0275] "min_confidence": 0.6,

[0276] "max_alternatives": 3,

[0277] "allow_multi_mapping": false

[0278] }

[0279] }

[0280] The following JSON output strictly follows the terminology of the claims, where confidence refers to overall similarity and match_type indicates the type of match source (secondary match).

[0281] json

[0282] {

[0283] "mapping_result": {

[0284] "source_field": "evt_ts",

[0285] "target_field": "event_time",

[0286] "confidence": 0.97,

[0287] "match_type": "Secondary matching (semantic matching)"

[0288] },

[0289] "reasoning": {

[0290] "step1_name_analysis": "evt_ts is broken down into evt + ts. evt is a common abbreviation for event, and ts is a common abbreviation for timestamp. Together, they represent "event timestamp".

[0291] "step2_data_analysis": "The sample values ​​are 13-digit numbers (e.g., 1705312456000), which conform to the characteristics of millisecond-level timestamps; the target field event_time is of type DATETIME, which can be converted to achieve compatibility."

[0292] "step3_context_analysis": "Sourced from user behavior logs, this table contains fields such as user_id and event_type, collectively describing user behavior events. The rule base contains 67 records of evt_ts→event_time, with a 97% accuracy rate, indicating high reliability."

[0293] "step4_candidates": "The event_time field in the target table is not mapped, and the semantics are a perfect match; although create_time and update_time are time fields, they have different meanings from the event time and have lower priority."

[0294] "step5_conclusion": "Field names are semantically clear, data format is validated as timestamp, rule base is fully supported, and the overall similarity is 0.97."

[0295] },

[0296] "alternatives": [

[0297] {"target_field": "create_time", "confidence": 0.25, "reason": "Creation time, which has a different meaning from event time"},

[0298] {"target_field": "update_time", "confidence": 0.20, "reason": "Update time, unrelated to event time"}

[0299] ],

[0300] "needs_human_review": false

[0301] }

[0302] After the above operations, the final mapping results are shown in the table below.

[0303] # Source field Recommended goals Overall similarity Is manual confirmation required? Match source type Reasoning Summary 1 srcaddr source_ip 0.95 No (Automatic mapping) Second-level matching (semantic matching) The rule base contains 89 records plus semantic matching. 2 dstaddr dest_ip 0.94 No (Automatic mapping) Second-level matching (semantic matching) Symmetrical to srcaddr, with ample experience 3 srcport src_port 0.97 No (Automatic mapping) Level 1 match (exact match) Field names are almost identical 4 dstport dst_port 0.97 No (Automatic mapping) Level 1 match (exact match) Field names are almost identical 5 proto protocol 0.88 Recommendation to confirm Second-level matching (semantic matching) proto is an abbreviation for protocol, and its overall similarity is between 0.7 and 0.9. 6 action none 0.42 Yes (cannot be determined) No match The meaning of "action" is ambiguous, with multiple candidates, and the overall similarity is lower than the fourth reference similarity. 7 severity alert_level 0.72 Recommendation to confirm Second-level matching (semantic matching) Semantically similar but with significantly different names, their overall similarity ranges from 0.7 to 0.9.

[0304] Specifically, if the overall similarity is ≥ 0.9 (second reference similarity): automatic mapping, no manual confirmation required; if 0.7 ≤ overall similarity < 0.9 (between the third and second reference similarities): automatic mapping and marked "recommendation for confirmation"; if 0.5 ≤ overall similarity < 0.7 (between the fourth and third reference similarities): no automatic mapping, and a list of candidates for manual selection is provided (this type does not appear in this table); if the overall similarity is < 0.5 (lower than the fourth reference similarity): marked as undetermined, manual mapping is required.

[0305] It should be noted that the order of the embodiments described above is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, specific embodiments have been described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims can be performed in a different order than that shown in the embodiments and still achieve the desired result. Additionally, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0306] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

Claims

1. A data mapping method, characterized by, Includes the following steps: Obtain the data information of the field to be mapped and the mapping rule library of the field to be mapped; The data information is matched against the mapping rule base at the first level; the first level match is an exact match between the data information and the mapping rule base. If the first-level matching fails, the data information will be matched with the mapping rule base for a second-level matching. The secondary matching includes: determining the comprehensive similarity and reference similarity between the field to be mapped and the mapping rule base; if the comprehensive similarity is greater than or equal to the reference similarity, the secondary matching is successful; If the first-level match or the second-level match is successful, the mapping result of the field to be mapped is output.

2. The data mapping method of claim 1, wherein, The process of obtaining the data information of the field to be mapped and the mapping rule library of the field to be mapped includes: Obtain the data information of the field to be mapped; the data information includes the data name; the data information includes the data type and / or comment information; Obtain several sample values ​​within the field to be mapped, and combine these sample values ​​to obtain the data features of the field to be mapped; the data features include the proportion of null values, the number of different values, and / or value format features; Obtain the field mapping database; the field mapping database includes the mapping rule library of the field to be mapped.

3. The data mapping method according to claim 1, characterized in that, The step of performing a first-level match between the data information and the mapping rule base; the first-level match is an exact match between the data information and the mapping rule base, including: The data information of the field to be mapped is retrieved from the mapping rule base; if the mapping rule base contains a mapping rule that completely corresponds to the data information, the first-level matching is successful.

4. The data mapping method according to claim 3, characterized in that, If the first-level matching fails, before performing a second-level matching between the data information and the mapping rule base, the method further includes: If the first-level matching fails, determine the highest similarity between the field to be mapped and all rules in the mapping rule base; Obtain the first reference similarity of the field to be mapped; The highest similarity is compared with the first reference similarity. If the highest similarity is greater than the first reference similarity, the mapping rule corresponding to the highest similarity is recorded as a candidate mapping rule.

5. The data mapping method according to claim 4, characterized in that, If the first-level matching fails, the data information is matched with the mapping rule base for a second-level matching; the second-level matching includes: determining the comprehensive similarity and reference similarity between the field to be mapped and the mapping rule base; if the comprehensive similarity is less than the reference similarity, the second-level matching is successful, including: If the first-level matching fails, the data information is processed to obtain a semantic vector, and vector retrieval is performed based on the semantic vector; By combining the vector retrieval results and data information of the field to be mapped, semantic reasoning is performed to obtain the target field; The highest similarity score, the result of the vector retrieval, and the result of the semantic reasoning are weighted and calculated to obtain a comprehensive similarity score. Obtain the second, third, and fourth reference similarities of the field to be mapped; If the overall similarity of the field to be mapped is greater than or equal to the second reference similarity, output the mapping result with the highest similarity of the field to be mapped in the mapping rule base; If the overall similarity of the field to be mapped is greater than the third similarity and less than or equal to the second similarity, output the mapping result with the highest similarity of the field to be mapped in the mapping rule base, and provide a similarity prompt. If the overall similarity of the field to be mapped is greater than the fourth similarity and less than or equal to the third similarity, output the mapping result with the highest similarity of the field to be mapped in the mapping rule base, and provide similarity prompts and / or list candidate mapping rules; If the overall similarity of the field to be mapped is less than or equal to the fourth similarity, a no-match result message will be displayed.

6. The data mapping method according to claim 1, characterized in that, If the first-level match or the second-level match is successful, the mapping result of the field to be mapped is output, including: The mapping result of the field to be mapped is standardized and encapsulated; the standardized encapsulation includes matching source encapsulation; the matching source type includes first-level matching result, second-level matching result, and no matching result; The output is a standardized and encapsulated mapping result; the standardized and encapsulated mapping result includes similarity and mapping process; when the matching source type includes a first-level matching result, the mapping process includes a first-level matching process; when the matching source type includes a second-level matching result, the mapping process includes a second-level matching process.

7. The data mapping method according to claim 6, characterized in that, After outputting the mapping result of the field to be mapped if the first-level match or the second-level match is successful, the method further includes: The system receives user feedback and updates the mapping rule base based on the feedback; the feedback includes confirmation, correction, and / or rejection.

8. The data mapping method according to claim 1, characterized in that, Before obtaining the data information of the field to be mapped and the mapping rule base of the field to be mapped, and / or after outputting the mapping result of the field to be mapped if the first-level match is successful or the second-level match is successful, the method further includes: Obtain the confirmation rate and correction rate of all mapping rules in the mapping rule base; The credibility rating of each rule in the mapping rule base is performed based on the confirmation rate and correction rate of each rule.

9. The data mapping method according to claim 8, characterized in that, The credibility rating of each rule in the mapping rule base based on the confirmation rate and correction rate of each rule in the mapping rule base includes: If the confirmation rate of the current rating mapping rule in the mapping rule base is greater than or equal to the first preset frequency, and the number of times the mapping rule is used is greater than the first preset number of times, the current rating mapping rule is rated as a highly reliable rule. If the correction rate of the current rating mapping rule in the mapping rule base is greater than or equal to the second preset frequency, and the number of times the mapping rule is used is greater than or equal to the second preset number of times, the current rating mapping rule is rated as a low-confidence rule. If the correction rate of the current rating mapping rule in the mapping rule library is greater than or equal to the third preset frequency, the current rating mapping rule will be removed from the mapping rule library. If none of the current rating mapping rules in the mapping rule base are used within a first preset time, the current rating mapping rule is marked as dormant.

10. The data mapping method according to claim 9, characterized in that, The method of rating the credibility of each rule in the mapping rule base based on the confirmation rate and correction rate of each rule in the mapping rule base also includes: After traversing all mapping rules in the mapping rule base, an optimization log for the mapping rule base is generated based on the credibility rating result of each rule; the optimization log includes the number of upgrades and downgrades of all mapping rules in the mapping rule base and / or the accuracy changes in the mapping rule base.

Images