A method for configuring data integration processes on a low-code platform
By launching field name and table structure inference channels in parallel within a low-code platform, generating and cross-validating prediction mappings, the complexity and accuracy issues of data field mapping in low-code platforms are resolved, achieving efficient and automated data integration configuration.
Patent Information
- Application Number
- CN202610969228.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2026-07-01
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2046-07-01
AI Technical Summary
The configuration complexity of data field mapping in low-code platforms is high, and manual operation by users is cumbersome and prone to errors. Especially when field names are inconsistent or table structures change in a complex way, the accuracy of existing automatic matching methods is low and cannot effectively reduce the workload of configuration.
By launching two mapping inference channels in parallel—one based on field name transformation patterns and the other based on table structure positional features—predictive mappings are generated. Through cross-validation and fusion, a set of recommended mappings is generated. Users only need to perform a mapping operation once to automatically complete the mapping of most fields.
It significantly reduces configuration complexity and operation time, improves mapping accuracy, and reduces manual user operations and the probability of errors.
Smart Images

Figure CN122489055B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of low-code platform technology, specifically relating to a method for configuring a data integration process on a low-code platform. Background Technology
[0002] Low-code platforms allow users to quickly build applications through a visual configuration interface using drag-and-drop and point-and-click methods. A key aspect of the data integration process is mapping fields between different data sources. For example, mapping user-defined data tables (source tables) within the platform to the response message structure of external business system interfaces (target tables).
[0003] Currently, data field mapping in low-code platforms primarily relies on manual configuration. Users need to manually search for and specify the corresponding target field from the target table's field list for each field in the source table within the configuration interface. When the source and target tables contain dozens or even hundreds of fields, this manual matching process is extremely tedious and time-consuming, and prone to mapping errors due to user fatigue or negligence. Although some existing technologies offer single automatic matching suggestions based on field name similarity, these methods suffer from significantly reduced matching accuracy in complex scenarios such as inconsistent field naming conventions or overall table structure reorganization (e.g., multiple fields from the source table being merged or split into different locations in the target table), failing to effectively reduce the user's configuration workload.
[0004] Therefore, how to reduce the configuration complexity of field mapping in the data integration process of low-code platforms and reduce the amount of manual operation for users is a technical problem that urgently needs to be solved. Summary of the Invention
[0005] In view of the above-mentioned defects or deficiencies in the prior art, a low-code platform data integration process configuration method is provided, characterized by including the following steps: In response to the first mapping operation performed by the user in the configuration interface of the low-code platform, a first mapping pair is obtained; the first mapping pair includes a first source field in the source data table and a first target field in the target data table. Using the first mapping pair as the anchor point, multiple mapping inference channels, including at least the first mapping inference channel and the second mapping inference channel, are launched in parallel. in, The first mapping inference channel is used to extract the field name transformation pattern between the field name of the first source field and the field name of the first target field, and apply the field name transformation pattern to other source fields in the source data table to obtain the first set of prediction mappings; The second mapping inference channel is used to compare the first structural position feature of the first source field in the table structure of the source data table with the second structural position feature of the first target field in the table structure of the target data table, identify the global table structure transformation pattern, and apply the global table structure transformation pattern to the other source fields to obtain the second set of prediction mappings. The first set of prediction mappings and the second set of prediction mappings are cross-validated and fused to generate a set of fused recommended mappings. The cross-validation and fusion include: for the same source field, if the two sets of prediction mappings are consistent, the result is adopted; if the two sets of prediction mappings are inconsistent or a certain channel does not provide a prediction, the prediction mapping with higher confidence is selected or a supplementary mapping strategy is triggered according to the preset confidence evaluation rules. The fusion recommendation mapping set is displayed in batches in the configuration interface in list form, and a one-click confirmation control is provided; In response to the user's triggering of the one-click confirmation control, the mapping pairs in the fusion recommendation mapping set are adopted in batches as formal data docking process configurations.
[0006] According to the technical solution provided in this application, the first mapping inference channel includes: The field names of the first source field and the first target field are aligned to identify the common and different parts. At least two candidate transformation rules are generated based on the differences, including: prefix replacement rule, suffix replacement rule, character case conversion rule, or delimiter conversion rule; Each candidate transformation rule is applied to all field names in the source data table, and the matching success rate with the actual field names in the target data table is calculated. If, for any source field name, the predicted field name obtained after applying the candidate transformation rule exists in the target data table with an edit distance less than a preset distance, it is determined to be a successful match. Select the candidate transformation rule with the highest matching success rate as the field name transformation mode.
[0007] According to the technical solution provided in this application, the second mapping inference channel includes: Obtain data samples from at least two fields in the source data table and data samples from at least two fields in the target data table; For each sampled field, extract its numerical distribution morphological feature vector, which includes a combination of at least two of the following: statistical distribution of data type, quantile profile, unique value density, null value rate, and data pattern signature based on locality-sensitive hashing. Map all sampled fields of the source data table to the first point set in the source distribution morphology space, and map all sampled fields of the target data table to the second point set in the target distribution morphology space; Using the corresponding points of the first source field and the first target field in the first mapping pair in the first point set and the second point set, respectively, as registration anchor points, the spatial transformation function from the first point set to the second point set is calculated; the spatial transformation function is used to characterize the overall transformation of the data distribution pattern between the source data table and the target data table at the field level. The spatial transformation function is applied to the corresponding points of the other source fields in the first point set to obtain the predicted positions of each other source field in the target distribution morphology space. In the second set of points, the target field corresponding to the point closest to each of the predicted locations is found and used as the corresponding target field prediction for each of the other source fields, thus forming the second set of prediction mappings.
[0008] According to the technical solution provided in this application, the cross-validation and fusion of the first set of prediction mappings and the second set of prediction mappings includes the following steps: For each source field to be mapped, obtain the first candidate target field and its first confidence level output by the first mapping inference channel, and the second candidate target field and its second confidence level output by the second mapping inference channel; If the first candidate target field is consistent with the second candidate target field, then the target field is used as the recommended mapping target, and the harmonic mean of the first confidence and the second confidence is used as the fusion confidence. If there is a discrepancy, a local mapping conflict graph is constructed. The local mapping conflict graph uses the source fields in the source data table that have not yet determined the mapping relationship and the target fields in the target data table that have not yet been mapped as nodes, and the candidate mapping relationships given by the first mapping inference channel and the second mapping inference channel as weighted directed edges, with the weight of the edge being the confidence level of the corresponding channel. A graph cut algorithm based on confidence propagation is executed on the local mapping conflict graph to maximize global mapping consistency. The global optimal solution is calculated for the conflicting mapping relationships to obtain the final recommended mapping target for each source field to be mapped. If the graph cut algorithm fails to converge within a preset number of iterations, or if the maximum global mapping consistency score obtained is lower than a first preset threshold, the supplementary mapping strategy is triggered. The supplementary mapping strategy includes presenting the unresolved mapping relationships in a visually distinct manner in the configuration interface and waiting for the user's manual confirmation. The objective function of the graph cut algorithm simultaneously penalizes the cost of violating a single channel's high-confidence mapping, as well as the cost of conflicting many-to-one mappings between the source and target fields.
[0009] According to the technical solution provided in this application, before performing the graph cut algorithm based on confidence propagation on the local mapping conflict graph, the following steps are further included: For each directed edge in the local mapping conflict graph, obtain the candidate mapping relationship, including the source field and the target field involved in the candidate mapping relationship. Calculate the actual data content matching degree between the source field and the target field. The actual data content matching degree includes at least one of the following: the overlap rate between the value range of the source field and the value range of the target field, the morphological similarity between the data distribution of the source field and the data distribution of the target field, and the functional dependency strength between the source field and the target field based on the sampled data. A verification confidence correction factor is generated based on the actual data content matching degree. The verification confidence correction factor is then fused with the original confidence of the corresponding edge to obtain the post-verification confidence. The post-verification confidence score is used as the weight of the corresponding edge in the graph cut algorithm.
[0010] According to the technical solution provided in this application, the graph cut algorithm is executed iteratively: Perform a graph cut solution with the current weights of each edge to obtain the current optimal mapping allocation scheme, and calculate the corresponding global energy function value; For each mapping relationship adopted in the current optimal mapping allocation scheme, recalculate the actual data content matching degree between the corresponding source field and target field, and update its verification confidence correction factor and edge weight. The above steps are repeated based on the updated edge weights until the change in the global energy function value between two adjacent iterations is less than the second preset threshold, or the number of iterations reaches the preset maximum number of iterations. The current optimal mapping allocation scheme obtained in the last iteration is taken as the final recommended mapping target.
[0011] According to the technical solution provided in this application, the source data table is a set of fields for user-defined columns in a low-code platform, and the target data table is a set of fields obtained after parsing the response messages of an external system interface; Before extracting the field name transformation pattern in the first mapping inference channel, the following steps are also included: Obtain the column metadata configured in the low-code platform for each source field in the source data table. The column metadata includes at least one of the following: field display format configuration, field validation rule configuration, field interaction behavior configuration, and field enumeration option configuration. The data features of each target field in the target data table are obtained by sampling the response message sample, and include at least one of the following: the format mode of the field value, the set of values of the field value, and the numerical range of the field value. Perform semantic-level matching between the column metadata and the data features to determine at least one high-confidence anchor mapping pair; The high-confidence anchor mapping pairs are recommended to the user as the target of the first mapping operation.
[0012] According to the technical solution provided in this application, after batch adopting the mapping pairs in the fusion recommendation mapping set as formal data docking process configurations, the following steps are also included: In response to a change in the response message structure of the external system interface, a sample of the changed response message is obtained; Parse the modified response message sample and extract the modified target field set; For each adopted mapping pair in the fusion recommendation mapping set, calculate the semantic consistency score between the column metadata of its source field and the data features of the changed target field; If the semantic consistency score of a mapping pair is lower than the third preset threshold, the mapping pair is determined to be invalid, and a mapping failure warning is generated in the configuration interface. For invalid mapping pairs, semantic-level matching is performed again in the changed target field set based on the column metadata of their source fields to generate alternative recommended mapping pairs and push them to users.
[0013] According to the technical solution provided in this application, the source fields in the source data table are organized into a tree structure according to the hierarchical structure of the columns in the low-code platform; the parent node in the tree structure corresponds to the parent column, and the child node corresponds to the child column; Before extracting the field name transformation pattern in the first mapping inference channel, the following steps are also included: Obtain the first-level path of the first source field in the tree structure. The first-level path is formed by combining the names of the various level columns to which the first source field belongs in hierarchical order. Obtain the field name of the first target field in the target data table; Parse the field name of the first target field to identify whether it contains a level identifier that matches the first level path; the level identifier includes: the complete string, abbreviation string, or preset encoding mapping value of the name of each level column in the first level path; If a matching hierarchical identifier is identified, the hierarchical expansion transformation rules from the hierarchical path of the tree structure to the target field name are extracted. The hierarchical expansion transformation rules include: connectors between hierarchies, case conversion methods for each level column name, abbreviation mapping tables for each level column name, and methods for preserving or adjusting the hierarchical order.
[0014] According to the technical solution provided in this application, after extracting the hierarchical expansion transformation rules from the hierarchical path of the tree structure to the target field name, the method further includes the following steps: The hierarchical expansion transformation rule is applied to the hierarchical paths of other source fields in the source data table within the tree structure to generate the prediction target field names of each other source field, forming a third set of prediction mappings. Assign a hierarchical channel confidence score to each prediction result in the third set of prediction mappings. The hierarchical channel confidence score is determined based on the degree of matching between the field name of the first target field and the first hierarchical path. The third set of prediction mappings and their hierarchical channel confidence scores, together with the first set of prediction mappings and the second set of prediction mappings, participate in the cross-validation and fusion. The graph cut algorithm executed in cross-validation and fusion is configured such that the weights of the directed edges between nodes are determined by a weighted sum of the confidence levels of the candidate mapping relationships given by the first, second, and third mapping inference channels, so as to consider the prediction results from the three channels simultaneously during the global optimization solution process.
[0015] Compared with the prior art, the beneficial effects of this application are as follows: First, this application allows users to automatically launch at least two mapping inference channels in parallel by performing only one mapping operation (i.e., providing a first mapping pair) as an anchor point. The first channel performs predictive mapping of the target field for the remaining source fields in the source data table independently, based on the text transformation of field names and the global position transformation of the table structure. This approach frees users from the repetitive work of searching and matching one by one. With only a clear example, the system can understand and automatically complete the mapping inference of the vast majority of remaining fields, significantly reducing configuration complexity and operation time.
[0016] Second, this application effectively filters out erroneous or conflicting predictions that may arise from a single inference mechanism by cross-validating and fusing the prediction mapping results generated independently by the two channels. For example, when field names are similar but their structural positions do not match, or when their structural positions correspond but their field names differ significantly, the fusion mechanism can select the optimal result or trigger a supplementary strategy based on preset confidence rules, rather than simply adopting unreliable recommendations from a single dimension. This ensures that the final generated fused recommendation mapping set has high accuracy and reliability. Attached Figure Description
[0017] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 A flowchart illustrating the steps of the low-code platform data integration configuration method provided in this application. Detailed Implementation
[0018] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.
[0019] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.
[0020] As mentioned in the background section, this application proposes a low-code platform data integration process configuration method, such as... Figure 1 As shown, it includes the following steps: S1. In response to the first mapping operation performed by the user in the configuration interface of the low-code platform, obtain a first mapping pair; the first mapping pair includes a first source field in the source data table and a first target field in the target data table; S2. Using the first mapping pair as the anchor point, start multiple mapping inference channels in parallel, including at least the first mapping inference channel and the second mapping inference channel; in, The first mapping inference channel is used to extract the field name transformation pattern between the field name of the first source field and the field name of the first target field, and apply the field name transformation pattern to other source fields in the source data table to obtain the first set of prediction mappings; The second mapping inference channel is used to compare the first structural position feature of the first source field in the table structure of the source data table with the second structural position feature of the first target field in the table structure of the target data table when the number of fields in the source data table is greater than or equal to 2, identify the global table structure transformation pattern, and apply the global table structure transformation pattern to the other source fields to obtain the second set of prediction mappings. S3. Perform cross-validation and fusion on the first set of prediction mappings and the second set of prediction mappings to generate a fused recommended mapping set. Cross-validation and fusion include: for the same source field, if the two sets of prediction mapping results are consistent, then the result is adopted; if the two sets of prediction mapping results are inconsistent or a certain channel does not provide a prediction, then according to the preset confidence evaluation rules, select the prediction mapping with higher confidence or trigger the supplementary mapping strategy. S4. Display the fusion recommendation mapping set in batches in the configuration interface in a list format, and provide a one-click confirmation control; S5. In response to the user's triggering of the one-click confirmation control, the mapping pairs in the fusion recommendation mapping set are adopted in batches as formal data docking process configurations.
[0021] Specifically, the key terms involved in the solution are explained in detail first. A low-code platform refers to a software development platform that allows users to quickly build applications with little or no traditional code through a graphical user interface and visual configuration. In this platform, users can define data models and business logic by dragging and dropping components and configuring properties. The configuration interface is a visual interactive area in the low-code platform used to set data integration process rules. This area typically displays source and target data table fields in a tree or list format and provides interactive controls for establishing mapping relationships. The first mapping operation refers to the interactive action performed by the user in the configuration interface to explicitly specify the correspondence between a field in the source data table and a field in the target data table. Specific implementations of this operation include, but are not limited to: dragging and dropping the source field icon with the left mouse button and then releasing it; clicking the source and target fields sequentially; or selecting the target field through a right-click menu after selecting the source field. Regardless of the specific interaction method used by the user, the system will parse this operation into a specific field mapping pair, i.e., the first mapping pair. The first mapping pair consists of two elements: the first source field in the source data table and the corresponding first target field in the target data table.
[0022] The source data table refers to the data structure definition corresponding to the data source in the data flow process. In typical low-code platform application scenarios, the source data table is usually a set of fields created by users within the platform through custom column functions, such as a data structure named "Customer Information Table," which includes fields such as customer name, contact number, and region. The target data table refers to the data structure definition corresponding to the data destination in the data flow process. This structure usually comes from the application programming interface (API) of an external business system, and is the set of fields extracted after the platform parses its response message. For example, the list of fields parsed from the JSON data returned by the customer query interface of an external customer relationship management system may have field names such as custName, contactPhone, and regionCode. The mapping inference channel refers to an independently running software function module in the system that uses a specific algorithm model to predict and infer the mapping relationship of unknown fields. In this embodiment, the system includes at least two such channels, referred to as the first mapping inference channel and the second mapping inference channel. With the support of computer multi-threading or asynchronous task processing mechanisms, these two channels can be started in parallel, meaning that their execution processes basically overlap in time and do not block each other, thereby shortening the overall response waiting time. The fusion recommendation mapping set refers to a set of field mapping suggestions that the system ultimately determines and prepares to display to the user after a series of processes, including cross-comparison, consistency verification, conflict resolution, and confidence assessment, of multiple sets of predicted mappings generated by various mapping inference channels. Each mapping relationship in this set has been screened and confirmed by the system's internal intelligent algorithm, ensuring a high degree of accuracy.
[0023] Regarding the specific implementation steps, the execution process of this method is as follows. Step S1: The system continuously listens for user operation events in the configuration interface. When it detects that the user drags a field named user_real_name from the source data table to a field named fullName in the target data table and releases the mouse button, the system determines that a first mapping operation has occurred. In response to this operation, the system extracts the dragged source field identifier and the pointed-to target field identifier from the interface data model, constructs and stores a first mapping pair object, the data structure of which can be represented as {sourceField: user_real_name, targetField: fullName}.
[0024] In step S2, the system uses the first mapping pair obtained in step S1 as the anchor point for algorithm startup. The anchor point is the only known correct reference point for the entire automatic inference process. The system's main control module simultaneously sends startup commands to both the first and second mapping inference channels, passing the content of the first mapping pair as input parameters to both channels. Due to the parallel startup mechanism, the two channels will begin executing subsequent inference logic in their respective computational contexts, with their execution progress independent of each other.
[0025] Upon receiving the start command, the first mapping inference channel focuses its internal processing logic on analyzing the text attribute of the field name. The channel first obtains the field name string of the first source field, `user_real_name`, and the field name string of the first target field, `fullName`. By comparing and analyzing these two strings, the channel attempts to summarize the transformation pattern between the source and target field naming styles; this pattern is defined as the field name transformation pattern. For example, the channel might identify that the source field name uses a lowercase letter-separated naming convention, while the target field name uses a camelCase naming convention with the first letter capitalized, and that certain words in the source field name, such as `user`, are omitted or replaced. After identifying this pattern, the channel assumes that all other source fields to be mapped in the source data table follow the same transformation pattern when converted to the target field. Subsequently, the channel iterates through every other source field in the source data table except the first source field, such as the user_phone_number field, and applies the aforementioned field name transformation pattern to generate a predicted target field name string, such as phoneNumber. Then, it searches the actual field list of the target data table to see if there exists a field that exactly matches or is highly similar to the predicted field name. If it does, the pairing of the source field and the target field is taken as a prediction mapping, and all prediction mappings constitute the first set of prediction mappings.
[0026] After receiving the start command, the second mapping inference channel focuses its internal processing logic on the structural level of the data table. This channel analyzes the positional characteristics of the first source field, `user_real_name`, within the overall architecture of its source data table. For example, it considers the field's position within the table, the data types of its adjacent fields, and whether it exists within a nested structure or field group. This positional characteristic is defined as the first structural positional characteristic. Similarly, the channel extracts the second structural positional characteristic from the first target field, `fullName`, within the target data table's structure. By comparing the first and second structural positional characteristics, the channel attempts to discover potential overall structural reorganization patterns between the source and target data tables. This pattern is defined as a global table structure transformation pattern. For example, the channel might find that address-related fields scattered across multiple locations in the source data table are consolidated into a single composite object named `address` in the target data table. After identifying this global transformation pattern, the channel applies this pattern logic to other source fields in the source data table, predicting their possible positions or corresponding fields in the target data table after undergoing the same structural reorganization transformation, and forming a second set of predictive mappings accordingly.
[0027] Step S3: After both the first and second mapping inference channels complete their respective prediction tasks, or if one of the channels terminates prematurely and returns an empty result due to unmet data conditions, the system's main control module collects the first and second sets of prediction mappings and enters the cross-validation and fusion processing stage. The core logic of this stage is to check which target field each source field to be mapped in the source data table is assigned to in each of the two sets of prediction mappings. If both sets of prediction mappings give completely identical target field prediction results for the same source field, the system determines that the mapping relationship is mutually corroborated on two independent inference dimensions, and its credibility is extremely high, directly incorporating the mapping relationship into the fusion recommendation mapping set. If the two sets of prediction mappings give different target field prediction results, or if one set of prediction mappings does not give any prediction for the source field (e.g., the first channel leaves it empty due to inability to match the naming rule, while the second channel gives a prediction), the system will call a pre-set confidence evaluation rule. This rule will calculate the confidence score assigned to the first and second channels for their respective prediction results. The calculation method for the confidence score will be detailed later; its essence is to quantify the probability of the prediction being correct. The system compares the two confidence scores and selects the prediction mapping with the higher confidence score as the recommendation result. If both confidence scores are below a preset minimum confidence threshold, or if the comparison result is in an ambiguous range where it is difficult to decide, the system triggers a supplementary mapping strategy. Specifically, this strategy marks these field mapping relationships that cannot be automatically determined as pending, leaving them for the user to make a final decision manually in subsequent steps.
[0028] In step S4, after cross-validation and fusion processing, the system organizes the confirmed recommended mappings and the mappings marked as pending into a unified fused recommended mapping set. The system dynamically generates a mapping recommendation list view in the configuration interface, displaying the contents of this set in a visually intuitive table format. Each row of the list corresponds to a source field, typically displaying the source field name, a directional indicator icon, the name of the recommended or pending target field, and a visual identifier reflecting the credibility of the mapping relationship, such as different colored labels or star icons with different fill levels. Below this list view, the system renders a button control. The text label of this control can be set to "Confirm All Recommendations with One Click" or similar prompt text, i.e., a one-click confirmation control.
[0029] In step S5, after viewing the recommendation list, the user can quickly check whether the mapping relationships in each row are correct. Since most mappings have been accurately inferred by the system, the user usually only needs to focus on a few items marked as pending. After confirming that everything is correct, the user executes the trigger operation by clicking the one-click confirmation control. The system responds to the click event of this control and executes the batch adoption program. This program will traverse each mapping pair in the fusion recommendation mapping set and formally write the correspondence between its source field identifier and target field identifier into the configuration storage area of the data docking process, such as saving it to the platform's process definition database table. Subsequently, when the data docking process is actually executed, the data flow engine will automatically extract the values of each field in the source data table according to this formal configuration and accurately fill them into the corresponding fields in the target data table.
[0030] The technical principle behind this embodiment lies in treating a single mapping operation performed by the user as a key to revealing the complex mapping patterns between the source and target tables. By launching two independent inference channels based on different dimensions of information in parallel, the system can simultaneously capture and model this pattern from both the textual level of field names and the positional level of table structures. The first channel excels at handling surface feature transformations such as naming convention conversions, while the second channel excels at handling deep layout transformations such as structural reorganization. The two complement each other. Subsequent cross-validation and fusion steps act as referees and arbitrators, using the mutual verification of the results from the two channels to improve recommendation accuracy and making rational decisions based on confidence levels when contradictions arise. The resulting technical effect is that the N manual pairing operations that users originally needed to perform on N fields are simplified to only one exemplary operation plus one batch confirmation operation. This change in working mode significantly reduces the configuration complexity of data integration tasks on low-code platforms, greatly shortens the time required for users to complete configuration, and effectively reduces the probability of mapping configuration errors caused by visual fatigue or negligence by reducing the manual item-by-item comparison step.
[0031] In a preferred embodiment, the first mapping inference channel includes: The field names of the first source field and the first target field are aligned to identify the common and different parts. At least two candidate transformation rules are generated based on the differences, including: prefix replacement rule, suffix replacement rule, character case conversion rule, or delimiter conversion rule; Each candidate transformation rule is applied to all field names in the source data table, and the matching success rate with the actual field names in the target data table is calculated. If, for any source field name, the predicted field name obtained after applying the candidate transformation rule exists in the target data table with an edit distance less than a preset distance, it is determined to be a successful match. Select the candidate transformation rule with the highest matching success rate as the field name transformation mode.
[0032] Specifically, string alignment refers to an algorithmic process that calculates the similarity correspondence between two strings, aiming to find the most closely matched common character sequences in the two strings. In the application scenario of this embodiment, the alignment operation targets the field names of the first source field and the first target field. For example, for the first source field name src_customer_name and the first target field name customerName, the alignment algorithm will identify that character fragments such as customer and name exist in both, while the src_ prefix and underscore separator are the differences. The common part refers to the set of consecutive characters or character fragments that are completely identical and in the same order in the two field names after the alignment algorithm. The difference part refers to the different character sequences or format symbols remaining in the two field names after removing the common part. Candidate transformation rules refer to a series of string processing hypotheses automatically generated by the system based on the difference parts. Each hypothesis represents a possible way to edit or transform field names. The specific types of candidate transformation rules include, but are not limited to, the following: Prefix replacement rules, which describe how to replace a specific character sequence at the beginning of the source field name with another character sequence. For example, replacing src_ with an empty string achieves prefix removal. The rules include: Suffix replacement rules, which describe how to replace a specific sequence of characters at the end of a source field name with another sequence, such as replacing _name with Name; Case conversion rules, which describe how to change the case of letters in a string, such as converting all lowercase letters to camelCase with the first letter of each word capitalized; and Delimiter conversion rules, which describe how to handle or replace special symbols used to connect words in field names, such as removing the underscore character _ and converting the following letter to uppercase. Match success rate is a quantitative evaluation metric used to measure the universality and correctness of a candidate transformation rule across the entire source data table. The specific calculation method is as follows: apply the candidate transformation rule to the field names of every other source field in the source data table except the first source field to obtain the corresponding predicted field names. Then, search the list of actual field names in the target data table to see if there is an item that matches the predicted field name. Finally, divide the number of successfully matched source fields by the total number of other source fields involved in the application; the resulting ratio is the match success rate. Edit distance is a commonly used algorithmic metric for measuring the similarity between two strings. It is precisely defined as the minimum number of single-character edit operations required to transform one string into another. Single-character edit operations include inserting a character at any position, deleting a character, or replacing one character with another. The preset distance is an integer threshold value that can be configured by the system administrator or developer based on the data standardization requirements of the actual application scenario; for example, it can be set to 2 in this embodiment.When the edit distance between the predicted field name and an actual field name in the target data table is less than or equal to the preset distance, the two can be considered to be successfully matched.
[0033] In terms of specific implementation steps, the internal workflow of the first mapping inference channel can be divided into the following five stages.
[0034] The first step involves performing field name string alignment analysis. The system takes the field names of the first source field and the first target field as input and calls the longest common subsequence solver based on a dynamic programming algorithm. This solver scans the two strings, generates an alignment matrix, and backtracks to obtain the common and differing parts. For example, if the first source field name is `order_created_date` and the first target field name is `createdAt`, the common part after alignment can be identified as the semantic association between `create`, `date`, and `At`. The differing parts include the prefix `order_`, the underscore separator, the full spelling of `date`, and the camelCase concatenation and the abbreviation `At` on the target side.
[0035] The second step involves generating multiple candidate transformation rules. The system has a built-in rule generation template library covering common naming convention conversion scenarios. The system uses the differences identified in the first step as parameters and attempts to instantiate each rule in the template library. For example, the system might generate rule R1: remove all underscores and capitalize the first letter after each underscore; rule R2: remove all characters before the first underscore in the field name as a prefix removal operation; and rule R3: replace the complete word "date" with its abbreviation "Dt". The purpose of generating multiple candidate rules at once is to form a hypothesis set, because from a single mapping pair example, the system cannot uniquely determine which rule(s) or combination thereof will be effective globally.
[0036] The third step involves calculating the matching success rate of each candidate transformation rule. The system iterates through all source fields in the source data table except for the first source field. For each other source field, the system obtains its field name string and applies the currently evaluated candidate transformation rule to obtain a predicted field name. Subsequently, the system iterates through the list of actual field names in the target data table, calculating the edit distance between the predicted field name and each actual field name. The system uses the standard Levenshtein distance algorithm for this calculation. If, during the iteration, the system finds an actual field name whose edit distance is less than or equal to a preset distance of 2, the system determines that the current other source field has successfully matched under this rule and immediately stops iterating through the target field list, moving on to the next other source field. If no item satisfying the edit distance condition is found after iterating through the entire target field list, the matching is considered a failure. After all other source fields have been processed, the system counts the number of successfully matched source fields and divides it by the total number of other source fields participating in the matching; the quotient is the matching success rate of the candidate transformation rule. The system temporarily stores each rule and its corresponding success rate in memory.
[0037] The fourth step involves selecting the optimal transformation mode. The system sorts all candidate transformation rules generated in the third step in descending order of their matching success rate. If the matching success rate of the rule ranked first is significantly higher than that of the rule ranked second (e.g., the first rule has a success rate of 92% while the second only has 45%), the system directly selects the first rule as the final field name transformation mode. If multiple rules have the same or very similar matching success rates, the system can determine the winner based on a preset priority strategy, such as prioritizing rules with lower complexity or fewer character deletion operations.
[0038] In the fifth step, the system applies a transformation pattern to generate the first set of prediction maps. The system formally applies the optimal field name transformation pattern selected in the fourth step to each other source field in the source data table, generating its corresponding final predicted target field name. Then, using this final predicted target field name as a benchmark, the system searches for the best-matching actual field in the target data table's actual field list by calculating the edit distance, and pairs this source field with the matched target field. All such generated mapping pairs constitute the first set of prediction maps. For the very few source fields that, even after applying the transformation pattern, cannot find any actual field in the target data table that meets the edit distance requirement, the system returns a null value or a specific unmatched identifier in this set of prediction maps.
[0039] In a preferred embodiment, the second mapping inference channel includes: Obtain data samples from at least two fields in the source data table and data samples from at least two fields in the target data table; For each sampled field, extract its numerical distribution morphological feature vector, which includes a combination of at least two of the following: statistical distribution of data type, quantile profile, unique value density, null value rate, and data pattern signature based on locality-sensitive hashing. Map all sampled fields of the source data table to the first point set in the source distribution morphology space, and map all sampled fields of the target data table to the second point set in the target distribution morphology space; Using the corresponding points of the first source field and the first target field in the first mapping pair in the first point set and the second point set, respectively, as registration anchor points, the spatial transformation function from the first point set to the second point set is calculated; the spatial transformation function is used to characterize the overall transformation of the data distribution pattern between the source data table and the target data table at the field level. The spatial transformation function is applied to the corresponding points of the other source fields in the first point set to obtain the predicted positions of each other source field in the target distribution morphology space. In the second set of points, the target field corresponding to the point closest to each of the predicted locations is found and used as the corresponding target field prediction for each of the other source fields, thus forming the second set of prediction mappings.
[0040] Specifically, the execution of the second mapping inference channel must adhere to an important precondition. Before the channel officially starts its core computation, the system first checks the number of fields actually available for sampling in the source and target data tables. Specifically, the system requires that the source data table contain at least one additional source field besides the first source field, and the target data table contain at least one additional target field besides the first target field. In other words, the total number of fields participating in the spatial transformation calculation must be at least two for each. If this condition is not met, for example, if the source data table has only one field, the second mapping inference channel will terminate its operation directly and return an empty prediction set or an execution failure flag to the main control module because a spatial point set with sufficient constraints cannot be constructed. Under the condition that this precondition is met, the internal workflow of the second mapping inference channel can be divided into the following six steps.
[0041] The first step is to perform data sampling. The system connects to the data sources containing both the source and target data tables, and executes data query statements to obtain sample data. To balance computational accuracy and performance overhead, the number of sampled rows can be configured to the default value of one thousand rows. The sampled fields are all fields in either the source or target data table. The sampling results are stored in memory in a columnar storage structure for subsequent processing.
[0042] The second step is to extract the numerical distribution feature vector for each field. The system initiates a feature extraction task for each sampled field. Taking a field in the source data table as an example, the system iterates through all data values in its sample column. First, it performs statistical distribution calculations on the data type, recording the frequency of string occurrences, numerical occurrences, and null value occurrences, and normalizes these to a proportion. Second, if the field is determined to be numeric overall, the non-null values are sorted, and their minimum, 25th percentile, median, 75th percentile, and maximum value are calculated. Third, the number of unique values in the field samples is calculated and divided by the total number of valid samples to obtain the unique value density. Subsequently, the number of null values is calculated and divided by the total number of samples to obtain the null value rate. Finally, for string values, a locality-sensitive hash function is called to calculate their pattern signature. The system concatenates all the above calculation results into a fixed-length floating-point array, which is the numerical distribution feature vector for the field. This process is repeated for all sampled source and target fields.
[0043] The third step is to construct a point set in the distribution morphology space. The system treats all source field feature vectors generated in the second step as coordinate values in a high-dimensional coordinate system, thus drawing the first point set in the source distribution morphology space. Similarly, all target field feature vectors are drawn as the second point set in the target distribution morphology space. Internally, the system maintains a mapping relationship between each point and the original field identifier.
[0044] The fourth step is to calculate the spatial transformation function. The system extracts the point corresponding to the first source field from the first point set, denoted as point A; and extracts the point corresponding to the first target field from the second point set, denoted as point B. Points A and B form a registration anchor pair. The system calls the spatial transformation function solver to calculate the mapping function from the space containing the first point set to the space containing the second point set. Depending on different assumptions about the transformation complexity, the solver can use different mathematical models. For example, if it is assumed that the transformation is mainly linear, an affine transformation based on the least squares method can be used; if it is assumed that there is local nonlinear distortion, a thin-plate spline interpolation function can be used. The optimization goal of the solver is to find a set of transformation parameters such that point A can be accurately mapped to the position of point B after the transformation, while the sum of the distances between the positions of other points in the first point set and their intuitively corresponding points in the second point set after the same transformation is minimized. The obtained spatial transformation function is saved in the form of a parameter matrix.
[0045] The fifth step is to predict the positions of other source fields in the target space. For each source field in the source data table other than the first source field, the system first finds its corresponding point in the first point set in memory, and then inputs its coordinates into the spatial transformation function obtained in the fourth step. Through matrix operations, a new coordinate is obtained, which is the predicted position of the source field in the target distribution morphology space.
[0046] Step 6: Find the nearest neighbor target field. In the second point set, the system finds the nearest point to each predicted location calculated in step 5. The distance metric can be Euclidean distance or cosine similarity. To improve search efficiency in high-dimensional space, the system can pre-build a KD-tree or ball tree index structure for the second point set. For each predicted location, the system quickly returns one or K nearest points through the index. The system selects the nearest point and, through the mapping relationship between the point and the field identifier, retrieves the actual field in the target data table corresponding to that point. This target field is then predicted as the corresponding target field for the other source fields. All such prediction results are aggregated to form the second set of prediction mappings.
[0047] In a preferred embodiment, the cross-validation and fusion of the first set of prediction mappings and the second set of prediction mappings includes the following steps: For each source field to be mapped, obtain the first candidate target field and its first confidence level output by the first mapping inference channel, and the second candidate target field and its second confidence level output by the second mapping inference channel; If the first candidate target field is consistent with the second candidate target field, then the target field is used as the recommended mapping target, and the harmonic mean of the first confidence and the second confidence is used as the fusion confidence. If there is a discrepancy, a local mapping conflict graph is constructed. The local mapping conflict graph uses the source fields in the source data table that have not yet determined the mapping relationship and the target fields in the target data table that have not yet been mapped as nodes, and the candidate mapping relationships given by the first mapping inference channel and the second mapping inference channel as weighted directed edges, with the weight of the edge being the confidence level of the corresponding channel. A graph cut algorithm based on confidence propagation is executed on the local mapping conflict graph to maximize global mapping consistency. The global optimal solution is calculated for the conflicting mapping relationships to obtain the final recommended mapping target for each source field to be mapped. If the graph cut algorithm fails to converge within a preset number of iterations, or if the maximum global mapping consistency score obtained is lower than a first preset threshold, the supplementary mapping strategy is triggered. The supplementary mapping strategy includes presenting the unresolved mapping relationships in a visually distinct manner in the configuration interface and waiting for the user's manual confirmation. The objective function of the graph cut algorithm simultaneously penalizes the cost of violating a single channel's high-confidence mapping, as well as the cost of conflicting many-to-one mappings between the source and target fields.
[0048] In terms of specific implementation steps, the detailed process of cross-validation and fusion includes the following six steps.
[0049] Initialization and Information Gathering. The system iterates through each source field to be mapped in the source data table. For the currently iterated source field, the system queries the first set of prediction mappings and the second set of prediction mappings. If the first set of prediction mappings contains a prediction for the source field, the target field identifier is extracted as the first candidate target field, and its associated confidence score is extracted as the first confidence score. If the first set of prediction mappings does not contain a prediction for the source field, both the first candidate target field and the first confidence score are marked as invalid. The same query operation is performed on the second set of prediction mappings to obtain the second candidate target field and the second confidence score.
[0050] In cases of consistency, the system compares the first and second candidate target fields to see if they are valid and identical. If both are valid and have the same identifier, it indicates that the two independent inference channels have reached a consensus on the mapping relationship of the source field. In this case, the system will directly adopt the consistent target field as the final recommended mapping target for the source field. Simultaneously, the system calculates the harmonic mean of the first and second confidence levels and uses this harmonic mean as the fused confidence level for this mapping relationship, which will be provided to the user for reference in the user interface later. After processing the source field, the system marks it as determined and marks its corresponding target field as occupied in the available field pool of the target data table to prevent conflicts in subsequent mapping allocations.
[0051] A local mapping conflict graph is constructed. After the system has traversed all source fields to be mapped, it filters out those source fields that were not marked as determined in step S402, i.e., source fields where candidate target fields are inconsistent or a certain channel has not provided a prediction. These source fields constitute the left-hand node set of the conflict graph. The system also filters out target fields in the target data table that are not occupied by any determined mapping, forming the right-hand node set of the conflict graph. Subsequently, for each undetermined source field, the system checks whether the first mapping inference channel has provided a valid first candidate target field. If so, a directed edge is drawn from the source field node to the corresponding first candidate target field node, and the edge weight is initialized to the first confidence level. Similarly, it checks whether the second mapping inference channel has provided a valid second candidate target field. If so, another directed edge is drawn, and the weight is initialized to the second confidence level. Through this process, a complete local mapping conflict graph is constructed in memory.
[0052] The graph cut algorithm is executed to solve the problem. The system calls a pre-implemented graph cut algorithm solver, taking the locally mapped conflict graph as input. The solver's objective function is designed to minimize a global energy function, which includes two penalty terms. The first penalty term corresponds to the cost of violating a single-channel high-confidence mapping. Mathematically, it is added to the energy function by a penalty value proportional to C if the final solution does not adopt an edge with confidence C. The second penalty term corresponds to the conflict cost of many-to-one mappings between the source and target fields. Mathematically, it is applied by imposing a large penalty constant if the final solution maps two or more source field nodes to the same target field node, thus preventing such solutions from appearing in the actual solution process. The graph cut algorithm iteratively calculates the minimum cut of the graph and propagates and updates node confidence information, gradually converging to a mapping allocation scheme that minimizes the global energy function value. This scheme explicitly specifies which target field node each source field node in the conflict graph should correspond to.
[0053] The system evaluates the solution results and triggers a supplementary mapping strategy. After the graph cut algorithm completes its solution, the system obtains its final global energy function value and converts it into a positive global mapping consistency score, for example, by taking the reciprocal or normalizing it after subtracting from a benchmark value. The system compares this score with a pre-configured first preset threshold. If the score is greater than or equal to the first preset threshold, it indicates that the algorithm has found a globally optimal solution with high credibility, and the system adopts the target field specified for each source field in this solution as the final recommended mapping target. If the algorithm fails to converge after reaching the maximum number of iterations, or if it converges but the global mapping consistency score is lower than the first preset threshold, it indicates that the complexity of the current mapping conflict has exceeded the reliable range of automatic processing by the algorithm. At this time, the system triggers a supplementary mapping strategy. The system records the identifiers of all source fields involved in the conflict graph and applies special visual styles to the entries corresponding to these fields when displaying the list. The system also records the pending state of these fields and suspends their automatic processing in the background.
[0054] Output Results. The system will summarize the adopted mapping relationships, the mapping relationships successfully solved by the graph cut algorithm, and the mapping relationships marked as pending due to the triggering of the supplementary mapping strategy, and form the final cross-validation and fusion processing results, which will be passed to the main control module to generate the fusion recommendation mapping set.
[0055] In a preferred embodiment, before performing the confidence-based graph cut algorithm on the locally mapped conflict graph, the method further includes the following steps: For each directed edge in the local mapping conflict graph, obtain the candidate mapping relationship, including the source field and the target field involved in the candidate mapping relationship. Calculate the actual data content matching degree between the source field and the target field. The actual data content matching degree includes at least one of the following: the overlap rate between the value range of the source field and the value range of the target field, the morphological similarity between the data distribution of the source field and the data distribution of the target field, and the functional dependency strength between the source field and the target field based on the sampled data. A verification confidence correction factor is generated based on the actual data content matching degree. The verification confidence correction factor is then fused with the original confidence of the corresponding edge to obtain the post-verification confidence. The post-verification confidence score is used as the weight of the corresponding edge in the graph cut algorithm.
[0056] Specifically, the first step is for the system to traverse every directed edge that has been generated in the local mapping conflict graph. Each directed edge represents a specific candidate mapping relationship, that is, a source field is considered by a mapping inference channel to point to a target field. The system extracts the source field identifier and target field identifier involved from the data structure of the directed edge.
[0057] The second step involves the system calculating the actual data content matching degree between each pair of source and target fields. This actual data content matching degree is not a single indicator, but a comprehensive evaluation comprised of at least one of the following three dimensions. The first dimension is the overlap rate between the source and target field value ranges. The system first obtains the set of all non-empty unique values of the source field in the data sample, denoted as set S; similarly, it obtains the set of all non-empty unique values of the target field in the data sample, denoted as set T. Then, it calculates the intersection of set S and set T divided by the size of set S; the resulting ratio is the overlap rate. The higher this ratio, the greater the proportion of data values in the source field appearing in the target field, and the more likely there is a correspondence in their content. The second dimension is the morphological similarity between the source and target field data distributions. The system calculates the morphological feature vectors of the numerical distributions of the source and target field sample data respectively, using the same calculation method as the feature vector extraction method. After obtaining the two feature vectors, the system calculates the cosine similarity or Pearson correlation coefficient between them as the quantified value of the morphological similarity. The closer this value is to 1, the more similar the data distribution patterns of the two fields. The third dimension is the strength of the functional dependency between the source and target fields based on the sampled data. The system aligns the source and target field sample data by row; that is, for the same row of sample data records, it takes the source field value and the target field value to form a value pair. The system checks whether a functional dependency exists, that is, whether a given source field value always corresponds to a unique target field value. The system calculates the degree of determination of the source field value on the target field value, that is, the ratio of the number of unique values in the source field to the number of unique values in the combination of the source and target fields. The closer this ratio is to 1, the stronger the functional dependency and the closer the association between the two fields.
[0058] Third, the system generates a verification confidence correction factor based on the actual data content matching degree calculated in the second step. This correction factor can be calculated by taking the weighted average of the scores from the three dimensions, or by taking the maximum or minimum value among them; the specific strategy can be pre-configured according to the application scenario. The value range of the correction factor is usually normalized to between 0 and 1. Subsequently, the system merges this verification confidence correction factor with the original confidence of the directed edge. The fusion method can be multiplying the two, or taking their arithmetic mean or harmonic mean. The value obtained after fusion is the post-verification confidence. For example, the original confidence of an edge from the first mapping inference channel was 0.9, but after verification with actual data content, it was found that the source field value range and the target field value range had almost no overlap, and the verification confidence correction factor was only 0.2. After multiplying the two, the post-verification confidence decreased to 0.18, more realistically reflecting the fact that the mapping relationship is actually unreliable.
[0059] Fourth, the system reassigns the calculated post-verification confidence score to the directed edge as its new weight value. After all directed edges in the locally mapped conflict graph have undergone the above confidence score correction process, the system uses this updated conflict graph as input and passes it to the graph cut algorithm for subsequent global optimal solution calculation.
[0060] In a preferred embodiment, the graph cut algorithm is executed iteratively: Perform a graph cut solution with the current weights of each edge to obtain the current optimal mapping allocation scheme, and calculate the corresponding global energy function value; For each mapping relationship adopted in the current optimal mapping allocation scheme, recalculate the actual data content matching degree between the corresponding source field and target field, and update its verification confidence correction factor and edge weight. The above steps are repeated based on the updated edge weights until the change in the global energy function value between two adjacent iterations is less than the second preset threshold, or the number of iterations reaches the preset maximum number of iterations. The current optimal mapping allocation scheme obtained in the last iteration is taken as the final recommended mapping target.
[0061] Specifically, in the initial step of the first iteration, the system performs a standard graph cut solution with the current edge weights. The current edge weights in this first iteration are the final verified confidence scores generated in the previous steps. The objective function of the graph cut solver simultaneously penalizes the cost of violating high-confidence mappings and the conflict cost of many-to-one mappings. After internal calculation, the solver outputs a current optimal mapping allocation scheme. This scheme explicitly indicates which target field node in the local mapping conflict graph should establish a final mapping relationship with each source field node under the current edge weights. Simultaneously, the solver calculates and returns the global energy function value corresponding to this scheme. This energy function value is a quantitative measure of the overall quality of the current allocation scheme; a lower value indicates a better scheme.
[0062] In the second step of the first iteration, the system performs post-validation and weight updates on the current optimal mapping allocation scheme obtained in the first step. The system iterates through each adopted mapping relationship in the scheme. For each adopted mapping relationship, the system re-acquires the source and target fields involved and recalculates the actual data content matching degree between the source and target fields using the same method as before. The purpose of this recalculation is to utilize the new information from the preliminarily determined mapping relationships to perform a more refined evaluation of the data content matching degree. For example, biases that may arise from sample randomness during the initial calculation can be more thoroughly verified after the mapping relationship is confirmed as a formal mapping relationship. Based on the recalculated actual data content matching degree, the system generates a new validation confidence correction factor and updates the weight of the directed edge in the conflict graph using this new factor. For those directed edges that were not adopted in the current optimal mapping allocation scheme, their weights can remain unchanged or be reduced proportionally.
[0063] After the first iteration, the system compares the global energy function value calculated in this iteration with that of the previous iteration. In the first iteration, the value of the previous iteration can be set to infinity or a very large initial value. The system calculates the change in the global energy function value between two adjacent iterations, i.e., taking the absolute value of the difference. The system compares this change with a pre-set second threshold. The second threshold is a very small positive number, such as 0.01, used to determine whether the algorithm has converged to a stable state. If the change is greater than or equal to the second threshold, it indicates that the current iteration still shows a significant improvement over the previous one, and the system will continue to execute the next iteration. The next iteration will use the updated edge weights as input and re-execute the graph cut solution from the first step. If the change is less than the second threshold, it indicates that the solutions obtained in the two consecutive iterations are basically consistent, the energy function value tends to stabilize, and the algorithm has converged. At this point, the system terminates the iteration loop. Furthermore, to prevent the algorithm from getting stuck in an infinite loop in extreme cases, the system also checks whether the current iteration count has reached the preset maximum iteration count, such as 20 iterations. If the maximum number of iterations is reached, the system will forcibly terminate the iteration regardless of whether it has converged.
[0064] After the iteration loop terminates, the system outputs the current optimal mapping allocation scheme calculated in the last iteration as the final recommended mapping target, which will be used by subsequent steps to generate the fused recommended mapping set.
[0065] In a preferred embodiment, the source data table is a set of fields for user-defined columns in a low-code platform, and the target data table is a set of fields obtained after parsing the response messages of an external system interface; Before extracting the field name transformation pattern in the first mapping inference channel, the following steps are also included: Obtain the column metadata configured in the low-code platform for each source field in the source data table. The column metadata includes at least one of the following: field display format configuration, field validation rule configuration, field interaction behavior configuration, and field enumeration option configuration. The data features of each target field in the target data table are obtained by sampling the response message sample, and include at least one of the following: the format mode of the field value, the set of values of the field value, and the numerical range of the field value. Perform semantic-level matching between the column metadata and the data features to determine at least one high-confidence anchor mapping pair; The high-confidence anchor mapping pairs are recommended to the user as the target of the first mapping operation.
[0066] Specifically, the application scenario of this embodiment is first clarified. The source data table specifically refers to the collection of business columns defined by the user through the visual column designer in the low-code platform. For example, when a user builds a customer management application on the platform, they will create columns such as customer name, contact number, email address, and industry. The field definitions of these columns constitute the source data table. The target data table specifically refers to the collection of fields obtained after the platform parses the response message of an external business system interface. For example, the platform needs to parse the JSON format message returned by the customer query interface of a third-party customer relationship management system and extract the list of fields as the target data table.
[0067] In terms of specific implementation steps, before the first mapping inference channel begins to extract the field name transformation pattern, the system will execute a preliminary anchor point recommendation process, which includes the following steps.
[0068] Retrieve category metadata from the source data table. When designing categories, low-code platforms typically allow users to configure richer metadata information in addition to defining field names and data types. The system reads the configured category metadata for each source field in the source data table by calling the platform's internal metadata interface. This metadata specifically includes at least one of the following: Field display format configuration, such as a date field being set to display in YYYY year MM month DD day format, and a numeric field being set to display thousands separators and retain two decimal places. Field validation rule configuration, such as a mobile phone number field being set to require an 11-digit regular expression, and an email address field being set to require the at symbol and period. Field interaction behavior configuration, such as whether the field is required in the form, whether it is read-only, and whether it has a default value. Field enumeration option configuration, such as an industry field being configured with a dropdown list of options, including information technology, finance, manufacturing, etc. This category metadata contains the user's explicit definition of the business meaning of the field and is a valuable clue for semantic understanding.
[0069] The system acquires the data characteristics of the target data table. Since the target data table originates from an external system interface, the platform typically cannot access its design metadata. Instead, the system uses data sampling analysis to infer its field characteristics. The system initiates test calls to the external interface containing the target data table, obtaining several response message samples, such as 10 or 20. These message samples are then parsed to extract the actual data value sample set for each target field in the target data table. For each target field's data sample, the system extracts its data characteristics, specifically including at least one of the following: Field value format pattern: The system uses regular expression matching to determine which format the field's values generally conform to, such as mobile phone number format, email format, date and time format, ID card number format, etc. The set of possible field values: For all unique values appearing in the sample, the system compiles them into a set, paying particular attention to whether the set presents a finite enumeration of values. The numerical range of the field values: For numeric fields, the system records their minimum and maximum values in the sample. These data characteristics reflect the business type and constraints of the target field at the content level.
[0070] Semantic-level matching is performed. The system compares and correlates the acquired category metadata with data features. This semantic-level matching is not a simple string comparison, but rather inference based on a preset matching rule base. The matching rule base defines the correspondence between metadata and data features. For example, rule one: if the validation rule of the source field is configured as a regular expression for a mobile phone number, and the data format pattern of the target field is also recognized as a mobile phone number format, then the two constitute a high semantic match. Rule two: if the enumeration options of the source field are configured to include options A, B, and C, and the value set of the target field, after deduplication, is exactly A, B, C, or their synonyms, then the two constitute a high semantic match. Rule three: if the display format of the source field is configured as a date format, and the data format pattern of the target field is recognized as a date format, then the two constitute a high semantic match. The system iterates through all source fields and all target fields, calculates the semantic matching score for each pair of combinations, and those with scores exceeding a preset threshold are identified as high-confidence anchor mapping pairs.
[0071] Recommended Anchor Mapping Pairs. The system will visually recommend one or more high-confidence anchor mapping pairs to the user. For example, at the top or sidebar of the configuration interface, the system may display a prompt: "The system has detected a possible correspondence between the source field 'phone number' and the target field 'contactPhone'. Click here to quickly establish a mapping." The user simply clicks on this recommended entry, and the system will use it as the first mapping operation target, automatically establishing the first mapping pair, thus saving the user the tedious process of manually searching for matching fields from a lengthy list. If the system recommends multiple high-confidence anchor mapping pairs, the user can choose one as the anchor to initiate the subsequent automatic inference process. If the user does not adopt the system's recommendation but manually establishes a first mapping pair, the system can still execute the subsequent steps described above normally; this pre-recommendation process exists only as an enhanced auxiliary function.
[0072] In a preferred embodiment, after batch adopting the mapping pairs in the fusion recommendation mapping set as formal data integration process configurations, the following steps are further included: In response to a change in the response message structure of the external system interface, a sample of the changed response message is obtained; Parse the modified response message sample and extract the modified target field set; For each adopted mapping pair in the fusion recommendation mapping set, calculate the semantic consistency score between the column metadata of its source field and the data features of the changed target field; If the semantic consistency score of a mapping pair is lower than the third preset threshold, the mapping pair is determined to be invalid, and a mapping failure warning is generated in the configuration interface. For invalid mapping pairs, semantic-level matching is performed again in the changed target field set based on the column metadata of their source fields to generate alternative recommended mapping pairs and push them to users.
[0073] Specifically, this embodiment extends the definition of the operation and maintenance phase after the data integration process is configured and launched. The core objective of this embodiment is to address the issue that some or all of the originally configured mapping relationships become invalid when the response message structure of external system interfaces changes due to version upgrades or changes in requirements. It provides an automated mapping failure early warning and alternative recommendation mechanism to ensure the long-term stable operation of the data integration process.
[0074] In terms of specific implementation, the processing flow described in this embodiment occurs at a certain point in time after the one-click confirmation control is triggered and the fused recommendation mapping set is adopted in batches as the official configuration. This flow can be triggered periodically by a scheduled task or by an interface change notification event actively pushed by an external system. The specific steps of the flow are as follows.
[0075] In response to changes in the response message structure of external system interfaces, the system retrieves a sample of the modified response message. The system can detect changes in several ways. One method is that the system periodically, for example, every morning, initiates a test call to the configured external system interface to obtain the latest response message and compares it with a previously stored snapshot of the message structure. If the structure has been added, deleted, or altered, a change is determined. Another method is that the external system provides a subscription notification mechanism for interface changes, which is passively triggered by the platform upon receiving the notification. Regardless of the triggering method, the system will obtain a sample of the latest modified response message as a basis for analysis.
[0076] The system parses the modified response message sample to extract the changed target field set. Following the same message parsing logic as during initial configuration, the system parses a completely new list of fields from the modified response message. This list of fields constitutes the changed target field set. This set may contain newly added fields, or some existing fields may have been deleted or renamed.
[0077] For each adopted mapping pair in the fusion recommendation mapping set, its semantic consistency score is calculated one by one. The system iterates through each formal mapping relationship stored in the configuration storage area. For each mapping relationship, the system extracts its source field identifier and retrieves the category metadata of the source field from the platform metadata database, using the same method as described in the previous steps. Simultaneously, the system searches the extracted set of modified target fields to see if the target field from the original mapping relationship exists. If the target field no longer exists, the semantic consistency score of the mapping pair is directly recorded as 0. If the target field still exists, the system re-extracts the data features of the target field in the modified message sample, using the same extraction method as described in the previous steps. Then, the system uses the same semantic matching rules as before to calculate the semantic consistency score between the category metadata of the source field and the data features of the modified target field. This score reflects whether the original mapping relationship still holds semantically under the new message structure.
[0078] The system determines whether a mapping pair is invalid and generates an alert. It compares the semantic consistency score of each mapping pair with a pre-set third threshold. This third threshold can be set to 0.5 or 0.6. If the semantic consistency score of a mapping pair is lower than the third threshold, the system determines that the mapping pair is invalid or highly questionable. The system then generates a mapping failure alert notification in the low-code platform's configuration interface. This alert notification can be displayed on the monitoring panel of the data integration process, highlighting the affected source and target fields in a prominent color, such as orange or red. The alert content can be specifically described as: "The target interface structure has been changed. The mapping from the source field: Customer Name to the target field: custName may be invalid. Please check promptly."
[0079] For invalid mapping pairs, alternative recommendations are generated. For each mapping pair deemed invalid, the system doesn't simply issue a warning and stop. Based on the category metadata of the source field in the invalid mapping pair, the system re-performs semantic-level matching within the changed set of target fields. The system attempts to find a new, semantically matching target field for the source field within the changed set of target fields. If a suitable alternative target field is found, and the semantic matching score exceeds a certain recommendation threshold, the system generates an alternative mapping pair and pushes it to the user. The push can take the form of displaying a repair suggestion button next to the warning notification. Clicking this button allows the user to see the system's recommended alternative mapping and apply it with a single click to fix the process configuration.
[0080] In a preferred embodiment, the source fields in the source data table are organized into a tree structure according to the hierarchical structure of the columns in the low-code platform; the parent node in the tree structure corresponds to the parent column, and the child node corresponds to the child column; Before extracting the field name transformation pattern in the first mapping inference channel, the following steps are also included: Obtain the first-level path of the first source field in the tree structure. The first-level path is formed by combining the names of the various level columns to which the first source field belongs in hierarchical order. Obtain the field name of the first target field in the target data table; Parse the field name of the first target field to identify whether it contains a level identifier that matches the first level path; the level identifier includes: the complete string, abbreviation string, or preset encoding mapping value of the name of each level column in the first level path; If a matching hierarchical identifier is identified, the hierarchical expansion transformation rules from the hierarchical path of the tree structure to the target field name are extracted. The hierarchical expansion transformation rules include: connectors between hierarchies, case conversion methods for each level column name, abbreviation mapping tables for each level column name, and methods for preserving or adjusting the hierarchical order.
[0081] Specifically, in the application scenario of this embodiment, before extracting the field name transformation mode in the first mapping inference channel, the system also performs the following steps.
[0082] The first step is to obtain the first-level path of the first source field in the tree structure. Starting from the node containing the field, the system traces back up the tree structure to the root node, collecting the names of each level of the field along the path and combining them into a hierarchical path string in a preset order. For example, if the source field "Customer Name" is located in the "Basic Information" parent field under the "Customer Management" root field, its first-level path would be "Customer Management.Basic Information.Customer Name".
[0083] The second step is to obtain the field name of the first target field in the target data table. This field name may be in the form of "custBasicInfoName", etc.
[0084] The third step involves parsing the field name of the first target field to identify whether it contains a level identifier that matches the first-level path. The system sequentially attempts three strategies: complete string matching, abbreviated string matching, and encoded mapping value matching. If a component corresponding to the name of each level column in the first-level path is identified in the target field name, then a matching level identifier is determined to exist.
[0085] The fourth step involves extracting the hierarchical expansion and transformation rules if a matching hierarchical identifier is identified. These rules include: the type of connector between hierarchical levels (e.g., camelCase without connectors); the case conversion method for each level's category names (e.g., capitalizing the first letter); an abbreviation mapping table for each level's category names (e.g., from "Basic Information" to "BasicInfo"); and the method for preserving or adjusting the hierarchical order. The system stores these rules as a structured object for use in subsequent steps.
[0086] This embodiment analyzes the transformation rules from tree-level paths to target field names, and obtains a third mapping inference capability based on hierarchical semantic structure, providing a technical foundation for handling the conversion scenario of nested data models to flat interface messages.
[0087] In a preferred embodiment, after extracting the hierarchical expansion transformation rules from the hierarchical path of the tree structure to the target field name, the method further includes the following steps: The hierarchical expansion transformation rule is applied to the hierarchical paths of other source fields in the source data table within the tree structure to generate the prediction target field names of each other source field, forming a third set of prediction mappings. Assign a hierarchical channel confidence score to each prediction result in the third set of prediction mappings. The hierarchical channel confidence score is determined based on the degree of matching between the field name of the first target field and the first hierarchical path. The third set of prediction mappings and their hierarchical channel confidence scores, together with the first set of prediction mappings and the second set of prediction mappings, participate in the cross-validation and fusion. The graph cut algorithm executed in cross-validation and fusion is configured such that the weights of the directed edges between nodes are determined by a weighted sum of the confidence levels of the candidate mapping relationships given by the first, second, and third mapping inference channels, so as to consider the prediction results from the three channels simultaneously during the global optimization solution process.
[0088] Specifically, the first step is to generate the third set of prediction mappings by applying hierarchical expansion transformation rules. The system traverses each source field in the source data table except for the first source field, obtains its hierarchical path, applies the hierarchical expansion transformation rules to the path, generates the prediction target field name, and searches for matching items in the actual field name list of the target data table to form the third set of prediction mappings.
[0089] The second step is to assign a hierarchical channel confidence score to each prediction result in the third set of prediction mappings. This confidence score is determined based on a quantitative score of the matching degree between the first target field name and the first hierarchical path, and is assigned to all prediction mappings derived from this rule after normalization.
[0090] The third step involves cross-validation and fusion of the third set of prediction mappings and their confidence scores. During the consistency assessment phase, the system simultaneously checks the prediction results of the three channels for the same source field, processing them according to preset rules based on combinations such as consistency among the three, consistency among two, or differences among all three. In the conflict graph construction phase, candidate relations from the third set of prediction mappings are added to directed edges, with their weights being the hierarchical channel confidence scores. In the subsequent graph cut algorithm solution, the objective function considers the confidence information from all three channels simultaneously, and the solver makes the optimal decision by integrating three dimensions: field name transformation mode, table structure transformation mode, and hierarchical path expansion transformation mode.
[0091] If the graph cut algorithm fails to converge or the consistency score is lower than the first preset threshold, when the supplementary mapping strategy is triggered, the source fields with serious discrepancies in the prediction results of the three channels are marked as pending for manual decision by the user.
[0092] This embodiment extends the mapping inference from a bimodal approach to a trimodal fusion decision, significantly improving the automated configuration capability for mapping complex hierarchical nested data structures.
[0093] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the inventive concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.
Claims
1. A method for configuring a data integration process on a low-code platform, characterized in that, Includes the following steps: In response to the first mapping operation performed by the user in the configuration interface of the low-code platform, a first mapping pair is obtained; the first mapping pair includes a first source field in the source data table and a first target field in the target data table. Using the first mapping pair as the anchor point, multiple mapping inference channels, including at least the first mapping inference channel and the second mapping inference channel, are launched in parallel. in, The first mapping inference channel is used to extract the field name transformation pattern between the field name of the first source field and the field name of the first target field, and apply the field name transformation pattern to other source fields in the source data table to obtain the first set of prediction mappings; The second mapping inference channel is used to compare the first structural position feature of the first source field in the table structure of the source data table with the second structural position feature of the first target field in the table structure of the target data table, identify the global table structure transformation pattern, and apply the global table structure transformation pattern to the other source fields to obtain the second set of prediction mappings. The first set of prediction mappings and the second set of prediction mappings are cross-validated and fused to generate a set of fused recommended mappings. The cross-validation and fusion include: for the same source field, if the two sets of prediction mappings are consistent, the result is adopted; if the two sets of prediction mappings are inconsistent or a certain channel does not provide a prediction, the prediction mapping with higher confidence is selected or a supplementary mapping strategy is triggered according to the preset confidence evaluation rules. The fusion recommendation mapping set is displayed in batches in the configuration interface in list form, and a one-click confirmation control is provided; In response to the user's triggering of the one-click confirmation control, the mapping pairs in the fusion recommendation mapping set are batch adopted as formal data docking process configurations; The first mapping inference channel includes: The field names of the first source field and the first target field are aligned to identify the common and different parts. At least two candidate transformation rules are generated based on the differences, including: prefix replacement rule, suffix replacement rule, character case conversion rule, or delimiter conversion rule; Each candidate transformation rule is applied to all field names in the source data table, and the matching success rate with the actual field names in the target data table is calculated. If, for any source field name, the predicted field name obtained after applying the candidate transformation rule exists in the target data table with an edit distance less than a preset distance, it is determined to be a successful match. Select the candidate transformation rule with the highest matching success rate as the field name transformation mode; Furthermore, the cross-validation and fusion of the first set of prediction mappings and the second set of prediction mappings includes the following steps: For each source field to be mapped, obtain the first candidate target field and its first confidence level output by the first mapping inference channel, and the second candidate target field and its second confidence level output by the second mapping inference channel; If the first candidate target field is consistent with the second candidate target field, then the target field is used as the recommended mapping target, and the harmonic mean of the first confidence and the second confidence is used as the fusion confidence. If there is a discrepancy, a local mapping conflict graph is constructed. The local mapping conflict graph uses the source fields in the source data table that have not yet determined the mapping relationship and the target fields in the target data table that have not yet been mapped as nodes, and the candidate mapping relationships given by the first mapping inference channel and the second mapping inference channel as weighted directed edges, with the weight of the edge being the confidence level of the corresponding channel. A graph cut algorithm based on confidence propagation is executed on the local mapping conflict graph to maximize global mapping consistency. The global optimal solution is calculated for the conflicting mapping relationships to obtain the final recommended mapping target for each source field to be mapped. If the graph cut algorithm fails to converge within a preset number of iterations, or if the maximum global mapping consistency score obtained is lower than a first preset threshold, the supplementary mapping strategy is triggered. The supplementary mapping strategy includes presenting the unresolved mapping relationships in a visually distinct manner in the configuration interface and waiting for the user's manual confirmation. The objective function of the graph cut algorithm simultaneously penalizes the cost of violating a single channel high-confidence mapping and the conflict cost of many-to-one mapping between the source field and the target field; the conflict cost of the many-to-one mapping is a preset penalty constant, which is applied when two or more source field nodes are mapped to the same target field node.
2. The low-code platform data integration process configuration method according to claim 1, characterized in that, The second mapping inference channel includes: Obtain data samples from at least two fields in the source data table and data samples from at least two fields in the target data table; For each sampled field, extract its numerical distribution morphological feature vector, which includes a combination of at least two of the following: statistical distribution of data type, quantile profile, unique value density, null value rate, and data pattern signature based on locality-sensitive hashing. Map all sampled fields of the source data table to the first point set in the source distribution morphology space, and map all sampled fields of the target data table to the second point set in the target distribution morphology space; Using the corresponding points of the first source field and the first target field in the first mapping pair in the first point set and the second point set, respectively, as registration anchor points, the spatial transformation function from the first point set to the second point set is calculated; the spatial transformation function is used to characterize the overall transformation of the data distribution pattern between the source data table and the target data table at the field level. The spatial transformation function is applied to the corresponding points of the other source fields in the first point set to obtain the predicted positions of each other source field in the target distribution morphology space. In the second set of points, the target field corresponding to the point closest to each of the predicted locations is found and used as the corresponding target field prediction for each of the other source fields, thus forming the second set of prediction mappings.
3. The low-code platform data integration process configuration method according to claim 1, characterized in that, Before performing the confidence-based graph cut algorithm on the locally mapped conflict graph, the following steps are also included: For each directed edge in the local mapping conflict graph, obtain the candidate mapping relationship, including the source field and the target field involved in the candidate mapping relationship. Calculate the actual data content matching degree between the source field and the target field. The actual data content matching degree includes at least one of the following: the overlap rate between the value range of the source field and the value range of the target field, the morphological similarity between the data distribution of the source field and the data distribution of the target field, and the functional dependency strength between the source field and the target field based on the sampled data. A verification confidence correction factor is generated based on the actual data content matching degree. The verification confidence correction factor is then fused with the original confidence of the corresponding edge to obtain the post-verification confidence. The post-verification confidence score is used as the weight of the corresponding edge in the graph cut algorithm.
4. The low-code platform data integration process configuration method according to claim 3, characterized in that, The graph cut algorithm is executed iteratively. Perform a graph cut solution with the current weights of each edge to obtain the current optimal mapping allocation scheme, and calculate the corresponding global energy function value; For each mapping relationship adopted in the current optimal mapping allocation scheme, recalculate the actual data content matching degree between the corresponding source field and target field, and update its verification confidence correction factor and edge weight. The above steps are repeated based on the updated edge weights until the change in the global energy function value between two adjacent iterations is less than the second preset threshold, or the number of iterations reaches the preset maximum number of iterations. The current optimal mapping allocation scheme obtained in the last iteration is taken as the final recommended mapping target.
5. The low-code platform data integration process configuration method according to claim 1, characterized in that, The source data table is a set of fields for user-defined columns in the low-code platform, and the target data table is a set of fields obtained after parsing the response messages of external system interfaces. Before extracting the field name transformation pattern in the first mapping inference channel, the following steps are also included: Obtain the column metadata configured in the low-code platform for each source field in the source data table. The column metadata includes at least one of the following: field display format configuration, field validation rule configuration, field interaction behavior configuration, and field enumeration option configuration. The data features of each target field in the target data table are obtained by sampling response message samples, and include at least one of the following: the format mode of the field value, the set of values of the field value, and the numerical range of the field value. Perform semantic-level matching between the column metadata and the data features to determine at least one high-confidence anchor mapping pair; The high-confidence anchor mapping pairs are recommended to the user as the target of the first mapping operation.
6. The low-code platform data integration process configuration method according to claim 5, characterized in that, After batch adopting the mapping pairs in the fusion recommendation mapping set as formal data integration process configurations, the following steps are also included: In response to a change in the response message structure of the external system interface, a sample of the changed response message is obtained; Parse the modified response message sample and extract the modified target field set; For each adopted mapping pair in the fusion recommendation mapping set, calculate the semantic consistency score between the column metadata of its source field and the data features of the changed target field; If the semantic consistency score of a mapping pair is lower than the third preset threshold, the mapping pair is determined to be invalid, and a mapping failure warning is generated in the configuration interface. For invalid mapping pairs, semantic-level matching is performed again in the changed target field set based on the column metadata of their source fields to generate alternative recommended mapping pairs and push them to users.
7. The low-code platform data integration process configuration method according to claim 5, characterized in that, The source fields in the source data table are organized into a tree structure according to the hierarchical structure of the columns in the low-code platform; the parent node in the tree structure corresponds to the parent column, and the child node corresponds to the child column. Before extracting the field name transformation pattern in the first mapping inference channel, the following steps are also included: Obtain the first-level path of the first source field in the tree structure. The first-level path is formed by combining the names of the various level columns to which the first source field belongs in hierarchical order. Obtain the field name of the first target field in the target data table; Parse the field name of the first target field to identify whether it contains a level identifier that matches the first level path; the level identifier includes: the complete string, abbreviation string, or preset encoding mapping value of the name of each level column in the first level path; If a matching hierarchical identifier is identified, the hierarchical expansion transformation rules from the hierarchical path of the tree structure to the target field name are extracted. The hierarchical expansion transformation rules include: connectors between hierarchies, case conversion methods for each level column name, abbreviation mapping tables for each level column name, and methods for preserving or adjusting the hierarchical order.
8. The low-code platform data integration process configuration method according to claim 7, characterized in that, After extracting the hierarchical expansion transformation rules from the hierarchical path of the tree structure to the target field name, the method further includes the following steps: The hierarchical expansion transformation rule is applied to the hierarchical paths of other source fields in the source data table within the tree structure to generate the prediction target field names of each other source field, forming a third set of prediction mappings. Assign a hierarchical channel confidence score to each prediction result in the third set of prediction mappings. The hierarchical channel confidence score is determined based on the degree of matching between the field name of the first target field and the first hierarchical path. The third set of prediction mappings and their hierarchical channel confidence scores, together with the first set of prediction mappings and the second set of prediction mappings, participate in the cross-validation and fusion. The graph cut algorithm executed in cross-validation and fusion is configured such that the weights of the directed edges between nodes are determined by a weighted sum of the confidence levels of the candidate mapping relationships given by the first, second, and third mapping inference channels, so as to consider the prediction results from the three channels simultaneously during the global optimization solution process.
Citation Information
Patent Citations
Multi-source heterogeneous data adaptive fusion and governance method
CN122153789A
Schema alignment and structural data mapping of database objects
US20200349129A1
