Method for data import and field intelligent matching for low-code platform

Abstract

The application discloses a method for data import and field intelligent matching of a low-code platform, and particularly relates to the technical field of low-code platform data processing; an initial feature vector of a field is constructed, a standard field and its historical matching features are extracted from a target data model, a semantic matching score matrix is calculated by using a multi-source feature fusion model; an initial mapping table is screened based on a dynamic threshold, and statistical deviation analysis is performed in combination with a field value distribution difference measurement feature delta, when the delta exceeds a preset threshold, a value mapping correction model is called to complete re-matching, and an optimized mapping table is obtained; if it is detected that there is user historical adjustment of the same source file type, then re-correction is performed based on user behavior logs, and a final field mapping table MP3 is output; finally, the MP3 is applied to data import, automatic matching and binding are realized; the application improves the accuracy and automation level of field matching, and is suitable for multi-source heterogeneous data integration scenarios.

Description

Technical Field

[0001] This invention relates to the field of low-code platform data processing technology, and more specifically to a method for data import and intelligent field matching for low-code platforms. Background Technology

[0002] As enterprises accelerate their digital transformation, low-code platforms are being widely adopted in industries such as healthcare, education, and manufacturing, serving as crucial tools for rapidly building business systems. During system construction, users often need to import structured data tables (such as Excel, CSV, and database export files) from diverse, heterogeneous systems and match them against the pre-defined data model within the low-code platform. This process suffers from inconsistent data sources, inconsistent field naming, and messy formatting, resulting in low field matching accuracy and a large amount of manual adjustment required, significantly impacting the platform's intelligence and usability.

[0003] Existing methods mostly rely on static rules or pre-trained fixed models for field matching, which makes it difficult to adapt to complex situations such as implicit field meanings, unit differences, ambiguous time fields, and decomposition of composite fields. Especially when importing data for the first time without sample reference data, they cannot achieve intelligent matching with high confidence. Summary of the Invention

[0004] The purpose of this invention is to provide a method for data import and intelligent field matching for low-code platforms, in order to address the shortcomings of the prior art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a method for data import and intelligent field matching for low-code platforms, comprising:

[0006] Obtain the original field set F1 and its corresponding field sample value set V1 from the data table D to be imported, and construct the initial feature vector set X1 of the fields;

[0007] Extract the standard field set F2 and its historical field matching feature set X2 from the target data model M;

[0008] Construct a field matching model WR based on multi-source feature fusion, and calculate the F1 score for each field based on WR. With the field in F2 The semantic matching score matrix S is a weighted calculation result of comprehensive feature similarity;

[0009] Perform a dynamic confidence analysis on S to select initial matching pairs that meet the matching confidence threshold θ. And use this to construct the initial field matching mapping table MP1;

[0010] Extract the difference measure feature δ of each field pair in MP1 on the data value distribution, and perform statistical bias analysis by combining the sample data value distributions in V1 and M;

[0011] Based on field pairs whose δ values ​​exceed the set deviation threshold ε, a value mapping correction model is used to rematch and correct them, resulting in an optimized field mapping table MP2.

[0012] If the same source file type is detected in the past and the user has manually adjusted the field mapping relationship, then MP2 is adjusted based on the user behavior log, and the final field mapping result MP3 is output.

[0013] Apply MP3 to the data import process to achieve automatic field matching and data binding.

[0014] Preferably, the initial feature vector set X1 of the field includes a field name character similarity vector, value type distribution features, field position encoding information, and cell context labels.

[0015] Preferably, the field historical matching feature set X2 includes historical mapping records, field labels, business semantic embedding vectors, and corresponding confidence distributions.

[0016] Preferably, the construction of the field matching model WR based on multi-source feature fusion includes:

[0017] The initial feature vector set X1 and the historical matching feature set X2 of the field are input into a bidirectional feature encoding network, and feature dimensionality reduction and semantic normalization are performed respectively to obtain the field embedding representation under a unified semantic space.

[0018] For embedded field pairs Perform feature similarity calculations, calculating the cosine similarity, Euclidean distance, and label matching score between vectors respectively, and then combining the three scores in a weighted manner to form a semantic matching score. ;

[0019] Semantic matching score for all field pairs Form a two-dimensional matrix S, where each row of S corresponds to a character to be matched. Columns correspond to standard fields The matrix elements represent the matching similarity between two fields.

[0020] Preferably, a dynamic confidence analysis is performed on S, including:

[0021] For each field to be imported in matrix S Extract the corresponding matching score row vector. Calculate the maximum matching score. and corresponding standard fields , as the optimal matching candidate for the field;

[0022] Combined with standard fields Statistical mean in the historical match confidence distribution and variance Dynamically generate confidence threshold ;

[0023] Only when At that time, confirm the matching pair Establish the mapping and add the field pairs to the initial field mapping table MP1;

[0024] When there are multiple Mapped to the same standard field When sorting by total score, retain the pair of field mappings with the highest confidence, and mark the remaining mappings as conflict items.

[0025] Preferably, the extraction of the difference measure feature δ of each field pair in MP1 on the data value distribution includes:

[0026] For each field pair in the initial matching mapping table MP1, Extract the corresponding sample data sets respectively. and ,in Sourced from table D, Field sample library derived from the target data model M;

[0027] Based on the data type of the fields, the corresponding distribution modeling method is used to... and Perform normalization and construct the probability distribution function. and ;

[0028] Based on the distribution difference assessment index, calculate and Difference measurement between This is used to characterize the statistical deviation of a field at the data content level;

[0029] Measurement of differences Tolerance threshold for field deviation When comparing, Exceed When this happens, the corresponding field pairs will be marked as having inconsistent distributions.

[0030] Preferably, the rematch correction using the value mapping correction model includes:

[0031] Filter out all field pairs in the initial field matching mapping table MP1 whose δ values ​​are greater than the set deviation threshold ε, and construct the abnormal field pair set C, where ε is the preset field value distribution difference tolerance;

[0032] Pair each field in set C The input-to-value mapping correction model RVM performs cross-mapping scoring based on the value range, cell similarity, and statistical relationship of historical values ​​of candidate fields between sample values.

[0033] For each exception field Several candidate fields are reselected from the target field set F2. Construct value matching score vector The field with the highest value mapping score and a δ value lower than ε is selected as the alternative pairing field. ;

[0034] Original pairing Replace with the updated pairing This forms the optimized field mapping table MP2.

[0035] Preferably, MP2 is adjusted based on user behavior logs to output the final field mapping result MP3, including:

[0036] Retrieve task instances with the same file type as the currently imported file from historical import task records, and extract operation logs of user manual adjustment of field mappings. The operation logs include field pairs before adjustment, field pairs after adjustment, and adjustment timestamps.

[0037] The extracted log data is structured to build a user behavior rule base (UB), where each rule consists of a source field identifier, a target field identifier, an adjustment frequency, and a user confirmation flag.

[0038] For the field pairs in MP2 If a corresponding historical correction rule exists in UB, and the adjustment frequency is greater than the preset threshold λ, then the current field pair will be automatically replaced with the target field pair after historical user confirmation. ;

[0039] Perform consistency checks on all field pairs that have been modified by user rules, and write the modified field pairs into the final field mapping result table MP3.

[0040] The technical effects and advantages provided by the present invention in the above technical solution are as follows:

[0041] 1. This invention considers not only semantic features such as field names, tags, and context during field matching, but also introduces a data value distribution difference metric δ, a value mapping correction model, and a user behavior log-driven re-optimization mechanism, realizing a three-layer collaborative field mapping strategy of "semantic features—data content—user knowledge." This multi-dimensional fusion approach effectively reduces the probability of incorrect matching caused by semantic similarity but data conflict, while ensuring a balance between automation and business consistency in field matching results, significantly improving the intelligence and usability of the data import process.

[0042] 2. This invention has the following advantages: First, the dynamic threshold mechanism and distribution difference measurement improve the accuracy and robustness of field matching; second, the value mapping correction model can automatically correct data-level anomalies, enhancing the system's adaptability to complex heterogeneous data sources; third, the introduction of user behavior logs enables the system to inherit and reuse users' knowledge and experience, achieving human-machine collaborative optimization; fourth, the final generated field mapping result MP3 has traceability and auditability, ensuring data security and compliance in enterprise-level application scenarios. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0044] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0046] For examples, please refer to Figure 1 As shown in this embodiment, the method for data import and intelligent field matching for low-code platforms includes:

[0047] Obtain the original field set F1 and its corresponding field sample value set V1 from the data table D to be imported, and construct the initial feature vector set X1 of the fields;

[0048] Extract the standard field set F2 and its historical field matching feature set X2 from the target data model M;

[0049] Construct a field matching model WR based on multi-source feature fusion, and calculate the F1 score for each field based on WR. With the field in F2 The semantic matching score matrix S is a weighted calculation result of comprehensive feature similarity;

[0050] Perform a dynamic confidence analysis on S to select initial matching pairs that meet the matching confidence threshold θ. And use this to construct the initial field matching mapping table MP1;

[0051] Extract the difference measure feature δ of each field pair in MP1 on the data value distribution, and perform statistical bias analysis by combining the sample data value distributions in V1 and M;

[0052] Based on field pairs whose δ values ​​exceed the set deviation threshold ε, a value mapping correction model is used to rematch and correct them, resulting in an optimized field mapping table MP2.

[0053] If the same source file type is detected in the past and the user has manually adjusted the field mapping relationship, then MP2 is fine-tuned based on the user behavior log, and the final field mapping result MP3 is output.

[0054] Apply MP3 to the data import process to achieve automatic field matching and data binding.

[0055] In the first stage of this method, the original field set F1 and its corresponding field sample value set V1 need to be obtained from the user-uploaded data table D, and the initial feature vector set X1 of the fields needs to be constructed based on this. The construction process of X1 includes the following steps:

[0056] Parse data table D and extract the header of each column in the row containing the table header as the original field set F1, where , where n is the number of fields in the data table.

[0057] For each field Extract the non-empty cells from the first k rows of this column to form a sample value set. This ultimately forms a field sample set. The default value for k is 50. If there are fewer than 50 rows in the data table, all available data will be used.

[0058] X1 is a set of multidimensional feature combination vectors, for each field Construct its corresponding feature vector , It consists of the following four types of features:

[0059] Field Name Character Similarity Vector: The system uses word encoding algorithms from natural language processing (such as word segmentation based on Byte-Pair Encoding) to vectorize field names, and calculates cosine similarity using word vectors constructed with the field names in the standard field set F2 to obtain the character semantic similarity between each field name and the standard field names. This feature dimension is the number of standard fields m, corresponding to an m-dimensional vector. This vector is used as one of the input features of the subsequent matching model WR.

[0060] Value type distribution characteristics: for each field sample set The system identifies the data type distribution, such as integers, floating-point numbers, dates, booleans, and text, using regular expressions and type inference algorithms. After statistically analyzing the frequency of different types, a normalized type probability vector is generated. For example, if... If a vector contains 70% date and 30% text, the corresponding vector is [0.7, 0.3, 0, 0, 0]. The length of this vector is the preset number of standard types (e.g., 5).

[0061] Field position encoding information: This feature reflects the relative position of the field in the table header. Assume the field... If the i-th field is a given field, its position code can be calculated using a normalization method as i divided by the total number of fields n, with a value ranging from 0 to 1. This feature is used to help determine the relationship between field priority and table structure.

[0062] Cell context labels: The system analyzes the columns immediately above and below the column containing the sample value (one column before and one after), extracts their field names, data types, and sample values, and constructs a context label embedding vector. This vector is then processed by a shallow convolutional neural network (CNN) to analyze the field... The adjacent context of the column is encoded to represent its semantic environment. This feature helps to identify complex information fields, such as "address" and "contact information".

[0063] Ultimately, each field eigenvectors It is composed of the above four types of features and is denoted as: Where: E_name is the field name character embedding vector with dimension m, V_type is the value type distribution vector with dimension t, P_pos is the positional encoding scalar with dimension 1, and C_ctx is the context label embedding vector with dimension c. The total dimension is m + t + 1 + c.

[0064] After constructing the initial feature vector set X1 for the field, the system normalizes all feature vectors to ensure that the values ​​of each dimension are within the range of 0 to 1. The processed X1 is then used as the input to the next stage field semantic matching model WR, entering the field matching calculation process.

[0065] To improve the accuracy and stability of field matching, the method of this invention further extracts a standard field set F2 and its corresponding historical field matching feature set X2 from the target data model M.

[0066] The target data model M is typically a predefined set of data entities in a low-code platform, characterized by complete structure, clear fields, and stable semantics. The standard field set F2 is represented as: , where m is the total number of standard fields.

[0067] For each standard field The system constructs its historical matching feature vector. ∈ X2. The field history matching feature set X2 includes the following four information features, which are described below:

[0068] Historical mapping records: The system records historical decision information regarding user and system field matching during the import of various data sources on the platform, including field name, source file type, mapping result, and whether user confirmation was required. For each standard field... Extract the set of source fields that have been successfully matched in the past. The frequency of each source field mapping is counted to generate a weighted mapping probability distribution. This distribution vector has a length of k, and its value reflects the relevance weight of the field to common aliases.

[0069] In terms of implementation, the matching frequency is defined. For: a field name and The historical matching vector is formed by dividing the number of successful matches by the total number of matches and then normalizing the sum. When a new field name is close to a historically high-frequency alias, the system can increase its initial match confidence.

[0070] Field Labels: System Standard Fields Maintain a set of field labels The tags originate from metadata such as semantic definitions, data types, and business uses during platform modeling. For example, the field "Date of Birth" can have tags like {Time-based, Personal Information, Date Format}; the field "Address" can have tags like {Text-based, Geographic Information, Decomposable}. These tags are encoded as multi-hot vectors to express the semantic classification and business attributes of the fields. The dimensions are determined by the size of the predefined tag set within the platform, typically several tens of dimensions.

[0071] Business semantic embedding vector: To achieve semantic-level understanding of fields, this invention performs embedding calculations on field description text based on domain-adaptive pre-trained language models (such as FinBERT, MedBERT, etc.). For each standard field... Semantic text is extracted from field names, tags, and business definition descriptions (such as notes, form names, and usage locations), and semantic embedding vectors are generated using a pre-trained model. . The dimension depends on the language model structure, and commonly used dimensions are 256 or 512. This semantic vector is used to represent the conceptual position of a field within the entire business system.

[0072] To improve contextual understanding during matching, the semantic embedding model uses the field mapping training dataset accumulated by the platform during the fine-tuning stage to perform a supervised semantic similarity learning task, further aligning with domain knowledge.

[0073] Confidence distribution characteristics: The system constructs a field confidence distribution curve by archiving the matching confidence scores of each previous field match. This feature records the system score (such as probability, similarity score, etc.) corresponding to successful matching of each standard field under different import source scenarios, and forms a distribution function graph. For example, the field The average matching confidence score is 0.92 in "Medical Import Template A" and 0.81 in "City Interface Import Template B". This feature is used to dynamically adjust the expected score threshold in new matching scenarios.

[0074] The system uses nonparametric methods such as Gaussian kernel density estimation (KDE) to fit and model the confidence level, thereby achieving continuous matching signal distribution estimation.

[0075] For each standard field Its complete feature vector Defined as: ,in: This is the historical mapping probability distribution vector (with dimension k). This is a multi-hot encoded vector for the field label (with dimension t). It is a semantic embedding vector (with dimension s). The confidence level statistical distribution characteristics (dimension d).

[0076] all The historical matching feature set X2 is used as the input to the subsequent field matching model WR, and together with X1, it participates in the calculation of field pair matching scores.

[0077] First, obtain the initial feature vector set X1 (from field F1 in the import table) and the historical matching feature set X2 (from field F2 in the target model) constructed above. Since the feature dimensions and distributions in the two sets are different, X1 and X2 need to be mapped to the same semantic space in order to realize subsequent similarity calculation.

[0078] To this end, this invention employs a Bi-Feature Encoder Network, which consists of two identical deep neural network encoders sharing weights, performing semantic normalization on the field feature vectors in X1 and X2, respectively. This encoder network includes the following hierarchical structure:

[0079] Input layer: Accepts the original feature vector, with dimension 1. .

[0080] Projection layer: Maps the input vector to a fixed dimension through a fully connected layer. (e.g., 128), to achieve feature dimensionality reduction.

[0081] Activation function layer: The ReLU activation function is used to enhance the nonlinear expressive power.

[0082] Residual connection layer: Preserves the proportion of numerical fields in the original features, enhancing the model's sensitivity to field value types.

[0083] Output layer: Outputs embedding vectors in a unified semantic space. .

[0084] After processing, two sets of semantic embedding vectors are obtained: and Each of them All vectors are A standardized semantic vector of dimension.

[0085] After obtaining the semantic embedding vector, this invention performs a process for each pair of fields to be matched. Feature similarity is calculated, and the results from multiple channels are fused to generate the final semantic matching score.

[0086] Specifically, it includes the following three similarity channels:

[0087] Semantic embedding vector cosine similarity: Calculation and The cosine similarity between the two fields reflects the angular distance between them in the semantic space, ranging from -1 to 1.

[0088] Euclidean distance similarity: The L2 norm of the vector difference is calculated and then scaled using a negative exponential function to transform it into a similarity form, so that the smaller the distance, the higher the similarity.

[0089] Field label matching score: The intersection-union ratio (IUU) is calculated for the multi-hot vectors of labels in two fields. The size of the intersection is divided by the size of the union, which is the label similarity score. The value range is 0 to 1.

[0090] To provide a unified model for the results from the three channels, this invention introduces a weighted fusion mechanism, which dynamically weights the three similarity scores according to field type to generate the final matching score. The formula is: Where α, β, and γ are weighting coefficients, satisfying α + β + γ = 1;

[0091] When the field is text-based, semantic similarity is prioritized (α=0.6). When the field is numeric, type and position are prioritized (β=0.5). When the field has a clear label, γ is increased (e.g., γ=0.4).

[0092] Based on the above calculations, a score matrix S is constructed, with dimensions n × m. Each element in the matrix... Indicates the fields to be imported With standard fields The matching confidence level.

[0093] In real-world scenarios, semantic dependencies often exist between fields. For example, "ID number" often appears alongside "name," and "longitude" often exists in pairs with "latitude." Therefore, to enhance the model's overall ability to perceive the context between fields, this invention introduces a feature enhancement module based on a self-attention structure onto the score matrix S.

[0094] This module uses a multi-head attention mechanism, with attention focused on each row (i.e., the import field). The query vector is defined as , and the remaining fields are context fields. The weights of these context fields in the table are calculated to influence the current match. This structure consists of the following steps:

[0095] Construct query vector key vector value vector All are derived from the embedding vector.

[0096] Calculate the weights: Use a scaled dot product attention mechanism to output the weight coefficients. .

[0097] Weighted summation: combining the original scores It is fused with attention-weighted values ​​to generate context-enhanced matching scores.

[0098] This attention mechanism can improve the model's ability to recognize composite fields, split fields, and semantically implicit fields, and it remains robust even when field names are ambiguous or samples are insufficient.

[0099] The field matching model WR described in this invention can be trained using supervised learning. The training dataset consists of manually confirmed field pairing samples from historical import tasks, including positive samples (correctly mapped field pairs) and negative samples (incorrectly matched field pairs). Parameters are optimized by minimizing the cross-entropy loss function.

[0100] First, for each row vector in the score matrix S , indicating the field to be imported With all standard fields Match the score list. This step aims to find each... The target field that is most likely to match.

[0101] The specific steps are as follows:

[0102] For i = 1 to n, iterate through all fields to be imported. ;

[0103] exist Find the maximum value in the middle. ;

[0104] Record the standard field index corresponding to this maximum value. ,Right now ;

[0105] Constructing the optimal field pair of candidates .

[0106] This operation ensures that every Each pair has a target field with the best initial semantic score, but whether the pair can enter the final matching mapping table MP1 still needs to be analyzed by the next step of confidence dynamic analysis.

[0107] To avoid high-scoring mismatches caused by field semantic drift, non-standard field naming, or inconsistent data samples, this invention designs a dynamic threshold mechanism to evaluate the "acceptable lower limit of matching quality" for each standard field.

[0108] This mechanism is based on the historical matching behavior of fields recorded in the platform, and counts the results for each standard field. From the confidence score data of successfully matched pairs, construct their confidence distribution features. Specifically, this includes:

[0109] Statistical fields In the historical successful matching records, the average model score ;

[0110] Statistical analysis of the sample variance of the corresponding scores ;

[0111] Based on a conservative strategy, dynamic threshold Set as: .

[0112] The dynamic threshold The design philosophy is as follows: for fields with stable confidence distribution and consistently high matching quality, the threshold will be increased accordingly to reduce the probability of mismatches; while for fields with large historical data fluctuations, the threshold will be relatively low with higher tolerance, thus preserving potentially semantically ambiguous field pairs. This field-specific threshold control strategy is significantly superior to traditional uniform threshold schemes, possessing stronger generalization ability and domain adaptability.

[0113] After obtaining each field Optimal matching pair Then, the system will use dynamic thresholds. The confidence level of this field is determined to filter out preliminarily reliable pairs.

[0114] The judgment rules are as follows:

[0115] like Then it is considered that the field is a pair Acceptable;

[0116] Add the paired field to the initial field matching mapping table MP1;

[0117] Otherwise, ignore the pairing and mark it as "insufficient confidence" for subsequent candidate evaluation.

[0118] The final generated MP1 is a one-to-one mapping table that records the initial reliable field pairing results and serves as the basis for subsequent data binding, mapping preview, and conflict handling.

[0119] In actual import scenarios, there may be multiple fields to be imported. Determined to match the same standard field To maintain the uniqueness and consistency of the field mapping table, this invention further designs a conflict detection and sorting strategy:

[0120] Traverse all field pairs in MP1 and perform statistics. Number of times it appears;

[0121] If a certain standard field by multiple Mapping, labeled as "conflict mapping";

[0122] For field pairs in a conflict group, rank them according to their matching score. Sort in descending order;

[0123] Keep the field pair with the highest score The remaining mappings are set to "candidate retention" and proceed to the next round of context or rule-assisted matching.

[0124] For each field pair in the initial matching field mapping table MP1 Extract sample value sets for corresponding fields from the imported data table D and the target data model M respectively, denoted as:

[0125] From field The sample values ​​collected in table D are sampled by default from the first 50 non-empty cells;

[0126] : From standard fields The representative sample values ​​extracted from historical data or sample database in M ​​are also limited to 50 samples, or can be flexibly adjusted according to the amount of available data.

[0127] To ensure the comparability of data distributions, the system must first determine the value types of the two fields and then select an appropriate distribution modeling strategy accordingly. Field value types are categorized into the following four types:

[0128] Numeric fields: such as amount, quantity, age;

[0129] Categorical fields: such as gender, department, and type label;

[0130] Date field (Datetime): such as date of birth, creation time;

[0131] Textual fields: such as notes, addresses, descriptions, etc.

[0132] The system employs differentiated modeling methods for different types of fields:

[0133] For numerical fields, kernel density estimation is used to model the probability density of sample values, forming a continuous probability distribution function. and ;

[0134] For categorical fields, construct a category frequency histogram and perform normalization to generate a probability distribution vector;

[0135] For date fields, the date is converted into a numerical timestamp before segmented statistical processing is performed;

[0136] For text fields, the system does not perform delta analysis by default, but this can be handled through subsequent semantic consistency mechanisms.

[0137] Ultimately, each field pair will correspond to a set of standardized distribution functions or vectors, ensuring that quantitative analysis can be performed.

[0138] This invention uses distribution distance indicators from statistics to quantify the differences between field sample distributions, mainly including the following three types of indicators, which the system can flexibly select according to the field type:

[0139] Kullback-Leibler divergence (KL divergence): Applicable to continuous or discrete distributions, it measures the information difference between two probability distributions;

[0140] Wasserstein distance (also known as Earth Mover's Distance): measures the minimum "transportation cost" between two distributions and is stable in numerical data;

[0141] Bhattacharyya distance: Used to measure the degree of overlap between two probability distributions, applicable to categorical distributions.

[0142] Taking the Wasserstein distance as an example, for two one-dimensional numerical distributions... and Its first-order Wasserstein distance is defined as: , where F_P and F_Q are the cumulative distribution functions (CDF) of P and Q, respectively.

[0143] Ultimately, the system will pair each field. The corresponding distribution difference measurement results are recorded as follows .

[0144] To automatically identify anomalous field pairings, this invention introduces a field-level difference tolerance threshold θ_δ as a benchmark for judging the consistency of field value distribution. The threshold θ_δ can be set according to the business scenario, for example:

[0145] The recommended value for general numeric fields is 0.2;

[0146] A value of 0.1 is recommended for sensitive financial fields;

[0147] The categorization field can be set to indicate excessive deviation when the overlap rate of category distribution is less than 80%.

[0148] The judgment rules are as follows:

[0149] like If the field values ​​are basically consistent, then the field pairing is considered reasonable.

[0150] like If the field is not found to be a "distribution inconsistency field", the system will mark it as suspicious or require secondary verification.

[0151] For the marked fields, the system can trigger an automatic rematch process (e.g., based on the second matching score candidate), prompt the user for manual confirmation, or record it as model training feedback data.

[0152] Initially match the field pairs in table MP1. Filter the data to identify the characteristics that measure the difference in the distribution of all values. Pairing relationships that exceed the set threshold ε.

[0153] The threshold ε is defined as the upper limit of the acceptable range for data distribution deviation in the system. The value is configurable based on the field type and business context. The default recommended values ​​are as follows:

[0154] Numeric field: ε = 0.2;

[0155] Categorical field: ε = 0.15;

[0156] Date field: ε = 0.25.

[0157] This step constructs a set C of exception field pairs, in the form of: .

[0158] For each pair of abnormal field pairs in the set C, the system maps its input values ​​to the correction model RVM (Reference Value Mapper). The purpose of this model is to recommend more suitable target fields based on the statistical similarity and semantic consistency between field sample values.

[0159] The input to RVM is:

[0160] Source field sample value set ;

[0161] Original pairing target field sample value set ;

[0162] Target field candidate set F2;

[0163] The set of sample values ​​corresponding to each candidate field , k ∈ [1, m].

[0164] The RVM model analyzes the source fields based on the following three dimensions. With each target candidate field The similarity of the value range is calculated:

[0165] Value range overlap rate: If it is a numeric field, calculate the percentage of the intersection between the minimum and maximum values; if it is a categorical field, calculate the number of common categories divided by the total number of categories.

[0166] Value-frequency distribution similarity: Calculate the cosine similarity between the frequency vectors of sample values ​​of the source field and the target field.

[0167] Unit-level similarity matching: Calculates the proportion of approximately identical values ​​in sample pairs using fuzzy string matching (such as Jaccard similarity or edit distance), applicable to text / address / encoded fields.

[0168] Each dimension yields a score, and the final score is a composite score. k The calculation method is as follows: ;in, For the weighting coefficients, satisfying It is recommended to set it to [0.4, 0.4, 0.2]. For each abnormal field Fields of the same type that are not mapped in MP1;

[0169] For each candidate field calculate ;

[0170] right Sort by high to low, and test each one in turn. Is the value of δ less than or equal to ε?

[0171] Select the first field that meets the criteria. Replace the original pair Replace the original pair ;

[0172] If no field meets the conditions, the original pairing is retained and marked as "awaiting manual review".

[0173] This mechanism ensures that the rematched fields are not only semantically related, but also that their data values ​​are statistically consistent, thus more accurately reflecting the business correspondence between the fields.

[0174] After completing all field rematches, the system replaces the abnormal field pairs in the original mapping table MP1 with the corrected field pairs, generating the optimized field mapping table MP2.

[0175] To ensure the traceability of mapping relationships and the transparency of system behavior, the system will record all field pairing changes that occur during rematching; write the field matching audit log, including field name, replacement time, original δ value, replacement field ID, RVM score, etc.; and support subsequent user backtracking analysis and model retraining.

[0176] Determine whether the current import task and historical tasks belong to the same source file type. The specific implementation is as follows:

[0177] When the import task begins, the system generates a file type identifier File_ID based on the file's metadata information (including file name, file extension, data table structure summary, number of fields and sequential signature, etc.).

[0178] The system searches the log database for historical records with the same or highly similar File_ID as the current one. If a record exists, it indicates that the import task and the historical task belong to the same source file type.

[0179] Once a file from the same source is detected, the system will extract all manual adjustment operation logs for field mapping relationships from all users in that historical task. The log content includes: field pairs before adjustment, field pairs after adjustment, adjustment timestamps, user confirmation status, and number of operations.

[0180] This step enables the system to provide a reliable data source for subsequent automated adjustments.

[0181] Plain log data is unstructured and cannot be directly used for mapping table adjustments. Therefore, this invention proposes to structure user operation logs and construct a user behavior rule base (UB).

[0182] The construction process of the rule base UB includes:

[0183] For each log entry, extract the source field identifier, the original target field identifier, and the user-adjusted target field identifier;

[0184] The cumulative number of times this adjustment operation occurs across all historical tasks is recorded and denoted as the adjustment frequency f.

[0185] If the user has explicitly confirmed the pairing relationship after the adjustment, then a "user confirmation mark" will be added to the rule;

[0186] Store the rules as quadruples: .

[0187] The rule base UB constructed in this way has the characteristics of being queryable, statistical, and applicable, and can directly participate in the optimization of the subsequent mapping table MP2.

[0188] After obtaining the rule base (UB), the system checks and corrects each entry in the optimized field mapping table (MP2). The specific process is as follows:

[0189] Iterate through each field pair in MP2 ;

[0190] Search in UB to see if there is a corresponding correction rule, i.e., the source field equals... The original target field equals If a rule exists, it is further determined whether the adjustment frequency f of that rule exceeds the threshold λ. λ is the adoption threshold set by the system, with a recommended value of 3, meaning that the system will automatically apply the rule only if the user makes the same adjustment in at least 3 tasks.

[0191] If the condition is met, then the current field will be... Replace with ,in The target field for user confirmation recorded in UB;

[0192] If the conditions are not met, the original mapping remains unchanged, and the candidate rule is recorded in the candidate correction rule set for manual review.

[0193] Through this mechanism, the system can intelligently inherit the user's domain knowledge and gradually form a highly personalized matching logic.

[0194] After rule-based corrections are completed, multiple source fields may be simultaneously mapped to the same target field, leading to conflicts. This invention proposes a consistency verification mechanism to ensure the uniqueness and correctness of the final output MP3 field mapping result.

[0195] The system analyzes the mapping relationships after MP2 correction and checks whether there are multiple source fields pointing to the same target field.

[0196] If conflicts exist, the following priority levels will be used for trimming:

[0197] First, retain the field pairs where the user's confirmation flag is true (Confirm_Flag = True);

[0198] If conflicts still exist, select the field pair with the higher adjustment frequency (f).

[0199] If the conflicting fields have the same frequency, the pairing relationship based on the most recent user operation will be selected.

[0200] The removed field pairs will be added back to the queue for correction, for subsequent manual confirmation or system candidate recommendation.

[0201] After completing the consistency check, the system outputs the final conflict-free mapping relationship as MP3 and writes it to the audit log, recording the application status of all correction rules.

[0202] After generating the final field mapping table MP3, the method of this invention enters the data import and field binding stage. The goal of this stage is to establish a one-to-one correspondence between the fields in the data table D to be imported and the standard fields in the target data model M, and to achieve data content binding and persistent storage.

[0203] Read the final mapping table MP3, where MP3 defines the source fields. With target field A unique mapping relationship. For each source field... The system locates the corresponding target field based on the mapping relationship. And establish a mapping link. For the source field data value set Preprocessing is performed, including missing value imputation, data type conversion, and illegal character removal. For example, when the source field is a string but the target field is defined as a date, the system will automatically convert it using a date regular expression parsing algorithm. Then, the system performs data binding according to the mapping relationship: for each source field... cell value Write it to the target field The data storage area is configured to form row-level bindings. If a value is encountered that fails to be mapped (such as a parsing error), the system marks it as an exception and logs it.

[0204] Finally, the system performs a consistency check on the binding results, verifying whether the number of imported data records matches the original records and whether the field value types conform to the target model definition. When the binding error rate exceeds the threshold η (default η=0.05, i.e., 5%), the system will trigger an alarm and stop the import to ensure data quality.

[0205] Through the above steps, the present invention can realize a fully automatic field matching and data binding process based on MP3, ensuring that the imported data is highly consistent with the target model in terms of semantics, type and content.

[0206] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A method for data import and intelligent field matching for low-code platforms, characterized by: include: Obtain the original field set F1 and its corresponding field sample value set V1 from the data table D to be imported, and construct the initial feature vector set X1 of the fields; Extract the standard field set F2 and its historical field matching feature set X2 from the target data model M; Construct a field matching model WR based on multi-source feature fusion, and calculate the F1 score for each field based on WR. With the field in F2 The semantic matching score matrix S is a weighted calculation result of comprehensive feature similarity; The initial feature vector set X1 and the historical matching feature set X2 of the field are input into a bidirectional feature encoding network, and feature dimensionality reduction and semantic normalization are performed respectively to obtain the field embedding representation under a unified semantic space. For embedded field pairs Perform feature similarity calculations, calculating the cosine similarity, Euclidean distance, and label matching score between vectors respectively, and then combining the three scores in a weighted manner to form a semantic matching score. The formula is: Where α, β, and γ are weighting coefficients, satisfying α + β + γ = 1; where, Cosine similarity of semantic embedding vectors: The similarity is determined by Euclidean distance. Score for matching field labels; score for semantic matching of all field pairs. Form a two-dimensional matrix S, where each row of S corresponds to a field to be matched. Columns correspond to standard fields The matrix elements represent the matching similarity between two fields; A dynamic confidence analysis of S is performed, including: For each field to be imported in matrix S Extract the corresponding matching score row vector. Calculate the maximum matching score. and corresponding standard fields , as the optimal matching candidate for the field; Combined with standard fields Statistical mean in the historical match confidence distribution and variance Dynamically generate confidence threshold ; Only when At that time, confirm the matching pair Establish the mapping and add the field pairs to the initial field mapping table MP1; When there are multiple Mapped to the same standard field When sorting by total score, retain the pair of field mappings with the highest confidence, and mark the remaining mappings as conflict items; Perform a dynamic confidence analysis on S to select initial matching pairs that meet the matching confidence threshold θ. And use this to construct the initial field matching mapping table MP1; Extract the difference measure feature δ of each field pair in MP1 on the data value distribution, and perform statistical bias analysis by combining the sample data value distributions in V1 and M; For field pairs whose δ values ​​exceed a set deviation threshold ε, a value mapping correction model is used to re-match and correct them, resulting in an optimized field mapping table MP2, which includes: Filter out all field pairs in the initial field matching mapping table MP1 whose δ values ​​are greater than the set deviation threshold ε, and construct the abnormal field pair set C, where ε is the preset field value distribution difference tolerance; Pair each field in set C The input-to-value mapping correction model RVM performs cross-mapping scoring based on the value range, cell similarity, and statistical relationship of historical values ​​of candidate fields between sample values. For each exception field Several candidate fields are reselected from the target field set F2. Construct value matching score vector The field with the highest value mapping score and a δ value lower than ε is selected as the alternative pairing field. ; Original pairing Replace with the updated pairing And form the optimized field mapping table MP2; If the same source file type is detected in the past and the user has manually adjusted the field mapping relationship, then MP2 is adjusted based on the user behavior log, and the final field mapping result MP3 is output. Apply MP3 to the data import process to achieve automatic field matching and data binding.

2. The method for data import and intelligent field matching for low-code platforms according to claim 1, characterized in that: The initial feature vector set X1 of the field includes the field name character similarity vector, value type distribution features, field position encoding information, and cell context labels.

3. The method for data import and intelligent field matching for low-code platforms according to claim 1, characterized in that: The field historical matching feature set X2 includes historical mapping records, field labels, business semantic embedding vectors, and corresponding confidence distributions.

4. The method for data import and intelligent field matching for low-code platforms according to claim 1, characterized in that: The extraction of the difference measure feature δ for each field pair in MP1 on the data value distribution includes: For each field pair in the initial matching mapping table MP1, Extract the corresponding sample data sets respectively. and ,in Sourced from table D, Field sample library derived from the target data model M; Based on the data type of the fields, the corresponding distribution modeling method is used to... and Perform normalization and construct the probability distribution function. ; Based on the distribution difference assessment index, calculate Difference measurement between This is used to characterize the statistical deviation of a field at the data content level; Measurement of differences Tolerance threshold for field deviation When comparing, Exceed When this happens, the corresponding field pairs will be marked as having inconsistent distributions.

5. The method for data import and intelligent field matching for low-code platforms according to claim 1, characterized in that: Based on user behavior logs, MP2 is adjusted to output the final field mapping result MP3, including: Retrieve task instances with the same file type as the currently imported file from historical import task records, and extract operation logs of user manual adjustment of field mappings. The operation logs include field pairs before adjustment, field pairs after adjustment, and adjustment timestamps. The extracted log data is structured to build a user behavior rule base (UB), where each rule consists of a source field identifier, a target field identifier, an adjustment frequency, and a user confirmation flag. For the field pairs in MP2 ; Perform consistency checks on all field pairs that have been modified by user rules, and write the modified field pairs into the final field mapping result table MP3.

Images