Standardized data processing method based on large model, electronic device, storage medium and computer program product

Abstract

The application provides a large model-based standardized data processing method, an electronic device, a storage medium and a computer program product. The method comprises the following steps: inputting a field name string, a field value sample list and context structure information of a field through a large model coding field, generating a standard field semantic vector; calculating a semantic offset degree of a to-be-standardized field semantic vector and a standardized field semantic vector, generating a successful matching field pair set and a normalization failure field set; obtaining a historical standard version, generating an optimal field and a maximum score version mapping set; performing semantic positioning in the maximum score version according to the optimal field, calculating a matching field; obtaining standard mapping field information of the matching field in a current version based on a mapping rule table of the matching field, and outputting. The application avoids the problems that a traditional rule system is dependent on fixed fields and is not sensitive to version changes.

Description

Technical Field

[0001] This invention belongs to the field of standardized data processing, and particularly relates to standardized data processing methods, electronic devices, storage media and computer program products based on large models. Background Technology

[0002] In typical information-intensive industries such as government affairs, healthcare, and finance, data sources exhibit high heterogeneity. Structured data, semi-structured forms, business interfaces, and textual records coexist. Different systems lack unified standards in terms of field naming conventions, field meanings, value encoding, and structural placement. This often results in fields with similar semantic expressions but completely different names, making it impossible to directly compare, aggregate, or model the same business concept during data governance. For example, "birth time," "birth date," and "birth month and year" are often scattered across multiple systems; even if their meanings are similar, differences in expression prevent direct merging and unification. Furthermore, the standards used by different systems or at different historical stages are constantly evolving. For instance, standard fields in medical insurance, finance and taxation, and social security have undergone naming changes, enumeration adjustments, and unit conversions in different years, making it difficult to integrate historical data with the current rule system. In existing technologies, standardization processing largely relies on manually constructed field mapping tables or fixed rule engines, which are not only costly to build and maintain but also severely limited in handling semantic shifts or the evolution of standard versions. In recent years, although some systems have attempted to use keyword- or similarity-based matching algorithms to achieve field normalization, these methods lack the ability to understand contextual structure and version migration mechanisms, and often cannot adapt to complex business contexts and dynamically updated standard systems in actual use.

[0003] With the advancements in large model technology for language understanding, structural encoding, and knowledge reasoning, it has become possible to build a field standardization system that is adaptive, scalable, and semantically accurate. Especially in scenarios where standards evolve frequently and semantic boundaries are ambiguous, there is an urgent need for an automated processing mechanism that integrates context modeling, semantic distance judgment, version attribution identification, and standard mapping output to solve the problem of insufficient standard unification capability of existing technologies in multi-source heterogeneous scenarios. Summary of the Invention

[0004] The purpose of this invention is to propose a standardized data processing method, electronic device, storage medium, and computer program product based on a large model to solve the above-mentioned problems.

[0005] To achieve the above objectives, a standardized data processing method based on a large model is provided in a first aspect of the present invention, the method comprising the following steps:

[0006] S1. Generate standard field semantic vectors by inputting field name strings, field value sample lists, and field context structure information through large model encoding. Each field has its own vector; the set of all fields is combined into a vector set.

[0007] S2. Calculate the semantic offset between the semantic vector of the field to be standardized and the semantic vector of the standardized field. Combine the context weight and confidence regularization term to make a matching judgment. The successfully matched seats are the set of successfully matched field pairs, and the unsuccessful ones are the set of normalized failed fields.

[0008] S3. Obtain historical standard versions. For the set of normalized failure fields, calculate the semantic distance, time system prior, and version uniqueness score between the set of normalized failure fields and the historical standard version to identify the historical standard version to which the current set of normalized failure fields belongs, and generate the optimal field and maximum score version mapping set, wherein the maximum score version is the optimal determination result of the version uniqueness score.

[0009] S4. Based on the optimal field, perform semantic positioning in the version with the highest score, calculate the matching field; based on the matching field matching mapping rule table, obtain its standard mapping field information in the current version, and output it.

[0010] Furthermore, the large model adopts a 12-layer Transformer encoder structure, with each layer including a multi-head self-attention mechanism and a feedforward neural network sublayer.

[0011] Furthermore, the step of generating a standard field semantic vector by inputting the field name string, the field value sample list, and the field's contextual structure information through a large model encoding is specifically as follows:

[0012] The input sequence is constructed from the field name string input, the field value sample list, and the field context structure information;

[0013] The input sequence is transformed into an embedding vector through a word embedding function;

[0014] The embedded vector is input into the Transformer for semantic modeling, and a standard field semantic vector is generated in the [CLS] position and then output.

[0015] Furthermore, S2 specifically refers to:

[0016] A context weight is introduced to represent the co-occurrence weight of a field with a standard field in terms of structural context; the context weight is obtained by the co-occurrence frequency of the context field name and the relative position of the field.

[0017] Based on the context weights, the cosine similarity between the semantic vector of the field to be standardized and the semantic vector of the standardized field is calculated and normalized, and used as the matching score.

[0018] Furthermore, S2 also includes:

[0019] Once the matching score is calculated, the index of the standard field with the highest score is found for each semantic vector of the field to be standardized, and it is determined whether the matching score exceeds the normalization threshold. If the condition is met, it is directly normalized; otherwise, it is left for the next step.

[0020] If the matching score exceeds the normalization threshold, the match is successful, a normalized pair is generated, and a set of successfully matched field pairs is formed.

[0021] Otherwise, add the semantic vector of the field to be normalized to the set of fields that failed to be normalized.

[0022] Furthermore, S3 specifically refers to:

[0023] Introduce field metadata information, including the data record collection time to which the field belongs, the data source system number or version number to which the field belongs, and descriptive information about the field;

[0024] Obtain the field embedding set under each standard version, wherein each standard field vector in the field embedding set is obtained by encoding the large model;

[0025] Obtain the semantic vector of the field to be standardized from the set of fields that failed to be normalized, and calculate the minimum semantic distance based on the standard version; and introduce the attribution probability obtained by the prior matching function and the version sparsity regularization term based on the version uniqueness score into the calculation of the minimum semantic distance; wherein the larger the version sparsity regularization term is, the more distinctive the standardized field semantic vector of the current standard version is.

[0026] The overall scoring function for each version is calculated based on the semantic vector, the version sparsity regularization term, and the prior matching function.

[0027] Based on the overall scoring function, the version with the highest score is identified, and combined with the corresponding standardized field semantic vector, an optimal field and maximum score version mapping set is generated.

[0028] Furthermore, S4 specifically refers to:

[0029] Using the optimal field, semantic localization is performed in the version with the highest score to find the standard field vector that is most semantically similar to it;

[0030] Once a matching field is found, the mapping rule table is queried to obtain its standard mapping field information in the current version, including: mapping field name, data type, unit, and enumeration value mapping dictionary;

[0031] Construct the final output field structure object based on the standard mapping field information and output it.

[0032] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of any of the above-described standardized data processing methods based on a large model.

[0033] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above-described standardized data processing methods based on a large model.

[0034] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described standardized data processing methods based on a large model.

[0035] The beneficial technical effects of the present invention are at least as follows:

[0036] This invention proposes a standardized data processing method driven by a large model. Addressing the challenges of semantic alignment difficulties in heterogeneous data and the coexistence of historical standard version evolution, it constructs a four-stage processing mechanism encompassing semantic representation, normalization matching, version identification, and structural output. A unified coding model generates field semantic representation vectors, which are then normalized within a standard semantic space. Context enhancement and confidence regularization control matching accuracy. For fields that cannot be directly normalized, a multi-factor scoring function is constructed to identify their historical standard version by combining source metadata and cross-version standard field sets. Based on the assigned version and the original semantic vector, a structured rule base is invoked to generate field names, types, units, and value mappings for the current version, ultimately achieving standard field structure output. This invention proposes several innovative designs at the structural level for standardized scenarios, including context co-occurrence weighting, version identification sparsity regularization, and a version-field semantic joint mapping mechanism. These constitute a closed-loop automated standardization process, avoiding the problems of traditional rule systems' dependence on fixed fields and insensitivity to version changes. It possesses high accuracy, high scalability, and high feasibility, making it suitable for large-scale normalization processing needs of heterogeneous fields in fields such as government affairs, healthcare, and finance. Attached Figure Description

[0037] The present invention will be further described with reference to the accompanying drawings, but the embodiments in the drawings do not constitute any limitation on the present invention. For those skilled in the art, other drawings can be obtained based on the following drawings without creative effort.

[0038] Figure 1 This is a flowchart of the standardized data processing method based on a large model according to the present invention.

[0039] Figure 2 This is a schematic diagram of the physical structure of an electronic device according to the present invention. Detailed Implementation

[0040] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0041] like Figure 1 As shown in the embodiment of the present invention, the standardized data processing method based on a large model includes:

[0042] S1. Generate standard field semantic vectors by inputting field name strings, field value sample lists, and field context structure information through large model encoding. Each field has its own vector; the set of all fields is combined into a vector set.

[0043] Specifically, the input for this step is the structural information of each field in the set of fields to be standardized. Each field consists of the following three parts:

[0044] Field names: such as "date of birth" or "registration time", derived from the data table structure definition, interface field identifier, or field name in the API schema;

[0045] Field value examples: obtained by sampling from the original database or interface. The system collects 3 to 10 typical non-null values ​​for each field and stores them as a list of strings, such as "1989 / 10 / 22" and "October 1989".

[0046] Field context structure information: This includes the field's relative position in the original table (e.g., column number), names of adjacent fields, and the type of the parent form. This information originates from the database metadata management system or the interface field structure specification document. All field structure information will be automatically injected by the system during the ETL phase.

[0047] First, the three input parts are concatenated into a description in natural language format for subsequent semantic modeling. This concatenation template is fixed, and an example is: "The field name is 'Date of Birth', typical values ​​include 'October 22, 1989' and 'October 1989', and it often appears alongside the fields 'Gender' and 'Place of Origin'."

[0048] The description is converted into a token sequence by the BPE (BytePairEncoding) segmenter, embedded, and then input into the model.

[0049] The model employs a 12-layer Transformer encoder structure, with each layer including a multi-head self-attention mechanism (hiddensize768, 12 attention heads) and a feedforward neural network sublayer. The entire input sequence is processed by the Embed(·) module before being fed into the Transformer for semantic modeling, ultimately generating a semantic vector E at the [CLS] position. field .

[0050] The calculation is expressed as follows:

[0051] E field =Encoder CLS (TransformerEncoder(Embed(x name ,x samples ,x context ))) (1)

[0052] Where: x name : Field name string input, such as "date of birth", obtained from the database field definition (such as SQL's COLUMN_NAME); x samples : A sample list of field values, such as "October 22, 1989" and "October 1989", obtained by sampling through a database query interface or data collection program;

[0053] x context : Contextual structure information of the field, such as adjacent fields "gender" and "place of origin", extracted from metadata of the data source structure or form template structure; Embed(): Word embedding function, using WordPiece or BPE encoder to convert text into embedding vectors; TransformerEncoder(): 12-layer Transformer encoder module, semantically modeling the embedded sequence; Encoder CLS (): Extracts the CLS bit representation from the Transformer output as the semantic vector for the entire field.

[0054] In actual deployment, each field will undergo the above operations to generate its semantic vector E. fieldThe semantic vectors of all fields are uniformly combined into a vector set:

[0055]

[0056] in: This represents the semantic vector of the i-th field; n represents the number of fields in this processing; all vectors have a uniform dimension of 768, which can be used for vector space matching operations in subsequent steps.

[0057] For example, for the field "Initial Consultation Time" in the healthcare industry, the collected sample values ​​are "2022-01-01" and "January 2022", with the context fields being "Patient" and "Department Number". The model will construct a natural language input: "Field name is 'Initial Consultation Time', typical values ​​include 'January 2022', appearing alongside fields such as 'Patient' and 'Department Number'." This text, after embedding and model encoding, will output E. field This is the unified semantic representation of the field.

[0058] This representation vector possesses characteristics such as fixed dimensions, structural uniformity, and context awareness, allowing it to directly participate in vector space similarity calculations for subsequent normalization discrimination and standard field mapping. The model's fine-tuning data comes from field samples from multiple real-world systems in the target domain, demonstrating strong domain generalization capabilities. Particularly in structural scenarios, this representation vector not only reflects the semantics of the field ontology but also includes its structural role in business forms, such as the distribution patterns of time-based, identifier-based, and grouping-based fields.

[0059] The innovation of this step lies in the fact that by unifying the three types of input data—field names, value samples, and context—a semantic encoder is used to achieve a structured and comparable semantic representation output of heterogeneous fields. For the first time, the role of field structure is explicitly introduced into the standardized pre-semantic processing flow, establishing a unified and high-quality field representation foundation for subsequent steps.

[0060] Output: The output is a field semantic vector E. field Each field represents a vector; the set of all fields is denoted as . This will be used in the next step to calculate semantic offset and determine matching with standard fields.

[0061] S2. Calculate the semantic offset between the semantic vector of the field to be standardized and the semantic vector of the standardized field. Combine the context weight and confidence regularization term to make a matching judgment. The successfully matched seats are the set of successfully matched field pairs, and the unsuccessful ones are the set of normalized failed fields.

[0062] Specifically, the input for this step is the set of field semantic vectors generated in step one. Each Let represent the semantic representation of the i-th field to be standardized, where d = 768, and is the unified embedding dimension output by the Transformer encoder. Another key input is the set of standard field vectors. The set of semantic embeddings of fields under the standard system is also derived from the standard field names, standard value samples and their structural context information through the same model structure processing in step one.

[0063] The objective of this step in the invention is to, in a given embedding vector space, find the normalization path for each field E to be normalized. field Find the most suitable standard field vector It then determines whether the match satisfies the normalization condition. To achieve this goal, this step proposes a semantic offset discrimination method based on context-weighted cosine matching and semantic signal-to-noise ratio adjustment mechanism, which enhances the model's ability to distinguish boundary cases, and is particularly suitable for data standardization problems with strong ambiguity and high business context dependence in patent scenarios.

[0064] This invention first calculates the cosine similarity between each field vector and all standard field vectors. To enhance sensitivity to structural context information in business scenarios, a context weight α is introduced. i,j This represents the contextual co-occurrence weight between field i and standard field j in terms of structure. This weight is composed of information such as the co-occurrence frequency of contextual field names and the difference in relative field positions, and is defined as follows:

[0065]

[0066] in: This represents the name of the k-th context field of field i (from the structure information in step one); This represents the k-th context field of the standard field j; I() is an indicator function, 1 for equality and 0 for otherwise; K is the maximum comparison depth of the context fields, defaulted to 3; α i,j ∈[0,1], used for weighted similarity scoring.

[0067] Then, the improved context-weighted cosine similarity is defined as follows:

[0068]

[0069] Where: the first term is the weighted cosine similarity, ∈ is a stabilizing factor to prevent division by zero; the second term R j This is the normalized confidence regularization term for the standard field j, derived from its historical normalization success rate, matching consistency records, etc., reflecting the semantic clarity of the standard field; λ is the regularization adjustment factor, which defaults to 0.05 and controls the influence weight of the normalized confidence.

[0070] The entire similarity calculation result Simi,j ∈[0,1], can be directly used for matching scores.

[0071] The innovation of this structure lies in combining contextual structure information with semantic space, and introducing a normalized reliability prior from the business domain (via R) into the matching function. j This modeling approach is particularly suitable for scenarios where field names are highly diverse but their structural positions are relatively stable, such as in government affairs, healthcare, and finance. Traditional methods only consider the character distance of field names or the dot product of semantic vectors, ignoring the role of contextual stability, which can easily lead to misclassification of "birthplace" as "birth date." This method can effectively reduce such "mismatching of high-frequency fields in the same domain."

[0072] Once the scoring matrix Sim i,j After the calculation is complete, this invention finds the standard field index j with the highest score for each field i. * And determine whether its score exceeds the normalization threshold δ th If the conditions are met, the value is directly normalized; otherwise, it is left for processing in the next step.

[0073] like If the match is successful, a pair will be generated.

[0074] Otherwise, Add to the set of fields that failed to normalize.

[0075] Normalized threshold δ th The settings combine domain experience with model parameter tuning results. For example, in an insurance business system, the threshold for name fields can be set to 0.85, while the threshold for code fields can be set to a higher 0.92 to improve accuracy.

[0076] For example, in an enterprise customer management system, a field "Customer Source Time" might have values ​​like "October 2022" or "10 / 2022," with the context being "Customer Type" and "Registered Location." In standard fields, both "First Contact Time" and "Registration Time" are potential matches, and traditional methods often fail to distinguish between them. However, this solution uses a contextual co-occurrence weight α. i,j Strengthen the structural relationship between "registration location" and "registration time", and combine the standard field "registration time" in historical data. j The high (multiple successful normalization) information correctly matches the field as "registration time".

[0077] Output: The output consists of two parts: a set of successfully matched field pairs. Collection of fields that failed to normalize: These fields will be further processed in the next step, incorporating version information.

[0078] S3. Obtain historical standard versions. For the set of normalized failure fields, calculate the semantic distance, time system prior, and version uniqueness score between the set of normalized failure fields and the historical standard version to identify the historical standard version to which the current set of normalized failure fields belongs, and generate the optimal field and maximum score version mapping set, wherein the maximum score version is the optimal determination result of the version uniqueness score.

[0079] Specifically, this step follows immediately after step two, and its input is the set of fields that failed to normalize. Each of them This is the semantic vector obtained through the Transformer encoder in step one, which failed to match the standard field in step two. In addition, to improve the accuracy of determining the version attribution of a field, this step also introduces field metadata information.

[0080] t k : Indicates the data record collection time to which field k belongs, which is written into the data timestamp field by the data source system;

[0081] s k : Indicates the data source system number or version number of field k, which is recorded by the ETL acquisition program or data integration platform;

[0082] m k : This refers to descriptive information about the fields (such as form names and field comments), which comes from the business table structure definition or data interface documentation.

[0083] The core objective of this step in the invention is to identify the historical standard version v to which each unnormalized field belongs. k This is to provide a prerequisite for generating mapping rules from a specific version to the current version in the next step. Traditional methods often assume that all fields use the current standard, ignoring the widespread problem of "data version misalignment" in reality, especially in environments where medical insurance, finance, and government standards are frequently iterated. For example, there are significant changes in field names, encoding, and interpretation rules for medical insurance diagnostic fields between V2017 and V2021. Directly normalizing these fields to the current version would cause serious semantic distortion.

[0084] Therefore, this step proposes a multi-dimensional scoring mechanism that combines cross-version semantic distance analysis, temporal system prior estimation, and version sparsity control, specifically designed to address data standardization environments with multiple versions coexisting and high field overlap.

[0085] First, this invention constructs a set of field embeddings for each standard version v∈V. Each standard field vector is encoded using the same Transformer structure as in step one, ensuring consistency in semantic representation. Next, for each field to be processed... This invention defines the minimum semantic distance between this field and version v as:

[0086]

[0087] Where: ∈ is a very small number (e.g., 10). -6 To prevent division by zero; the cosine distance between vectors reflects the minimum proximity of field semantics across different versions; The smaller the value, the more likely the field is to have a corresponding item in that version.

[0088] Since semantic similarity itself cannot fully reflect field version attribution (because some fields may have semantically similar terms in multiple versions), this invention introduces a time and system prior matching function. Used based on collection time t k and data source s k Give the probability of version attribution:

[0089]

[0090] Where: TimeScore(t) k `,v)` represents the score based on the distance between the field collection time and the version release time. For example, if the field was collected in October 2020, but version `v` was released in January 2021, the score will be low. `SourceScore(s)` k ,v) represents the historical compatibility score between the data source system and the standard version (e.g., a business system deployed in 2019 using Medical Insurance V2017); w t ,w s For time and system weight parameters (such as w) t =0.6, w s =0.4); σ() is the sigmoid function, used to normalize the score to the [0,1] interval;

[0091] To further avoid incorrect field attribution due to semantic overlap across multiple versions, this invention introduces a version sparsity regularization term Ω(v), defined as the uniqueness score of all fields under version v in the global standard system:

[0092]

[0093] The larger this value is, the more distinctive the version v field is. After regularization, it encourages the attribution judgment to prioritize the version with higher semantic distinctiveness.

[0094] Finally, the present invention constructs the overall scoring function for this field for each version as follows:

[0095]

[0096] Among them: the first term is the semantic matching score (the smaller the distance, the higher the score); the second term is the time system prior; the third term is the version sparsity regularization to avoid ambiguous version attribution; λ and η are adjustable weights (e.g., λ = 0.5, η = 0.2).

[0097] The version identification result is the version with the highest score:

[0098]

[0099] This structure integrates an innovative weighting mechanism across three levels, making it particularly suitable for the "standardization of historical multi-source heterogeneous data" problem addressed in this patent. Compared to general multi-classification models, this solution does not require the construction of version-labeled samples. Instead, it utilizes existing metadata and standard fields to self-supervise the generation of a comparison structure, greatly reducing the deployment threshold.

[0100] Taking a real-world business scenario as an example: the "ID Card Type" field in the government archives management system had different encoding methods between 2018 and 2022 (early using "0-Other" and "1-Resident ID Card", later using "01-Resident ID Card" and "99-Other"). Traditional normalization strategies struggled to determine the field's origin. This solution uses time t... k =2019, System s k =“Service Hall V2.1”, automatically identifies that this field belongs to the 2017 version, thereby guiding the subsequent standard mapping generation.

[0101] The output is a collection of field-version mappings. Used in the next step to build the standard field mapping path for the current version. Each v k All results are derived from the optimal judgment of the scoring function, possessing semantic interpretability and version traceability capabilities.

[0102] S4. Based on the optimal field, perform semantic positioning in the version with the highest score, calculate the matching field; based on the matching field, query the mapping rule table to obtain its standard mapping field information in the current version, and output it.

[0103] Specifically, this step aims to... In version v k The meaning of "under" is transformed into the standard field structure under the current version v0 through mapping rules, including field name, unit, data type, enumeration value mapping, etc., to achieve the final normalized output.

[0104] First, using In the original version v k Semantic localization is performed in [the context of the sentence]. Find the standard field vector that has the closest semantics.

[0105]

[0106] in: For version v k The standard field vector set; ∈ is a small constant to avoid division by zero. Find the matching field j. * Then, query the mapping rule table. Retrieve its standard mapping field information in the current version, including: mapping field name Data types (such as strings, floating-point numbers, booleans, etc.); units (e.g., yuan, day, %); enumeration value mapping dictionary Used for standard conversion of field values.

[0107] Construct the final output field structure object f k :

[0108]

[0109] This structure contains all the definition information of the field in the current version system, and can be directly used for updating business database fields and standardizing API interface returned fields.

[0110] For example, the original field is "Diagnosis Classification", the version is Medical Insurance V2017, and the actual value is "01-Primary Diagnosis". This field is in v k =In V2017, the standard field "Diagnosis Category" is matched. In V2021, the standard field is "Diagnosis Type," with the unit remaining the same. The value "01-Primary Diagnosis" needs to be mapped to "Primary." The final generated field structure is as follows:

[0111]

[0112] The innovation of this step is reflected in the following points:

[0113] Define the standard field structured output as a field object f. k It is not limited to field name mapping, but includes type, unit, and value conversion logic, and supports direct deployment;

[0114] By aligning the versions semantically and then executing the mapping rules, redundant modeling is avoided and efficiency is improved.

[0115] The mapping rules support dynamic expansion (such as fine-tuning through inter-version migration learning) and have long-term adaptability.

[0116] The final output is a structured standard field set F std =f kEach object contains the structural definition of its fields under the current standard version. This collection can be used for unified data warehouse fields, standard interface data models, or AI model feature dictionaries.

[0117] Figure 2 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 2 As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430, and a communication bus 440, wherein the processor 410, the communication interface 420, and the memory 430 communicate with each other through the communication bus 440. The processor 410 can call logical instructions in the memory 430 to execute a standardized data processing method based on a large model. This method includes: generating standard field semantic vectors by inputting field name strings encoded in the large model, a list of field value samples, and contextual structure information of the fields, where each field has its own vector; combining all fields into a vector set; calculating the semantic offset between the semantic vector of the field to be standardized and the semantic vector of the standardized field, and performing matching determination by combining context weights and confidence regularization terms; assigning successfully matched field pairs to the set of successfully matched pairs, and assigning unsuccessful pairs as a set of normalized failed field pairs; obtaining historical standard versions; for the set of normalized failed field pairs, calculating the semantic distance, time system prior, and version uniqueness score between the set of normalized failed field pairs and the historical standard versions to identify the historical standard version to which the current set of normalized failed field pairs belongs; generating an optimal field and a maximum-score version mapping set, where the maximum-score version is determined by the optimal determination result of the version uniqueness score; performing semantic positioning in the maximum-score version based on the optimal field, and calculating the matching field; querying the mapping rule table based on the matching field to obtain its standard mapping field information under the current version, and outputting it.

[0118] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0119] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the large-model-based standardized data processing method provided by the above methods. This method includes: generating standard field semantic vectors by inputting field name strings encoded by the large model, a list of field value samples, and contextual structure information of the fields, wherein each field has one vector; combining the set of all fields into a vector set; calculating the semantic offset between the semantic vector of the field to be standardized and the semantic vector of the standardized field, and performing matching by combining context weights and confidence regularization terms. The process involves several steps: First, a matching set is established. Successfully matched seats are identified as a set of matched field pairs, while unsuccessful matches are classified as a set of normalized failed field pairs. Then, a historical standard version is retrieved. For each normalized failed field pair, the semantic distance, temporal system prior, and version uniqueness score between the normalized failed field pair and the historical standard version are calculated to identify the historical standard version to which the current normalized failed field pair belongs. This generates a mapping set of optimal fields and a version with the highest score, where the version with the highest score is determined by the optimal judgment result of the version uniqueness score. Next, semantic positioning is performed within the version with the optimal fields, and a matching field is calculated. Finally, the mapping rule table is queried based on the matching field to obtain its standard mapping field information under the current version, and this information is output.

[0120] Furthermore, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, this computer program implements the standardized data processing method based on a large model provided by the methods described above. This method includes: generating standard field semantic vectors by inputting field name strings encoded in the large model, a list of field value samples, and contextual structure information of the fields, wherein each field has one vector; combining the set of all fields into a vector set; calculating the semantic offset between the semantic vector of the field to be standardized and the semantic vector of the standardized field; performing matching determination by combining context weights and confidence regularization terms; and determining the successfully matched seats. The matching field set is configured, and those that fail are treated as a set of normalized failed fields. Historical standard versions are obtained. For each set of normalized failed fields, the semantic distance, temporal system prior, and version uniqueness score between the set of normalized failed fields and the historical standard version are calculated to identify the historical standard version to which the current set of normalized failed fields belongs. An optimal field and a maximum-scoring version mapping set are generated, where the maximum-scoring version is determined by the optimal version uniqueness score. Semantic positioning is performed within the maximum-scoring version based on the optimal field, and a matching field is calculated. Based on the matching field, the mapping rule table is queried to obtain the standard mapping field information for the current version, and this information is output.

[0121] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0122] In the embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of apparatuses or units may be electrical, mechanical, or other forms.

[0123] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0124] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. A standardized data processing method based on a large model, characterized in that, The method includes the following steps: S1. Generate standard field semantic vectors by inputting field name strings, field value sample lists, and field context structure information through large model encoding. Each field has its own vector; the set of all fields is combined into a vector set. S2. Calculate the semantic offset between the semantic vector of the field to be standardized and the semantic vector of the standardized field. Combine the context weight and confidence regularization term to make a matching judgment. The successfully matched fields are the set of successfully matched field pairs, and the unmatched fields are the set of fields that have failed to be normalized. S3. Obtain historical standard versions. For the set of normalized failure fields, calculate the semantic distance, time system prior, and version uniqueness score between the set of normalized failure fields and the historical standard version to identify the historical standard version to which the current set of normalized failure fields belongs, and generate the optimal field and maximum score version mapping set, wherein the maximum score version is the optimal determination result of the version uniqueness score. S4. Based on the optimal field, perform semantic positioning in the version with the highest score, calculate the matching field; based on the matching field matching mapping rule table, obtain its standard mapping field information in the current version, and output it; Specifically, S2 refers to: A context weight is introduced to represent the co-occurrence weight of a field with a standard field in terms of structural context; the context weight is obtained by the co-occurrence frequency of the context field name and the relative position of the field. Based on the context weights and confidence regularization terms, the cosine similarity between the semantic vector of the field to be standardized and the semantic vector of the standardized field is calculated and normalized as a matching score. Specifically, S3 is: Introduce field metadata information, including the data record collection time to which the field belongs, the data source system number or version number to which the field belongs, and descriptive information about the field; Obtain the field embedding set under each standard version, wherein each standard field vector in the field embedding set is obtained by encoding the large model; Obtain the semantic vector of the field to be standardized from the set of fields that failed to be normalized, and calculate the minimum semantic distance based on the standard version; and introduce the attribution probability obtained by the prior matching function and the version sparsity regularization term based on the version uniqueness score into the calculation of the minimum semantic distance; wherein the larger the version sparsity regularization term, the more distinctive the standardized field semantic vector of the current standard version is. The overall scoring function for each version is calculated based on the semantic vector, the version sparsity regularization term, and the prior matching function. Based on the overall scoring function, the version with the highest score is identified, and combined with the corresponding standardized field semantic vector, an optimal field and maximum score version mapping set is generated.

2. The standardized data processing method based on a large model according to claim 1, characterized in that, The large model adopts a 12-layer Transformer encoder structure, with each layer including a multi-head self-attention mechanism and a feedforward neural network sublayer.

3. The standardized data processing method based on a large model according to claim 1, characterized in that, The process of generating a standard field semantic vector by inputting the field name string, the field value sample list, and the field's contextual structure information using a large model encoding method is as follows: The input sequence is constructed from the field name string input, the field value sample list, and the field context structure information; The input sequence is transformed into an embedding vector through a word embedding function; The embedded vector is input into the Transformer for semantic modeling, and a standard field semantic vector is generated in the [CLS] position and then output.

4. The standardized data processing method based on a large model according to claim 1, characterized in that, The S2 further includes: Once the matching score is calculated, the index of the standard field with the highest score is found for each semantic vector of the field to be standardized, and it is determined whether the matching score exceeds the normalization threshold. If the condition is met, it is directly normalized; otherwise, it is left for the next step. If the matching score exceeds the normalization threshold, the match is successful, a normalized pair is generated, and a set of successfully matched field pairs is formed. Otherwise, add the semantic vector of the field to be normalized to the set of fields that failed to be normalized.

5. The standardized data processing method based on a large model according to claim 1, characterized in that, Specifically, S4 is: Using the optimal field, semantic localization is performed in the version with the highest score to find the standard field vector that is most semantically similar to it; Once a matching field is found, the mapping rule table is queried to obtain its standard mapping field information in the current version, including: mapping field name, data type, unit, and enumeration value mapping dictionary; Construct the final output field structure object based on the standard mapping field information and output it.

6. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the standardized data processing method as described in any one of claims 1 to 5.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the standardized data processing method based on a large model as described in any one of claims 1 to 5.

8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the standardized data processing method based on a large model as described in any one of claims 1 to 5.

Images