Data tubular processing method and device, equipment, storage medium and program product

Abstract

The invention relates to a data tubular processing method and device, equipment, a storage medium and a program product. Comprising the steps of obtaining a to-be-processed word set; determining the context length of the word set to be processed; if the context length is greater than a window threshold value of the large language model, performing slicing processing and embedded vector conversion processing on the word set to be processed to obtain a plurality of vector features carrying context information; storing the vector features into a vector database; for any field in the to-be-processed word set, obtaining a target vector feature corresponding to the field from a vector database; and generating a tubular name corresponding to the target vector feature by using a large language model. According to the method, the reasoning ability of the large language model is utilized, the context semantics are fully utilized to carry out high-precision tubular bidding, the optimal tubular bidding strategy is automatically selected according to the context length, information loss or reasoning precision reduction caused by window limitation of the large language model is avoided, and then tubular bidding accuracy is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a data standardization processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Technology

[0002] Currently, data standardization (also known as data standard naming, semantic standardization, or semantic unification) mainly relies on the following methods:

[0003] 1. Rule-based matching: This method uses regular expressions, keyword mapping, and other techniques to map non-standardized field names to a standard thesaurus. While simple to implement, it has limited scalability and semantic understanding capabilities.

[0004] 2. Classification models based on traditional machine learning: This method trains a classifier to map field names to standard vocabulary labels. However, it relies on a large amount of manually labeled training data and struggles to handle new domains and new vocabulary.

[0005] 3. Vector-based semantic search methods: Field names are encoded into vectors using an embedding model and then compared with a standard vocabulary for similarity retrieval. While this approach addresses the semantic matching problem to some extent, it still falls short in handling contextual relevance and polysemous words.

[0006] 4. Language model-based assisted generation methods: Some studies have attempted to use large language models (LLM) to directly generate standard field names, but when the context is too long or the data volume is too large, it will be limited by the model context window, resulting in information truncation or a decrease in inference accuracy. Summary of the Invention

[0007] Therefore, it is necessary to provide a data standardization processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product to address the above-mentioned technical problems. This method can automatically select the optimal standardization strategy based on the context length, avoiding information loss or decreased inference accuracy caused by the limitation of large language model windows, thereby improving the accuracy of standardization.

[0008] Firstly, this application provides a data standardization processing method, including:

[0009] Get the set of words to be processed;

[0010] Determine the context length of the set of words to be processed;

[0011] When the context length is greater than the window threshold of a large language model, the set of words to be processed is sliced ​​and the embedding vector is transformed to obtain several vector features carrying context information.

[0012] Store several vector features carrying contextual information into a vector database;

[0013] For any field in the set of words to be processed, obtain the target vector feature corresponding to the field from the vector database;

[0014] Using a large language model, the standard name corresponding to the target vector feature is generated.

[0015] In one embodiment, for any field in the set of words to be processed, the target vector feature corresponding to the field is obtained from the vector database, including:

[0016] For any field in the set of words to be processed, search the vector database for multiple vectorized representations corresponding to the field;

[0017] Multiple vectorized representations are input into the reordering model for semantic reordering to obtain the reordering result;

[0018] Based on the re-sorting results, select one or more of the top-ranked vectorized representations as the target vector features corresponding to the fields.

[0019] In one embodiment, after generating the standard name corresponding to the target vector feature using a large language model, the method further includes:

[0020] Obtain the audit results corresponding to the standard implementation name;

[0021] Based on the audit results, the fields and standard names will be stored as training samples;

[0022] The parameters of the re-ranking model are adjusted based on several stored training samples.

[0023] In one embodiment, a large language model is used to generate the standard name corresponding to the target vector features, including:

[0024] Generate the first input prompt based on the target vector features and the preset prompt template;

[0025] Using a large language model, the standard name corresponding to the target vector feature is generated based on the first input prompt;

[0026] Accordingly, after obtaining the audit results corresponding to the standard implementation name, the method further includes:

[0027] Based on the audit results for each standard implementation name, determine the error type;

[0028] Add prompts corresponding to the error type to the preset prompt template to obtain the updated preset prompt template.

[0029] In one embodiment, after determining the context length of the set of words to be processed, the method further includes:

[0030] If the context length is less than or equal to the window threshold of the large language model, generate a second input prompt based on the set of words to be processed;

[0031] Using a large language model, based on the second input prompt, generate the standard name corresponding to the set of words to be processed.

[0032] In one embodiment, the method further includes:

[0033] Record the window configuration and performance metrics for each task processed by the large language model;

[0034] Based on the pre-configured expected performance data, as well as the window configuration and performance metrics of each recorded task, determine the window threshold for the current large language model.

[0035] Secondly, this application also provides a data standardization processing apparatus, comprising:

[0036] The acquisition module is used to acquire the set of words to be processed;

[0037] The determination module is used to determine the context length of the set of words to be processed;

[0038] The chunking module is used to slice the set of words to be processed and perform embedding vector transformation when the context length is greater than the window threshold of the large language model, so as to obtain several vector features carrying context information.

[0039] The storage module is used to store several vector features carrying context information into the vector database;

[0040] The standardization module is used to retrieve the target vector features corresponding to any field in the set of words to be processed from the vector database; and to generate the standardization name corresponding to the target vector features using a large language model.

[0041] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method described in the first aspect above.

[0042] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the first aspect above.

[0043] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method described in the first aspect above.

[0044] The aforementioned data standardization processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product acquire a set of words to be processed; determine the context length of the set of words to be processed; when the context length is greater than the window threshold of the large language model, slice the set of words to be processed and perform embedding vector transformation to obtain several vector features carrying context information; store these vector features in a vector database; for any field in the set of words to be processed, retrieve the target vector feature corresponding to that field from the vector database; and use the large language model to generate the standardization name corresponding to the target vector feature. By utilizing the reasoning power of the large language model, high-precision standardization can be achieved by fully leveraging contextual semantics, and it can adapt to new standardization domains. A dynamic adaptation mechanism is proposed to automatically select the optimal standardization strategy based on the context length, avoiding information loss or decreased reasoning accuracy caused by the window limitation of the large language model, thereby improving standardization accuracy. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 This is an application environment diagram of the data standardization processing method in one embodiment;

[0047] Figure 2 This is a flowchart illustrating a data standardization process in one embodiment;

[0048] Figure 3 This is a structural block diagram of a data standardization processing device in one embodiment;

[0049] Figure 4 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0051] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0052] The data standardization processing method provided in this application embodiment can be applied to, for example, Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or placed on a cloud or other network server. Server 104 is configured with a Large Language Model (LLM). Terminal 102 sends a set of words to be processed to server 104 via a client or webpage. Server 104 uses the data standardization processing method provided in this application embodiment to process the set of words submitted by terminal 102, generate standardization names, and return them to terminal 102 for display. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, drones, low-altitude aircraft, IoT devices, and portable wearable devices. IoT devices can be smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, projection devices, etc. Portable wearable devices can be smartwatches, smart bracelets, head-mounted devices, etc. Head-mounted devices can be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. Server 104 can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides cloud computing services.

[0053] In one exemplary embodiment, such as Figure 2 As shown, a data standardization processing method is provided, which is applied to... Figure 1 Taking server 104 as an example, the explanation includes:

[0054] Step 202: Obtain the set of words to be processed.

[0055] The set of words to be processed includes multiple fields that need to be standardized. For example, it might include several fields in an enterprise master data management system that need to be unified. Or, it might include several industry-standard terms that need to be normalized. Or, it might include several data table fields in the banking and finance sectors that need to be mapped. Or, it might include multilingual vocabularies that require semantic standardization.

[0056] Step 204: Determine the context length of the set of words to be processed.

[0057] Optionally, the set of words to be processed is converted into a token sequence, and the length of the token sequence is calculated as the context length of the set of words to be processed.

[0058] Step 206: If the context length is greater than the window threshold of the large language model, the set of words to be processed is sliced ​​and the embedding vector is transformed to obtain several vector features carrying context information.

[0059] Step 208: Store several vector features carrying contextual information into the vector database.

[0060] Step 210: For any field in the set of words to be processed, obtain the target vector feature corresponding to the field from the vector database.

[0061] Step 212: Using a large language model, generate the standard name corresponding to the target vector features.

[0062] The process involves comparing the context length with the window threshold of the large language model. If the context length exceeds the window threshold, a chunked retrieval standardization strategy is employed. This strategy includes a chunked storage phase and a retrieval standardization phase.

[0063] In the block storage stage, an embedding model (such as text-embedding-3-small or bge-large) is used to slice the set of words to be processed and convert them into vectors. The vector features of each slice are then stored in a vector database (such as Milvus, FAISS).

[0064] During the standardization retrieval phase, for the current field, similar fields are retrieved from the vector database, and the corresponding standardization name is generated by the large language model.

[0065] Optionally, the target vector features and a preset prompt template are used as input to the large language model to obtain the standardization name output by the large language model. The preset prompt template is used to prompt the large language model to perform the standardization task.

[0066] The above data standardization processing method involves: obtaining a set of words to be processed; determining the context length of the set of words; if the context length exceeds the window threshold of the large language model, slicing and embedding vector transformation of the set of words to be processed to obtain several vector features carrying context information; storing these vector features in a vector database; for any field in the set of words to be processed, retrieving the target vector feature corresponding to that field from the vector database; and using the large language model to generate the standardization name corresponding to the target vector feature. This approach leverages the reasoning capabilities of the large language model to fully utilize contextual semantics for high-precision standardization, adapting to new standardization domains. A dynamic adaptation mechanism is proposed to automatically select the optimal standardization strategy based on the context length, avoiding information loss or decreased reasoning accuracy caused by the window limitation of the large language model, thereby improving standardization accuracy.

[0067] In an exemplary embodiment, step 210 includes: for any field in the set of words to be processed, searching in a vector database for multiple vectorized representations corresponding to the field; inputting the multiple vectorized representations into a re-ranking model for semantic re-ranking to obtain a re-ranking result; and selecting one or more top-ranked vectorized representations as the target vector features corresponding to the field based on the re-ranking result.

[0068] The process involves receiving the field to be standardized, retrieving similar fields from the vector database, obtaining Top-K approximate matches, and generating K vectorized representations. The results are then input into a reranking model (i.e., a Rerank model, such as bge-reranker or cross-encoder) for semantic reranking. Top-1 or Top-N high-scoring results are input into a large language model, which then generates the final standardized name.

[0069] This embodiment provides a hierarchical standardization mechanism that combines embedding vector retrieval with Rerank semantic reordering. Rerank filtering significantly reduces semantic drift and greatly improves the quality of the candidate set input to the large language model. Based on this, the large language model performs the final decision and standardization output, ensuring high consistency and accuracy of the standardization results.

[0070] In an exemplary embodiment, after step 212, the method further includes: obtaining the audit result corresponding to the standard implementation name; storing the field and the standard implementation name as training samples according to the audit result; and adjusting the parameters of the re-ranking model according to the stored training samples.

[0071] The process can involve manual review of the standardization results, or the use of a pre-trained evaluation model to evaluate the results based on the fields to be standardized and the standardization name. The evaluation results indicate whether the standardization was correct or incorrect. Positive training samples are generated based on correctly standardized fields and names, and negative training samples are generated based on incorrectly standardized fields and names. These positive and negative training samples are stored. The parameters of the re-ranking model are dynamically adjusted based on several updated training samples. By calculating the loss gradient between the model prediction and the manual standard, the model parameters (such as attention weights) are dynamically updated with a small learning rate, enabling it to accurately identify and correct previous errors. The re-ranking model with adjusted parameters is then used to re-rank the standardization retrieval results.

[0072] In this embodiment, by adjusting the parameters of the reordering model, the accuracy of standardization can be improved.

[0073] In an exemplary embodiment, step 212 includes: generating a first input prompt based on the target vector features and a preset prompt template; using a large language model, generating a standardization name corresponding to the target vector features based on the first input prompt; accordingly, after obtaining the audit results corresponding to the standardization name, the method further includes: determining the error type based on the audit results of each standardization name; adding prompt words corresponding to the error type to the preset prompt template to obtain an updated preset prompt template.

[0074] The preset prompt template is a predefined text structure used to guide the large language model in performing the standardization task and generating specific types of output. The first input prompt is a complete prompt text generated based on the target vector features and the preset prompt template, which is then passed to the large language model. In the specific implementation, the first input prompt is input into the large language model, which uses its pre-trained knowledge to generate a standardization name that meets the prompt requirements.

[0075] Error types are specific error categories categorized based on the review results. They help identify problems in the generated names, such as formatting errors, semantic errors, and duplication errors. Semantic errors indicate that the name does not match the meaning of the target feature; formatting errors indicate that the name does not conform to the preset structure (e.g., missing prefix); duplication errors indicate that the name conflicts with an existing name.

[0076] Cue words are specific words or phrases added to the cue template to guide the LLM to avoid specific errors. These words are designed based on error types to enhance the explicitness and constraint of the cue, thereby improving the quality of model generation. For example, if the error type is a semantic error, the cue word could be: "Ensure the name accurately reflects the core features"; or if the error type is a formatting error, the cue word could be: "The standard prefix must be included."

[0077] The updated preset prompt template is a new template obtained by adding prompt words corresponding to the error type on the basis of the original preset prompt template. It is used in the subsequent generation process to reduce similar errors and achieve iterative optimization.

[0078] In this embodiment, based on the determined error type, additional prompts are added to the original preset prompt template. These prompts are designed to constrain or guide the LLM to avoid repeating similar errors. For example, if the error type is a semantic error, the prompts may emphasize accuracy; if it is a formatting error, the prompts may specify formatting rules. The updated template will be used in subsequent generation cycles to progressively improve generation quality.

[0079] In an exemplary embodiment, after step 204, the method further includes: generating a second input prompt based on the set of words to be processed when the context length is less than or equal to the window threshold of the large language model; and generating a standard name corresponding to the set of words to be processed based on the second input prompt using the large language model.

[0080] If the context length is less than or equal to the window threshold of the large language model, a second input prompt is generated based on the set of words to be processed and the prompt template (which can be the same as or different from the aforementioned preset prompt template); the corresponding standard implementation name is generated based on the second input prompt using the large language model; the output result is used as a standard implementation mapping table.

[0081] In an exemplary embodiment, the method further includes: recording the window configuration and performance metrics of the large language model for each task; and determining the current window threshold of the large language model based on pre-configured expected performance data and the recorded window configuration and performance metrics for each task.

[0082] Here, window configuration refers to the window size input to the large language model when processing a specific task. Performance metrics are quantitative standards for measuring the effectiveness of the large language model in handling the task, including accuracy and response time. Expected performance data refers to pre-set performance targets, such as an accuracy higher than 90% and an average response time lower than 2 seconds. Optionally, multiple window configurations that meet the expected performance data are selected from the recorded data, and the window size with the largest value is selected as the current window threshold for the large language model. Optionally, a prediction model of window size and task performance is trained based on the recorded window configurations and performance metrics. The prediction model is used to calculate the prediction performance for the current task under different window sizes; based on the expected performance data, the optimal window threshold is selected. In this way, better window thresholds can be continuously determined to ensure that the performance of the large language model remains at a high level.

[0083] In this embodiment, an adaptive context-aware strategy is adopted. For inputs of appropriate length, processing is performed directly within the model's capabilities to ensure efficiency. When the input is excessively long, a block-based retrieval mechanism is automatically triggered to improve the processing efficiency of the large language model. A hierarchical processing architecture consisting of embedding, Rerank, and LLM is used to achieve dual enhancement of semantic understanding and contextual relevance. Rerank filtering significantly reduces semantic drift and significantly improves the quality of the candidate set input to the large language model. This method has strong versatility and is applicable to vocabularies in any domain. It is not only suitable for internal field standardization within enterprises but can also be applied to scenarios such as industry data governance and knowledge graph construction.

[0084] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0085] Based on the same inventive concept, this application also provides a data standardization processing apparatus for implementing the data standardization processing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more data standardization processing apparatus embodiments provided below can be found in the limitations of the data standardization processing method described above, and will not be repeated here.

[0086] In one exemplary embodiment, such as Figure 3 As shown, a data standardization processing device is provided, comprising:

[0087] Module 302 is used to obtain the set of words to be processed;

[0088] Module 304 is used to determine the context length of the set of words to be processed;

[0089] The segmentation module 306 is used to slice the set of words to be processed and perform embedding vector transformation processing when the context length is greater than the window threshold of the large language model, so as to obtain several vector features carrying context information.

[0090] Storage module 308 is used to store several vector features carrying context information into a vector database;

[0091] The standardization module 310 is used to obtain the target vector features corresponding to any field in the set of words to be processed from the vector database; and to generate the standardization name corresponding to the target vector features using a large language model.

[0092] The aforementioned data standardization processing device leverages the reasoning capabilities of a large language model to achieve high-precision standardization by fully utilizing contextual semantics, and can adapt to new standardization domains. A dynamic adaptation mechanism is proposed, which automatically selects the optimal standardization strategy based on the context length, avoiding information loss or decreased reasoning accuracy caused by the window limitations of the large language model, thereby improving standardization accuracy.

[0093] In an exemplary embodiment, the segmentation module 306 is further configured to search for multiple vectorized representations corresponding to any field in the set of words to be processed in the vector database; input the multiple vectorized representations into the reordering model for semantic reordering to obtain the reordering result; and select one or more top-ranked vectorized representations as the target vector features corresponding to the field based on the reordering result.

[0094] In an exemplary embodiment, the data standardization processing device further includes an adjustment module, which is used to obtain the audit results corresponding to the standardization name; store the fields and standardization names as training samples according to the audit results; and adjust the parameters of the re-ranking model according to the stored training samples.

[0095] In an exemplary embodiment, the standardization module 310 is further configured to generate a first input prompt based on the target vector features and the preset prompt template; generate a standardization name corresponding to the target vector features based on the first input prompt using a large language model; correspondingly, the adjustment module is further configured to determine the error type based on the review result of each standardization name; and add prompt words corresponding to the error type to the preset prompt template to obtain an updated preset prompt template.

[0096] In an exemplary embodiment, the standardization module 310 is further configured to generate a second input prompt based on the set of words to be processed when the context length is less than or equal to the window threshold of the large language model; and to generate a standardization name corresponding to the set of words to be processed based on the second input prompt using the large language model.

[0097] In an exemplary embodiment, the determining module 304 is further configured to record the window configuration and performance metrics of the large language model for each task; and to determine the current window threshold of the large language model based on the pre-configured expected performance data and the recorded window configuration and performance metrics of each task.

[0098] Each module in the aforementioned data standardization processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0099] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores vector features. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When executed by the processor, the computer program implements a data standardization processing method.

[0100] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0101] In one exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to perform the following steps: obtaining a set of words to be processed; determining the context length of the set of words to be processed; if the context length is greater than the window threshold of a large language model, performing slicing and embedding vector transformation on the set of words to be processed to obtain several vector features carrying context information; storing the several vector features carrying context information into a vector database; for any field in the set of words to be processed, obtaining the target vector feature corresponding to the field from the vector database; and using a large language model to generate the standard name corresponding to the target vector feature.

[0102] In one embodiment, when the processor executes the computer program, it further performs the following steps: for any field in the set of words to be processed, it searches in the vector database for multiple vectorized representations corresponding to the field; it inputs the multiple vectorized representations into the reordering model for semantic reordering to obtain the reordering result; and based on the reordering result, it selects one or more top-ranked vectorized representations as the target vector features corresponding to the field.

[0103] In one embodiment, when the processor executes the computer program, it also performs the following steps: obtaining the audit result corresponding to the standard implementation name; storing the field and the standard implementation name as training samples based on the audit result; and adjusting the parameters of the re-ranking model based on the stored training samples.

[0104] In one embodiment, when the processor executes the computer program, it further performs the following steps: generating a first input prompt based on the target vector features and a preset prompt template; using a large language model, generating a standardization name corresponding to the target vector features based on the first input prompt; determining the error type based on the review result of each standardization name; and adding prompt words corresponding to the error type to the preset prompt template to obtain an updated preset prompt template.

[0105] In one embodiment, when the processor executes the computer program, it further performs the following steps: generating a second input prompt based on the set of words to be processed when the context length is less than or equal to the window threshold of the large language model; and generating a standard name corresponding to the set of words to be processed based on the second input prompt using the large language model.

[0106] In one embodiment, when the processor executes the computer program, it also performs the following steps: recording the window configuration and performance metrics of the large language model for each task; and determining the current window threshold of the large language model based on pre-configured expected performance data and the recorded window configuration and performance metrics for each task.

[0107] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, it performs the following steps: obtaining a set of words to be processed; determining the context length of the set of words to be processed; if the context length is greater than the window threshold of a large language model, performing slicing and embedding vector transformation on the set of words to be processed to obtain several vector features carrying context information; storing the several vector features carrying context information into a vector database; for any field in the set of words to be processed, obtaining the target vector feature corresponding to the field from the vector database; and using a large language model to generate the standard name corresponding to the target vector feature.

[0108] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: for any field in the set of words to be processed, searching in the vector database for multiple vectorized representations corresponding to the field; inputting the multiple vectorized representations into the reordering model for semantic reordering to obtain the reordering result; and selecting one or more top-ranked vectorized representations as the target vector feature corresponding to the field based on the reordering result.

[0109] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: obtaining the audit result corresponding to the standardization name; storing the field and the standardization name as training samples based on the audit result; and adjusting the parameters of the re-ranking model based on the stored training samples.

[0110] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: generating a first input prompt based on the target vector features and a preset prompt template; using a large language model, generating a standardization name corresponding to the target vector features based on the first input prompt; determining the error type based on the review result of each standardization name; and adding prompt words corresponding to the error type to the preset prompt template to obtain an updated preset prompt template.

[0111] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: generating a second input prompt based on the set of words to be processed when the context length is less than or equal to the window threshold of the large language model; and generating a standard name corresponding to the set of words to be processed based on the second input prompt using the large language model.

[0112] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: recording the window configuration and performance metrics of the large language model for each task; and determining the current window threshold of the large language model based on pre-configured expected performance data and the recorded window configuration and performance metrics for each task.

[0113] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps: obtaining a set of words to be processed; determining the context length of the set of words to be processed; if the context length is greater than the window threshold of a large language model, performing slicing and embedding vector transformation on the set of words to be processed to obtain several vector features carrying context information; storing the several vector features carrying context information into a vector database; for any field in the set of words to be processed, obtaining the target vector feature corresponding to the field from the vector database; and using a large language model to generate the standard name corresponding to the target vector feature.

[0114] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: for any field in the set of words to be processed, searching in the vector database for multiple vectorized representations corresponding to the field; inputting the multiple vectorized representations into the reordering model for semantic reordering to obtain the reordering result; and selecting one or more top-ranked vectorized representations as the target vector feature corresponding to the field based on the reordering result.

[0115] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: obtaining the audit result corresponding to the standardization name; storing the field and the standardization name as training samples based on the audit result; and adjusting the parameters of the re-ranking model based on the stored training samples.

[0116] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: generating a first input prompt based on the target vector features and a preset prompt template; using a large language model, generating a standardization name corresponding to the target vector features based on the first input prompt; determining the error type based on the review result of each standardization name; and adding prompt words corresponding to the error type to the preset prompt template to obtain an updated preset prompt template.

[0117] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: generating a second input prompt based on the set of words to be processed when the context length is less than or equal to the window threshold of the large language model; and generating a standard name corresponding to the set of words to be processed based on the second input prompt using the large language model.

[0118] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: recording the window configuration and performance metrics of the large language model for each task; and determining the current window threshold of the large language model based on pre-configured expected performance data and the recorded window configuration and performance metrics for each task.

[0119] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0120] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0121] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0122] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A data tag processing method, characterized by, The method comprises: acquiring a set of words to be processed; determining the context length of the set of words to be processed; in the case where the context length is greater than the window threshold of a large language model, performing slicing processing and embedding vector conversion processing on the set of words to be processed to obtain a plurality of vector features carrying context information; storing the plurality of vector features carrying context information in a vector database; for any one field in the set of words to be processed, acquiring the target vector feature corresponding to the field from the vector database; using the large language model to generate the title corresponding to the target vector feature.

2. The method of claim 1, wherein, The method further comprises: for any one field in the set of words to be processed, searching for a plurality of vectorized representations corresponding to the field in the vector database; inputting the plurality of vectorized representations into a reordering model for semantic reordering to obtain a reordering result; according to the reordering result, selecting one or more vectorized representations ranked at the front as the target vector feature corresponding to the field.

3. The method of claim 2, wherein, After the large language model is used to generate the title corresponding to the target vector feature, the method further comprises: acquiring an audit result corresponding to the title; storing the field and the title as training samples according to the audit result; adjusting the parameters of the reordering model according to a plurality of stored training samples.

4. The method of claim 3, wherein, The method further comprises: according to the target vector feature and a preset prompt template, generating a first input prompt; using the large language model to generate the title corresponding to the target vector feature according to the first input prompt; Accordingly, after the audit result corresponding to the title is acquired, the method further comprises: determining an error type according to the audit result of each title; adding a prompt word corresponding to the error type in the preset prompt template to obtain an updated preset prompt template.

5. The method of claim 1, wherein, After the context length of the set of words to be processed is determined, the method further comprises: in the case where the context length is less than or equal to the window threshold of the large language model, generating a second input prompt according to the set of words to be processed; using the large language model to generate the title corresponding to the set of words to be processed according to the second input prompt.

6. The method according to any one of claims 1 to 5, characterized in that, The method further comprises: recording the window configuration and performance indicators of the large language model processing each task; determining the window threshold of the current large language model according to the preconfigured expected performance data and the recorded window configuration and performance indicators of each task.

7. A data tag processing apparatus, characterized by comprising: The device comprises: an acquisition module for acquiring a set of words to be processed; a determination module for determining the context length of the set of words to be processed; The block module is configured to perform slicing processing and embedding vector conversion processing on the set of words to be processed to obtain a plurality of vector features carrying context information in a case where the context length is greater than a window threshold of a large language model. The storage module is configured to store the plurality of vector features carrying context information in a vector database. The tag module is configured to, for any one field in the set of words to be processed, acquire a target vector feature corresponding to the field from the vector database, and generate a tag name corresponding to the target vector feature by using the large language model.

8. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, The processor executes the computer program to implement the steps of the method in any one of claims 1 to 6.

9. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 6.

10. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 6.

Images