Training data processing method and device, equipment, storage medium and product

Abstract

The invention provides a training data processing method and device, equipment, a storage medium and a product. Comprises: obtaining a question set of a target domain; for any question in the question set, a first response and a second response are generated, the first response is a response that the pre-training language model is not enhanced, and the second response is a response that the pre-training language model is combined with an external knowledge source to be enhanced; obtaining a similarity value of the first response and the second response; and generating a training data set according to the similarity value. According to the method, the human input and cost are reduced, and the training data generation efficiency is improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, device, storage medium, and product for processing training data. Background Technology

[0002] In the target domain, training large-scale industry models requires highly specialized knowledge support. However, while general-purpose pre-trained language models possess broad knowledge, their training data is mostly geared towards general scenarios and lacks sufficient coverage of specialized knowledge in the vertical target domain. Directly applying general-purpose pre-trained language models can lead to output results that deviate from the requirements, or even produce erroneous conclusions.

[0003] Currently, existing methods rely on manual annotation or static rules to generate training data for the target domain. For example, designing financial questions and annotating answers, or using templated rules to generate question-answer pairs in the financial field. However, this approach is labor-intensive, costly, and inefficient. Summary of the Invention

[0004] This application provides methods, apparatus, devices, storage media, and products for processing training data, in order to reduce manpower and costs and improve the efficiency of generating training data.

[0005] In a first aspect, embodiments of this application provide a method for processing training data, applied to an electronic device, comprising: acquiring a set of questions in a target domain; generating a first response and a second response for any question in the set of questions, wherein the first response is an unenhanced response of a pre-trained language model and the second response is an enhanced response of a pre-trained language model combined with external knowledge sources; acquiring a similarity value between the first response and the second response; and generating a training dataset based on the similarity value.

[0006] In one possible implementation, obtaining the similarity value between the first response and the second response includes: obtaining the semantic similarity value between the first response and the second response; extracting multiple first named entities from the first response; extracting multiple second named entities from the second response; determining the named entity similarity value between the first response and the second response based on the multiple first named entities and the multiple second named entities; and obtaining the similarity value between the first response and the second response based on the semantic similarity value and the named entity similarity value.

[0007] In one possible implementation, obtaining the semantic similarity value between the first response and the second response includes: inputting the first response into a trained semantic embedding model, such that the trained semantic embedding model outputs the feature vector of the first response; inputting the second response into the trained semantic embedding model, such that the trained semantic embedding model outputs the feature vector of the second response; and determining the cosine similarity between the feature vector of the first response and the feature vector of the second response as the semantic similarity value between the first response and the second response.

[0008] In one possible implementation, determining the named entity similarity value between a first response and a second response based on a plurality of first named entities and a plurality of second named entities includes: obtaining a plurality of named entity types based on the plurality of first named entities and a plurality of second named entities; matching the plurality of first named entities with the plurality of second named entities according to the plurality of named entity types to obtain a plurality of named entity pairs; wherein each named entity pair includes one first named entity and one second named entity, and the named entity types of the first named entity and the second named entity in each named entity pair are the same; calculating the similarity value of each named entity pair; and obtaining the named entity similarity value between the plurality of first named entities and the plurality of second named entities based on the similarity value of each named entity pair.

[0009] In one possible implementation, obtaining the similarity value between the first response and the second response based on the semantic similarity value and the named entity similarity value includes: obtaining the similarity value between the first response and the second response based on the semantic similarity value, the named entity similarity value, the preset weight of the semantic similarity value, and the preset weight of the named entity similarity value.

[0010] In one possible implementation, generating training data based on similarity values ​​includes: obtaining multiple questions from a question set whose similarity values ​​are lower than a preset similarity threshold; determining each question and its corresponding second response as a first training dataset; generating a third response for each question using prompts pre-designed for the target domain; and determining each question and its corresponding third response as a second training dataset.

[0011] Secondly, embodiments of this application provide a training data processing apparatus applied to an electronic device, comprising: a first acquisition module for acquiring a question set in a target domain; a first generation module for generating a first response and a second response for any question in the question set, wherein the first response is an unenhanced response of a pre-trained language model, and the second response is an enhanced response of a pre-trained language model combined with external knowledge sources; a second acquisition module for acquiring a similarity value between the first response and the second response; and a second generation module for generating a training dataset based on the similarity value.

[0012] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor;

[0013] The memory stores instructions that the computer executes;

[0014] The processor executes computer execution instructions stored in memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0015] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible implementations of the first aspect.

[0016] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.

[0017] The training data processing method, apparatus, device, storage medium, and product provided in this application generate two responses for any given question: a first response without enhancement from the pre-trained language model, and a second response enhanced by combining the pre-trained language model with external knowledge sources. By quantifying the difference between the first and second responses using their similarity values, the degree of knowledge mastery corresponding to any given question can be assessed when the pre-trained language model is not enhanced. Based on the similarity values, a training dataset is generated. The entire process replaces manually preset rules with a similarity mechanism, reducing manpower, lowering costs, and improving generation efficiency. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0019] Figure 1 A schematic diagram illustrating a scenario for a training data processing method provided in an embodiment of this application;

[0020] Figure 2 A flowchart illustrating the training data processing method provided in the embodiments of this application;

[0021] Figure 3 A schematic diagram of the structure of the training data processing device provided in the embodiments of this application;

[0022] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0023] The accompanying drawings have illustrated specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to specific embodiments. Detailed Implementation

[0024] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0025] Figure 1 This is a schematic diagram of a scenario for the training data processing method provided in the embodiments of this application, such as... Figure 1 As shown, it includes: a receiving device 101, a processing device 102, and a display device 103.

[0026] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the method of processing training data. In other feasible embodiments of this application, the above architecture may include more or fewer components than illustrated, or combine some components, or split some components, or arrange different components, which can be determined according to the actual application scenario and is not limited here. Figure 1 The components shown can be implemented in hardware, software, or a combination of both.

[0027] In the specific implementation process, the receiving device 101 can be an input / output interface or a communication interface, and can acquire the problem set of the target domain.

[0028] The processing device 102 can generate a first response and a second response for any question in the question set. The first response is the response of the pre-trained language model without enhancement, and the second response is the response of the pre-trained language model enhanced by external knowledge sources. It can also obtain the similarity value between the first response and the second response and generate a training dataset based on the similarity value.

[0029] The display device 103 can be used to display the training dataset.

[0030] It should be understood that the aforementioned processor can be implemented by reading instructions from memory and executing those instructions, or it can be implemented through chip circuitry.

[0031] Furthermore, the network architecture and business scenarios described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the evolution of network architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0032] To address the aforementioned technical challenges, this application proposes the following technical concept: how to accurately identify knowledge that the model has not mastered and generate high-value training data. By analogy with the human learning process, the model autonomously learns the knowledge it lacks, quantifying this unmastered knowledge. Specifically, for any given question, two responses are generated: a first response from the pre-trained language model without enhancement, and a second response from the pre-trained language model enhanced by external knowledge sources. The difference between the first and second responses is quantified by the similarity value, allowing assessment of the pre-trained language model's mastery of the corresponding knowledge for any given question without enhancement. Based on the similarity value, a training dataset is generated. The entire process replaces manually preset rules with a similarity mechanism, reducing manpower, lowering costs, and improving generation efficiency.

[0033] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0034] Figure 2 A flowchart illustrating the training data processing method provided in the embodiments of this application is shown below. Figure 2 As shown, the method includes:

[0035] S201: Obtain the problem set of the target domain.

[0036] In this embodiment, the scope of the vertical field is first defined by the target field, such as the financial field, the legal field, and the medical field, so as to clarify the boundaries of the core knowledge of the field.

[0037] In this embodiment, a multi-channel problem collection approach is adopted for the target domain. Channel 1: Expert-designed questions, designed by multiple senior experts in the target domain, focusing on core knowledge, business pain points, and high-frequency scenarios, covering more than 80% of the core knowledge points; Channel 2: User log extraction questions, extracting questions from real user interaction logs in the target domain, such as bank customer service logs and medical consultation records, and using deduplication algorithms to remove duplicate questions, retaining real and high-frequency consultations; Channel 3: Model generation questions, constructing a knowledge graph of the target domain, including core entities, such as credit reports, entity attributes, such as credit reports containing 24 months of repayment records and entity relationships, such as credit card applications relying on credit reports; designing prompts, and calling a large model to generate questions.

[0038] For example, taking the financial sector as the target domain, the financial expert team provides 50 core questions for expert-designed questions, such as: how to identify group misconduct in credit card applications, and which financial indicators should be focused on in the risk assessment of micro and small enterprise loans; for user log extraction questions, 300 real user inquiries are extracted from the bank's customer service system, such as: why was my loan application automatically rejected, and which factors in my credit report affect my credit card limit; for model generation questions, the prompt words are:

[0039] You are a senior financial risk control expert. Please generate professional questions based on the following knowledge graph:

[0040] Knowledge graph fragment:

[0041] Entity: The credit report includes 24 months of repayment history, inquiry history, public records, etc.

[0042] Relationship: Credit card application - (dependency) -> Credit report

[0043] Entity: The rule attribute is to require monitoring large suspicious transactions.

[0044] Please generate 10 professional financial risk control questions covering these knowledge points.

[0045] Example of a model-generated problem: Based on rule attributes, which characteristics of large transactions require manual review?

[0046] In this embodiment, the questions from the three channels are merged and deduplicated again to form a set of 420 questions in the financial field.

[0047] S202: For any question in the question set, generate a first response and a second response respectively. The first response is the response without enhancement from the pre-trained language model, and the second response is the response enhanced by the pre-trained language model in combination with external knowledge sources.

[0048] In this embodiment, a general pre-trained language model is selected to ensure that the pre-trained language model has basic language capabilities, but has not deeply internalized the target domain knowledge. Any question is directly input into the pre-trained language model without adding any external knowledge support or enabling enhancement functions such as retrieval and search; the pre-trained language model answers only based on its own internalized knowledge, and the output result is the first response.

[0049] In this embodiment, the pre-trained language model remains unchanged. When answering the same question, external knowledge source enhancement is enabled, and any one or a combination of the following methods can be selected:

[0050] RAG retrieval enhancement: Call upon a professional vector database in the target domain to retrieve the top 3 to 5 core corpora related to the question, use the corpora as context input to the model, and generate a response;

[0051] Enhanced online search: The latest industry news, regulatory updates, and other information are obtained by calling search engines via API and added to the input of the pre-trained language model;

[0052] External professional knowledge base: Directly connects to authoritative knowledge bases in the target field, extracts relevant fragments as the basis for pre-trained language models to answer questions;

[0053] In this embodiment, the output result is the second response, which must be more accurate and comprehensive than the first response.

[0054] For example, in response to the question: According to the rule attributes, which characteristics of large transactions require manual review, the first response only mentioned that transactions exceeding 50,000 yuan require review. The second response provided an answer based on the bank's internal risk control rules and the scenario of monitoring large transactions in personal savings accounts and corporate accounts.

[0055] S203: Obtain the similarity value between the first response and the second response.

[0056] In this embodiment, the purpose of step S203 is to calculate the similarity value in two dimensions using semantic similarity value and named entity similarity value, quantify the difference between the first response and the second response, and determine the degree of knowledge mastery of the pre-trained language model for any question.

[0057] Specifically, step S203 includes S2031 to S2035:

[0058] S2031: Obtain the semantic similarity value between the first response and the second response.

[0059] Specifically, the first response is input into the trained semantic embedding model, so that the trained semantic embedding model outputs the feature vector of the first response; the second response is input into the trained semantic embedding model, so that the trained semantic embedding model outputs the feature vector of the second response; the cosine similarity between the feature vector of the first response and the feature vector of the second response is determined as the semantic similarity value between the first response and the second response.

[0060] Alternatively, the trained semantic embedding model can be BERT.

[0061] Optionally, it is necessary to fine-tune the data in advance using general corpora and target domain corpora to ensure that the specialized semantics of the target domain can be captured.

[0062] In this embodiment, the semantic similarity value between the first response and the second response is calculated using the vector cosine similarity formula. The semantic similarity value ranges from 0 to 1, with the closer the similarity is to 1.

[0063] S2032: Extract multiple first named entities from the first response.

[0064] In this embodiment, a model that supports entity recognition in the target domain is selected. The first response is input into the entity recognition model, which automatically identifies and extracts named entities, such as cross-border payments in the financial field and hypertension in the medical field. Meaningless entities are removed, and only core entities in the domain are retained, such as law names, business scenarios, and indicator names.

[0065] S2033: Extract multiple second named entities from the second response.

[0066] In this embodiment, the implementation process of step S2033 is the same as that of step S2032, and will not be described again here.

[0067] S2034: Determine the name entity similarity value between the first response and the second response based on multiple first named entities and multiple second named entities.

[0068] Specifically, based on multiple first named entities and multiple second named entities, multiple named entity types are obtained; the multiple first named entities and multiple second named entities are matched according to the multiple named entity types to obtain multiple named entity pairs; each named entity pair includes one first named entity and one second named entity, and the first named entity and the second named entity in each named entity pair have the same named entity type; the similarity value of each named entity pair is calculated; based on the similarity value of each named entity pair, the named entity similarity value of the multiple first named entities and the multiple second named entities is obtained.

[0069] In this embodiment, multiple first named entities and multiple second named entities are classified according to their named entity types, such as regulatory name types, business scenario types, etc.; within the same type, named entities of the same type are paired to form named entity pairs. In this step, only named entity pairs with the same named entity type are considered.

[0070] For example, we get m pairs of named entities (e1_a1, e1_a2), ..., (em_a1, em_a2).

[0071] In this embodiment, if the named entities in a named entity pair are completely identical, the similarity value of the named entity pair is 1; if the entity descriptions are different but the core meanings are the same, the similarity value of the named entity pair can be calculated using edit distance; if they are of the same type but have different meanings, the similarity value of each named entity pair is 0. The named entity similarity values ​​of multiple first named entities and multiple second named entities are calculated by weighted summation.

[0072] S2035: Obtain the similarity value between the first response and the second response based on the semantic similarity value and the named entity similarity value.

[0073] Specifically, the similarity values ​​of the first response and the second response are obtained based on the semantic similarity value, the named entity similarity value, the preset weight of the semantic similarity value, and the preset weight of the named entity similarity value.

[0074] In this embodiment, the preset weights for semantic similarity and named entity similarity are both preset. For example, the preset weight for semantic similarity is 0.7, and the preset weight for named entity similarity is 0.3.

[0075] In this embodiment, the similarity value between the first response and the second response = semantic similarity value × preset weight of semantic similarity value + named entity similarity value × preset weight of named entity similarity value.

[0076] Optionally, the preset weights of semantic similarity values ​​and named entity similarity values ​​can be adjusted according to actual circumstances.

[0077] In this embodiment, the knowledge mastery of the unenhanced pre-trained language model in the target domain is quantified by the similarity value between the first response and the second response.

[0078] S204: Generate a training dataset based on the similarity values.

[0079] In this embodiment, the training dataset includes a first training dataset and a second training dataset.

[0080] Specifically, step S204 includes S2041 to S2045:

[0081] S2041: From the question set, obtain multiple questions whose similarity values ​​are lower than a preset similarity threshold.

[0082] In this embodiment, the similarity value is lower than the preset similarity threshold, indicating that there is a large difference between the first response and the second response, and the unenhanced pre-trained language model has a low level of knowledge in the target domain.

[0083] Optionally, the preset similarity threshold is 0.55, which can be adjusted according to the actual situation.

[0084] For example, if the similarity between the first and second responses to any question is 0.575, which is greater than 0.55, then it is determined that the unenhanced pre-trained language model has mastered the knowledge in the target domain.

[0085] For example, if the similarity between the first and second responses to any question is 0.48, which is less than 0.55, then the unenhanced pre-trained language model is determined to lack knowledge in the target domain.

[0086] S2042: Determine each question and its corresponding second response as the first training dataset.

[0087] In this embodiment, for multiple questions with similarity values ​​lower than a preset similarity threshold, if the second response is better than the first response, then each question and the corresponding second response will continue to be used as a prediction for training.

[0088] S2043: Generate a third response for each question by using pre-designed prompts for the target domain.

[0089] Among them, prompt words are instructions, task descriptions or guiding words given to the model. Their core function is to clearly tell the model what to do, how to do it, and what standards to achieve, so that the model can output accurate results as required.

[0090] In this embodiment, for multiple questions with similarity values ​​below a preset similarity threshold, the second response includes knowledge in the target domain that the unenhanced pre-trained language model does not possess. A third response for each question can be generated by pre-designing prompts based on the first training dataset.

[0091] For example, the prompt word is:

[0092] Please generate a question-and-answer pair based on the following professional materials:

[0093] Materials: {Group misconduct in credit card applications}

[0094] Require:

[0095] 1. Generate 3 professional questions

[0096] 2. Each question is accompanied by a detailed answer.

[0097] 3. Answers must cite specific clauses.

[0098] S2044: Determine each question and its corresponding third question and answer as the second training dataset.

[0099] In this embodiment, both the first training dataset and the second training dataset are in the form of multiple question-answer pairs, each of which includes a question and a response.

[0100] In this embodiment, the first training dataset is used as training data for further training of the pre-trained language model, and the second training dataset is used as training data for supervised fine-tuning of the pre-trained language model. The pre-trained language model trained using the first and second training datasets can be used in professional consulting, business risk control, and intelligent decision-making scenarios in the target domain, addressing the problem of insufficient professional knowledge in unenhanced pre-trained language models. Optionally, the target domain includes, but is not limited to, the financial, legal, and medical fields.

[0101] In this embodiment, experiments have shown that, compared to full training, step S204 reduces redundant training data by 71%, shortens training time from 120 hours to 35 hours, and improves the performance of the pre-trained language model. Furthermore, it saves on manual annotation costs, approximately 150,000 yuan per model iteration, reduces the deployment cycle from 2 months to 3 weeks, and lowers the misclassification rate by 43%.

[0102] In summary, for any given question, two responses are generated: a first response without enhancement from the pre-trained language model, and a second response enhanced by the pre-trained language model incorporating external knowledge sources. By quantifying the difference between the first and second responses using their similarity scores, the degree of knowledge mastery for any given question can be assessed when the pre-trained language model is not enhanced. Based on these similarity scores, a training dataset is generated. The entire process replaces manually preset rules with a similarity mechanism, reducing manpower, lowering costs, and improving generation efficiency.

[0103] Figure 3 This is a schematic diagram of the structure of the training data processing device provided in the embodiments of this application, as shown below. Figure 3 As shown, the training data processing device provided in this embodiment includes: a first acquisition module 301, a first generation module 302, a second acquisition module 303, and a second generation module 304.

[0104] The first acquisition module 301 is used to acquire a set of questions in the target domain.

[0105] The first generation module 302 is used to generate a first response and a second response for any question in the question set. The first response is the response of the pre-trained language model without enhancement, and the second response is the response of the pre-trained language model enhanced by external knowledge sources.

[0106] The second acquisition module 303 is used to acquire the similarity value between the first response and the second response.

[0107] The second generation module 304 is used to generate a training dataset based on similarity values.

[0108] In one possible implementation, the second acquisition module 303 is specifically used for: acquiring the semantic similarity value between the first response and the second response; extracting multiple first named entities from the first response; extracting multiple second named entities from the second response; determining the named entity similarity value between the first response and the second response based on the multiple first named entities and the multiple second named entities; and acquiring the similarity value between the first response and the second response based on the semantic similarity value and the named entity similarity value.

[0109] In one possible implementation, the second acquisition module 303 is further configured to: input the first response into the trained semantic embedding model, so that the trained semantic embedding model outputs the feature vector of the first response; input the second response into the trained semantic embedding model, so that the trained semantic embedding model outputs the feature vector of the second response; and determine the cosine similarity between the feature vector of the first response and the feature vector of the second response as the semantic similarity value between the first response and the second response.

[0110] In one possible implementation, the second acquisition module 303 is further configured to: acquire multiple named entity types based on multiple first named entities and multiple second named entities; match the multiple first named entities with the multiple second named entities according to the multiple named entity types to obtain multiple named entity pairs; wherein each named entity pair includes one first named entity and one second named entity, and the first named entity and the second named entity in each named entity pair have the same named entity type; calculate the similarity value of each named entity pair; and acquire the named entity similarity value of the multiple first named entities and the multiple second named entities based on the similarity value of each named entity pair.

[0111] In one possible implementation, the second acquisition module 303 is further configured to: acquire the similarity value between the first response and the second response based on the semantic similarity value, the named entity similarity value, the preset weight of the semantic similarity value, and the preset weight of the named entity similarity value.

[0112] In one possible implementation, the second generation module 304 is specifically used to: obtain multiple questions with similarity values ​​lower than a preset similarity threshold from the question set; determine each question and its corresponding second response as a first training dataset; generate a third response for each question using prompt words pre-designed for the target domain; and determine each question and its corresponding third question and answer as a second training dataset.

[0113] The training data processing device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0114] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 4 As shown, the electronic device provided in this embodiment includes at least one processor 401 and a memory 402. Optionally, the electronic device further includes a communication component 403. The processor 401, memory 402, and communication component 403 are connected via a bus.

[0115] In a specific implementation, at least one processor 401 executes computer execution instructions stored in memory 402, causing at least one processor 401 to perform the above-described method.

[0116] The specific implementation process of processor 401 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0117] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0118] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0119] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0120] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0121] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0122] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0123] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0124] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0125] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0126] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0127] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0128] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0129] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A method for processing training data, characterized in that, Applied to electronic devices, including: Obtain the question set in the target domain; For any question in the question set, a first response and a second response are generated respectively. The first response is the response without enhancement of the pre-trained language model, and the second response is the response enhanced by the pre-trained language model in combination with external knowledge sources. Obtain the similarity value between the first response and the second response; A training dataset is generated based on the similarity values.

2. The method according to claim 1, characterized in that, Obtaining the similarity value between the first response and the second response includes: Obtain the semantic similarity value between the first response and the second response; Extract multiple first named entities from the first response; Extract multiple second named entities from the second response; Based on the plurality of first named entities and the plurality of second named entities, determine the named entity similarity value between the first response and the second response; Based on the semantic similarity value and the named entity similarity value, the similarity value between the first response and the second response is obtained.

3. The method according to claim 2, characterized in that, The step of obtaining the semantic similarity value between the first response and the second response includes: The first response is input into the trained semantic embedding model, so that the trained semantic embedding model outputs the feature vector of the first response; The second response is input into the trained semantic embedding model, so that the trained semantic embedding model outputs the feature vector of the second response; The cosine similarity between the feature vectors of the first response and the feature vectors of the second response is determined as the semantic similarity value between the first response and the second response.

4. The method according to claim 2, characterized in that, The step of determining the name entity similarity value between the first response and the second response based on the plurality of first named entities and the plurality of second named entities includes: Based on the plurality of first named entities and the plurality of second named entities, a plurality of named entity types are obtained; The plurality of first named entities and the plurality of second named entities are matched according to the plurality of named entity types to obtain a plurality of named entity pairs; wherein each named entity pair includes a first named entity and a second named entity, and the first named entity and the second named entity in each named entity pair have the same named entity type; Calculate the similarity value for each named entity pair; Based on the similarity values ​​of each named entity pair, the named entity similarity values ​​of the plurality of first named entities and the plurality of second named entities are obtained.

5. The method according to claim 2, characterized in that, The step of obtaining the similarity value between the first response and the second response based on the semantic similarity value and the named entity similarity value includes: The similarity values ​​of the first response and the second response are obtained based on the semantic similarity value, the named entity similarity value, the preset weight of the semantic similarity value, and the preset weight of the named entity similarity value.

6. The method according to any one of claims 1-5, characterized in that, The step of generating training data based on the similarity value includes: From the set of questions, obtain multiple questions whose similarity values ​​are lower than a preset similarity threshold; Each question, and the second response corresponding to each question, are determined as the first training dataset; By using pre-designed prompts for the target domain, a third response is generated for each question. The questions and their corresponding third answers are used as the second training dataset.

7. A training data processing apparatus, characterized in that, Applied to electronic devices, including: The first acquisition module is used to acquire a set of questions in the target domain; The first generation module is used to generate a first response and a second response for any question in the question set, wherein the first response is the response of the pre-trained language model without enhancement, and the second response is the response of the pre-trained language model enhanced by combining external knowledge sources. The second acquisition module is used to acquire the similarity value between the first response and the second response; The second generation module is used to generate a training dataset based on the similarity value.

8. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-6.

10. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method described in any one of claims 1-6.

Images