Sample verification method, electronic equipment, storage medium and program product

Abstract

The invention discloses a sample verification method, electronic equipment, a storage medium and a program product. The method comprises the following steps: acquiring a to-be-verified evolution sample; performing differential analysis of questions and answers on the basis of the seed sample and the generated sample of the to-be-verified evolution sample, and determining sample change in the evolution process; performing causal effect evaluation on the evolution process according to the sample change to determine a first causal effect evaluation result, performing analysis based on the evolution instruction of the to-be-verified evolution sample and the sample change, performing causal effect evaluation on the evolution process according to the evolution instruction to determine a second causal effect evaluation result; according to the first causal effect evaluation result and the second causal effect evaluation result, the verification result of the to-be-verified evolution sample is determined, the problem that the verification result is inaccurate is solved, and the accuracy and reliability of logic verification are improved by identifying and analyzing the complex causal relationship among the question, the answer and the evolution instruction; the method adapts to different evolution instructions, and verification is more flexible.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a sample verification method, electronic equipment, storage medium and program product. Background Art

[0002] With the development of large-scale language models, researchers have found that these models have powerful instruction-following capabilities. However, creating high-quality instruction-output pairs for specific tasks is still a time-consuming and expensive process. Traditional manual annotation methods are difficult to meet the needs of large-scale data sets, so researchers began to explore using the capabilities of large models themselves to generate instruction samples. The main technical implementation methods include: self-guided instruction generation, adversarial instruction generation, multi-model collaborative generation, human-machine collaborative generation, knowledge graph-based instruction generation, etc., which can greatly reduce the cost of manual annotation and quickly generate large-scale and diverse instruction-output pairs.

[0003] However, the quality of training samples is equally important for large models. How to accurately evaluate and verify the quality of samples generated by large models is also a focus of the industry. The existing technology for sample verification usually evaluates and verifies the generated samples from the perspective of results, ignoring the process of sample generation. It is difficult to evaluate the consistency between instructions and generated results. Some instructions may obtain correct results in unexpected or inappropriate ways; therefore, the accuracy of the verification results is low. Summary of the invention

[0004] The present invention provides a sample verification method, electronic equipment, storage medium and program product to improve the accuracy of the verification result.

[0005] According to one aspect of the present invention, there is provided a sample verification method, comprising:

[0006] Acquire an evolution sample to be verified, wherein the evolution sample to be verified includes a seed sample, an evolution instruction, and a generated sample, the seed sample includes a first question and a first answer, and the generated sample includes a second question and a second answer;

[0007] Perform differential analysis of questions and answers based on the seed sample and the generated sample of the evolution sample to be verified to determine sample changes during the evolution process;

[0008] Performing a causal effect evaluation on the evolution process according to the sample change to determine a first causal effect evaluation result, performing analysis based on the evolution instruction of the evolution sample to be verified and the sample change, performing a causal effect evaluation on the evolution process according to the evolution instruction to determine a second causal effect evaluation result;

[0009] The verification result of the evolution sample to be verified is determined according to the first causal effect evaluation result and the second causal effect evaluation result.

[0010] According to another aspect of the present invention, there is provided an electronic device, the electronic device comprising:

[0011] at least one processor, and a memory communicatively coupled to the at least one processor;

[0012] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the sample verification method described in any embodiment of the present invention.

[0013] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the sample verification method described in any embodiment of the present invention when executed.

[0014] According to another aspect of the present invention, a computer program product is provided. The computer program product includes a computer program. When the computer program is executed by a processor, the sample verification method according to any embodiment of the present invention is implemented.

[0015] The technical solution of the embodiment of the present invention is to obtain an evolution sample to be verified, wherein the evolution sample to be verified includes a seed sample, an evolution instruction and a generated sample, the seed sample includes a first question and a first answer, and the generated sample includes a second question and a second answer; based on the seed sample and the generated sample of the evolution sample to be verified, differential analysis of questions and answers is performed to determine sample changes in the evolution process; according to the sample changes, a causal effect evaluation is performed on the evolution process to determine a first causal effect evaluation result, based on the evolution instruction of the evolution sample to be verified and the sample changes, a causal effect evaluation is performed on the evolution process according to the evolution instruction to determine a second causal effect evaluation result; based on the first causal effect evaluation result and the second causal effect evaluation result, the verification result of the evolution sample to be verified is determined, thereby solving the problem of inaccurate verification results. The seed sample and the generated sample of the evolution sample are used to perform differential analysis on the questions and answers, and the sample changes generated in the evolution process are determined. The causal effect of the evolution process is evaluated through the sample changes to determine whether there is a causal relationship between the sample changes in the evolution process, and obtain a first causal effect evaluation result; based on the evolution instructions and sample changes, analysis is performed to determine whether there is a causal effect between the evolution instructions and the sample changes, and a second causal effect evaluation result is obtained. Finally, the verification result of the evolution sample to be verified is determined by combining the first causal effect evaluation result and the second causal effect evaluation result; the sample verification method provided in the present application can identify and analyze the complex causal relationship between questions, answers and evolution instructions, thereby improving the accuracy and reliability of logical verification; it can adapt to different evolution instructions, and the verification of samples is more flexible, so that the verification process is more in line with practical applications, and the practicality and ease of use of the verification method are improved.

[0016] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0018] Figure 1 is a flow chart of a sample verification method provided according to Embodiment 1 of the present invention;

[0019] Figure 2 is a flow chart of a sample verification method provided according to Embodiment 2 of the present invention;

[0020] Figure 3 is an example diagram of an implementation of a verification sample provided according to Embodiment 2 of the present invention;

[0021] Figure 4 is a schematic diagram of the structure of a sample verification device provided according to Embodiment 3 of the present invention;

[0022] Figure 5 It is a schematic diagram of the structure of an electronic device for implementing the sample verification method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0024] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0025] Embodiment 1

[0026] Figure 1 This is a flow chart of a sample verification method provided in the first embodiment of the present invention. This embodiment is applicable to the case where the evolved sample is accurately verified. The method can be executed by a sample verification device. The sample verification device can be implemented in the form of hardware and / or software. The sample verification device can be configured in an electronic device. Figure 1 As shown, the method includes:

[0027] S101. Obtain an evolution sample to be verified, where the evolution sample to be verified includes a seed sample, an evolution instruction, and a generated sample. The seed sample includes a first question and a first answer, and the generated sample includes a second question and a second answer.

[0028] In this embodiment, the evolution sample to be verified can be understood as a sample that needs to be verified, which is obtained through evolution. The seed sample can be understood as the original sample, which is used for evolution; the evolution instruction can be understood as an instruction to instruct the sample to evolve; the generated sample can be understood as the sample generated after the seed sample is evolved. The first question and the first answer are a pair of question-answer pairs, and the second question and the second answer are a pair of question-answer pairs. The seed sample and the generated sample in the embodiment of the present application both include question-answer pairs, i.e., questions and answers. The question and answer in the seed sample are respectively recorded as the first question and the first answer, and the question and answer in the generated sample are respectively recorded as the second question and the second answer.

[0029] The seed sample is evolved in advance according to the evolution instruction to obtain the generated sample, and the evolved sample is formed based on the seed sample, the evolution instruction and the generated sample. The evolved sample can be verified immediately after it is generated, or it can be saved to the sample library after it is generated. When the verification condition is met, the evolved sample is read from the sample library and verified. The verification condition can receive the user's trigger operation, receive the verification instruction, or detect that the time and other information meet the verification condition, etc. The embodiment of the present application obtains the evolved sample that needs to be verified and uses it as the evolved sample to be verified.

[0030] The embodiment of the present application provides a method for generating a sample library: one or more seed samples and one or more evolution instructions are pre-set. In order to enrich the samples and improve the diversity of the samples, the number of seed samples and evolution instructions can be set to multiple. A seed sample and an evolution instruction are selected according to a certain rule or randomly, and the seed sample is instructed to evolve through the evolution instruction to obtain a generated sample, and the seed sample, the evolution instruction and the generated sample are stored in the sample library as an evolved sample. The verification method provided in the embodiment of the present application can verify each evolved sample when it is generated, or it can be verified after all evolved samples are generated.

[0031] For example, seed sample S = {Q, A}, evolution instruction Evol Instruction, generated sample S' = {Q', A'}, evolution sample S e={S->Evol Instruction->S'}. The general large language model (referred to as base LLM) is an ordinary generation model. It inputs questions to generate answers. The evolution model Evol LLM is responsible for generating sample evolution and outputs seed samples as evolved samples following the evolution instruction (Evol Instruction). In this step, the seed sample can be obtained by sampling from the original sample library or generated by the base LLM. After obtaining the seed sample, the seed sample S = {Q, A} and the evolution instruction Evol Instruction can be input into the evolution model (Evol LLM) to obtain the generated sample S' = {Q', A'}. Repeat the above generation work to finally complete the construction of the sample library.

[0032] S102: Perform differential analysis of questions and answers based on the seed sample and the generated sample of the evolution sample to be verified, and determine the sample changes during the evolution process.

[0033] In this embodiment, the sample change can be understood as information describing the change of the sample before and after the evolution, which can be the change of the question, the change of the answer, etc. The questions and answers of the seed sample and the generated sample are analyzed to determine the difference before and after the change. For example, the first question of the seed sample and the second question of the generated sample are analyzed to determine the difference change of the question before and after the evolution, and the first answer of the seed sample and the second answer of the generated sample are analyzed to determine the difference change of the answer before and after the evolution. The sample change generated in the evolution process is determined based on the difference analysis results.

[0034] S103, performing causal effect evaluation on the evolution process according to the sample changes to determine a first causal effect evaluation result, performing causal effect evaluation on the evolution process based on the evolution instructions and sample changes of the evolution sample to be verified to determine a second causal effect evaluation result.

[0035] In this embodiment, the first causal effect evaluation result and the second causal effect evaluation result can both be understood as the results obtained by evaluating the causal effect relationship of the evolution process.

[0036] According to the sample changes, we analyze whether there is a causal relationship between the changes in questions and answers during the evolution process. We can directly compare the changes in questions and answers. We can also convert the changes in questions into changes in answers and then compare them with the actual changes in answers, or convert the changes in answers into changes in questions and then compare them with the actual changes in questions, etc. By analyzing the sample changes, we evaluate the causal effect of the evolution process and obtain the first causal effect evaluation result, which can be the existence of a causal relationship, the non-existence of a causal relationship, etc.

[0037] Instruct the sample to evolve according to the evolution instruction to generate a new sample, compare the changes in the sample before and after the evolution, and compare it with the sample change to determine whether the changes are consistent, so as to determine whether there is a causal relationship between the evolution instruction and the sample change, and obtain a second causal effect evaluation result, which can be the existence of a causal relationship, the non-existence of a causal relationship, etc.

[0038] The existing technologies mainly verify samples in the following ways:

[0039] 1. Manual evaluation: This method relies on human experts to evaluate the quality of the generated instructions. Evaluators score the instructions based on predefined criteria (such as clarity, relevance, difficulty, etc.). Its advantage is that it can provide high-quality and fine-grained evaluation results; its disadvantages are that it is time-consuming and labor-intensive, difficult to apply to large-scale data sets, and may have subjective biases.

[0040] 2. Automated metric evaluation: This method uses predefined metrics to automatically evaluate the quality of instructions. Common metrics include perplexity, BLEU score, ROUGE score, etc. Its advantages are that it can quickly process a large amount of data and the evaluation results are objective and consistent; its disadvantage is that these metrics may not fully capture the semantics and task relevance of the instructions.

[0041] 3. Model performance feedback: This method evaluates the quality of instructions by testing the model's performance on the generated instructions. If the model can correctly execute the instruction and produce the expected output, the instruction is considered effective. The advantage is that it is directly related to the actual use effect of the instruction; the disadvantage is that it may be biased towards the task type that the model is already good at.

[0042] 4. Contrastive learning evaluation: This method compares the generated instructions with high-quality instructions created manually. The quality of the generated instructions is evaluated by training a discriminator to distinguish between manual instructions and generated instructions. The advantage is that subtle quality differences can be learned; the disadvantage is that a large number of high-quality manual instructions are required as reference.

[0043] 5. Multi-model consistency check: This method uses multiple different models to execute the same instruction and compares their output consistency. Highly consistent results indicate that the instruction may be clearer and more specific. Its advantage is that the universality of the instruction can be evaluated; its disadvantage is that the computational cost is high and innovative instructions may be ignored.

[0044] The embodiment of the present application evaluates the evolution process from the perspectives of sample changes and evolution instructions, identifies and analyzes the complex causal relationship between questions, answers and evolution instructions, thereby improving the accuracy and reliability of logic verification; it can adapt to different evolution instructions and make sample verification more flexible. It does not require human participation, is applicable to different evolution instructions, and has low computational cost.

[0045] S104. Determine the verification result of the evolution sample to be verified according to the first causal effect evaluation result and the second causal effect evaluation result.

[0046] Comprehensively analyze the first causal effect evaluation result and the second causal effect evaluation result to determine the verification result of the evolution sample to be verified. For example, if the first causal effect evaluation result and the second causal effect evaluation result both indicate that there is a causal relationship, the verification result of the evolution sample to be verified is determined to be passed; if at least one of the first causal effect evaluation result and the second causal effect evaluation result indicates that there is no causal relationship, the verification result of the evolution sample to be verified is determined to be failed; or, if at least one of the first causal effect evaluation result and the second causal effect evaluation result indicates that there is a causal relationship, the verification result of the evolution sample to be verified is determined to be passed; otherwise, it is failed. The evolution samples to be verified that fail the verification can be eliminated, and the evolution samples to be verified that pass the verification can be retained, and the retained samples can be used for model training tasks.

[0047] The sample verification method of the embodiment of the present invention solves the problem of inaccurate verification results. It determines the sample changes generated in the evolution process by performing differential analysis of questions and answers on the seed sample and the generated sample of the evolution sample to be verified, and performs causal effect evaluation on the evolution process through the sample changes to determine whether there is a causal relationship between the sample changes in the evolution process, thereby obtaining a first causal effect evaluation result; it determines whether there is a causal effect between the evolution instruction and the sample change based on the analysis of the evolution instruction, thereby obtaining a second causal effect evaluation result, and finally determines the verification result of the evolution sample to be verified by combining the first causal effect evaluation result and the second causal effect evaluation result; the sample verification method provided by the present application can identify and analyze the complex causal relationship between questions, answers and evolution instructions, thereby improving the accuracy and reliability of logical verification; it can adapt to different evolution instructions, and the verification of samples is more flexible, so that the verification process is more in line with practical applications, thereby improving the practicality and ease of use of the verification method.

[0048] Embodiment 2

[0049] Figure 2 This is a flow chart of a sample verification method provided in Embodiment 2 of the present invention. This embodiment is refined on the basis of the above embodiment. Figure 2 As shown, the method includes:

[0050] S201. Obtain an evolution sample to be verified, where the evolution sample to be verified includes a seed sample, an evolution instruction, and a generated sample. The seed sample includes a first question and a first answer, and the generated sample includes a second question and a second answer.

[0051] S202: Perform differential analysis of questions and answers based on the seed sample and the generated sample of the evolution sample to be verified, and determine sample changes in the evolution process, where the sample changes include: first question changes and first answer changes.

[0052] Compare the difference between the first question and the second question to obtain the change of the first question; compare the difference between the first answer and the second answer to obtain the change of the first answer.

[0053] Optionally, the differential analysis includes at least one of the following:

[0054] Added words;

[0055] Added sentences;

[0056] deleted words;

[0057] deleted sentences;

[0058] Replacement words;

[0059] Replace the sentence.

[0060] Taking the analysis of the differences in the problems as an example, when analyzing the differences in the changes before and after the problems, one or more of the added words, sentences, deleted words, sentences, replaced words, sentences, etc. can be analyzed.

[0061] Exemplarily, the differential analysis of the embodiment of the present application can be implemented based on a rule processing system or an instruction generation mode, and the difference record types of the text can be divided into the following types:

[0062] Add word / sentence, Δ=[-\0,+word]

[0063] Delete words / sentences, Δ=[-word,+\0]

[0064] Replace word / sentence, Δ=[-word,+word]

[0065] Among them, word is an abstraction of the difference value, which can be a word or a sentence. The output after difference analysis is recorded as ΔQ1 = {Δq1,…,Δqn}, ΔA1 = {Δa1,…,Δan}.

[0066] S203: Evolve the first question according to the change of the first question to generate a third question.

[0067] In this embodiment, the third question can be understood as a question obtained by evolving the first question under the guidance of the change of the first question. According to the change of the first question, the first question is instructed how to evolve, for example, according to the change of the first question, the instruction of the evolution is summarized and output, and the first question is instructed to evolve based on the instruction of the evolution to generate the third question.

[0068] As an optional embodiment, this optional embodiment further evolves the first problem according to the change of the first problem to generate a third problem, which is optimized as follows:

[0069] A1. Input the first question change into the large language model, generate a prompt of a predetermined effect instruction as a prompt, instruct the large language model to perform an evolutionary effect analysis, and determine the effect instruction according to the output of the large language model.

[0070] In this embodiment, the effect instruction generation prompt can be understood as information for prompting the large language model to generate an effect instruction. The effect instruction can be understood as an instruction formed by summarizing the impact of changes in the evolution process on the answer.

[0071] Large Language Model (LLM) is an artificial intelligence technology based on deep learning. It can understand and generate natural language text by learning from a large amount of text data. The core idea of ​​LLM is to learn the grammar, semantics and context information of the language by training a large amount of text data. These models usually use neural networks to learn patterns and regularities in text data. The large language model used in this step can be a model service such as GPT.

[0072] Predetermine the effect instruction generation prompt, which can be set according to different business types. For the evolution samples of the same business type, the effect instruction generation prompt used when verifying them can be the same. Input the first problem change into the large language model, use the predetermined effect instruction generation prompt as a prompt, and guide the large language model to perform evolution effect analysis. For example, through the effect instruction generation prompt, the large language model is required to analyze the evolution effect and summarize and output the corresponding effect instruction. The large language model performs the analysis and then outputs the effect instruction.

[0073] Exemplarily, the effect instruction generates a prompt in the form of "Please analyze the changes in the following text {ΔQ1}, analyze the impact of these changes on the topic, argument or emotional color of the question or answer, summarize and output the corresponding effect instructions, and elaborate on how to adjust the text to be processed based on these changes."

[0074] A2. Input the first question into the large language model, use the effect instruction as a prompt, instruct the large language model to generate a question, and determine the third question based on the output of the large language model.

[0075] The first question is input into the large language model, and the effect instruction is used as a prompt. The effect instruction instructs the large language model to generate questions, guides the large language model to generate questions, and the large speech model evolves the first question according to the first question and the effect instruction, and generates and outputs the third question. For example, the effect instruction is to adjust the emotion expressed by the text from happy to depressed. When evolving the first question, the first question is adjusted based on the instruction of the effect instruction.

[0076] S204: Perform differential analysis of answers based on the first question and the third question to determine changes in the second answer.

[0077] In this embodiment, the second answer change can be understood as information describing the change between the two answers. The answers corresponding to the first question and the third question are determined respectively, and the two answers are differentially analyzed to obtain the second answer change.

[0078] As an optional embodiment, this optional embodiment further performs differential analysis of the answers based on the first question and the third question, determines the change of the second answer, and optimizes it to:

[0079] B1. Input the first question into the large language model for question answering, and determine the first answer to be analyzed based on the output of the large language model.

[0080] In this embodiment, the first answer to be analyzed can be understood as the answer that needs to be analyzed. The first question is input into the large language model for question answering, and the large language model answers the first question and outputs an answer, and the answer output by the large language model is used as the first answer to be analyzed.

[0081] B2. Input the third question into the large language model for question answering, and determine the second answer to be analyzed based on the output of the large language model.

[0082] In this embodiment, the second answer to be analyzed can be understood as the answer that needs to be analyzed. The third question is input into the large language model for question and answering, and the large language model answers the third question and outputs an answer, and the answer output by the large language model is used as the second answer to be analyzed.

[0083] B3. Perform differential analysis on the first answer to be analyzed and the second answer to be analyzed to determine the change of the second answer.

[0084] The first answer to be analyzed and the second answer to be analyzed are subjected to differential analysis, and the differential analysis includes one or more of the words, sentences added, words, sentences deleted, words, sentences replaced, etc. in the analysis answer. The change of the second answer is determined according to the analysis result. Among them, the specific analysis principle and process of the differential analysis can refer to the above-mentioned differential analysis, and the embodiment of the present application will not be described in detail here.

[0085] S205. Perform a causal effect evaluation based on the first answer change and the second answer change to determine a first causal effect evaluation result.

[0086] Compare whether the change in the first answer is consistent with the change in the second answer, for example, calculate the similarity of the text, compare through a pre-trained model, etc. If they are consistent, it is determined that there is a causal effect between the change in the question and the answer; if they are inconsistent, it is determined that there is no causal effect between the change in the question and the answer.

[0087] As an optional embodiment, this optional embodiment further performs a causal effect evaluation based on the first answer change and the second answer change to determine the first causal effect evaluation result, which is optimized as follows:

[0088] C1. Perform similarity analysis on the changes in the first answer and the second answer to determine the similarity coefficient.

[0089] In this embodiment, the similarity coefficient can be understood as information used to indicate the degree of similarity.

[0090] The similarity analysis of the change of the first answer and the change of the second answer can be performed by calculating the similarity between the two, and the similarity can be cosine similarity, Euclidean distance, etc. The similarity can be directly used as a similarity coefficient, or the similarity can be calculated, for example, the similarity is substituted into a preset calculation formula for calculation to obtain a similarity coefficient. Alternatively, a neural network model is pre-trained, and the change of the first answer and the change of the second answer are input into the neural network model, and the neural network model performs similarity analysis based on the learned knowledge to determine the similarity coefficient, etc.

[0091] C2. If the similarity coefficient is greater than the similarity threshold, the first causal effect evaluation result is determined to be that there is a causal effect in the changes in the question and the answer.

[0092] In this embodiment, the similarity threshold can be preset and can be set according to the business type, accuracy requirements, etc. The similarity coefficient and the similarity threshold are compared. If the similarity coefficient is greater than the similarity threshold, it can be considered that the change in the first answer is consistent with the change in the second answer, and the first causal effect evaluation result is determined to be that there is a causal effect between the change in the question and the answer. If the similarity coefficient is not greater than the similarity threshold, the first causal effect evaluation result is determined to be that there is no causal effect between the change in the question and the answer.

[0093] During the sample evolution process, the changes before and after the sample are affected by both the explicit influence of the evolutionary instructions and the implicit influence of the model. Therefore, the effect instruction approximates the equivalent of "explicit + implicit influence". When the effect instruction evolves a problem in a new model, the change in the second answer represents the impact result of "explicit instruction + implicit evolutionary model + implicit new model". When the change in the second answer is similar to the change in the first answer, it shows that under the condition of the evolutionary instruction, the implicit influence brought by the model itself is always stable. Therefore, there is a causal effect between the change in the problem and the change in the answer.

[0094] When performing causal effect analysis on sample changes, the embodiment of the present application aims to compare whether the change in the first question is consistent with the change in the first answer. However, since it is impossible to directly compare the change in the first question and the change in the first answer, the present application applies the change in the first question to the first question, evolves and generates a third question, and uses a large language model to ask and answer the third question to obtain a second answer change. The change in the first question is converted into a second answer change, and finally the first causal effect evaluation result is determined by comparing the change in the first answer and the change in the second answer.

[0095] S206. Obtain question and answer samples.

[0096] In this embodiment, the question and answer sample can be understood as a sample formed by a question and an answer. In this embodiment of the application, a sample library can be generated in advance, a sample can be selected from the sample library as a question and answer sample, or a question and answer sample can be generated according to certain rules during the sample verification process, and so on.

[0097] As an optional embodiment, this optional embodiment further optimizes obtaining question and answer samples as follows:

[0098] D1. Determine the length of the question text and the length of the answer text based on the evolution sample to be verified.

[0099] In this embodiment, the question text length can be understood as the text length of the question in the sample; the answer text length can be understood as the text length of the answer in the sample. The text lengths of the questions and answers of the seed sample and / or the generated sample in the evolution sample to be verified are analyzed, and the question text length and the answer text length are determined according to the text lengths of the questions and answers. For example, the text length of the question of the seed sample is used as the question text length, and the text length of the answer of the seed sample is used as the answer text length; or, the average of the text length of the question of the seed sample and the text length of the question of the generated sample is used as the question text length, and the average of the text length of the answer of the seed sample and the text length of the answer of the generated sample is used as the answer text length.

[0100] D2. Filter out question and answer samples from the pre-generated sample library according to the question text length and the answer text length.

[0101] The samples in the sample library are screened to select samples with similar question text length and answer text length as question and answer samples. For example, firstly, the samples that meet the requirements are screened according to the question text length, such as selecting n samples with the closest length, or selecting samples with length differences within a certain range, etc.; then the remaining samples are further screened according to the answer text length, and the samples with the closest length are selected as question and answer samples.

[0102] S207, input the evolution instructions of the question and answer sample and the evolution sample to be verified into the evolution model, and determine the evolution sample according to the output of the evolution model, where the evolution sample includes a first evolution question and a first evolution answer.

[0103] In this embodiment, the evolved sample can be understood as a sample obtained by evolving the question and answer sample; the first evolved question is the question in the evolved sample, and the first evolved answer is the answer in the evolved sample.

[0104] The evolution model can be pre-trained, and the evolution model can be a large language model. The evolution instructions of the question and answer sample and the evolution sample to be verified are input into the evolution model, and the evolution model evolves the question and answer sample according to the evolution instruction and outputs the evolution sample, which includes the first evolution question and the first evolution answer.

[0105] S208. Perform evolution instruction effect analysis based on the evolution samples and sample changes to determine the second causal effect evaluation result.

[0106] Analyze the changes of the evolution samples, compare the changes of the evolution samples with the changes of the samples, and judge the influence of the evolution instructions on the evolution process. If the changes of the evolution samples are consistent with the changes of the samples, it means that there is a causal relationship between the evolution instructions and the results of the evolution, and the second causal effect evaluation result is determined to be that there is a causal relationship between the evolution instructions and the sample changes; otherwise, the second causal effect evaluation result is determined to be that there is no causal relationship between the evolution instructions and the sample changes.

[0107] As an optional embodiment, this optional embodiment further performs an evolution instruction effect analysis based on the evolution sample and the sample change to determine a second causal effect evaluation result, including:

[0108] E1. Input the first evolved question into the large language model for question answering, and determine the second evolved answer based on the output of the large language model.

[0109] In this embodiment, the second evolved answer can be understood as the answer obtained by answering the first evolved question. The first evolved question is input into the large language model for question answering, and the large language model answers the question based on prior knowledge, generates an answer and outputs it, and the answer output by the large language model is used as the second evolved answer.

[0110] E2. Perform differential analysis on the first evolved question and the questions in the question and answer sample to determine the changes in the second question.

[0111] In this embodiment, the second question change can be understood as the change information of the question before and after the evolution. The first evolved question and the question and answer sample are subjected to differential analysis, and the differential analysis includes analyzing one or more of the words, sentences added, words, sentences deleted, words, sentences replaced, etc. in the answer. The second question change is determined based on the analysis results. Among them, the specific analysis principle and process of the differential analysis can refer to the above-mentioned differential analysis, and the embodiment of this application will not be described in detail here.

[0112] E3. Perform differential analysis on the first evolution answer and the second evolution answer to determine the change in the third answer.

[0113] In this embodiment, the third answer change can be understood as answer change information. The first evolved answer and the second evolved answer are subjected to differential analysis, and the differential analysis includes analyzing one or more of the words, sentences added, words, sentences deleted, words, sentences replaced, etc. in the answer. The third answer change is determined based on the analysis results. Among them, the specific analysis principle and process of the differential analysis can refer to the above-mentioned differential analysis, and the embodiment of this application will not be described in detail here.

[0114] Since the change in the question is the cause of the change in the answer (which can be determined by the result of the causal effect evaluation of the evolution process based on the sample change), when performing the causal effect analysis of the evolution instruction, the difference between the first evolved answer and the second evolved answer is compared, rather than comparing the difference between the first evolved answer and the answer of the question and answer sample. In this step, the generation process of the second question change and the third answer change has nothing to do with the first question change and the first answer change, but is only related to the evolution instruction EvolInstruction, the evolution model Evol LLM and the general model base LLM. Therefore, the second question change and the third answer change can be used as objective evaluation indicators of the instruction effect of the evolution sample Evol Instruction.

[0115] E4. Analyze whether the categories of the changes in the first question and the second question are equivalent, and analyze whether the categories of the changes in the first answer and the third answer are equivalent, and determine the second causal effect evaluation result based on the analysis results.

[0116] A specific analysis is performed on the first question change and the second question change to determine whether the categories are equivalent. The categories can refer to the pattern, intention, specific changes that cause the changes, etc., that is, the first question change and the second question change are analyzed to determine whether the two question changes are the same, whether they have similar intentions, and whether the reasons for the changes, instructions, etc. are the same. Based on the analysis results, determine whether the categories of the first question change and the second question change are equivalent. Similarly, analyze whether the categories of the first answer change and the third answer change are equivalent. Determine the second causal effect evaluation result based on the analysis results. If the categories of the first question change and the second question change are equivalent, and the categories of the first answer change and the third answer change are equivalent, determine that the second causal effect evaluation result is that there is a causal relationship between the evolutionary instruction and the sample change. The second causal effect evaluation result is determined based on the analysis results. For example, if the categories of the first question change and the second question change are equivalent and the categories of the first answer change and the third answer change are equivalent, the second causal effect evaluation result is determined to be that there is a causal relationship between the evolution instruction and the sample change; otherwise, the second causal effect evaluation result is determined to be that there is no causal relationship between the evolution instruction and the sample change; or, if one of the categories of the first question change and the second question change and the categories of the first answer change and the third answer change is equivalent, the second causal effect evaluation result is determined to be that there is a causal relationship between the evolution instruction and the sample change; otherwise, the second causal effect evaluation result is determined to be that there is no causal relationship between the evolution instruction and the sample change, and so on.

[0117] As an optional embodiment, this optional embodiment further analyzes whether the categories of the first question change and the second question change are equivalent, and is optimized as follows: the first question change and the second question change are input into the large language model, a predetermined category judgment prompt is used as a prompt to instruct the large language model to perform category judgment, and whether the categories are equivalent is determined based on the output of the large language model.

[0118] In this embodiment, the category judgment prompt can be understood as information for prompting when judging the category. The category judgment prompt can be generated in advance, and different category judgment prompts can be set for samples of different business types. When performing sample verification, the corresponding category judgment prompt is obtained, the first question change and the second question change are input into the large language model, and the predetermined category judgment prompt is used as a prompt to instruct the large language model to make a category judgment on the first question change and the second question change, and output whether the categories are equivalent, and determine whether the categories are equivalent according to the output of the large language model.

[0119] As an optional embodiment, this optional embodiment further analyzes whether the categories of the first answer change and the third answer change are equivalent, and is optimized as follows: the first answer change and the third answer change are input into the large language model, a predetermined category judgment prompt is used as a prompt to instruct the large language model to perform category judgment, and whether the categories are equivalent is determined based on the output of the large language model.

[0120] When performing sample verification, obtain corresponding category judgment prompts, input the first answer change and the third answer change into the large language model, use the predetermined category judgment prompts as prompts, instruct the large language model to make category judgments on the first answer change and the third answer change, and output whether the categories are equivalent, and determine whether the categories are equivalent based on the output of the large language model.

[0121] The category judgment prompt includes at least the following information:

[0122] Identify changes in text;

[0123] Analyze whether the pattern or intent of the changes is similar;

[0124] Analyze whether the instructions or intentions that caused the changes are the same;

[0125] Analyze the reasoning process corresponding to different instructions or intentions that cause changes.

[0126] Among them, identifying changes in text can instruct the large language model to identify changes in text during the process of category judgment. Analyzing whether the change pattern or intention is similar can instruct the large language model to analyze whether the change pattern or intention of two answers or questions is similar during the process of category judgment; analyzing whether the instructions or intentions causing the change are the same can instruct the large language model to analyze whether the instructions or intentions causing the change are the same during the process of category judgment; analyzing the corresponding reasoning process when the instructions or intentions causing the change are different can instruct the large language model to provide a reasoning process if it is inferred that the instructions or intentions causing the change are different during the process of category judgment.

[0127] Exemplarily, the category judgment prompt is: "Please analyze the following two sets of text change sequence pairs and determine whether they may be caused by the same type of instruction. It is known that text change sequence pair 1 is {Δ1} and text change sequence pair 2 is {Δ2}. Please analyze according to the following steps: 1. Compare the texts in the two sets of sequence pairs and identify the specific changes that occurred in each set. 2. Analyze whether these changes show similar patterns or intentions, such as whether they are all for enhancing clarity, correcting errors, updating information, or changing tone. 3. Based on your analysis, determine whether the two sets of text changes may be caused by the same editing instruction or intention. If possible, please speculate what this common instruction or intention may be. 4. If the two sets of changes are not caused by the same instruction, please explain why and give your reasoning process. Please provide your analysis results and explain your reasoning process as detailed as possible."

[0128] To analyze whether categories are equivalent, we can also use the discrimination method of the classification model. The input of the model is the text difference sequence, and the output is the evolutionary segmentation type.

[0129] If the change in the first question is of the same type as the change in the second question, and the change in the first answer is of the same type as the change in the third answer, then there is a causal relationship between the evolutionary instruction Evol Instruction and the sample change. If Evol Instruction can always produce a stable and consistent instruction effect (not a specific result, but an effect type) for different samples, then there is a causal effect between the evolutionary instruction EvolInstruction and the sample change.

[0130] S209: Determine the verification result of the evolution sample to be verified according to the first causal effect evaluation result and the second causal effect evaluation result.

[0131] For example, Figure 3 An implementation example diagram of a verification sample is provided. After the question Q is processed by the general model base LLM, the answer A is generated. A seed sample S{Q, A} is formed based on the question Q and the answer A. The seed sample S{Q, A} and the evolution instruction Evol Instruction are input into the evolution model Evol LLM for evolution to obtain a generated sample S'{Q', A'}. Based on the samples before and after evolution and the evolution instruction, the evolved sample S_e{S->Evol Instruction->S'} is generated to generate a sample library, wherein the sample library includes: seed sample S{Q, A}, generated sample S'{Q', A'} and S_e{S->Evol Instruction->S'}.

[0132] Select an evolution sample {S1->Evol Instruction->S1'} to be verified from the sample library, where S1{Q1, A1} is a seed sample and S1'{Q1', A1'} is an evolution sample. Perform differential analysis through the differential analysis module to determine the sample change, which includes: the first question change ΔQ1 = |Q1'-Q1| and the first answer change ΔA1 = |A1'-A1|.

[0133] In the first stage, the evolution process is analyzed for causal effects (ΔQ->ΔA): the first question Q1 and ΔQ1 are input into the Query generation module, and the Query generation module evolves Q1 based on ΔQ1 to generate the third question Query'. The first question Q1 and the third question Query' are question-answered by the base LLM model to obtain the corresponding first answer Answer to be analyzed and the second answer Answer_new to be analyzed. The first answer Answer to be analyzed and the second answer Answer_new to be analyzed are analyzed for differentiation to obtain the second answer change ΔA1'=|Answer_new–Answer|. The similarity analysis module is used to perform similarity analysis on the first answer change and the second answer change. If the similarity coefficient is greater than the similarity threshold, it is determined that the first causal effect evaluation result is that there is a causal effect between ΔQ and ΔA.

[0134] In the second stage, the causal effect analysis (ΔQ->ΔA) of the evolution instruction is performed: the question and answer sample {Query1, Answer 1} is obtained, and the question and answer sample {Query1, Answer 1} and the evolution instruction EvolInstruction of the evolution sample to be verified are input into the Evol LMM for evolution to obtain the evolution sample {Query2, Answer2}, and the first evolution question Query2 is input into the base LLM for question and answer to obtain the second evolution answer Answer2', and the first evolution question Query2 and the question Query1 of the question and answer sample are differentially analyzed through the differential analysis module to determine the second question change ΔQ1'=|Query2–Query1|, and the first evolution answer Answer2 and the second evolution answer Answer2' are differentially analyzed through the differential analysis module to determine the third answer change ΔA1'=|Answer2'–Answer2|. The category judgment module is used to perform category judgment, and analyze whether the categories of the first question change ΔQ1 and the second question change ΔQ1' are equivalent, and analyze whether the categories of the first answer change ΔA1 and the third answer change ΔA1' are equivalent; if the categories of the first question change ΔQ1 and the second question change ΔQ1' are equivalent, the second causal effect evaluation result is determined to be that there is a causal effect between the question change ΔQ1 and the evolution instruction EvolInstruction; if the categories of the first answer change ΔA1 and the third answer change ΔA1' are equivalent, the second causal effect evaluation result is determined to be that there is a causal effect between the question change ΔQ1 and the evolution instruction Evol Instruction. The verification result of the evolution sample to be verified is determined by combining the first causal effect evaluation result and the second causal effect evaluation result.

[0135] In the embodiment of the present application, when performing verification, the two stages of causal analysis can also be performed successively. For the verification process of performing causal analysis successively, if the result of the first causal analysis is that there is no causal relationship, the verification result can be considered as failing the verification, and the subsequent causal analysis is no longer performed.

[0136] The embodiment of the present invention provides a sample verification method, which solves the problem of inaccurate verification results, can identify and analyze the complex causal relationship between questions, answers and evolution instructions, thereby improving the accuracy and reliability of logic verification; it can adapt to different evolution instructions. The verification method provided by the embodiment of the present application can be applied to any type of samples and evolution instructions, has universality, and is highly logical, which enhances the adaptability and flexibility of the method, makes the verification process more suitable for practical applications, and improves the practicality and ease of use of the verification method. Through the analysis of strong causal rule constraints, potential logical errors can be quickly identified and corrected, thereby improving the quality of the entire instruction generation sample and effectively improving the efficiency of large-scale corpus quality assessment.

[0137] Embodiment 3

[0138] Figure 4 This is a schematic diagram of the structure of a sample verification device provided by Embodiment 3 of the present invention. Figure 4 As shown, the device includes: a to-be-verified sample acquisition module 31, a sample change determination module 32, a causal evaluation module 33 and a verification result determination module 34.

[0139] The sample acquisition module 31 to be verified is used to obtain the evolution sample to be verified, wherein the evolution sample to be verified includes a seed sample, an evolution instruction and a generated sample, the seed sample includes a first question and a first answer, and the generated sample includes a second question and a second answer;

[0140] A sample change determination module 32, configured to perform differential analysis of questions and answers based on the seed sample and the generated sample of the evolution sample to be verified, and determine the sample change during the evolution process;

[0141] A causal evaluation module 33 is used to perform a causal effect evaluation on the evolution process according to the sample change to determine a first causal effect evaluation result, and to perform a causal effect evaluation on the evolution process based on the evolution instruction of the evolution sample to be verified and the sample change to determine a second causal effect evaluation result;

[0142] The verification result determination module 34 is used to determine the verification result of the evolution sample to be verified according to the first causal effect evaluation result and the second causal effect evaluation result.

[0143] An embodiment of the present invention provides a sample verification device, which solves the problem of inaccurate verification results. By performing differential analysis of questions and answers on the seed sample and the generated sample of the evolution sample to be verified, the sample changes generated in the evolution process are determined, and the evolution process is causally evaluated through the sample changes to determine whether there is a causal relationship between the sample changes in the evolution process, thereby obtaining a first causal effect evaluation result; based on the evolution instructions and sample changes, analysis is performed to determine whether there is a causal effect between the evolution instructions and the sample changes, thereby obtaining a second causal effect evaluation result; finally, the verification result of the evolution sample to be verified is determined by combining the first causal effect evaluation result and the second causal effect evaluation result; the sample verification method provided in the present application can identify and analyze the complex causal relationship between questions, answers and evolution instructions, thereby improving the accuracy and reliability of logical verification; it can adapt to different evolution instructions, and the verification of samples is more flexible, so that the verification process is more in line with practical applications, and the practicality and ease of use of the verification method are improved.

[0144] Optionally, the sample change includes: a first question change and a first answer change;

[0145] Causal Assessment Module 33, including:

[0146] A third question generating unit, configured to evolve the first question according to a change of the first question to generate a third question;

[0147] A second answer change unit, configured to perform differential analysis of answers based on the first question and the third question to determine a second answer change;

[0148] The first causal effect evaluation unit is used to perform a causal effect evaluation based on the first answer change and the second answer change to determine a first causal effect evaluation result.

[0149] Optionally, a third question generating unit is specifically used to: input the first question change into the large language model, use a predetermined effect instruction generation prompt as a prompt, instruct the large language model to perform evolutionary effect analysis, and determine the effect instruction according to the output of the large language model; input the first question into the large language model, use the effect instruction as a prompt, instruct the large language model to generate a question, and determine the third question according to the output of the large language model.

[0150] Optionally, the second answer change unit is specifically used to: input the first question into the large language model for question and answering, and determine the first answer to be analyzed according to the output of the large language model; input the third question into the large language model for question and answering, and determine the second answer to be analyzed according to the output of the large language model; perform differential analysis on the first answer to be analyzed and the second answer to be analyzed to determine the change of the second answer.

[0151] Optionally, the first causal evaluation unit is specifically used to: perform similarity analysis on the first answer change and the second answer change to determine a similarity coefficient; if the similarity coefficient is greater than a similarity threshold, determine that the first causal effect evaluation result is that there is a causal effect between the changes in the question and the answer.

[0152] Optionally, the causal assessment module 33 includes:

[0153] A question and answer sample acquisition unit, used for acquiring question and answer samples;

[0154] An evolution sample determination unit, used for inputting the evolution instructions of the question and answer sample and the evolution sample to be verified into the evolution model, and determining the evolution sample according to the output of the evolution model, wherein the evolution sample includes a first evolution question and a first evolution answer;

[0155] The second causal effect evaluation unit is used to perform evolution instruction effect analysis based on the evolution sample and the sample change to determine a second causal effect evaluation result.

[0156] Optionally, the question and answer sample acquisition unit is specifically used to: determine the question text length and the answer text length based on the evolution sample to be verified; and filter out question and answer samples from a pre-generated sample library based on the question text length and the answer text length.

[0157] Optionally, the sample change includes: a first question change and a first answer change;

[0158] The second causal evaluation unit is specifically used to: input the first evolved question into a large language model for question and answering, and determine the second evolved answer according to the output of the large language model; perform differential analysis on the first evolved question and the question and answer sample to determine the change of the second question; perform differential analysis on the first evolved answer and the second evolved answer to determine the change of the third answer; analyze whether the categories of the first question change and the second question change are equivalent, and analyze whether the categories of the first answer change and the third answer change are equivalent, and determine the second causal effect evaluation result according to the analysis result.

[0159] Optionally, the analyzing whether the categories of the first problem change and the second problem change are equivalent includes:

[0160] Inputting the first question variation and the second question variation into a large language model, using a predetermined category judgment prompt as a prompt to instruct the large language model to perform category judgment, and determining whether the categories are equivalent according to an output of the large language model;

[0161] Optionally, analyzing whether the categories of the first answer change and the third answer change are equivalent includes:

[0162] Inputting the first answer variation and the third answer variation into a large language model, using a predetermined category judgment prompt as a prompt, instructing the large language model to perform category judgment, and determining whether the categories are equivalent according to an output of the large language model;

[0163] The category judgment prompt includes at least the following information:

[0164] Identify changes in text;

[0165] Analyze whether the pattern or intent of the changes is similar;

[0166] Analyze whether the instructions or intentions that caused the changes are the same;

[0167] Analyze the reasoning process corresponding to different instructions or intentions that cause changes.

[0168] Optionally, the differential analysis includes at least one of the following:

[0169] Added words;

[0170] Added sentences;

[0171] deleted words;

[0172] deleted sentences;

[0173] Replacement words;

[0174] Replace the sentence.

[0175] The sample verification device provided in the embodiment of the present invention can execute the sample verification method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0176] Embodiment 4

[0177] Figure 5 A schematic diagram of an electronic device 50 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0178] like Figure 5 As shown, the electronic device 40 includes at least one processor 41, and a memory connected to the at least one processor 41, such as a read-only memory (ROM) 42, a random access memory (RAM) 43, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 41 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 42 or the computer program loaded from the storage unit 48 to the random access memory (RAM) 43. In the RAM 43, various programs and data required for the operation of the electronic device 40 can also be stored. The processor 41, the ROM 42, and the RAM 43 are connected to each other through a bus 44. An input / output (I / O) interface 45 is also connected to the bus 44.

[0179] A number of components in the electronic device 40 are connected to the I / O interface 45, including: an input unit 46, such as a keyboard, a mouse, etc.; an output unit 47, such as various types of displays, speakers, etc.; a storage unit 48, such as a disk, an optical disk, etc.; and a communication unit 49, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 49 allows the electronic device 40 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0180] The processor 41 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 41 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 41 performs the various methods and processes described above, such as the sample verification method.

[0181] In some embodiments, the sample verification method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 48. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 40 via the ROM 42 and / or the communication unit 49. When the computer program is loaded into the RAM 43 and executed by the processor 41, one or more steps of the sample verification method described above may be performed. Alternatively, in other embodiments, the processor 41 may be configured to perform the sample verification method in any other appropriate manner (e.g., by means of firmware).

[0182] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0183] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0184] An embodiment of the present invention provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the sample verification method described in any embodiment of the present invention is implemented.

[0185] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0186] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0187] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0188] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0189] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0190] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A sample verification method, characterized in that: include: Acquire an evolution sample to be verified, wherein the evolution sample to be verified includes a seed sample, an evolution instruction, and a generated sample, the seed sample includes a first question and a first answer, and the generated sample includes a second question and a second answer; Perform differential analysis of questions and answers based on the seed sample and the generated sample of the evolution sample to be verified to determine sample changes during the evolution process; Performing a causal effect evaluation on the evolution process according to the sample change to determine a first causal effect evaluation result, and performing a causal effect evaluation on the evolution process based on the evolution instruction of the evolution sample to be verified and the sample change to determine a second causal effect evaluation result; The verification result of the evolution sample to be verified is determined according to the first causal effect evaluation result and the second causal effect evaluation result.

2. The method according to claim 1, characterized in that The sample change includes: a first question change and a first answer change. The causal effect evaluation of the evolution process is performed according to the sample change to determine a first causal effect evaluation result, including: Evolving the first question according to the change of the first question to generate a third question; Perform differential analysis of answers based on the first question and the third question to determine a change in the second answer; A causal effect assessment is performed based on the first answer change and the second answer change to determine a first causal effect assessment result.

3. The method according to claim 2, characterized in that The step of evolving the first question according to the change of the first question to generate a third question includes: Inputting the first question variation into the large language model, using a predetermined effect instruction generation prompt as a prompt, instructing the large language model to perform an evolution effect analysis, and determining an effect instruction according to an output of the large language model; The first question is input into the large language model, the effect instruction is used as a prompt to instruct the large language model to generate a question, and a third question is determined according to an output of the large language model.

4. The method according to claim 2, characterized in that: The step of performing differential analysis of answers based on the first question and the third question to determine a change in the second answer includes: Inputting the first question into a large language model for question answering, and determining a first answer to be analyzed according to an output of the large language model; Inputting the third question into the large language model for question answering, and determining a second answer to be analyzed according to an output of the large language model; Perform differential analysis on the first answer to be analyzed and the second answer to be analyzed to determine a change in the second answer.

5. The method according to claim 2, characterized in that: The step of performing a causal effect assessment based on the first answer change and the second answer change to determine a first causal effect assessment result includes: Performing a similarity analysis on the first answer change and the second answer change to determine a similarity coefficient; If the similarity coefficient is greater than the similarity threshold, it is determined that the first causal effect evaluation result is that there is a causal effect in the change of the question and the answer.

6. The method according to claim 1, characterized in that The step of performing a causal effect evaluation on the evolution process based on the evolution instruction of the evolution sample to be verified and the sample change to determine a second causal effect evaluation result includes: Get sample questions and answers; Inputting the question and answer sample and the evolution instructions of the evolution sample to be verified into the evolution model, and determining the evolution sample according to the output of the evolution model, wherein the evolution sample includes a first evolution question and a first evolution answer; An evolution instruction effect analysis is performed based on the evolution sample and the sample change to determine a second causal effect evaluation result.

7. The method according to claim 6, characterized in that The obtaining of the question and answer sample includes: Determining the question text length and the answer text length according to the evolution sample to be verified; Question and answer samples are screened out from a pre-generated sample library according to the question text length and the answer text length.

8. The method according to claim 6, characterized in that The sample change includes: a first question change and a first answer change, and the evolution instruction effect analysis is performed according to the evolution sample and the sample change to determine the second causal effect evaluation result, including: Inputting the first evolved question into a large language model for question answering, and determining a second evolved answer according to an output of the large language model; Performing differential analysis on the first evolved question and the question of the question and answer sample to determine a change in the second question; Performing differential analysis on the first evolved answer and the second evolved answer to determine a change in the third answer; Analyze whether the categories of the first question change and the second question change are equivalent, and analyze whether the categories of the first answer change and the third answer change are equivalent, and determine the second causal effect evaluation result based on the analysis results.

9. The method according to claim 8, characterized in that The analyzing whether the categories of the first problem change and the second problem change are equivalent includes: Inputting the first question variation and the second question variation into a large language model, using a predetermined category judgment prompt as a prompt to instruct the large language model to perform category judgment, and determining whether the categories are equivalent according to an output of the large language model; The analyzing whether the categories of the first answer change and the third answer change are equivalent includes: Inputting the first answer variation and the third answer variation into a large language model, using a predetermined category judgment prompt as a prompt to instruct the large language model to perform category judgment, and determining whether the categories are equivalent according to an output of the large language model; The category judgment prompt includes at least the following information: Identify changes in text; Analyze whether the pattern or intent of the changes is similar; Analyze whether the instructions or intentions that caused the changes are the same; Analyze the reasoning process corresponding to different instructions or intentions that cause changes.

10. The method according to any one of claims 1 to 9, characterized in that: The differential analysis includes at least one of the following: Added words; Added sentences; deleted words; deleted sentences; Replacement words; Replace the sentence.

11. An electronic device, characterized in that: The electronic device comprises: at least one processor, and a memory communicatively coupled to the at least one processor; The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the sample verification method according to any one of claims 1 to 10.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the sample verification method according to any one of claims 1 to 10 when executed.

13. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the computer program implements the sample verification method according to any one of claims 1 to 10.

Images