Method and device for verifying output data of processing model, equipment and medium

Abstract

The invention provides a method, a device, equipment and a medium for verifying output data of a processing model. In one method, in response to receiving input data for a processing model, output data corresponding to the input data is determined from the processing model. A verification result associated with the input data and the output data is acquired from a verification model corresponding to the processing model, the verification result indicating a degree of matching between the output data and the input data. A final result corresponding to the input data is provided based on the verification result and the output data. By means of the exemplary implementation mode, the verification model can judge whether the output data are correct or not, and then a corresponding final result is output to a user. In this way, the accuracy of the processing model can be improved without introducing an additional knowledge base.

Description

Technical Field

[0001] Exemplary implementations of the present disclosure relate generally to machine learning, and more particularly to methods, apparatuses, devices, and computer-readable storage media for validating output data of a processing model. Background Art

[0002] Machine learning techniques have been widely used to perform a variety of processing tasks. For example, a variety of processing models can be constructed to perform various processing tasks. Input data can be sent to the processing model and output results from the processing model can be received. However, due to various reasons such as training data, model structure, and training process, the results of the processing model output may not fully match the input data. At this time, it is expected to improve the accuracy of the processing model and enable the processing model to provide a final result that is more compatible with the input data. Summary of the invention

[0003] In a first aspect of the present disclosure, a method for verifying output data of a processing model is provided. In the method, in response to receiving input data for the processing model, output data corresponding to the input data is determined from the processing model. A verification result associated with the input data and the output data is obtained from a verification model corresponding to the processing model, the verification result indicating a degree of match between the output data and the input data. Based on the verification result and the output data, a final result corresponding to the input data is provided.

[0004] In a second aspect of the present disclosure, a device for verifying output data of a processing model is provided. The device comprises: a determination module configured to determine output data corresponding to the input data from the processing model in response to receiving input data for the processing model; an acquisition module configured to obtain a verification result associated with the input data and the output data from a verification model corresponding to the processing model, the verification result indicating a degree of match between the output data and the input data; and a providing module configured to provide a final result corresponding to the input data based on the verification result and the output data.

[0005] In a third aspect of the present disclosure, an electronic device is provided. The electronic device includes: at least one processing unit; and at least one memory, the at least one memory is coupled to the at least one processing unit and stores instructions for execution by the at least one processing unit, and when the instructions are executed by the at least one processing unit, the electronic device executes the method according to the first aspect of the present disclosure.

[0006] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the processor implements the method according to the first aspect of the present disclosure.

[0007] It should be understood that the content described in this content section is not intended to limit the key features or important features of the implementation of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easy to understand through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] In the following, in conjunction with the accompanying drawings and with reference to the following detailed description, the above and other features, advantages and aspects of various implementations of the present disclosure will become more apparent. In the accompanying drawings, the same or similar figure labels represent the same or similar elements, wherein:

[0009] Figure 1 A block diagram showing an application environment of a processing model according to an exemplary implementation of the present disclosure;

[0010] Figure 2 A block diagram illustrating output data for validating a processing model according to some implementations of the present disclosure is shown;

[0011] Figure 3 A block diagram for obtaining a verification model according to some implementations of the present disclosure is shown;

[0012] Figure 4 A block diagram for obtaining negative samples according to some implementations of the present disclosure is shown;

[0013] Figure 5 A block diagram for obtaining positive samples according to some implementations of the present disclosure is shown;

[0014] Figure 6 A block diagram for obtaining positive samples and negative samples according to some implementations of the present disclosure is shown;

[0015] Figure 7 A block diagram illustrating the structure of a data portion of a reference sample according to some implementations of the present disclosure;

[0016] Figure 8 A block diagram of a page for providing a final result according to some implementations of the present disclosure is shown;

[0017] Fig. 9 A block diagram of a page for providing a final result according to some implementations of the present disclosure is shown;

[0018] Fig.10 A flowchart showing a method for validating output data of a processing model according to some implementations of the present disclosure is shown;

[0019] Fig.11 A block diagram showing an apparatus for verifying output data of a processing model according to some implementations of the present disclosure; and

[0020] Fig.12 A block diagram of a device capable of implementing multiple implementations of the present disclosure is shown. DETAILED DESCRIPTION

[0021] The implementation of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain implementations of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be interpreted as being limited to the implementations described herein. On the contrary, these implementations are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and implementations of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0022] In the description of the implementation methods of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "an implementation" or "the implementation" should be understood as "at least one implementation". The term "some implementations" should be understood as "at least some implementations". The following may also include other explicit and implicit definitions. As used herein, the term "model" can represent the association relationship between various data. For example, the above-mentioned association relationship can be obtained based on a variety of technical solutions currently known and / or to be developed in the future.

[0023] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0024] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0025] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0026] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information is sent to the user in a manner such as a pop-up window, in which the prompt information can be presented in text form. In addition, the pop-up window can also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0027] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0028] The term "in response to" as used herein refers to a state in which a corresponding event occurs or a condition is satisfied. It will be understood that the timing of executing a subsequent action executed in response to the event or condition is not necessarily strongly related to the time when the event occurs or the condition is satisfied. For example, in some cases, the subsequent action may be executed immediately when the event occurs or the condition is satisfied; while in other cases, the subsequent action may be executed some time after the event occurs or the condition is satisfied.

[0029] Example Environment

[0030] Machine learning techniques have been widely used to perform a variety of processing tasks. Figure 1 Describes an application environment according to an example implementation of the present disclosure. Figure 1 A block diagram 100 of an application environment of a processing model according to an exemplary implementation of the present disclosure is shown. Figure 1 As shown, a processing model 130 may be pre-built, and input data 110 may be provided to the processing model 130. For example, in the context of a language processing model, the input data 110 may be "what is a programming language", etc. At this time, the processing model 130 may provide output data 120 matching the input data 110, for explaining the meaning of the programming language.

[0031] Due to various reasons such as training data, model structure, and training process, the results of processing model output may not completely match the input data. Specifically, large model-generated hallucination refers to the results that may appear inconsistent with reality or have no practical basis when using large deep learning models (such as large language models (LLMs), image generation models, etc.) to generate new content or answer questions. This phenomenon usually stems from factors such as data bias during model training, inaccurate training data, or lack of logical constraints.

[0032] There are many reasons that may cause large models to generate hallucinations, the main reasons are as follows. (1) Data bias: If the training data is biased or repeatedly repeats existing false and misleading information, the model may learn these false cognitions, resulting in hallucinations when generating content. (2) Overfitting: The model may over-learn the noise and information in the training data and produce fictitious or inaccurate results when generating content. These results may appear to be real, but in fact they are caused by the model overfitting the data. (3) Insufficient training data: Models require a large amount of data to obtain more comprehensive knowledge during training. If the amount of training data is insufficient or the quality is low, the model may not be able to fully understand the information about a certain entity or concept, resulting in hallucinations when generating content. (4) Lack of logical constraints: Because deep learning models are often based on statistics and probabilities, they may lack consistent logic when generating content. This may cause the model to produce inconsistent or illogical hallucinations in the output.

[0033] To address the problem of large model generation hallucinations, solutions have been proposed from both the data and model levels. For example, at the data level, the following methods can be used to alleviate hallucinations. (1) Data cleaning: Generate training data for large models using more detailed and clear annotations to minimize bias, errors, and misleading content. Although this method can reduce the generation of large model hallucinations at the source, the manpower required is very huge. (2) Vector database: Retrieve domain or encyclopedia knowledge through a vector database and input the retrieved content into the model so that the answers generated by the model have a source and are traceable. However, the cost of maintaining and developing a vector database is huge, requiring more R&D manpower. (3) Integration into a knowledge base: Each vertical domain knowledge can have a knowledge base in that domain. The user's input data and the model's output data can be searched for similarities with the content in the knowledge base to identify which content involves model hallucinations and which content is within the scope of the domain knowledge base. This method also relies on the knowledge base of the vertical domain, and building this knowledge base also requires R&D costs.

[0034] For example, the following methods can be used at the model / strategy level. (1) Based on RLHF (Reinforce Learning From Human Feedback): This reinforcement learning method based on human feedback can align with human knowledge and adjust parameters at the model level. However, implementing the RLHF framework itself is also difficult and cumbersome. In addition, process-based supervision mainly solves the problem of model reasoning tasks. Supervision of each reasoning step can reduce the generation of model hallucinations, and this method is better than the method of simply supervising the generated results. However, in question-answering where there is no or no reasoning process, the advantages of this method are difficult to reflect. (2) Data retrieval: This method is to vectorize the user input and then search for relevant data in the training set or vertical field. It also requires additional engineering support related to data and vector retrieval. (3) Model generation and search engine cross-validation: This method is to send the question to the large model and search engine at the same time after the user enters the question, and then verify the returned content to determine whether the model generation is an illusion. This also increases the complexity of the process and resource overhead.

[0035] Although the above methods can alleviate the problem of hallucination, the data-level method requires additional operations to be performed on the training data or requires an additional knowledge base, and the model and strategy-level method requires additional training steps or execution processes. This greatly increases the difficulty and complexity of data processing and increases time consumption. At this time, it is expected to improve the accuracy of the processing model and enable the processing model to provide a final result that is more matched to the input data.

[0036] Overview of the Verification Process

[0037] In order to at least partially address the deficiencies in the prior art, according to an exemplary implementation of the present disclosure, a method for verifying output data of a processing model is proposed. Figure 2 An overview of an exemplary implementation of the present disclosure is described. Figure 2 A block diagram 200 is shown for verifying output data of a processing model according to some implementations of the present disclosure. Figure 2 As shown, a verification model 210 may be added downstream of the processing model 130, and the processing system 240 at this time may include the processing model 130 and the verification model 210. The verification model 210 may be used to determine whether the output data 120 from the processing model 130 matches the input data 110 (that is, whether the output data 120 is correct), and then provide a final result 230 based on the output data 130 and the verification result 220.

[0038] Specifically, the user can construct input data 110 and submit the input data 110 to the processing system 240. In the case of receiving the input data 110 for the processing model, the processing system 240 can obtain the output data 120 corresponding to the input data 110 from the processing model 130. Further, the input data 110 and the output data 120 can be input to the downstream verification model 210, so as to obtain the verification result 220 associated with the input data 110 and the output data 120 from the verification model 210 corresponding to the processing model 130. It should be understood that the verification model 210 here can be a pre-constructed model for describing the association relationship between the input data and the output data. For example, the verification result 220 can represent the matching degree between the output data and the input data, that is, whether the output data is the correctness of the answer to the input data.

[0039] According to an example implementation of the present disclosure, the verification result 220 may indicate the degree of match between the output data and the input data. For example, the verification model 210 may be a binary classification model that describes correctness or error, that is, the verification result 220 may be represented by 1 (hallucination) and 0 (non-hallucination). Alternatively and / or additionally, the verification model 210 may also be represented by a real number in the range of [0,1], and a larger value may indicate a greater probability that the output data is a hallucination. Further, based on the verification result 220 and the output data 120, the processing system 240 may provide a final result 230 for the input data 110.

[0040] In this way, the verification model 210 can determine whether the output data 120 directly provided by the processing model 130 is correct (that is, whether it is an illusion), and then generate the final result 230. For example, assuming that the verification result 220 indicates that the output data 120 has a high degree of match (that is, non-illusion), the output data 120 can be directly provided as the final result 230. Assuming that the verification result 220 indicates that the output data 120 has a low degree of match (that is, an illusion), the user can be reminded that the output data 120 may not be accurate, and / or the user can be reminded to check whether the input data 110 includes typos, etc. At this time, the processing system 240 can receive the input data 110 as a whole and provide a more accurate final result 230.

[0041] By using the exemplary implementation of the present disclosure, the verification system 210 can quickly determine whether the output data 120 is correct, and then output the corresponding final result 230 to the user. In this way, the accuracy of the processing system 240 can be improved without introducing an additional knowledge base.

[0042] Detailed process of verification process

[0043] Already seen Figure 2The following describes an overview of an example implementation according to the present disclosure. Figure 2 The processing model 130 is described by taking a language model as an example. Alternatively and / or additionally, the processing model 130 may include a multimodal processing model. In this case, the input data 110, the output data 120, and the final result 230 may span multiple modalities, thereby processing a variety of tasks with richer expressions.

[0044] According to an example implementation of the present disclosure, the verification model 210 may be a pre-built model. It should be understood that the processing model 130 may involve a large number of fields. For example, when the processing model 130 is a large language model, the processing model 130 may involve a large number of tasks in a large field such as query, translation, and analysis. At this time, the verification model 210 can be obtained for each segmented vertical field. In this way, the difficulty of generating a verification model for all fields can be reduced, and the accuracy of the processing model 130 in the segmented vertical field can be improved.

[0045] According to an exemplary implementation of the present disclosure, the verification model 210 may be established based on a method that has been proposed currently and / or will be developed in the future. Further, the verification model 210 may be updated using the input data 110 and the output data 120. Specifically, a reference sample set for training the verification model 210 may be constructed. Figure 3 Describes more details about the reference sample set. Figure 3 A block diagram 300 for obtaining a verification model according to some implementations of the present disclosure is shown. Figure 3 As shown, the reference sample set 310 may include a large number of reference samples, such as positive samples 314 and negative samples 312 .

[0046] The data portion of each reference sample may include reference input data and reference output data, and the annotation portion of each sample may indicate the degree of match between the output data and the input data (for example, represented by 0 and 1). Further, the verification model 210 may be trained using these samples respectively, so that the verification model 210 may describe the degree of match between the output data and the input data, that is, determine whether the output data is an illusion.

[0047] According to an example implementation of the present disclosure, reference input data may be generated. The reference input data may be input to the processing model 130 so as to obtain reference output data corresponding to the reference input data from the processing model 130. Further, the reference input data, the reference output data, and the degree of matching between the reference output data and the reference input data may be used to generate a reference sample for obtaining a verification model. Figure 4 Describe the process of generating negative samples. Figure 4A block diagram 400 for obtaining negative samples according to some implementations of the present disclosure is shown.

[0048] like Figure 4 As shown, reference input data 410 with semantic meaning can be obtained. For example, in the case where the processing model 130 is a language model, a text expressed in natural language with correct semantics can be used as the reference input data 410. The reference input data 410 can be input into the processing model 130, and reference output data 420 can be obtained. The annotated data 430 can be determined, for example, the annotated data 430 can be determined based on manual annotation; for another example, the annotated data 430 can be automatically determined. It should be understood that since the reference input data 410 has the correct semantic meaning, when the processing model 130 has been relatively fully trained, the reference output data 420 provided by the processing model 130 is the correct answer, and there is no hallucination. At this time, the annotated data 430 can be set to "0" to indicate non-hallucination.

[0049] It should be understood that in the process of generating negative samples, no additional knowledge base is required, nor is an additional manual annotation process required. In this way, negative samples can be generated in a simple and effective manner. At this time, when the processing model 130 has been fully trained, the processing model 130 can be reused to generate negative samples, thereby reducing the complexity of generating training samples and various resource overheads. Further, reference input data 410, reference output data 420 and annotation data 430 can be used to generate negative samples for training verification model 210. Table 1 below shows an example of generated negative samples.

[0050] Table 1 Examples of negative samples

[0051]

[0052]

[0053] According to an example implementation of the present disclosure, positive samples for training the validation model 210 may be generated. Figure 5 Describe in more detail the Figure 5 A block diagram 500 for obtaining negative samples according to some implementations of the present disclosure is shown. Figure 5 As shown, reference input data 510 without semantic meaning can be obtained. For example, when the processing model 130 is a language model, text expressed in natural language with incorrect semantics can be used as reference input data 510. The reference input data 510 can be input to the processing model 130, and reference output data 520 can be obtained.

[0054] At this time, the labeled data 530 can be determined based on knowledge such as concepts, inferences, question-and-answer content, etc. in the relevant field. For example, the labeled data 530 can be determined based on manual labeling; the labeled data 530 can be determined based on comparison with the above knowledge; and for another example, the labeled data 530 can be automatically determined. It should be understood that since the reference input data 510 has an incorrect semantic meaning, the processing model 130 cannot understand the query content expressed by the reference data 510, and the reference output data 520 provided by the processing model 130 will be the wrong answer, that is, hallucination. At this time, the labeled data 530 can be set to "1" to represent hallucination. Further, the reference input data 510, the reference output data 520, and the labeled data 530 can be used to generate a positive sample 314 for training the verification model 210.

[0055] It should be understood that in the process of generating positive samples, no additional knowledge base is required, nor is an additional manual labeling process required. In this way, positive samples can be generated in a simple and effective manner. According to an example implementation manner of the present disclosure, a large number of negative samples can be generated based on the method described above for training the verification model 210.

[0056] According to an example implementation manner of the present disclosure, the reference input data 410 with semantic meaning can be referred to as true input data, and the reference input data 510 without semantic meaning can be referred to as pseudo input data. The pseudo input data can be generated by modifying the true input data. For example, at least a part of the true input data can be replaced to create the pseudo input data. For example, the true input data can be "programming language", and any one or more of the above words can be replaced. For example, "编 (biān)" can be replaced with "偏 (piān)" with the same pronunciation and / or similar glyph, and "编程 (biān chéng)" can be replaced with "变成 (biàn chéng)" with a similar pronunciation, etc.

[0057] Alternatively and / or additionally, the pseudo input data can be generated by splicing. For example, two or more unrelated texts (such as "programming language" and "repo strategy") can be spliced, and the texts spliced here do not have correct semantics. Using the example implementation manner of the present disclosure, the pseudo input data can be determined quickly and effectively, thereby generating positive samples for detecting hallucinations. Table 2 below shows an example of the generated positive samples.

[0058] Table 2 Example of positive samples

[0059]

[0060] According to an exemplary implementation of the present disclosure, multiple true input data with semantic meanings can be obtained respectively, and multiple true output data corresponding to the multiple true input data can be obtained respectively from the processing model. Further, each input data and output data can be used to generate positive samples and / or negative samples. Figure 6 Describes more details about the generated samples. Figure 6 A block diagram 600 for obtaining positive samples and negative samples according to some implementations of the present disclosure is shown. Figure 6 As shown, multiple reference input data 410, 610, ..., 620 with correct semantics can be obtained. The above reference input data can be respectively input to the processing model 130, and multiple reference output data 420, 612, ..., 622 can be respectively received from the processing model 130.

[0061] like Figure 6 As shown, the legend 630 shown in solid lines represents the negative sample relationship between the two data, and the legend 632 shown in dotted lines represents the positive sample relationship between the two data. At this time, the negative sample may include: (reference input data 410, reference output data 420, 0), (reference input data 610, reference output data 612, 0), ..., (reference input data 620, reference output data 622, 0).

[0062] Further, for the first true input data among the multiple true input data, the first true input data and the second true output data among the multiple true output data corresponding to the second true input data other than the first true input data can be used to generate a positive sample. Alternatively and / or additionally, multiple positive samples can be generated, for example, the second true input data and the first true output data among the multiple true output data corresponding to the first true input data can be used to generate another positive sample in the reference sample. That is, the positive sample can be generated using the mismatched reference input data and reference output data. Using the example implementation of the present disclosure, a large number of positive samples can be generated in a combined manner, thereby improving the efficiency of the training process.

[0063] exist Figure 6In the example of , the positive sample may include: (reference input data 410, reference output data 612, 1), ..., (reference input data 410, reference output data 622, 1), (reference input data 610, reference output data 420, 1), ..., (reference input data 610, reference output data 622, 1), (reference input data 620, reference output data 420, 1), ..., (reference input data 620, reference output data 612, 1). In the case of only two true input data, the order of the two true input data can be exchanged (or the order of the output data corresponding to the two true input data can be exchanged) to generate the corresponding positive sample. Using the example implementation of the present disclosure, the process of generating positive samples can be simplified, thereby increasing the speed of generating positive samples. For example, the order of the reference input data in Table 1 can be exchanged, and a positive sample as shown in Table 3 below can be generated.

[0064] Table 3 Examples of positive samples

[0065]

[0066]

[0067] It should be understood that the above process is merely illustrative, and alternatively and / or additionally, the verification model 210 may be a model for detecting the correct answer. Figure 4 The labeled data 430 in is set to "1" to indicate the correct answer, and the generated reference sample can be used as a positive sample. Figure 5 The labeled data 530 shown is set to "0" to represent an incorrect answer, and the generated reference sample can be used as a negative sample. At this time, the generated positive sample and negative sample can be used to train the verification model 210 so that the trained verification model 210 outputs "1" when a correct answer is detected, and outputs "0" when an incorrect answer is detected.

[0068] The processes for generating positive samples and negative samples have been described above, respectively. A large number of samples can be generated based on the above processes to train the verification model 210. When the verification model 210 has been obtained, the trained verification model 210 can be used to determine whether the output of the processing model 130 is a hallucination. For example, the input data 110 and the output data 120 can be submitted to the verification model 210, and the verification result 220 can be received from the verification model 210.

[0069] It should be understood that although the above only shows the case where the data portion of the reference sample only includes reference input data and reference output data. Alternatively and / or additionally, the data portion may include more features. For example, the probability distribution of each element in the reference output data may be obtained from the processing model 130. Figure 7 Describes more details about the data section of the reference sample, Figure 7 Block diagram 700 illustrates the structure of a data portion of a reference sample according to some implementations of the present disclosure.

[0070] like Figure 7 As shown in Table 1, the data part 710 may include reference input data 711, reference output data 712, and probability distribution 713. Here, probability distribution 713 may be intermediate data of the processing model 130, for example, each element in the reference output data (for example, the probability of a basic language unit such as a character or a word) may be represented. At this time, the negative sample shown in Table 1 may be rewritten as shown in Table 4. Similarly, a new column "probability distribution" may be inserted into the examples of the positive samples shown in Tables 2 and 3.

[0071] Table 4 Examples of negative samples

[0072]

[0073] At this time, the verification model 210 can further describe the reference input data, the reference output data, the probability distribution of each element in the reference output data, and the correlation between the matching degrees. Since the probability distribution 713 can describe the expected occurrence probability of each element in the process of generating the reference output data 712 based on the reference input data 711, such as Figure 7 The format shown can more accurately describe the intermediate details of the processing process of the processing model 130, and further characterize the relationship between input and output in a more accurate and effective manner.

[0074] When using the verification model 210 involving probability distribution, a large number of samples can be generated based on the above process to train the verification model 210. In the case where the verification model 210 has been obtained, the trained verification model 210 can be used to determine whether the output of the processing model 130 is an illusion. For example, the input data 110, the output data 120, and the probability distribution of each element in the output data 120 can be submitted to the verification model 210, and the verification result 220 can be received from the verification model 210. In this way, it is possible to verify in a more accurate manner whether the output data 120 is an illusion. Then, the corresponding final result is provided to the user based on the verification result.

[0075] According to an exemplary implementation of the present disclosure, if it is determined that the verification result meets the predetermined condition, the output data can be directly provided. Figure 8 Describe in more detail the Figure 8 A block diagram 800 of a page for providing a final result according to some implementations of the present disclosure is shown. Figure 8As shown, in page 810, the user may ask question 820 "What is a programming language?" At this time, in the background of the processing system, "What is a programming language" may be input into the verification model and the output from the processing model "Programming language ..." may be input.

[0076] If the verification result output by the verification model satisfies a predetermined condition (for example, when the verification model is a binary classification model, the verification result is "0" (i.e., not a hallucination); or when the verification model outputs a predicted probability, the verification result is less than or equal to a predetermined value (e.g., 0.5 or other numerical values)), the output data from the processing model 130, i.e., the answer 830, can be directly provided to the user. In this way, it can be ensured that the answer 830 output to the user is not a hallucination and is a verified correct answer, thereby ensuring the accuracy of the final result output by the processing system.

[0077] According to an example implementation of the present disclosure, if it is determined that the verification result does not meet the predetermined conditions, at least one of the following is provided: output data, an indication of the degree of match between the output data and the input data. For example, assuming that the verification result output by the verification model does not meet the predetermined conditions (for example, when the verification model is a binary classification model, the verification result is "1" (that is, an illusion); or when the verification model outputs a predicted probability, the verification result is greater than a predetermined value (for example, 0.5 or other numerical values)), the user may be prompted that the correct answer matching the question cannot be found; or the user may be prompted that the answer provided may not be completely accurate. Further, the output data from the processing model may be provided after the prompt statement. In this way, the user's attention may be alerted, thereby preventing the user from making the next judgment based on an answer that may not be correct.

[0078] Alternatively and / or additionally, the user may be further prompted to modify the input data. Fig. 9 Describe in more detail the Fig. 9 A block diagram 900 of a page for providing a final result according to some implementations of the present disclosure is shown. Fig. 9 As shown, suppose the user inputs question 920, in which the term "programming language" is mistakenly input as "parallel programming language". At this time, page 910 may provide prompt 930 and ask the user whether the question inputted is correct. The user may check and find the typo in question 920, correct the typo and input the correct question 940. At this time, processing model 130 may output output data corresponding to question 940. When verification model 210 confirms that the output data is not an illusion, page 910 may output answer 940. In this way, it is easy to find errors in user questions, thereby reminding the user to correct the errors and output the correct answers.

[0079] According to an example implementation of the present disclosure, if it is determined that the verification result does not meet the predetermined condition, it means that there is a potential problem in the processing model when processing the input data. The problem may be caused by an error in the input data itself, or it may be caused by a problem with the model itself and / or a problem with the training data. At this time, the technicians of the processing system can be reminded to pay special attention to the input data, and then perform corresponding tests and adjustments. For example, the correct data corresponding to the input data can be obtained to update the processing model. In this way, the illusion in the processing model can be quickly and effectively discovered, and then the illusion can be eliminated.

[0080] It should be understood that although the above describes the specific implementation of the present disclosure based on the natural language model of Chinese, alternatively and / or additionally, the process of verifying the output data can be implemented in other language environments. For example, in environments such as English, French, German, and Japanese, positive samples and negative samples can be generated based on the above-described method, and then these samples can be used to train the verification model.

[0081] According to an example implementation of the present disclosure, in a multimodal environment, multimodal positive and negative samples can be constructed based on the method described above. Assuming that the processing model 130 can generate images based on text input, positive and negative samples can be generated based on the above method. For example, you can input "draw a cat" to obtain an image of a cat, and you can replace "cat" with "dog" to obtain an image of a dog. At this time, ("draw a cat", image of a cat, 0) can be used as a negative sample, ("draw a dog", image of a dog, 0) can be used as a negative sample, ("draw a cat", image of a dog, 1) can be used as a positive sample, and ("draw a dog", image of a cat, 1) can be used as a positive sample. Further, the above samples are used to train the verification model.

[0082] By using the exemplary implementation of the present disclosure, the verification model 210 downstream of the processing system 130 can quickly determine whether the output data 120 is correct, and then output the adjusted final result 230 to the user. In this way, the accuracy of the processing system 240 can be improved without introducing an additional knowledge base.

[0083] In the process of identifying hallucinations in large vertical model outputs, content that does not exist in the vertical field can be detected, and content that exists but the model outputs the wrong answer can be detected. For the first case, the verification model can identify content that does not exist in the field, and at this time, a conclusion can be obtained with a high probability whether the content is a hallucination. For the second case, the verification model can determine that there may be a mismatch between the question entered by the user and the answer output by the model.

[0084] Furthermore, when considering the probability distribution of elements in the output data, the processing model itself can also predict the probability value of the next element. Generally speaking, the more confident the processing model is in outputting a certain element, the larger the probability value; otherwise, the smaller it is. At this point, the probability values ​​of each element in the output data of the large model can be used as an auxiliary to input the verification model, which can help improve the accuracy of the detection model. The verification model in a vertical field may involve the following content: knowledge such as concepts, reasoning, question-and-answer content in the vertical field, hallucination detection cases of input data and output data, and the probability value output by the processing model itself.

[0085] In vertical fields, there is no need to perform manual processing on the constructed hallucination question-answer pairs, but they can be used directly for training. Specifically, the modeling process can be performed based on the following method. The encoder of the large model can be used to extract the vector V1 of the question and the vector V2 of the answer respectively. V1, V2 and the probability value vector V3 can be spliced ​​and input into the classifier. The real hallucination sample can be used as a test set, and the above classifier can be used to test the model performance index on the data set, and the verification model is continuously optimized to improve the accuracy of identifying hallucinations.

[0086] When the verification model identifies hallucinations in the processing model output, it can prompt the user by configuring a standard template: the model cannot solve the problem temporarily; if no hallucinations are detected, the content of the processing model output can be directly returned to the user after passing through other post-processing modules. In various vertical fields, it is possible to reduce the output of hallucinatory information and reduce user misleading without relying on external knowledge bases or retrieval frameworks, thereby improving the practicality of large vertical models.

[0087] Example Process

[0088] Fig.10 A flow chart of a method 1000 for verifying output data of a process model according to some implementations of the present disclosure is shown. At block 1010, it is determined whether input data for the process model is received. If input data is received, the method 1000 proceeds to block 1020 to determine output data corresponding to the input data from the process model. At block 1030, a verification result associated with the input data and the output data is obtained from a verification model corresponding to the process model, the verification result indicating a degree of match between the output data and the input data. At block 1040, a final result corresponding to the input data is provided based on the verification result and the output data.

[0089] According to an example implementation of the present disclosure, a verification model is obtained by: obtaining reference output data corresponding to reference input data from a processing model; generating a reference sample for obtaining the verification model using the reference input data, reference output data, and the degree of match between the reference output data and the reference input data; and obtaining the verification model based on the reference sample.

[0090] According to an example implementation of the present disclosure, the reference input data is true input data with semantic meaning, and the reference sample is a negative sample for verifying the model.

[0091] According to an example implementation of the present disclosure, the reference input data is pseudo input data without semantic meaning, and the reference sample is a positive sample for verifying the model.

[0092] According to an example implementation of the present disclosure, the pseudo input data is obtained by: obtaining real input data having semantic meaning; and replacing at least a portion of the real input data to create the pseudo input data.

[0093] According to an example implementation of the present disclosure, generating a reference sample includes: respectively acquiring a plurality of true input data having semantic meanings; respectively acquiring a plurality of true output data corresponding to the plurality of true input data from a processing model; and generating a positive sample for a first true input data among the plurality of true input data by utilizing the first true input data and a second true output data among the plurality of true output data corresponding to a second true input data other than the first true input data.

[0094] According to an example implementation of the present disclosure, the method 1000 further includes: generating another positive sample in the reference sample by using the second true input data and the first true output data corresponding to the first true input data among the plurality of true output data.

[0095] According to an example implementation of the present disclosure, the verification model further describes the reference input data, the reference output data, the probability distribution of each element in the reference output data, and the correlation relationship between the matching degrees.

[0096] According to an example implementation of the present disclosure, generating the reference sample further includes: generating the reference sample using the probability distribution of each element in the reference output data determined by the processing model.

[0097] According to an example implementation of the present disclosure, providing the final result includes: in response to determining that the verification result satisfies a predetermined condition, providing output data.

[0098] According to an example implementation of the present disclosure, providing a final result includes: in response to determining that the verification result does not meet a predetermined condition, providing at least any one of the following: output data, an indication of a degree of match between the output data and the input data, an indication for modifying the input data.

[0099] According to an example implementation of the present disclosure, the method 1000 further includes: in response to determining that the verification result does not satisfy a predetermined condition, updating the processing model using the input data and the output data corresponding to the input data.

[0100] According to an example implementation of the present disclosure, the processing model is a multimodal processing model, and the validation model corresponds to a domain of the input data.

[0101] Example devices and equipment

[0102] Fig.11 A block diagram of an apparatus 1100 for verifying output data of a processing model according to some implementations of the present disclosure is shown. The apparatus includes: a determination module 1110, configured to determine output data corresponding to the input data from the processing model in response to receiving input data for the processing model; an acquisition module 1120, configured to obtain a verification result associated with the input data and the output data from a verification model corresponding to the processing model, the verification result indicating a degree of match between the output data and the input data; and a providing module 1130, configured to provide a final result corresponding to the input data based on the verification result and the output data.

[0103] According to an example implementation of the present disclosure, a verification model is acquired by: a data acquisition module configured to acquire reference output data corresponding to reference input data from a processing model; a generation module configured to generate a reference sample for acquiring the verification model using reference input data, reference output data, and a degree of match between the reference output data and the reference input data; and a model acquisition module configured to acquire the verification model based on the reference sample.

[0104] According to an example implementation of the present disclosure, the reference input data is true input data with semantic meaning, and the reference sample is a negative sample for verifying the model.

[0105] According to an example implementation of the present disclosure, the reference input data is pseudo input data without semantic meaning, and the reference sample is a positive sample for verifying the model.

[0106] According to an example implementation of the present disclosure, pseudo input data is acquired by: a first data acquisition module configured to acquire real input data having semantic meaning; and a replacement module configured to replace at least a portion of the real input data to create pseudo input data.

[0107] According to an example implementation of the present disclosure, a generation module includes: a second data acquisition module, configured to respectively acquire multiple true input data with semantic meanings; a true output data acquisition module, configured to respectively acquire multiple true output data corresponding to the multiple true input data from a processing model; and a first positive sample generation module, configured to generate a positive sample for a first true input data among the multiple true input data, using the first true input data and second true output data among the multiple true output data corresponding to a second true input data other than the first true input data.

[0108] According to an example implementation of the present disclosure, the apparatus further includes: a second positive sample generation module configured to generate another positive sample in the reference sample using the second true input data and first true output data corresponding to the first true input data among the plurality of true output data.

[0109] According to an example implementation of the present disclosure, the verification model further describes the reference input data, the reference output data, the probability distribution of each element in the reference output data, and the correlation relationship between the matching degrees.

[0110] According to an example implementation of the present disclosure, the generation module further includes: a distribution-based generation module configured to generate reference samples using the probability distribution of each element in the reference output data determined by the processing model.

[0111] According to an example implementation of the present disclosure, a providing module includes: a first providing module configured to provide output data in response to determining that a verification result satisfies a predetermined condition.

[0112] According to an example implementation of the present disclosure, the providing module includes: a second providing module, in response to determining that the verification result does not meet a predetermined condition, provides at least any one of the following: output data, an indication of the degree of match between the output data and the input data, and an indication for modifying the input data.

[0113] According to an example implementation of the present disclosure, the apparatus further includes: an updating module configured to update the processing model using the input data and the output data corresponding to the input data in response to determining that the verification result does not satisfy the predetermined condition.

[0114] According to an example implementation of the present disclosure, the processing model is a multimodal processing model, and the validation model corresponds to a domain of the input data.

[0115] Fig.12 1 shows a block diagram of a device 1200 capable of implementing multiple implementations of the present disclosure. It should be understood that Fig.12 The computing device 1200 shown is merely exemplary and should not constitute any limitation on the functionality and scope of the implementations described herein. Fig.12 The computing device 1200 shown may be used to implement the methods described above.

[0116] like Fig.12 As shown, computing device 1200 is in the form of a general-purpose computing device. Components of computing device 1200 may include, but are not limited to, one or more processors or processing units 1210, memory 1220, storage device 1230, one or more communication units 1240, one or more input devices 1250, and one or more output devices 1260. Processing unit 1210 may be an actual or virtual processor and is capable of performing various processes according to a program stored in memory 1220. In a multi-processor system, multiple processing units execute computer executable instructions in parallel to increase the parallel processing capabilities of computing device 1200.

[0117] The computing device 1200 typically includes a plurality of computer storage media. Such media may be any available media accessible to the computing device 1200, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 1220 may be a volatile memory (e.g., registers, caches, random access memory (RAM)), a non-volatile memory (e.g., a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 1230 may be a removable or non-removable medium, and may include a machine-readable medium, such as a flash drive, a disk, or any other medium, which may be capable of being used to store information and / or data (e.g., training data for training) and may be accessed within the computing device 1200.

[0118] The computing device 1200 may further include additional removable / non-removable, volatile / non-volatile storage media. Fig.12 As shown in , a disk drive for reading or writing from a removable, non-volatile disk (e.g., a "floppy disk") and an optical drive for reading or writing from a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to the bus (not shown) by one or more data media interfaces. Memory 1220 may include a computer program product 1225 having one or more program modules that are configured to perform various methods or actions of various implementations of the present disclosure.

[0119] The communication unit 1240 enables communication with other computing devices via a communication medium. Additionally, the functions of the components of the computing device 1200 can be implemented in a single computing cluster or multiple computing machines that can communicate via a communication connection. Therefore, the computing device 1200 can operate in a networked environment using a logical connection with one or more other servers, a network personal computer (PC), or another network node.

[0120] Input device 1250 may be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 1260 may be one or more output devices, such as a display, speaker, printer, etc. Computing device 1200 may also communicate with one or more external devices (not shown) through communication unit 1240 as needed, such as storage devices, display devices, etc., communicate with one or more devices that allow a user to interact with computing device 1200, or communicate with any device that allows computing device 1200 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).

[0121] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, the computer program product is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is provided, on which a computer program is stored, and when the program is executed by a processor, the method described above is implemented.

[0122] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices, equipment, and computer program products implemented according to the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0123] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0124] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable data processing apparatus, or other device implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0125] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple implementations of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of a module, program segment or instruction includes one or more executable instructions for realizing the logical function of the specification. In some implementations as replacements, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.

[0126] The above descriptions of various implementations of the present disclosure are exemplary, non-exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described implementations. The selection of terms used herein is intended to best explain the principles of the implementations, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the various implementations disclosed herein.

Claims

1. A method for verifying output data of a processing model, comprising: In response to receiving input data for the process model, determining output data from the process model corresponding to the input data; obtaining, from a verification model corresponding to the processing model, a verification result associated with the input data and the output data, the verification result indicating a degree of match between the output data and the input data; as well as Based on the verification result and the output data, a final result corresponding to the input data is provided.

2. The method according to claim 1, wherein the verification model is obtained by: acquiring, from the processing model, the reference output data corresponding to the reference input data; Generate a reference sample for obtaining the verification model by using the reference input data, the reference output data, and the matching degree between the reference output data and the reference input data; and The verification model is obtained based on the reference sample.

3. The method according to claim 2, wherein the reference input data is true input data with semantic meaning, and the reference sample is a negative sample of the verification model. 4 . The method according to claim 2 , wherein the reference input data is pseudo input data having no semantic meaning, and the reference sample is a positive sample of the verification model.

5. The method according to claim 4, wherein the pseudo input data is obtained by: Obtaining real input data with semantic meaning; and At least a portion of the real input data is replaced to create the fake input data.

6. The method according to claim 4, wherein generating the reference sample comprises: Obtain multiple real input data with semantic meanings respectively; respectively obtaining from the processing model a plurality of true output data corresponding to the plurality of true input data; as well as For first true input data among the plurality of true input data, the positive sample is generated using the first true input data and second true output data among the plurality of true output data corresponding to second true input data other than the first true input data.

7. The method according to claim 6, further comprising: Another positive sample among the reference samples is generated by using the second true input data and first true output data among the plurality of true output data corresponding to the first true input data.

8. The method according to claim 2, wherein the verification model further describes the reference input data, the reference output data, the probability distribution of each element in the reference output data, and the correlation relationship between the matching degrees.

9. The method according to claim 8, wherein generating the reference sample further comprises: The reference sample is generated using the probability distribution of each element in the reference output data determined by the processing model.

10. The method of claim 1, wherein providing the final result comprises: In response to determining that the verification result satisfies a predetermined condition, the output data is provided.

11. The method of claim 1 , wherein providing the final result comprises: In response to determining that the verification result does not satisfy a predetermined condition, providing at least any one of the following: the output data, an indication of a degree of match between the output data and the input data, an indication for modifying the input data.

12. The method according to claim 1, further comprising: In response to determining that the verification result does not satisfy a predetermined condition, the process model is updated using the input data and output data corresponding to the input data.

13. The method of claim 1, wherein the processing model is a multimodal processing model and the validation model corresponds to a domain of the input data.

14. An apparatus for verifying output data of a processing model, comprising: a determination module configured to, in response to receiving input data for the process model, determine output data corresponding to the input data from the process model; an acquisition module configured to acquire, from a verification model corresponding to the processing model, a verification result associated with the input data and the output data, the verification result indicating a degree of match between the output data and the input data; as well as A providing module is configured to provide a final result corresponding to the input data based on the verification result and the output data.

15. An electronic device, comprising: at least one processing unit; as well as At least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to any one of claims 1 to 13 when executed by the at least one processing unit.

16. A computer-readable storage medium having a computer program stored thereon, the computer program causing a processor to implement the method according to any one of claims 1 to 13 when the computer program is executed by the processor.

Images