Sample classification method and device based on large model, electronic equipment and storage medium

Abstract

The invention discloses a sample classification method and device based on a large model, electronic equipment and a storage medium, and relates to the technical field of computers, in particular to the artificial intelligence fields of deep learning, large models, agents and the like. The specific implementation scheme is as follows: acquiring a target negative feedback sample; wherein the target negative feedback sample comprises a query statement and a model answer corresponding to the query statement, and the model answer is the answer of the query statement generated by using the first large model; obtaining a reference answer corresponding to the query statement; according to the model answer and the reference answer, classifying the target negative feedback sample by using a second large model to obtain a classification result of the target negative feedback sample; wherein the classification result comprises a problem category label of the target negative feedback sample and a judgment reason of the problem category label.

Description

Technical Field

[0001] The present application relates to the field of computer technology, in particular to artificial intelligence fields such as deep learning, large models, and intelligent agents, and specifically to a sample classification method, device, electronic device, and storage medium based on a large model. Background Art

[0002] In the field of artificial intelligence, sample classification is one of the core tasks of machine learning, especially supervised learning. Its goal is to determine the category information of samples and label them. For example, in question-answering systems, question-answer pairs can be classified. Summary of the Invention

[0003] This application provides a sample classification method, device, electronic device, and storage medium based on a large model. The specific solution is as follows:

[0004] According to one aspect of the present application, a sample classification method based on a large model is provided, comprising:

[0005] Obtaining a target negative feedback sample; wherein the target negative feedback sample includes a query statement and a model answer corresponding to the query statement, wherein the model answer is an answer to the query statement generated using the first model;

[0006] Obtaining a reference answer corresponding to the query statement;

[0007] According to the model answer and the reference answer, the target negative feedback sample is classified using the second largest model to obtain a classification result of the target negative feedback sample; wherein the classification result includes the question category label of the target negative feedback sample and the reason for determining the question category label.

[0008] According to another aspect of the present application, a sample classification device based on a large model is provided, comprising:

[0009] A first acquisition module is configured to acquire a target negative feedback sample; wherein the target negative feedback sample includes a query statement and a model answer corresponding to the query statement, wherein the model answer is an answer to the query statement generated using the first large model;

[0010] A second acquisition module is used to obtain a reference answer corresponding to the query statement;

[0011] A classification module is used to classify the target negative feedback sample using a second large model according to the model answer and the reference answer to obtain a classification result of the target negative feedback sample; wherein the classification result includes the question category label of the target negative feedback sample and the reason for determining the question category label.

[0012] According to another aspect of the present application, an electronic device is provided, including:

[0013] at least one processor; and

[0014] a memory communicatively connected to the at least one processor; wherein,

[0015] The memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to perform the method described in the above embodiment.

[0016] According to another aspect of the present application, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method according to the above embodiment.

[0017] According to another aspect of the present application, a computer program product is provided, including a computer program, which implements the steps of the method described in the above embodiment when executed by a processor.

[0018] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present application.

[0020] Figure 1 A schematic diagram of a flow chart of a sample classification method based on a large model provided in one embodiment of the present application;

[0021] Figure 2 A schematic flow chart of a large model-based sample classification method according to another embodiment of the present application;

[0022] Figure 3 A schematic flow chart of a large model-based sample classification method according to another embodiment of the present application;

[0023] Figure 4 A schematic flow chart of a large model-based sample classification method according to another embodiment of the present application;

[0024] Figure 5 A schematic diagram of a negative feedback sample classification process provided in an embodiment of the present application;

[0025] Figure 6 A schematic diagram of the structure of a sample classification device based on a large model provided in one embodiment of the present application;

[0026] Figure 7 It is a block diagram of an electronic device used to implement the large model-based sample classification method of an embodiment of the present application. DETAILED DESCRIPTION

[0027] The following description of exemplary embodiments of the present application is made in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0028] It should be noted that the acquisition, storage, use, and processing of data in the technical solution of this application comply with the relevant provisions of national laws and regulations and do not violate public order and good morals.

[0029] The following describes the large model-based sample classification method, device, electronic device and storage medium of the embodiments of the present application with reference to the accompanying drawings.

[0030] Figure 1 A flowchart of a large model-based sample classification method provided in one embodiment of the present application.

[0031] The large model-based sample classification method of the embodiment of the present application can be executed by the large model-based sample classification device of the embodiment of the present application, and the device can be configured in an electronic device.

[0032] Among them, the electronic device can be any device with computing capabilities, such as a personal computer, mobile terminal, server, etc. The mobile terminal can be, for example, a vehicle-mounted device, a mobile phone, a tablet computer, a personal digital assistant, a wearable device, and other hardware devices with various operating systems, touch screens and / or display screens.

[0033] like Figure 1 As shown, the sample classification method based on the large model includes:

[0034] Step 101: Obtain target negative feedback samples; wherein the target negative feedback samples include query statements and model answers corresponding to the query statements.

[0035] The model answer may be an answer to a query statement generated using the first model.

[0036] In this application, the target negative feedback sample may refer to a question-answer pair in which the answer output by the first largest model contains negative feedback.

[0037] For example, a user enters a query statement in the question-answering system, and the question-answering system uses the first model to output the answer. If the user clicks "no" on the answer, the question-answer pair consisting of the query statement and the answer can be considered a negative feedback sample.

[0038] Exemplarily, each question-answer pair and the feedback data of each question-answer pair can be obtained from the question-answer system that outputs answers using the first largest model, and the feedback type of the feedback data can be determined. Based on the question-answer pairs whose feedback type is negative feedback, candidate negative feedback samples can be determined, and then based on the candidate negative feedback samples, the target negative feedback samples can be determined.

[0039] The feedback data may refer to data used to represent the user's feedback on the answer in the question-answer pair, such as likes, dislikes, etc.

[0040] The feedback types of the feedback data may include positive feedback and negative feedback. For example, a like is positive feedback, and a dislike is negative feedback.

[0041] Exemplarily, all question-answer pairs whose feedback type is negative feedback can be taken as candidate negative feedback samples.

[0042] Exemplarily, the candidate negative feedback sample may be directly determined as the target negative feedback sample.

[0043] Exemplarily, samples that meet preset conditions among the candidate negative feedback samples may also be determined as target feedback samples.

[0044] The preset condition may be, for example, at least one of the following: the query statement belongs to a non-system question, the query statement belongs to a target domain question, etc.

[0045] Among them, non-system problems may refer to query statements that are not caused by technical or performance problems such as interface timeout, network delay, service unavailability, etc. of the question-answering system, which result in the question-answering system being unable to return answers normally.

[0046] The target domain may refer to the domain targeted by the question-answering system. For example, for a question-answering system in the transportation field, if the answer to a query statement has negative feedback, and the query statement belongs to a transportation field problem, then the question-answer pair consisting of the query statement and its answer can be used as a target negative feedback sample.

[0047] For example, if the query statement has no direct connection with core areas such as transportation, roads and travel, traffic regulations and safety, and traffic management, it can be determined that the query statement belongs to a non-transportation field problem.

[0048] For example, if the question-and-answer system cannot return answers normally due to technical or performance problems such as interface timeout, network delay, service unavailability, etc., the question-and-answer system can return information such as "tool execution error" and "interface timeout and no answer obtained". Therefore, it can be determined through keyword matching whether it is a system problem. For example, the answers in the candidate negative feedback samples can be matched with preset keywords to determine whether it is a system problem.

[0049] Exemplarily, if the answer in the candidate negative feedback sample does not match the preset keyword, and the domain to which the candidate negative feedback sample belongs matches the target domain corresponding to the question-answering system, the candidate feedback sample can be determined as the target negative feedback sample.

[0050] Exemplarily, if the candidate negative feedback sample belongs to a system problem, the category label of the candidate negative feedback sample may be determined to be a system problem.

[0051] For example, if the domain to which the candidate negative feedback sample belongs does not match the target domain corresponding to the question-answering system, it can be determined that the category label of the candidate negative feedback sample is a non-target domain question.

[0052] Therefore, the candidate negative feedback samples can be screened by whether they belong to non-system problems, whether the query statements in the candidate negative feedback samples belong to problems in the target field, etc., to obtain target negative feedback samples, thereby improving the quality of negative feedback samples used for classification.

[0053] Step 102: Obtain a reference answer corresponding to the query statement.

[0054] In this application, a retrieval agent can be used to search for answers to query statements, and the answers to the searched query statements can be used as reference answers to the query statements.

[0055] The retrieval agent is a retrieval system built on artificial intelligence technology. By simulating human search behavior, it efficiently and accurately retrieves the information users need from massive amounts of data. Therefore, using the retrieval agent to obtain reference answers ensures a reliable benchmark for comparing model answers.

[0056] Exemplarily, the query statement in the target feedback sample may be matched with the query statement in the question and answer library, and the answer corresponding to the matched query statement in the question and answer library may be used as a reference answer.

[0057] Step 103 : Classify the target negative feedback sample using the second largest model according to the model answer and the reference answer to obtain a classification result of the target negative feedback sample.

[0058] The classification result may include the problem category label of the target negative feedback sample and the reason for determining the problem category label.

[0059] Among them, the question category label can be used to indicate the category of the question in the answer of the target negative feedback sample.

[0060] Exemplarily, question category labels may include but are not limited to knowledge error, knowledge loss, partially correct, model error, slicing problem, model recall problem, query statement similarity, non-query statement similarity, query statement similarity degradation, normal answer, correct answer, etc.

[0061] For example, knowledge errors include factual errors, logical contradictions, and irrelevant answers. Factual errors refer to model answers that provide incorrect information and are inconsistent with generally accepted facts or data. Logical contradictions refer to internal inconsistencies in model answers that render them invalid. Irrelevant answers refer to answers that do not provide valid information.

[0062] For example, knowledge deficiency may refer to the question answering system failing to provide sufficient or complete information to answer the query statement.

[0063] For example, a model error may mean that the recalled document contains the correct answer, but the model answer is inconsistent with the document content.

[0064] For example, a slicing problem may refer to a situation where the recalled document contains the correct answer, but the similarity of the answers output by different models exceeds a preset threshold.

[0065] For example, the model recall problem may refer to the recalled documents being irrelevant to the query statement.

[0066] For example, query statement similarity may refer to transformed query statements, where the models all answer key questions in the query statements.

[0067] For example, non-query statement similarity may refer to transformed query statements where the model does not answer the key questions in the query statements.

[0068] Exemplarily, query statement similarity degradation means that after the query statement is changed, the model fails to answer the key questions in the query statement and the quality of the model answer after the query statement is changed is reduced relative to the model answer.

[0069] For example, a normal answer means that the model answer answers the key questions in the query statement.

[0070] For example, the query statement is "What gas station is the B service area gas station in City A?", and the model answer is "The B service area gas station in City A is C Petroleum Gas Station", and the reference answer is "The B service area gas station in City A includes D Chemical Gas Station and D Petroleum and Natural Gas Co., Ltd. A City Sales Branch B Service Area Gas Station". By using the second largest model, it is determined that the category label of the query statement is missing key information. The reason for this determination is that the model answer only mentions "C Petroleum Gas Station", but the reference answer mentions that the service area also includes D Chemical Gas Station, and the information is missing.

[0071] It should be noted that due to the subjectivity of user feedback on the answers output by the question-answering system, there may be a correct answer but a negative feedback from the user. Therefore, the target negative feedback sample's question category label may be a correct answer. If the target negative feedback sample's question category label is a correct answer, it can be added to the positive sample set.

[0072] In this application, the second largest model can be used to compare the reference answer with the reference answer used as a benchmark to classify the target feedback sample and obtain a classification result.

[0073] As an example, based on the query statement, the reference answer can be retrieved through the retrieval agent, which can ensure that there is a reliable benchmark to compare the model answer. Therefore, based on the combination of the retrieval agent and the large model, the target negative feedback samples can be classified, which can improve the classification accuracy and efficiency.

[0074] After obtaining the classification results of the target negative feedback samples, the first model can be improved or the knowledge base can be updated accordingly based on the question category labels in the classification results, so that the questions and answers of the question-answering system are more accurate.

[0075] For example, if the classification result of the target negative feedback sample is knowledge missing or knowledge error, the knowledge base can be adjusted in time to reduce redundant and erroneous information and make the answer more accurate; for model recall problems and model answer problems, find out the problems with the model answer and provide data support for subsequent model updates.

[0076] For example, the target feedback samples and their classification results can be added to the negative sample evaluation set to improve the completeness of the evaluation set. After completing the knowledge base update or model repair, the samples in the negative sample evaluation set can be used for regression testing to evaluate the repair effect of the negative feedback samples.

[0077] In the embodiment of the present application, by obtaining the reference answer corresponding to the query statement, and using the second largest model to classify the target negative feedback sample based on the model answer and the reference answer, a classification result is obtained. Thus, by using the reference answer as a reference for the model answer and using the large model to classify the target negative feedback sample, not only can the accuracy and efficiency of classification be improved, but the classification result obtained not only includes the question category label but also the reason for the judgment. Therefore, the classification result based on the negative feedback sample can accurately guide model improvement or update the knowledge base.

[0078] Figure 2 A flowchart of a large model-based sample classification method provided in another embodiment of the present application.

[0079] like Figure 2 As shown, the sample classification method based on the large model includes:

[0080] Step 201: Obtain target negative feedback samples; wherein the target negative feedback samples include query statements and model answers corresponding to the query statements.

[0081] Step 202: Obtain a reference answer corresponding to the query statement.

[0082] In the present application, steps 201 to 202 can be implemented in any of the embodiments of the present application, and therefore will not be described in detail here.

[0083] Exemplarily, the second largest model can be used to judge whether the query statement is ambiguous to obtain an ambiguity judgment result. If the ambiguity category label in the ambiguity judgment result is no ambiguity, that is, the query statement is not ambiguous, a reference answer is obtained based on the query statement and the retrieval agent.

[0084] The ambiguity judgment result may include the ambiguous category label, the reason for the determination of the ambiguous category label, etc. For example, the ambiguous category label may be ambiguous or non-ambiguous.

[0085] For example, the second largest model can be used to perform ambiguity analysis based on the query to obtain an ambiguity analysis result. If the ambiguity analysis result determines that the query is ambiguous, the second largest model can be used to perform ambiguity analysis based on the query and its context to obtain an ambiguity analysis result. If the ambiguity analysis result determines that the query is unambiguous, a reference answer can be obtained based on the query and the search agent. Alternatively, the second largest model can be used to perform ambiguity analysis directly based on the query and its context to obtain an ambiguity analysis result, and then the ambiguity analysis result can be used to determine whether the query is ambiguous.

[0086] For example, if the ambiguous category label in the ambiguity judgment result indicates ambiguity, that is, the query statement is ambiguous, the ambiguity judgment result can be used as the classification result. In other words, the ambiguous category label in the ambiguity judgment result is used as the problem category label of the target negative feedback sample, and the reason for determining the ambiguous category label is the reason for determining the problem category label.

[0087] Therefore, before using the retrieval agent to obtain the reference answer, the large model can be used to judge whether the query statement is ambiguous. If it is determined that the query statement is not ambiguous, the retrieval agent can be used to obtain the reference answer, thereby improving the accuracy of the reference answer.

[0088] Step 203: Use the second largest model to perform consistency evaluation on the query statement, the model answer, and the reference answer to obtain a first evaluation result.

[0089] The second largest model may be the same as or different from the first largest model, and there is no limitation on this.

[0090] Among them, the first evaluation result may include the answer consistency label corresponding to the target negative feedback sample, the reason for determining the answer consistency label, etc.

[0091] In this application, prompt information can be generated based on the query statement, model answer, and reference answer, and the prompt information can be processed using the second largest model to obtain the first evaluation result. The prompt information is used to instruct the second largest model to perform consistency evaluation on the query statement, model answer, and reference answer.

[0092] Exemplarily, a prompt template may be obtained, and the prompt model may be filled in according to the query statement, the model answer, and the reference answer to obtain the prompt information.

[0093] Among them, the prompt template may include answer consistency judgment rules, which are rules for evaluating answer consistency. The answer consistency judgment rules may include answer consistency labels, explanatory information of answer consistency labels, etc.

[0094] Exemplarily, answer consistency labels may include but are not limited to correct answer, partially correct (information enhancement), partially correct (ambiguous expression), missing key information, knowledge error (factual error), unable to judge (information source limitation), unable to judge (content inconsistency), unable to judge (time and space difference), etc.

[0095] The content in brackets in the above answer consistency labels is the specific type of answer consistency label or the specific reason for the answer consistency label. For example, "Partially Correct (Information Enhancement)" means that the answer is partially correct and belongs to the information enhancement type, and "Partially Correct (Ambiguous Expression)" means that the answer is partially correct and belongs to the ambiguous expression type.

[0096] For example, the explanation information for a correct answer may be that the model answer and the reference answer are semantically completely consistent, and even if the expressions are different, there is no missing or incorrect information.

[0097] Exemplarily, partially correct (information enhanced) explanation information may be that the model answer supplements the reference answer with correct and relevant information, making the answer more complete, and the supplemented content will not cause ambiguity.

[0098] For example, a partially correct (ambiguous) explanation might indicate that the model answer and the reference answer have similar meanings, but are expressed in a way that could easily lead to misunderstanding. For example, the model answer might be "There are no traffic restrictions in City F today," while the reference answer might be "Out-of-town vehicles in City F have no license plate restrictions, but they still need a permit to enter the city." Therefore, the model answer and the reference answer are ambiguous.

[0099] Illustratively, the explanation for missing key information may be that the model answer only partially answers the content of the reference answer, omitting key information, making it impossible for the user to obtain a complete answer. Even if the content of the model answer is correct, if key supplementary information is missing, it is also considered as missing key information, or the model answer does not fully answer the core question of the query statement itself. If the query statement has been fully answered, and the reference answer only provides additional information such as background knowledge, legal basis, etc., the model answer is not judged to be missing key information.

[0100] For example, knowledge errors (factual errors) may be errors in factual information provided by the model answer, which can be compared with the reference answer or common sense for confirmation.

[0101] For example, the explanation message for "cannot determine (information source limitation)" can be: if the reference answer clearly indicates "no information" or "unable to obtain data", and the model answer provides information that may be correct, then it cannot be determined whether the reference answer is correct and can be marked as "cannot determine (information source limitation)".

[0102] For example, the explanation information that cannot be judged (content inconsistency) may be that the model answer and the reference answer provide completely different contents, and there is insufficient basis to judge which one is correct.

[0103] An example of an unidentifiable explanation (spatiotemporal discrepancy) might be that both the model answer and the reference answer are correct, but they describe different time or location ranges, making direct comparison impossible. For example, the model answer provides the license plate number with restrictions in City F, while the reference answer provides the license plate number with restrictions in City G, making comparison difficult.

[0104] Therefore, prompt information is generated based on the prompt template containing the answer consistency judgment rules, so that the large model can determine the question category label and judgment reason of the target negative feedback sample based on the consistency judgment rules, thereby improving the accuracy of the negative feedback sample classification results.

[0105] Exemplarily, the prompt template may also include reference examples of answer consistency labels, answer consistency evaluation rules, model output requirements, etc.

[0106] For example, the answer consistency evaluation rules may include: 1. During the evaluation, the judgment is strictly based on the query statement. As long as the model answer completely and correctly answers the query statement, it can be judged as "correct"; 2. The model answer may provide additional information, but if this information exceeds the requirements of the query statement and the model answer does not mention it, it should not be judged as "missing key information"; 3. When the model answer omits key information of the query statement itself, it is judged as "missing key information."

[0107] It should be noted that the above-mentioned answer consistency judgment rules, answer consistency evaluation rules, etc. can be set according to actual needs and are not limited to this.

[0108] In order to further improve the accuracy of the answer consistency evaluation results, for example, prompt information is generated based on the query statement, model answer, reference answer and reference examples of the answer consistency label, so that the second largest model can combine the reference examples of the answer consistency label to determine the question category label and judgment reason of the target negative feedback sample, thereby further improving the accuracy of the classification results.

[0109] Step 204: Obtain a classification result based on the first evaluation result.

[0110] In this application, a classification result can be obtained based on the type of the answer consistency label in the first evaluation result, wherein the type of the answer consistency label can include an intermediate state label and a final state label.

[0111] For example, unable to judge (information source limitation), unable to judge (content inconsistency), unable to judge (time and space differences) are intermediate state labels, and correct answer, partially correct (information enhancement), partially correct (ambiguous expression), missing key information, knowledge error (factual error) are final state labels.

[0112] For example, if the answer consistency label corresponding to the target negative feedback sample belongs to the final state label, the first evaluation result can be directly used as the classification result, that is, the answer consistency label can be used as the question category label, and the determination reason of the answer consistency label can be used as the determination reason of the question category label; if the answer consistency label corresponding to the target negative feedback sample belongs to the intermediate state label, then the classification result can be further determined based on the first evaluation result.

[0113] Exemplarily, if the answer consistency label in the first evaluation result is the target label, the first large model can be used to obtain the recalled documents based on the query statement, and the classification result can be determined based on the query statement and the recalled documents.

[0114] Among them, the target label can be an intermediate state label, such as unable to judge (information source limitation), unable to judge (content inconsistency), unable to judge (time and space difference), etc.

[0115] For example, the classification result may be determined based on the relationship between the recalled document and the query statement.

[0116] Therefore, when the answer consistency label corresponding to the negative feedback sample is the target label, the recalled document can be obtained based on the query statement, and the classification result can be further obtained based on the query statement combined with the recalled document, thereby improving the accuracy of the classification result.

[0117] Exemplarily, the second largest model may be used to determine the relevance between the query statement and the recalled documents to obtain a relevance determination result, and the classification result may be determined based on the relevance determination result.

[0118] The relevance determination result may include a relevance category label, a determination reason for the relevance category label, etc. For example, the relevance category label may be relevant or irrelevant.

[0119] As an example, a prompt template for relevance judgment can be filled in according to the query statement and the recalled document to obtain prompt information for relevance judgment, and the prompt information can be processed using the second largest model to obtain a relevance judgment result.

[0120] The prompt template for correlation judgment may include correlation judgment rules, and the correlation judgment rules may include various correlation category labels, explanation information of various correlation category labels, and the like.

[0121] Therefore, by adopting the second largest model to obtain the correlation judgment result between the query statement and the recalled document, the accuracy of the correlation judgment result can be improved. Based on the correlation judgment result, the classification result is determined, which can improve the accuracy of the classification result.

[0122] For example, if the relevance category label is irrelevant, that is, the query statement is irrelevant to the recalled document, it can be determined that the question category label of the target negative feedback sample is a model recall question.

[0123] For example, if the relevance category label is related, that is, the query statement is related to the recalled document, the second largest model can be used to judge whether the recalled document completely covers the query statement to obtain a coverage judgment result, and then obtain a classification result based on the coverage judgment result.

[0124] The coverage determination result may include a coverage category label, a determination reason for the coverage category label, etc. For example, the coverage category label may be complete coverage or incomplete coverage, etc.

[0125] As an example, the prompt template for coverage judgment can be filled in according to the query statement and the recalled document to obtain prompt information for coverage judgment, and the prompt information can be processed using the second largest model to obtain the coverage judgment result.

[0126] The prompt template for coverage judgment may include coverage judgment rules, and the coverage judgment rules may include various coverage category labels, explanation information of various coverage category labels, and the like.

[0127] Therefore, when the query statement is related to the recalled document, the accuracy of the coverage judgment result can be improved by adopting the second largest model to obtain the coverage judgment result between the query statement and the recalled document. Based on the coverage judgment result, the classification result can be determined, which can improve the accuracy of the classification result.

[0128] For example, if the coverage category label is incomplete coverage, that is, the recalled document does not completely cover the query statement, it can be determined that the problem category label of the target negative feedback sample is knowledge loss.

[0129] For example, if the coverage category label is complete coverage, that is, the recalled document completely covers the query statement, the second largest model can be used to determine whether the model answer matches the recalled document to obtain a matching judgment result, and determine the classification result based on the matching judgment result.

[0130] The matching judgment result may include a matching category label, a reason for the matching category label, etc. For example, the matching category label may be a match or a mismatch.

[0131] As an example, the prompt template for matching judgment can be filled in according to the query statement and the recalled document to obtain prompt information for matching judgment, and the prompt information can be processed using the second largest model to obtain the matching judgment result.

[0132] The prompt template for matching judgment may include matching judgment rules, and the matching judgment rules may include various matching category labels, explanation information of various matching category labels, and the like.

[0133] Therefore, when the recalled document completely covers the query statement, the accuracy of the matching judgment result can be improved by adopting the second largest model to obtain the matching judgment result between the query statement and the recalled document. Based on the matching judgment result, the classification result can be determined, which can improve the accuracy of the classification result.

[0134] For example, if the matching category label is a match, that is, the model answer matches the recalled document, it can be determined that the question category label of the target negative feedback sample is a correct answer.

[0135] For example, if the matching category label is mismatched, that is, the model answer does not match the recalled document, the second largest model can be used to perform multi-model testing on the query statement to obtain the model test judgment result, and the classification result is determined based on the model test judgment result.

[0136] The model test result may include a model test category label, a reason for determining the model test category label, etc. For example, the model test category label may be a test or no test, etc.

[0137] As an example, the prompt template for model test judgment can be filled in according to the query statement and the recalled document to obtain prompt information for model test judgment, and the prompt information can be processed using the second largest model to obtain the model test judgment result.

[0138] The prompt template for model test judgment may include model test judgment rules, and the model test judgment rules may include each model test category label, explanation information of each model test category label, etc.

[0139] For example, the model test judgment rule may be that the recalled document contains the correct answer, and the model answer is inconsistent with the content of the recalled document. Therefore, if the model answer and the recalled document meet the test judgment rule, the model test judgment label is determined to not perform multi-model testing. If the model answer and the recalled document do not meet the test judgment rule, the model test judgment label is determined to perform multi-model testing.

[0140] Exemplarily, if the model test judgment label is not to perform multi-model testing, it can be determined that the problem category label of the target negative feedback sample is a model error.

[0141] For example, if the model test judgment label is to perform multi-model testing, the third largest model can be used to obtain the test answer of the query statement, and the second largest model can be used to judge the similarity between the model answer and the test answer to obtain the similarity judgment result, and then the classification result is obtained based on the similarity judgment result.

[0142] The similarity determination result may include a similarity test category label, a determination reason for the similarity category label, etc. For example, the similarity category label may be similar or dissimilar.

[0143] As an example, if the similarity category label is similar, that is, the model answer is similar to the test answer, it can be determined that the question category label of the target negative feedback sample is a slicing question; if the similarity category label is dissimilar, that is, the model answer is dissimilar to the test answer, it can be determined that the question category label of the target negative feedback sample is a model error.

[0144] Therefore, when the model answer does not match the recalled document, the accuracy of the model test judgment result can be improved by using the second largest model to obtain the judgment result of whether to perform multi-model testing on the query statement. Based on the model test judgment result, the classification result can be determined, which can improve the accuracy of the classification result.

[0145] In an embodiment of the present application, by utilizing the second largest model to perform consistency evaluation on the query statement, model answer and reference answer, the accuracy of the answer consistency evaluation results can be improved, and then based on the answer consistency evaluation results, the classification results of the negative feedback samples can be obtained, which can improve the accuracy of the negative feedback sample classification.

[0146] Figure 3 A flowchart of a large model-based sample classification method provided in another embodiment of the present application.

[0147] like Figure 3 As shown, the sample classification method based on the large model includes:

[0148] Step 301: Obtain target negative feedback samples; wherein the target negative feedback samples include query statements and model answers corresponding to the query statements.

[0149] Step 302: Obtain a reference answer corresponding to the query statement.

[0150] Step 303: Use the second largest model to perform consistency evaluation on the query statement, the model answer, and the reference answer to obtain a first evaluation result.

[0151] In this application, steps 301 to 303 can be implemented in any of the embodiments of this application, so they will not be described in detail here.

[0152] Step 304: rewrite the query statement to obtain a rewritten query statement.

[0153] In this application, the second largest model can be used to rewrite the query statement to obtain a rewritten query statement.

[0154] For example, a prompt template for rewriting the query statement can be filled in based on the query statement to obtain prompt information for rewriting the query statement, and the prompt information can be processed using the second largest model to obtain a rewritten query statement. The prompt information for rewriting the query statement can instruct the second largest model to rewrite the query statement.

[0155] Step 305: Use the first large model to generate an answer to the rewritten query statement.

[0156] In this application, the first model used in the question-answering system can be used to generate answers to rewritten query statements.

[0157] Step 306: Obtain a classification result based on the first evaluation result and the answer to the rewritten query statement.

[0158] In this application, the question category label related to the model answer can be obtained based on the first evaluation result, and the question category label related to the query statement can be obtained based on the answer to the rewritten query statement. Based on these two types of question category labels, the question category label of the target negative feedback sample can be obtained.

[0159] For example, it is possible to determine whether the model answer answers the key question of the query statement. If the model answer answers the key question of the query statement, the quality of the model answer is compared with that of the answer to the rewritten query statement, and the classification result is obtained based on the quality comparison result and the first evaluation result.

[0160] Among them, the key question can be understood as the core question of the query statement.

[0161] For example, if the quality comparison result shows that the answer to the rewritten query statement is of lower quality than the model answer, it can be determined that the question category label related to the query statement is query statement similarity degradation.

[0162] For example, if the model answer answers the key question of the query statement, the question category label associated with the model answer can also be determined as a normal answer.

[0163] Therefore, when the model answer answers the key questions of the query statement, combined with the quality comparison results between the model answer and the answer to the rewritten query statement, we can obtain problem category labels of multiple different dimensions of the negative feedback sample, so as to accurately locate the root cause of the problem and improve the labeling quality of the negative feedback sample.

[0164] For example, if the model answer does not answer the key question of the query statement, it can be determined whether the answer to the rewritten query statement answers the key question of the query statement, and a classification result is obtained based on the first evaluation result and the determination result.

[0165] For example, if the answer to the rewritten query statement answers the key question of the query statement, but the model answer does not answer the key question of the query statement, it means that the query statement is transformed and the model answer is consistent. The question category label related to the query statement can be determined as query statement similarity; if the answer to the rewritten query statement does not answer the key question of the query statement, the question category label related to the query statement can be determined as non-query statement similarity.

[0166] Therefore, when the model answer is to answer the key question of the query statement, the similarity of the query statement can be judged based on whether the answer to the rewritten query statement answers the key question, and question category labels of multiple different dimensions of the negative feedback sample can be obtained, so as to accurately locate the root cause of the problem and improve the labeling quality of the negative feedback sample.

[0167] In an embodiment of the present application, the query statement in the target negative feedback sample is rewritten to obtain a rewritten query statement, and the first large model is used to obtain the answer to the rewritten query statement. According to the answer to the rewritten query statement, the question category label related to the query statement is determined, and combined with the answer consistency evaluation result, multiple question category labels of different dimensions can be obtained, thereby improving the labeling accuracy of the negative feedback sample.

[0168] Figure 4 A flowchart of a large model-based sample classification method provided in another embodiment of the present application.

[0169] like Figure 4 As shown, the sample classification method based on the large model includes:

[0170] Step 401: Obtain target negative feedback samples; wherein the target negative feedback samples include query statements and model answers corresponding to the query statements.

[0171] Step 402: Obtain a reference answer corresponding to the query statement.

[0172] In the present application, steps 401 and 402 can be implemented in any of the embodiments of the present application, and therefore will not be described in detail here.

[0173] Step 403: Use the second largest model to perform consistency evaluation on the query statement and the model answer to obtain a second evaluation result.

[0174] Among them, the second evaluation result may include the question-answer consistency label corresponding to the model answer, the reason for determining the question-answer consistency label, etc.

[0175] Exemplarily, question-answer consistency labels may include but are not limited to complete relevance, partial relevance, missing key information, knowledge error (irrelevant error), knowledge error (logical contradiction), knowledge error (factual error), etc.

[0176] Exemplary, fully relevant explanation information may include that the model answer completely and accurately answers the query statement, the information is clear, not misleading or ambiguous, or the model answer does not lack any information and is consistent with generally accepted facts or official data.

[0177] Exemplarily, partially relevant explanatory information may include that the model answer basically conforms to the query statement, but may lack supplementary information, but the core information has been conveyed, or the expression is not precise enough, or may cause slight misunderstanding, but the overall answer is still relevant.

[0178] Exemplarily, the explanation information for missing key information may include that the model answer involves the query statement topic but does not provide the core content, resulting in the user being unable to obtain complete information, or the model answer cannot be judged as an irrelevant error because the explanation is "unable to provide". If it still involves the query statement topic, it should be classified as missing key information.

[0179] For example, the query statement is "Add the regional divisions of each brigade and squadron in map and text form", and the model answer is "I'm sorry, I can't directly provide a map of the regional divisions and a detailed text description of each brigade and squadron." Analysis: The model answer explains that it cannot be provided, but it still involves the topic of regional division. Therefore, it is a case of missing key information rather than an irrelevant error.

[0180] Exemplarily, the explanation information of knowledge errors (irrelevant errors) may include that the model answer does not answer the query statement at all, even if it contains keywords related to the query statement, or the information provided by the model answer is completely different from the topic and cannot help the user answer the question.

[0181] For example, the query statement is "Is Gaoxin Road a strictly controlled section of road?", and the model's answer is "F City's traffic restriction policy..." Analysis: The query statement asks "Is Gaoxin Road a strictly controlled section of road?", but the model's answer is "traffic restriction policy." The two topics are unrelated, so this is an irrelevant error among knowledge errors.

[0182] For example, the query is "What should be done when transporting hazardous chemicals without a dangerous goods transport certificate?" The model answer is "Your question may be beyond my business scope. I suggest consulting the relevant department." Analysis: The model answer does not answer the topic of the query, but only suggests consulting the relevant department. It does not provide substantive information and is an irrelevant error among knowledge errors.

[0183] For example, the query statement is "Learn the law to reduce points", and the model answer is "Thank you very much for your question. We focus on knowledge and services in the field of transportation." Analysis: The model answer is only a general answer and does not provide any valid information.

[0184] For example, the explanation information of knowledge error (logical contradiction) may include internal inconsistency of the model answer, resulting in invalid answer.

[0185] For example, explanation information of knowledge errors (factual errors) may provide erroneous information to the model answer, which is inconsistent with generally accepted facts or official data.

[0186] Exemplarily, the prompt template for question-answer consistency evaluation can be filled in according to the query statement and the model answer to obtain prompt information for question-answer consistency evaluation, and the prompt information can be processed using the second largest model to obtain a second evaluation result.

[0187] Among them, the prompt template used for question and answer consistency evaluation may include question and answer consistency judgment rules, and the question and answer consistency judgment rules include each question and answer consistency label, explanatory information of each question and answer consistency label, etc.

[0188] Optionally, the prompt template for question-answer consistency assessment may further include reference examples of question-answer consistency labels, model output requirements, etc. The model output requirements may include output content requirements, output format requirements, etc.

[0189] Step 404: Use the second largest model to determine the consistency between the query statement and the reference answer to obtain a third evaluation result.

[0190] Among them, the third evaluation result may include the question and answer consistency label corresponding to the reference answer, the reason for determining the question and answer consistency label, etc.

[0191] Among them, the explanatory information of each question-answer consistency label corresponding to the reference answer is similar to the explanatory information of each question-answer consistency label corresponding to the above-mentioned model answer, so it will not be repeated here.

[0192] In this application, the method of using the second largest model to judge the consistency between the query statement and the reference answer is similar to the above-mentioned method of using the second largest model to judge the consistency between the query statement and the model answer, so it will not be repeated here.

[0193] Step 405, in response to determining that the model answer is valid according to the second evaluation result and determining that the reference answer is valid according to the third evaluation result, use the second largest model to perform consistency evaluation on the query statement, the model answer and the reference answer to obtain the first evaluation result.

[0194] For example, if the question-answer consistency label corresponding to the model answer is completely relevant or partially relevant, the model answer can be determined to be valid. If the question-answer consistency label corresponding to the model answer is any of the following: missing key information, knowledge error (irrelevant error), knowledge error (logical contradiction), knowledge error (factual error), etc., the model answer can be determined to be invalid.

[0195] It can be understood that the method of determining whether the reference answer is valid based on the third evaluation result is similar to the method of determining whether the model answer is valid, so it will not be repeated here.

[0196] In this application, it is possible to determine whether the model answer is valid and whether the reference answer is valid. If both the model answer and the reference answer are valid, the second largest model is used to perform consistency evaluation on the query statement, the model answer and the reference answer to obtain the first evaluation result.

[0197] In an embodiment of the present application, by adopting the second largest model, the query statement and the model answer, and the query statement and the reference answer are evaluated for consistency, and whether the model answer and the reference answer are valid is determined based on the corresponding evaluation results, thereby improving the accuracy of the question-answer consistency evaluation, thereby improving the accuracy of judging whether the answer is valid, and then, when it is determined that the model answer and the reference answer are both valid, the second largest model is used to perform a consistency evaluation on the query statement, the model answer and the reference answer, which can improve the efficiency and accuracy of the answer consistency evaluation.

[0198] In order to facilitate the understanding of the solution of this application, the following is an example of a question-answering system in the field of transportation. Figure 5 To explain, Figure 5 A schematic diagram of a negative feedback sample classification process provided in an embodiment of the present application.

[0199] like Figure 5 As shown, negative feedback samples can be preprocessed to obtain processed negative feedback samples. This preprocessing includes format verification, deduplication, and invalid data removal, as well as filtering out negative feedback samples that are not related to system issues. Next, determine whether the processed negative feedback samples belong to the transportation field. For negative feedback samples in the transportation field, determine whether the query statement in the negative feedback sample is ambiguous.

[0200] For ambiguous queries, clarification can be performed based on the query context. If the clarification results in no ambiguity, the query is rewritten, and the rewritten query and the unambiguous query are classified using a retrieval agent and a large model. For example, a retrieval agent can be used to obtain reference answers, and the large model can be used to determine whether the model answers and reference answers in negative feedback samples are valid. If the model answers and reference answers are valid, the large model can be used to evaluate the consistency of the model answers and reference answers. For details, please refer to the explanation of the above embodiment.

[0201] The question category labels of the negative feedback samples output by the large model can include any of the following: knowledge error, correct answer, missing knowledge, and partially correct. For example, knowledge errors can include irrelevant errors, logical contradictions, factual errors, etc.

[0202] If the question category label of the negative feedback sample includes "unable to determine", then determine whether the recalled document is relevant to the query statement. If not, it can be determined that the question category label of the negative feedback sample is a model recall problem. If relevant, then determine whether the recalled document completely covers the query statement.

[0203] If the recalled document does not fully cover the query statement, the question category label of the negative feedback sample is determined to be knowledge missing. If the recalled document fully covers the query statement, it is determined whether the model answer matches the recalled document.

[0204] If the model answer matches the recalled document, determine the question category label of the negative feedback sample including the correct answer. If the model answer does not match the recalled document, determine whether to perform multi-model testing.

[0205] If multi-model testing is not performed, the question category labels for the negative feedback samples are determined to include model errors. If multi-model testing is performed, the two models' responses are determined to be similar. If they are not similar, the question category labels for the negative feedback samples are determined to include model errors. If they are similar, the category labels for the negative feedback samples are determined to include slicing issues.

[0206] Furthermore, for unambiguous queries, the model's answer is determined to be correct, specifically, whether it addresses the key question in the query. If the model's answer is correct, the negative feedback sample's question category label is determined to include a normal answer. If the model's answer is incorrect, the query is transformed to obtain a rewritten query, and the large model used by the question-answering system is then used to obtain the answer to the rewritten query.

[0207] Determine whether the answer to the rewritten query statement is correct. If correct, determine that the question category label of the negative feedback sample includes query statement similarity. If uncertain, determine that the question category label of the negative feedback sample includes non-query statement similarity.

[0208] If the model answer in the negative feedback sample is correct, the quality of the two answers can be compared. If the quality of the answer to the rewritten query statement is lower than that of the model answer, that is, the quality of the second answer is lower than that of the first answer, it can be determined that the question category label of the negative feedback sample includes the query statement similarity degradation.

[0209] In order to implement the above embodiment, the embodiment of the present application also proposes a sample classification device based on a large model. Figure 6 A schematic structural diagram of a large model-based sample classification device provided in one embodiment of the present application.

[0210] like Figure 6 As shown, the large model-based sample classification device 600 includes:

[0211] A first acquisition module 610 is configured to acquire a target negative feedback sample, wherein the target negative feedback sample includes a query statement and a model answer corresponding to the query statement, wherein the model answer is an answer to the query statement generated using the first large model;

[0212] A second acquisition module 620 is used to obtain a reference answer corresponding to the query statement;

[0213] The classification module 630 is used to classify the target negative feedback sample using the second largest model according to the model answer and the reference answer to obtain the classification result of the target negative feedback sample; wherein the classification result includes the question category label of the target negative feedback sample and the reason for determining the question category label.

[0214] Optionally, the classification module 630 is configured to:

[0215] Using the second largest model, performing a consistency evaluation on the query statement, the model answer, and the reference answer to obtain a first evaluation result;

[0216] The classification result is obtained according to the first evaluation result.

[0217] Optionally, the classification module 630 is configured to:

[0218] Obtaining a prompt template; wherein the prompt template includes an answer consistency judgment rule, and the answer consistency judgment rule includes an answer consistency label and explanation information of the answer consistency label;

[0219] Generate prompt information according to the query statement, the model answer, the reference answer and the prompt template;

[0220] The prompt information is processed using the second largest model to obtain the first evaluation result.

[0221] Optionally, the classification module 630 is configured to:

[0222] In response to the answer consistency label in the first evaluation result being a target label, obtaining recalled documents according to the query statement using the first large model;

[0223] The classification result is determined according to the query statement and the recalled document.

[0224] Optionally, the classification module 630 is configured to:

[0225] Using the second largest model, determining the relevance between the query statement and the recalled document to obtain a relevance determination result;

[0226] The classification result is determined according to the correlation judgment result.

[0227] Optionally, the relevance determination result includes a relevance category label, and the classification module 630 is configured to:

[0228] In response to the relevance category label being relevant, using the second largest model to determine whether the recalled document completely covers the query statement to obtain a coverage determination result;

[0229] The classification result is determined according to the coverage judgment result.

[0230] Optionally, the coverage judgment result includes a coverage category label, and the classification module 630 is configured to:

[0231] In response to the coverage category label being complete coverage, using the second largest model to determine whether the model answer matches the recalled document to obtain a matching determination result;

[0232] The classification result is determined according to the matching judgment result.

[0233] Optionally, the matching judgment result includes a matching category label, and the classification module 630 is configured to:

[0234] In response to the matching category label being mismatched, using the second largest model to perform a multi-model test on the query statement to obtain a model test judgment result;

[0235] The classification result is determined according to the model test judgment result.

[0236] Optionally, the classification module 630 is configured to:

[0237] Rewriting the query statement to obtain a rewritten query statement;

[0238] Using the first model, generate an answer to the rewritten query statement;

[0239] The classification result is obtained according to the first evaluation result and the answer to the rewritten query statement.

[0240] Optionally, the classification module 630 is configured to perform any of the following:

[0241] In response to the model answer answering the key question of the query, obtaining the classification result based on a quality comparison result between the model answer and the answer to the rewritten query and the first evaluation result;

[0242] In response to the model answer not answering the key question of the query statement, the classification result is obtained according to the first evaluation result and whether the answer to the rewritten query statement answers the key question.

[0243] Optionally, the classification module 630 is configured to:

[0244] Using the second largest model, performing a consistency evaluation on the query statement and the model answer to obtain a second evaluation result;

[0245] Using the second model, determining the consistency between the query statement and the reference answer to obtain a third evaluation result;

[0246] In response to determining that the model answer is valid based on the second evaluation result and determining that the reference answer is valid based on the third evaluation result, the second large model is used to perform a consistency evaluation on the query statement, the model answer and the reference answer to obtain the first evaluation result.

[0247] Optionally, the second obtaining module 620 is configured to:

[0248] Using the second model, determining whether the query statement is ambiguous to obtain an ambiguity determination result;

[0249] In response to the ambiguity category label in the ambiguity judgment result being that there is no ambiguity, the reference answer is obtained according to the query statement based on the retrieval agent.

[0250] It should be noted that the explanation of the aforementioned embodiment of the sample classification method based on a large model is also applicable to the sample classification device based on a large model in this embodiment, so it will not be repeated here.

[0251] In the embodiment of the present application, by obtaining the reference answer corresponding to the query statement, and using the second largest model to classify the target negative feedback sample based on the model answer and the reference answer, a classification result is obtained. Thus, by using the reference answer as a reference for the model answer and using the large model to classify the target negative feedback sample, not only can the accuracy and efficiency of classification be improved, but the classification result obtained not only includes the question category label but also the reason for the judgment. Therefore, the classification result based on the negative feedback sample can accurately guide model improvement or update the knowledge base.

[0252] According to an embodiment of the present application, the present application also provides an electronic device, a readable storage medium and a computer program product.

[0253] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement an embodiment of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or claimed herein.

[0254] like Figure 7 As shown, the device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a ROM (Read-Only Memory) 702 or a computer program loaded from a storage unit 708 into a RAM (Random Access Memory) 703. Various programs and data required for the operation of the device 700 can also be stored in the RAM 703. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An I / O (Input / Output) interface 705 is also connected to the bus 704.

[0255] Various components in device 700 are connected to I / O interface 705, including an input unit 706, such as a keyboard, mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a magnetic disk, optical disk, etc.; and a communication unit 709, such as a network card, modem, wireless communication transceiver, etc. The communication unit 709 allows device 700 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0256] The computing unit 701 can be various general-purpose and / or specialized processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a CPU (Central Processing Unit), a GPU (Graphic Processing Unit), various specialized AI (Artificial Intelligence) computing chips, various computing units that run machine learning model algorithms, a DSP (Digital Signal Processor), and any appropriate processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as the large model-based sample classification method. For example, in some embodiments, the large model-based sample classification method can be implemented as a computer software program that is tangibly embodied in a machine-readable medium, such as the storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the large model-based sample classification method described above can be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to execute a large model-based sample classification method in any other appropriate manner (eg, by means of firmware).

[0257] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, FPGAs (Field Programmable Gate Arrays), ASICs (Application-Specific Integrated Circuits), ASSPs (Application-Specific Standard Products), SOCs (System on Chips), CPLDs (Complex Programmable Logic Devices), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special-purpose or general-purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0258] The program code for implementing the methods of the present application can be written in any combination of one or more programming languages. Such program code can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the program code is executed by the processor or controller, the functions / operations specified in the flow charts and / or block diagrams are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0259] In the context of the present application, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or apparatus. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include an electrical connection based on one or more wires, a portable computer disk, a hard disk, RAM, ROM, EPROM (Electrically Programmable Read-Only-Memory) or flash memory, optical fiber, CD-ROM (Compact Disc Read-Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0260] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (Cathode-Ray Tube) or LCD (Liquid Crystal Display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0261] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: LAN (Local Area Network), WAN (Wide Area Network), the Internet, and blockchain networks.

[0262] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact via a communication network. The client-server relationship is established by computer programs running on the respective computers and establishing a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, a host product within the cloud computing service ecosystem that addresses the management difficulties and poor scalability of traditional physical hosts and VPS services. The server may also be a server in a distributed system or a server integrated with blockchain.

[0263] According to an embodiment of the present application, the present application also provides a computer program product, which, when an instruction processor in the computer program product executes, executes the large model-based sample classification method proposed in the above embodiment of the present application.

[0264] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this application can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this application can be achieved. This is not a limitation herein.

[0265] The above specific embodiments do not constitute a limitation on the scope of protection of this application. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application shall be included within the scope of protection of this application.

Claims

1. A sample classification method based on a large model, comprising: Obtaining a target negative feedback sample; wherein the target negative feedback sample includes a query statement and a model answer corresponding to the query statement, wherein the model answer is an answer to the query statement generated using the first model; Obtaining a reference answer corresponding to the query statement; According to the model answer and the reference answer, the target negative feedback sample is classified using the second largest model to obtain a classification result of the target negative feedback sample; wherein the classification result includes the question category label of the target negative feedback sample and the reason for determining the question category label.

2. The method according to claim 1, wherein The method of classifying the target negative feedback sample using the second largest model according to the model answer and the reference answer to obtain a classification result of the target negative feedback sample includes: Using the second largest model, performing a consistency evaluation on the query statement, the model answer, and the reference answer to obtain a first evaluation result; The classification result is obtained according to the first evaluation result.

3. The method according to claim 2, wherein: The using the second largest model to perform consistency evaluation on the query statement, the model answer, and the reference answer to obtain a first evaluation result includes: Obtaining a prompt template; wherein the prompt template includes an answer consistency judgment rule, and the answer consistency judgment rule includes an answer consistency label and explanation information of the answer consistency label; Generate prompt information according to the query statement, the model answer, the reference answer and the prompt template; The prompt information is processed using the second largest model to obtain the first evaluation result.

4. The method according to claim 3, wherein: The obtaining the classification result according to the first evaluation result includes: In response to the answer consistency label in the first evaluation result being a target label, obtaining recalled documents according to the query statement using the first large model; The classification result is determined according to the query statement and the recalled document.

5. The method according to claim 4, wherein: Determining the classification result according to the query statement and the recalled document includes: Using the second largest model, determining the relevance between the query statement and the recalled document to obtain a relevance determination result; The classification result is determined according to the correlation judgment result.

6. The method according to claim 5, wherein: The relevance judgment result includes a relevance category label, and determining the classification result based on the relevance judgment result includes: In response to the relevance category label being relevant, using the second largest model to determine whether the recalled document completely covers the query statement to obtain a coverage determination result; The classification result is determined according to the coverage judgment result.

7. The method according to claim 6, wherein: The coverage judgment result includes a coverage category label, and determining the classification result according to the coverage judgment result includes: In response to the coverage category label being complete coverage, using the second largest model to determine whether the model answer matches the recalled document to obtain a matching determination result; The classification result is determined according to the matching judgment result.

8. The method of claim 7, wherein: The matching judgment result includes a matching category label, and determining the classification result based on the matching judgment result includes: In response to the matching category label being mismatched, using the second largest model to perform a multi-model test on the query statement to obtain a model test judgment result; The classification result is determined according to the model test judgment result.

9. The method of claim 2, wherein: The obtaining the classification result according to the first evaluation result includes: Rewriting the query statement to obtain a rewritten query statement; Using the first model, generate an answer to the rewritten query statement; The classification result is obtained according to the first evaluation result and the answer to the rewritten query statement.

10. The method of claim 9, wherein: Obtaining the classification result based on the first evaluation result and the answer to the rewritten query statement includes any of the following: In response to the model answer answering the key question of the query, obtaining the classification result based on a quality comparison result between the model answer and the answer to the rewritten query and the first evaluation result; In response to the model answer not answering the key question of the query statement, the classification result is obtained according to the first evaluation result and whether the answer to the rewritten query statement answers the key question.

11. The method of claim 2, wherein: The using the second largest model to perform consistency evaluation on the query statement, the model answer, and the reference answer to obtain a first evaluation result includes: Using the second largest model, performing a consistency evaluation on the query statement and the model answer to obtain a second evaluation result; Using the second model, determining the consistency between the query statement and the reference answer to obtain a third evaluation result; In response to determining that the model answer is valid based on the second evaluation result and determining that the reference answer is valid based on the third evaluation result, the second large model is used to perform a consistency evaluation on the query statement, the model answer and the reference answer to obtain the first evaluation result.

12. The method according to any one of claims 1 to 11, wherein Obtaining the reference answer corresponding to the query statement includes: Using the second model, determining whether the query statement is ambiguous to obtain an ambiguity determination result; In response to the ambiguity category label in the ambiguity judgment result being no ambiguity, the reference answer is obtained.

13. A sample classification device based on a large model, comprising: A first acquisition module is configured to acquire a target negative feedback sample; wherein the target negative feedback sample includes a query statement and a model answer corresponding to the query statement, wherein the model answer is an answer to the query statement generated using the first large model; A second acquisition module is used to obtain a reference answer corresponding to the query statement; A classification module is used to classify the target negative feedback sample using a second large model according to the model answer and the reference answer to obtain a classification result of the target negative feedback sample; wherein the classification result includes the question category label of the target negative feedback sample and the reason for determining the question category label.

14. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 12.

15. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-12.

16. A computer program product comprising a computer program, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 12.

Images