Data processing method, device and computer program product

Abstract

The invention relates to a data processing method and device and a computer program product. The method comprises the following steps: processing at least one query object in a first data set to obtain a first query result; determining a target query object of which the first query result does not reach a preset index; analyzing associated information of the target query object, and determining a target event of which the first query result does not reach the preset index and an influence factor of the target event; and based on the target event and the influence factor of the target event, obtaining an evaluation result of the question and answer system. The labor cost and the time cost can be reduced; and the comprehensiveness of the evaluation result can be improved.

Description

Technical Field

[0001] The present disclosure relates to but is not limited to the field of information technology, and in particular to a data processing method, device, and computer program product. Background Art

[0002] Data query refers to querying and analyzing the data in the database based on the query object input by the user, and then obtaining the query results. For example, a question-answering system can perform a query operation on the input query object and obtain the answer corresponding to the query object. In order to make the content output by the question-answering system more in line with user needs, it is necessary to evaluate the question-answering effect and quality of the question-answering system.

[0003] In the related art, the content output by the question-answering system can be evaluated manually. However, the evaluation results obtained manually are not comprehensive and will increase labor costs and time costs. Summary of the invention

[0004] In order to overcome the problems existing in the related art, the present disclosure provides a data processing method, device and computer program product, which can reduce labor costs and time costs, and improve the comprehensiveness of evaluation results.

[0005] According to a first aspect of an embodiment of the present disclosure, there is provided a data processing method, including:

[0006] Processing at least one query object in the first data set to obtain a first query result;

[0007] Determine the target query object for which the first query result does not meet the preset indicator;

[0008] Analyze the associated information of the target query object to determine the target event for which the first query result does not meet the preset index, and the influencing factors of the target event;

[0009] Based on the target event and the influencing factors, an evaluation result is obtained.

[0010] According to a second aspect of an embodiment of the present disclosure, there is provided a data processing device, including:

[0011] an acquisition module, configured to process at least one query object in the first data set to obtain a first query result;

[0012] A first determining module is configured to determine that the first query result does not meet a target query object of a preset indicator;

[0013] A second determination module is configured to analyze the associated information of the target query object, determine the target event for which the first query result does not meet the preset index, and the influencing factors of the target event;

[0014] The evaluation module is configured to obtain an evaluation result based on the target event and the influencing factors.

[0015] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, including:

[0016] processor;

[0017] Memory for storing computer programs or instructions;

[0018] The processor executes the computer program or instructions to implement the steps of any one of the methods in the first aspect above.

[0019] According to a fourth aspect of an embodiment of the present disclosure, a non-temporary computer-readable storage medium is provided, wherein the storage medium stores a computer program or instructions. When the computer program or instructions in the storage medium are executed by a processor, the steps of the method described in any one of the first aspects are implemented.

[0020] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, including a computer program or instructions, which, when executed by a processor, implement the steps of any one of the methods described in the first aspect.

[0021] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:

[0022] The present disclosure can obtain a first query result obtained by processing at least one query object in a first data set, analyze the associated information of a target query object whose first query result does not meet a preset indicator, determine a target event whose first query result does not meet the preset indicator, and an influencing factor of the target event, and obtain an evaluation result based on the target event and the influencing factor of the target event.

[0023] On the one hand, the system evaluates itself automatically, which can reduce labor and time costs compared to manual evaluation of the system. On the other hand, it can not only determine the target event whose first query result does not meet the preset indicators, but also analyze the influencing factors of the target event to obtain the evaluation results of the system. By analyzing the target event from multiple dimensions, the comprehensiveness of the evaluation results can be improved.

[0024] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0026] Figure 1 A data processing method according to an exemplary embodiment is shown in FIG. Figure 1 .

[0027] Figure 2 A data processing method according to an exemplary embodiment is shown in FIG. Figure 2 .

[0028] Figure 3 A data processing method according to an exemplary embodiment is shown in FIG. Figure 3 .

[0029] Figure 4 A data processing method according to an exemplary embodiment is shown in FIG. Figure 4 .

[0030] Figure 5 A data processing method according to an exemplary embodiment is shown in FIG. Figure 5 .

[0031] Figure 6 It is a block diagram of a data processing device according to an exemplary embodiment.

[0032] Figure 7 The diagram is a schematic structural diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0033] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0034] Figure 1 A data processing method according to an exemplary embodiment is shown in FIG. Figure 1 .like Figure 1 As shown, the method mainly includes the following steps:

[0035] In step 101, at least one query object in a first data set is processed to obtain a first query result;

[0036] In step 102, it is determined that the first query result does not meet the target query object of the preset index;

[0037] In step 103, the associated information of the target query object is analyzed to determine the target event for which the first query result does not meet the preset index, and the influencing factors of the target event;

[0038] In step 104, an evaluation result is obtained based on the target event and the influencing factors.

[0039] The data processing method proposed in the present disclosure can be applied to electronic devices or servers. Here, the electronic device may include: terminal devices, for example, mobile terminals or fixed terminals. Among them, mobile terminals may include: mobile phones, tablet computers, laptop computers and other devices. Fixed terminals may include: desktop computers or smart TVs, etc. The data processing method in the embodiment of the present disclosure may be configured in a data processing device, which may be set in a server or in an electronic device, and the embodiment of the present disclosure does not limit this.

[0040] The execution subject of the embodiments of the present disclosure may be, for example, a central processing unit (CPU) in a server or electronic device in terms of hardware, or may be, for example, a related background service in a server or electronic device in terms of software, without limitation.

[0041] In some embodiments, the data processing method of the present disclosure can be applied to a system that needs to parse and respond to the intent of the query object input by the user. Exemplarily, the data processing method of the present disclosure can be applied to a question-answering system or a conversation system. That is, the present disclosure is a solution for the question-answering system or the conversation system to evaluate itself.

[0042] It should be noted that the question-answering system may be a system with a retrieval-augmented generation (RAG) function, that is, a query system including a RAG architecture, wherein the RAG architecture is a language model architecture that combines retrieval technology and generation technology.

[0043] In some embodiments, the question-answering system may include a retrieval module and a generation module. When performing a question-answering task, the retrieval module may be used to retrieve text segments related to the query object (question) to be processed from a preset corpus (knowledge base), and then the generation module may be used to generate an answer based on the retrieved text segments and context information of the query object to be processed. The query object to be processed may be a question input by a user, and the text segment may also be called a document segment.

[0044] Exemplarily, the question-answering system is a RAG system, which first uses a retrieval module to retrieve text fragments related to the question from a preset corpus (e.g., a large text library), and then uses a generation module to generate a final answer based on the retrieved text fragments and the context information of the question.

[0045] In some embodiments, the predetermined knowledge source can be preprocessed based on the question-answering system, and then the preprocessed data can be converted into vectors and stored as a data source that the question-answering system needs to refer to, that is, a knowledge base. After receiving the query object to be processed, the question-answering system can determine the user's question intention based on the query object to be processed, and perform word segmentation on the query object to be processed, and then retrieve relevant text fragments from the knowledge base based on the result of the word segmentation, sort the retrieved text fragments, and the question-answering system obtains the final answer based on the sorting result and the text fragments.

[0046] Exemplarily, the RAG system can first prepare knowledge: filter and / or segment the predetermined knowledge sources (e.g., documents), and convert the results of the filtering and / or segmentation into vectors and store them as a data source for the big model to answer references, i.e., the knowledge base; then the sentences input by the user can be processed, the big model can determine the user's question intention, and perform word segmentation on the sentences input by the user, and then perform retrieval based on the results of the word segmentation, and sort the retrieved text fragments, and obtain the final answer based on the sorting results and the text fragments.

[0047] In some embodiments, text fragments with higher relevance, timeliness, authority, etc. in the knowledge base can be recalled preferentially, the text fragments can be sorted, and the sorting results and the corresponding text fragments can be input into the big model to control the big model to answer based on the text fragments (recall fragments), thereby improving the accuracy, explainability and timeliness of the answers output by the model.

[0048] Since the RAG system needs to go through various steps such as document slicing, document vectorization and storage, retrieval in the knowledge base, and answer generation by a large model, the question-answering effect of the RAG system will be affected by each step. In addition, each step has many influencing factors and variables, which makes it difficult to evaluate and optimize the RAG system.

[0049] In the disclosed embodiment, after obtaining the question-answering system, the question-answering system can evaluate its own performance to obtain an evaluation result of the question-answering system. During the evaluation of the question-answering system, at least one query object in the first data set can be processed to obtain a first query result. It should be noted that the query object can be an object input by a user for performing a query operation, and the query object can include vocabulary, or the content included in the query object can be recognized and understood as including the target vocabulary, for example, the expression of a smiley face can be understood as happy or smiley face, and if the target vocabulary is happy, the expression of the smiley face can be understood as including the target vocabulary.

[0050] In some embodiments, the format and type of the query object are related to the entry point for the user to input the query object or the user's needs. The query object may include content in one format, or may be composed of a combination of content in different formats. Exemplarily, the query object may include at least one of the following: content in audio format; content in text format; content in image format. For example, the query object may include at least one of the following: audio, image, text, emoticon, or dynamic image.

[0051] In other embodiments, the query object may also be a query statement. Taking the query object as a query statement as an example, the query statement may include at least one of the following: a query statement in voice format, a query statement in text format, and a query statement in image format. It should be noted that the first data set may also be referred to as a test set. Since the evaluation is for a certain question-and-answer scenario, the same scenario may be evaluated multiple times. In order to ensure that the results before and after are comparable, a fixed first data set may be set. In some embodiments, the data in the first data set may be derived from a sampling of query objects (questions) in real scenarios that need to be evaluated to ensure that the query objects are representative of the real scenarios that need to be evaluated. The data structure in the first data set may be a query object Q (question) and a reference result A (standard answer), and the query object Q and the reference result A correspond one to one. The reference result A may be provided by personnel in the real scenario, and experts may be asked to review the reference result A to ensure that the reference result A is accurate, complete and easy to understand.

[0052] After obtaining the first data set, each query object in the first data set may be input into the question-answering system respectively, and the first query result may be output based on the question-answering system.

[0053] Exemplarily, the query object (question) is executed through the question-answering system to obtain the first query result A' and the corresponding second query result S (recall fragment) for each query object Q in the first data set. In some embodiments, the determination of the first query result can be achieved through a batch script of the question-answering system, or after manually inputting the query object, the corresponding data (including the first query result and the second query result) can be collected and sorted and summarized.

[0054] In some embodiments, after obtaining the first query results of each query object, a target query object whose first query result does not meet the preset index can be determined from at least one query object, and the associated information of the target query object can be analyzed to determine that the first query result is a target event that does not meet the preset index, as well as the influencing factors of the target event. Among them, the associated information of the target query object can include: information associated with the target query object generated in the process of processing the target query object. Taking the processing of the target query object based on the question-and-answer system as an example, the associated information of the target query object can be the associated information of the target query object processed by the question-and-answer system, that is, the information associated with the target query object generated by the question-and-answer system in the process of processing the target query object.

[0055] For example, after obtaining the question-answering system, the question-answering system can be evaluated by the target evaluation model to obtain an evaluation result of the question-answering system, wherein the evaluation result can be used to indicate the performance of the question-answering system. In some embodiments, the target evaluation model can be a model built into the question-answering system for evaluating the question-answering system, that is, the question-answering system implements its own evaluation. In other embodiments, the target evaluation model can also be a model independent of the question-answering system.

[0056] The target evaluation model may be an artificial intelligence model (e.g., a deep learning model) trained using large-scale data. Exemplarily, the deep learning model may include, but is not limited to, various neural networks, such as convolutional neural networks (CNN) and recurrent neural networks (RNN).

[0057] Exemplarily, after obtaining the first query result, the first query result and the query object may be input into a target evaluation model, and the target evaluation model may be used to determine, from at least one query object, a target query object for which the first query result does not meet a preset indicator.

[0058] In some embodiments, the preset indicator can be set as needed, for example, it can be determined in a customized manner or based on empirical values, as long as it can be used to evaluate the quality of the first query result.

[0059] Taking the preset indicator being a preset query result as an example, after obtaining the first query result, the similarity between the first query result and the preset query result can be determined. When the similarity between the first query result and the preset query result is less than the preset similarity threshold, it can be determined that the first query result does not meet the preset indicator.

[0060] Taking the preset indicator as a preset evaluation value set for the first query result as an example, after obtaining the first query result, an evaluation value of the first query result can be obtained. When the evaluation value of the first query result is less than the preset evaluation value, it can be determined that the first query result does not meet the preset indicator. In some embodiments, the evaluation value of the first query result can be determined based on the first query result and the reference result.

[0061] Taking the processing of the first query result based on the target evaluation model as an example, after the first query result and the corresponding query object are input into the target evaluation model, the target evaluation model can score each first query result, obtain the evaluation value of the first query result of each query object, and determine the query object corresponding to the first query result whose evaluation value is less than the preset evaluation value as the target query object.

[0062] In some embodiments, determining a target query object for which a first query result does not meet a preset indicator includes: for each query object, determining a first target evaluation value of the first query result of the query object based on the first query result of the query object and a reference result of the query object; and determining the query object corresponding to the first query result whose first target evaluation value is less than a first threshold as the target query object.

[0063] It should be noted that after obtaining the first query result of the query object, for each query object, a first target evaluation value of the query object may be determined based on the first query result of the query object and the reference result of the query object.

[0064] The first target evaluation value is used to indicate the difference between the first query result of the query object and the reference result of the query object, and the first target evaluation value is negatively correlated with the difference, that is, the smaller the first target evaluation value, the greater the difference between the first query result of the query object and the reference result of the query object; the larger the first target evaluation value, the smaller the difference between the first query result of the query object and the reference result of the query object. It should be noted that for each query object, there is a corresponding reference result, that is, the query object and the reference result correspond one to one.

[0065] In some embodiments, the difference between the first query result of the query object and the reference result of the query object can be compared based on the target evaluation model (referee model), and the first query result of the query object can be scored to obtain a first target evaluation value.

[0066] In some embodiments, based on the first query result of the query object and the reference result of the query object, a first target evaluation value of the first query result of the query object is determined, including: analyzing the first query result of the query object and the reference result of the query object from at least two dimensions to obtain at least two evaluation values; based on at least one of the at least two evaluation values, determining the first target evaluation value.

[0067] It should be noted that before analyzing the first query result, the dimensions for analyzing the first query result and the content information that needs to be output for the first query result can be pre-configured. In the disclosed embodiment, the first query result of the query object and the reference result of the query object can be analyzed from at least two dimensions to obtain at least two evaluation values, and based on at least one of the at least two evaluation values, a first target evaluation value can be determined. Among them, the dimensions for analyzing the first query result may include: accuracy, completeness, relevance, and redundancy.

[0068] Exemplarily, if the first query result is scored by the target evaluation model, the first query result, the query object, the pre-configured dimension, and the information content to be output can be input into the target evaluation model to obtain a first target evaluation value. After obtaining the first target evaluation value, the first query results of each query object can be divided based on the first target evaluation value, and a judgment of whether the answer is good or bad can be given.

[0069] In real business scenarios, since users do not require the first query result (AI-generated answer) to be completely consistent with the reference result (standard answer), but allow the first query result to reorganize the information or provide some other information. In the disclosed embodiment, the requirements for the answer quality of the question-answering system can be concentrated on the following dimensions: accuracy, completeness, relevance, and redundancy.

[0070] In the disclosed embodiment, at least two evaluation values ​​are obtained by analyzing the first query result of the query object and the reference result of the query object from multiple dimensions, and the first target evaluation value is determined based on at least one of the at least two evaluation values. The difference between the first query result and the reference result can be analyzed from multiple dimensions to improve the credibility of the obtained first target evaluation result, thereby improving the accuracy of the evaluation result of the question-answering system.

[0071] In some embodiments, the first query result of the query object and the reference result of the query object are analyzed from at least two dimensions to obtain at least two evaluation values, including: determining a first evaluation value based on the matching degree between the first query result of the query object and the reference result of the query object; determining a second evaluation value based on the inclusion relationship between the first query result of the query object and the reference result of the query object; determining a third evaluation value based on the relevance between the first query result of the query object and the query task indicated by the query object; determining a fourth evaluation value based on the interference data in the first query result of the query object and the relevance of the query object; wherein the interference data is data in the first query result other than the reference result.

[0072] It should be noted that the first evaluation value is used to indicate the accuracy of the first query result. Since the reference result is provided as the standard answer, the first evaluation value is determined based on the matching degree between the first query result of the query object and the reference result of the query object, so that it can be judged whether there is a factual conflict between the first query result of the query object and the reference result of the query object. Exemplarily, the first evaluation value is positively correlated with the matching degree, that is, the greater the matching degree between the first query result and the reference result, the smaller the possibility that there is a factual conflict between the first query result and the reference result, and the larger the first evaluation value (the higher the score); the smaller the matching degree between the first query result and the reference result, the greater the possibility that there is a factual conflict between the first query result and the reference result, and the smaller the first evaluation value (the lower the score).

[0073] The second evaluation value is used to indicate the completeness of the first query result. Based on the inclusion relationship between the first query result of the query object and the reference result of the query object, it can be determined whether the first query result of the query object contains all the key facts in the reference result of the query object. Exemplarily, the second evaluation value is positively correlated with the completeness, that is, the greater the completeness of the first query result and the reference result, the more key facts in the reference result included in the first query result, and the larger the second evaluation value (the higher the score); the smaller the completeness of the first query result and the reference result, the fewer key facts in the reference result included in the first query result, and the smaller the second evaluation value (the lower the score).

[0074] The third evaluation value is used to indicate the relevance between the first query result of the query object and the query object. Based on the relevance between the first query result of the query object and the query task indicated by the query object, the third evaluation value is determined, which can determine whether the first query result of the query object has formed an answer to the query object, and whether the query task in the query object is omitted. Exemplarily, the third evaluation value is positively correlated with the relevance, that is, the greater the relevance between the first query result and the query object, the fewer query tasks in the query object omitted by the first query result, and the greater the third evaluation value (the higher the score); the smaller the relevance between the first query result and the query object, the more query tasks in the query object omitted by the first query result, and the smaller the third evaluation value (the lower the score).

[0075] The fourth evaluation value is used to indicate the redundancy of the first query result. Based on the interference data in the first query result of the query object and the correlation of the query object, the fourth evaluation value is determined, which can determine whether the first query result of the query object contains other information (interference data) beyond the fact scope of the reference result, and determine whether the interference data conflicts with the existing facts and whether it is related to the query object. Exemplarily, the fourth evaluation value is negatively correlated with the redundancy, that is, the greater the redundancy of the first query result, the more interference data the first query result contains, the greater the possibility that the interference data conflicts with the existing facts, the smaller the correlation between the interference data and the query object, and the smaller the fourth evaluation value (the lower the score); the smaller the redundancy of the first query result, the less interference data the first query result contains, the smaller the possibility that the interference data conflicts with the existing facts, the greater the correlation between the interference data and the query object, and the larger the fourth evaluation value (the higher the score).

[0076] In some embodiments, the second evaluation value may be determined first, and the completeness of the first query result may be determined based on the second evaluation value; then the first evaluation value may be determined, and the accuracy of the first query result may be determined based on the first evaluation value; while determining the first evaluation value, a fourth evaluation value may also be determined, and the redundancy of the first query result may be determined based on the fourth evaluation value; and then the third evaluation value may be determined, and the relevance of the first query result and the query object may be determined based on the third evaluation value.

[0077] Exemplarily, from the above four dimensions, the judgment can be made in a preset order. First, the completeness of the first query result can be determined, that is, first sort out the information contained in the first query result and the reference result, whether all the information in the reference result is mentioned in the first query result, and then determine which of the information mentioned in the first query result is accurate and which is in conflict. In parallel, the redundancy of the first query result can be determined, that is, information that is not mentioned in the reference result is mentioned in the query result. These data can be called interference data, and then it can be determined whether the interference data causes a factual conflict with the reference result. Finally, it is generally determined whether the query object is logically related to the first query result, and whether there is a question and answer.

[0078] In other embodiments, the order of determining the first evaluation result, the second evaluation value, the third evaluation value, and the fourth evaluation value may be determined as needed and is not specifically limited herein.

[0079] In some embodiments, after obtaining the first evaluation value, the second evaluation value, the third evaluation value and the fourth evaluation value, a first target evaluation value may be determined based on at least one of the first evaluation value, the second evaluation value, the third evaluation value and the fourth evaluation value.

[0080] Exemplarily, the first target evaluation value may be determined based on an average of the first evaluation value, the second evaluation value, the third evaluation value, and the fourth evaluation value. Further exemplary, the first target evaluation value may be determined based on a median of the first evaluation value, the second evaluation value, the third evaluation value, and the fourth evaluation value. Further exemplary, the first target evaluation value may be determined based on a weighted sum of the first evaluation value, the second evaluation value, the third evaluation value, and the fourth evaluation value.

[0081] In the disclosed embodiment, taking the first target evaluation result obtained based on the target evaluation model (referee model) as an example, the reference result and the first query result are split into the dimensions of opinion and minimum fact through the target evaluation model for comparison and judgment, and the first query result of the query object and the reference result of the query object are analyzed from the four dimensions of accuracy, completeness, relevance, and redundancy, and the scoring result of the first query result for the query object is output based on the target evaluation model. The accuracy of the obtained first target evaluation result can be improved, thereby improving the accuracy of the evaluation result of the question-answering system.

[0082] In the disclosed embodiment, after obtaining the first target evaluation value, the query object whose first target evaluation value is less than the first threshold value can be determined as the target query object. The first threshold value can be determined according to needs or empirical values. Exemplarily, the obtained first target evaluation value is 3, and the first threshold value is 4, then the first target evaluation value is less than the first threshold value. Since when the first target evaluation value is less than the first threshold value, it indicates that the first query result corresponding to the first target evaluation value is inaccurate. At this time, the query object corresponding to the first query result whose first target evaluation value is less than the first threshold value can be determined as the target query object.

[0083] For example, if the first target evaluation value is 1, the corresponding first query result has a very low accuracy (the answer is particularly poor), if the first target evaluation value is 5, the corresponding first query result has a very high accuracy (the answer is particularly good), and if the first target evaluation value is 3, the corresponding first query result has an average accuracy. When the first threshold is 4, it can be determined that when the score is 4 and 5, the accuracy of the first query result is high, that is, the answer is a good answer. The first query results (answers) with first target evaluation values ​​of 1, 2, and 3 have low accuracy, and the query objects corresponding to these first query results can be determined as target query objects. The following is an example of determining the score of the first query result:

[0084] Target query object: How many products does our company currently have for enterprise users?

[0085] Reference result A: Our company currently has five products for enterprise users, namely ABCDE.

[0086] The first query result A': The company currently has four products for enterprise users, namely ABCD. If you need to know more details, you need to check the XX document.

[0087] The second query result S: Enterprise products: A product introduction XXXXXX; B introduction XXXXXX; C introduction XXXXX; D introduction XXXXX; E introduction XXXXX.......; personal user product F introduction XXXXX.

[0088] Among them, from the perspective of factual consistency, the first query result A' is not complete in listing product types compared to the reference result A. It only mentions ABCD but not E, which leads to factual differences. In terms of relevance, the content in the first query result A' meets the requirements of the question. In terms of redundancy, there is no conflict with the content in the reference result A. Therefore, this first query result can be rated 3 points in the previous scoring stage, and then enter the bad answer attribution analysis process. The above-mentioned first query result A' and the second query result S are the final results and recall fragments obtained by the question-and-answer system for the same query object. That is, a query can correspond to a first query result and a second query result.

[0089] In the disclosed embodiment, the query object, the first query result, and the reference result in the result of the question-answering system execution can be input into the big model, and the big model can score the factual consistency, factual completeness, logical relevance, positive and negative redundancy of the first query result, and give a judgment on whether the answer is a good answer or a bad answer, and explain it. By treating the low-scoring first query result as a bad answer, and then determining the corresponding target query object, and locating the analysis object to the target query object, it is possible to reduce the interference of the first query result with a relatively accurate answer on the evaluation result, thereby improving the accuracy of the evaluation of the question-answering system.

[0090] After determining the target query object, the associated information of the target query object processed by the question-answering system can be analyzed to determine the target event for which the first query result does not meet the preset index, as well as the influencing factors of the target event. The associated information of the target query object processed by the question-answering system may include: information related to the target query object generated by the question-answering system in the process of processing the target query object. For example, the associated information of the target query object processed by the question-answering system may include: text fragments recalled from a preset corpus based on the target query object, status information of the question-answering system, interface information when performing a recall operation, etc.

[0091] Exemplarily, the associated information of the target query object processed by the question-answering system can be analyzed based on the target evaluation model to determine the target event for which the first query result does not meet the preset indicators, as well as the influencing factors of the target event.

[0092] In some embodiments, the associated information of the target query object is analyzed to determine the target event for which the first query result does not meet the preset indicators, including: obtaining a second query result generated based on the target query object; wherein the amount of information in the second query result is greater than the amount of information in the first query result; determining the target event based on the degree of match between the second query result of the target query object and the reference result of the target query object; wherein different target events are determined by different degrees of match.

[0093] In some embodiments, when the application interface of the question-and-answer system and the network in which the question-and-answer system is located are in a normal state, a second query result generated based on the target query object can be obtained. Exemplarily, the second query result generated by the question-and-answer system based on the target query object is obtained. In the disclosed embodiment, after determining the target query object, the status information of the question-and-answer system can be determined, wherein the status information may include: network status information of the question-and-answer system and / or working status information of the question-and-answer system. After determining the status information of the question-and-answer system, it can be determined whether the application interface of the question-and-answer system is in a normal state based on the status information of the question-and-answer system, and whether the network in which the question-and-answer system is located is in a normal state.

[0094] Exemplarily, the interface can be used to automatically determine whether the network where the question-and-answer system is located is in a normal state. For example, if the question-and-answer system is a RAG system, since the RAG system is a complex system with multiple links nested internally, for problems such as service response or error reporting in some links, the network status information (e.g., gateway status code) automatically fed back by the interface can be used to determine whether the network where the question-and-answer system is located is in a normal state, that is, to determine whether the system service of the question-and-answer system is in a normal state. For example, the network status information can be used to indicate whether the gateway status code has been received. Exemplarily, if the gateway status code is not received, it can be determined that the system service of the question-and-answer system is in a normal state.

[0095] In other embodiments, when the system service of the question-answering system is in an abnormal state, the target event can be directly determined to be a system service problem based on the gateway status code. Exemplarily, if the gateway status code is received, it is determined that the large model service for question-answering in the question-answering system is unstable, and the system service of the question-answering system is in an abnormal state.

[0096] As another example, if the recall interface does not report an error during the retrieval phase and there is a second query result (recalled text fragment), it can be determined that the application interface of the question-answering system is in a normal state.

[0097] In other embodiments, if the recall interface reports an error during the retrieval phase, resulting in no second query result (recalled text fragment), the target event can be directly determined to be a document missing problem. In this case, the document information in the preset corpus (knowledge base) needs to be reviewed. It should be noted that in the case of a recall interface error during the retrieval phase, different reasons can be separated according to the interface granularity on the question-and-answer system development side, that is, the influencing factors causing the anomaly can be determined based on the interface granularity on the question-and-answer system development side. Among them, the interface granularity is used to indicate the degree of subdivision of the interface function in the question-and-answer system.

[0098] In the disclosed embodiment, when the application interface of the question-answering system and the network in which the question-answering system is located are in a normal state, a second query result generated by the question-answering system based on the target query object can be obtained; wherein the amount of information in the second query result is greater than the amount of information in the first query result. It should be noted that the second query result can be a text fragment recalled from a preset corpus based on the target query object. The application interface may include an interface for performing a recall operation in the retrieval phase.

[0099] After the second query result of the target query object is determined, the accuracy of the second query result may be determined based on the reference result of the target query object, that is, a recall accuracy judgment may be performed.

[0100] In the process of judging the recall accuracy, the target event can be determined based on the matching degree between the second query result of the target query object and the reference result of the target query object. Since the matching degree can indicate the difference between the second query result of the target query object and the reference result of the target query object, and the matching degree is negatively correlated with the difference between the second query result of the target query object and the reference result. That is, the greater the difference between the second query result of the target query object and the reference result of the target query object, the smaller the matching degree between the second query result of the target query object and the reference result of the target query object; the smaller the difference between the second query result of the target query object and the reference result of the target query object, the greater the matching degree between the second query result of the target query object and the reference result of the target query object.

[0101] Exemplarily, when it is determined based on the status information that the application interface has not reported an error and there is a second query result (recalled text fragment), the recall accuracy judgment process can be entered. In the process of recall accuracy judgment, the target event can be determined by comparing the second query result S (recalled fragment) and the reference result A (standard answer).

[0102] It should be noted that the first query result can be obtained from the second query result. For example, the first query result can be extracted from the second query result, or the first query result can be obtained based on analyzing the second query result.

[0103] Since the first query result A' is obtained by the large model by extracting information from the second query result S, if the information in the reference result A can be extracted or inferred from the second query result S, it can be judged that the recall is accurate, that is, the target event is not a recall problem. If the information in the reference result A cannot be extracted or inferred from the second query result S, it can be judged that the recall is inaccurate, that is, the target event is a recall problem.

[0104] In the disclosed embodiment, when it is determined that the application interface of the question-and-answer system is in a normal state based on the status information of the question-and-answer system, a second query result generated by the question-and-answer system based on the target query object is obtained, and the target event is determined based on the degree of match between the second query result of the target query object and the reference result of the target query object. After excluding network problems and interface problems, the recall accuracy of the recall module can be judged, and the target event that occurred during the recall stage can be determined.

[0105] In some embodiments, a target event is determined based on the degree of match between a second query result of a target query object and a reference result of the target query object, including: determining the existence of a first target event when the degree of match between the second query result of a target query object and the reference result of a target query object is less than or equal to a first matching threshold; wherein the first target event indicates that a recall module of a question-and-answer system is abnormal; determining the existence of a second target event when the degree of match between the second query result of a target query object and the reference result of a target query object is greater than the first matching threshold and less than the second matching threshold; wherein the second target event indicates that the corpus data of a question-and-answer system is abnormal.

[0106] It should be noted that, when the matching degree between the second query result of the target query object and the reference result of the target query object is less than or equal to the first matching degree threshold, all opinions and facts in the reference result A cannot be directly extracted or inferred from the second query result S, and the existence of the first target event is determined, and it is determined that there is a recall problem, that is, the recall module of the question-and-answer system is abnormal.

[0107] When the matching degree between the second query result of the target query object and the reference result of the target query object is greater than the first matching degree threshold and less than the second matching degree threshold, some opinions and facts in the reference result A can be extracted in the second query result S, but some cannot be extracted. It is determined that there is a second target event, and it is determined that there is an incomplete recall problem, that is, the corpus data of the question-answering system is abnormal. At this time, the document information of the knowledge base needs to be reviewed, and it can be determined that the second target event is a document missing problem or a recall fragment limitation problem.

[0108] In the disclosed embodiment, a first instruction may be input into the target evaluation model, and the first instruction may instruct the target evaluation model to split the second query result and the reference result of the target query object into the dimensions of opinion and minimum fact for comparative judgment, so as to make an abnormality judgment and identify different target events.

[0109] In some embodiments, determining a target event based on a matching degree between the second query result of the target query object and the reference result of the target query object includes:

[0110] When the matching degree between the second query result of the target query object and the reference result of the target query object is greater than a second matching degree threshold, determining the matching degree between the first query result of the target query object and the reference result of the target query object;

[0111] A target event is determined based on a matching degree between the first query result of the target query object and a reference result of the target query object.

[0112] It should be noted that, when the degree of match between the second query result of the target query object and the reference result of the target query object is greater than the second matching degree threshold, all opinions and facts in the reference result A can be completely extracted in the second query result S, the recall problem in the retrieval stage can be eliminated, the recall is determined to be accurate, and the accuracy judgment is generated. In the process of generating the accuracy judgment, the degree of match between the first query result of the target query object and the reference result of the target query object can be determined.

[0113] If the recall is completely accurate, but the answer is still not ideal, the accuracy judgment process is entered. By comparing the reference result A and the first query result A', the target event can be determined. It is known that the first query result A' and the reference result A rely on the same knowledge source, but the output is different, indicating that the large model may not answer based on the target query object Q and the second query result S.

[0114] In the disclosed embodiment, a second instruction may be input into the target evaluation model, and the second instruction may instruct the target evaluation model to split the first query result A' of the target query object and the reference result A of the target query object into the dimensions of opinion and minimum fact for comparative judgment, so as to make an abnormality judgment and identify different target events.

[0115] Exemplarily, compared with reference result A, the first query result A' contains all the key information of reference result A, then the target event is judged to be an error in the automatic scoring link, and feedback is required for the target query object, which is classified as good answer feedback. Compared with reference result A, the relevant information in the first query result A' is omitted, indicating that the question-answering system failed to find all the relevant information from the recalled fragments, and the answer is deficient, then the target event is judged to be insufficient reasoning ability of the large model. Compared with reference result A, the first query result A' has a difference in expression with the reference result A, indicating that the question-answering system has a fallacy in understanding the recalled part of the fragment, then the target event is judged to be a large model hallucination. Compared with reference result A, the first query result A' contains all the key information of reference result A, but does not directly face the target query object Q, indicating that the question-answering system did not further refine the information to answer, then the target event is judged to be a problem with the input of the large model prompt word. Exemplarily, the prompt word does not emphasize that the large model should answer completely based on the question, resulting in more independent play in the answer output by the large model. For target events where there are problems with the prompt words, the optimization method is to adjust the prompt words of the large model.

[0116] In the disclosed embodiment, the target evaluation model determines the first query result with a low score as a bad answer, and controls the target evaluation model to determine the target event that leads to the bad answer by judging the relationship between the target query object, the reference result, the first query result, and the second query result retrieved and recalled, and analyzes the cause of the target event. By disassembling the attribution analysis process, first judging whether there is a recall problem in the retrieval stage, and then judging whether there is a large model problem, through level-by-level investigation and step-by-step judgment, the target event can be comprehensively analyzed and the influencing factors of the target event can be determined.

[0117] In some embodiments, determining the influencing factors of the target event includes: when determining that a first target event exists, changing the target query object based on a reference result of the target query object to obtain a third query object; processing the third query object to obtain a second query result of the third query object; comparing the second query result of the third query object with the second query result of the target query object to obtain a comparison result, and determining the influencing factors of the first target event based on the comparison result.

[0118] It should be noted that, when it is determined that there is a first target event, a recall problem analysis can be performed. Based on the recall problem diversion in the previous link, it can be determined that there is a recall problem, but the influencing factors that cause the recall problem are not determined. The present disclosure can analyze the target query object Q, compare the target query object Q with the reference result A through the target evaluation model, rewrite the target query object Q through the target evaluation model, obtain the third query object, and again judge the accuracy of the second query result based on the recall of the third query object. In some embodiments, this can be achieved by continuously calling the interface of the question-and-answer system.

[0119] In some embodiments, rewriting the target query object may include: replacing or adjusting the vocabulary in the target query object so that the third query object is different from the target query object. The second query result is used to distinguish from the first query result, wherein the first query result is the final result output by the question-answering system, and the second query result is a recall fragment, and the first query result can be extracted from the second query result or inferred based on the second query result. The generation process of the second query result of the third query object is the same as the generation process of the second query result of the target query object.

[0120] In some embodiments, the second query result of the third query object and the second query result of the target query object may be compared to obtain a comparison result, and based on the comparison result, it is determined whether the second query result recalled based on the third query object is accurate. Exemplarily, the similarity between the second query result of the third query object and the second query result of the target query object may be determined, and if the similarity is greater than a preset similarity threshold, it is determined that the second query result recalled based on the third query object is accurate; if the similarity is less than or equal to the preset similarity threshold, it is determined that the second query result recalled based on the third query object is inaccurate.

[0121] In other embodiments, when the target query object Q is changed to obtain a third query object, and the second query result recalled based on the third query object is accurate, the determined influencing factor is: the question-answering system lacks the ability to generalize the query object. When the target query object Q is changed to obtain a third query object request, and the second query result recalled based on the third query object is inaccurate, the determined influencing factor is: a document missing problem or a slice parsing problem, wherein the slice parsing problem is a problem of inaccurate parsing of the units into which the document or text is segmented. In some embodiments, the document information in the knowledge base can be analyzed to determine whether it is a document missing problem or a slice parsing problem. Exemplarily, the document information in the knowledge base can be viewed through manual intervention to determine whether it is a document missing problem or a slice parsing problem.

[0122] In some embodiments, the evaluation results of the question-answering system may include: target events and influencing factors of the target events. After obtaining the evaluation results of the question-answering system, the evaluation results may be directly output, that is, the target events and influencing factors of the target events may be output, so as to facilitate users to understand the problems and make adjustments in a timely manner. Among them, the target events may also be called abnormal events, that is, events that may cause abnormalities in the question-answering system.

[0123] The present disclosure can obtain a first query result obtained by a question-and-answer system processing at least one query object in a first data set, analyze the associated information of a target query object whose first query result does not meet a preset indicator, determine a target event whose first query result does not meet the preset indicator, and influencing factors of the target event, and obtain an evaluation result of the question-and-answer system based on the target event and the influencing factors of the target event.

[0124] On the one hand, automatic evaluation of the question-answering system can reduce labor and time costs compared to manual evaluation of the question-answering system. On the other hand, it can not only determine the target event for which the first query result does not meet the preset indicators, but also analyze the influencing factors of the target event, and then obtain the evaluation result of the question-answering system. By analyzing the target event from multiple dimensions, the comprehensiveness of the evaluation result can be improved.

[0125] In some embodiments, the method also includes: determining a second target evaluation value of the question and answer system based on the number of target query objects and the total number of query objects; analyzing the associated information of the target query objects processed by the question and answer system to determine the target event for which the first query result does not meet the preset indicators, as well as the influencing factors of the target event, including: when the second target evaluation value is less than a second threshold, analyzing the associated information of the target query objects processed by the question and answer system to determine the target event; determining the influencing factors of the target event based on the type of the target event; wherein the influencing factors of different types of target events are different.

[0126] In the embodiment of the present disclosure, when the number of target query objects is determined, the number of non-target query objects can be determined based on the total number of query objects and the number of target query objects, and then the second target evaluation value can be determined based on the number of non-target query objects and the total number of query objects. For example, the second target evaluation value can be determined based on the ratio of the number of non-target query objects to the total number of query objects. The calculation formula of the second target evaluation value is as follows:

[0127]

[0128] In formula (1), A1 represents the second target evaluation value; M represents the number of non-target query objects; and N represents the total number of query objects. The total number of query objects refers to the total number of query objects in the first data set.

[0129] It should be noted that the second target evaluation value is used to indicate the accuracy of the current evaluation of the question-answering system. In some embodiments, non-target query objects may include query objects with a score greater than or equal to 4 points. By calculating the proportion of 4-point answers and 5-point answers, the accuracy of the current evaluation of the question-answering system can be accurately determined.

[0130] The second target evaluation value is positively correlated with the accuracy of the question-answering system, that is, the larger the second target evaluation value is, the higher the accuracy of the question-answering system is; the smaller the second target evaluation value is, the lower the accuracy of the question-answering system is.

[0131] In the disclosed embodiment, when the second target evaluation value is less than the second threshold, the associated information of the target query object processed by the question-answering system is analyzed to determine the target event, and based on the type of the target event, the influencing factors of the target event are determined.

[0132] Among them, the types of target events may include: document missing problems, lack of generalization ability of query objects, slice parsing problems, system service problems, recall fragment limitation problems, insufficient large model reasoning ability, large model hallucination, and large model encoding problems. Different types of target events will have different corresponding influencing factors.

[0133] In some embodiments, the target evaluation model can perform multiple steps to determine the specific cause of the bad answer for the first query result of a single target query object. For example, the query object Q, the reference result A, the first query result A' and the second query result S in the result of the previous execution can be compared for multiple rounds, and the recall accuracy, the generation accuracy, and the analysis of the query object Q can be determined in turn, and the specific cause can be output based on the multiple rounds of comparison.

[0134] After determining the type of the target event, the present disclosure can determine the influencing factors of the target event based on the type of the target event. Exemplarily, the influencing factors of the lack of generalization ability of the query object, slice parsing problems, system service problems, and recall fragment limitation problems are on the development side. Insufficient reasoning ability of large models, large model hallucinations, and large model encoding problems involve the selection of large question-answering models, which are related to both the effect and the configuration cost of the question-answering system. The influencing factors need to be jointly decided by multiple roles based on the application link.

[0135] Since there is no need to perform abnormal analysis and evaluation on the question-answering system when the accuracy of the question-answering system is high, the present invention analyzes the associated information of the target query object processed by the question-answering system when the accuracy of the question-answering system is low, determines the target event, and determines the influencing factors of the target event based on the type of the target event, thereby reducing invalid evaluations.

[0136] In some embodiments, the method further comprises:

[0137] Determine an adjustment strategy for the question-answering system based on the influencing factors of the target event and / or determine the type of the target event;

[0138] The question-answering system is adjusted based on the adjustment strategy until the second target evaluation value is greater than or equal to the second threshold.

[0139] In the disclosed embodiments, an adjustment strategy for the question-answering system may be determined based on influencing factors of the target event and / or the type of the target event, and the question-answering system may be adjusted based on the adjustment strategy.

[0140] Among them, the types of target events may include: document missing problems, lack of generalization ability of query objects, slice parsing problems, system service problems, recall fragment limitation problems, insufficient large model reasoning ability, large model hallucination, and large model encoding problems. Different types of target events will have different corresponding influencing factors.

[0141] After determining the type of the target event, the present disclosure can determine the influencing factors of the target event based on the type of the target event. Exemplarily, the influencing factors of the lack of generalization ability of the query object, slice parsing problems, system service problems, and recall fragment limitation problems are on the development side of the question-answering system, and need to be adjusted based on the influencing factors on the development side. Insufficient reasoning ability of large models, large model hallucinations, and large model encoding problems involve the selection of large question-answering models, which are related to both the effect and the configuration cost of the question-answering system. The influencing factors require joint decision-making by multiple roles based on the application link. The problem of missing documents requires supplementary documents on the user side of the question-answering system. The present disclosure can achieve a rough diversion of problems. The proportion of each type of problem in this evaluation can be obtained, and then targeted adjustments can be made based on the evaluation results.

[0142] After each adjustment of the corresponding question is completed, the question-answering system can be evaluated again using the same test set (first data set) to obtain a second evaluation value of the question-answering system, that is, the accuracy, and then the optimization effect can be checked until the second target evaluation value is greater than or equal to the second threshold, that is, a relatively satisfactory level of effect is achieved.

[0143] In other embodiments, the evaluation system can also be rechecked based on the automatic scoring link of the target evaluation model to obtain good answer feedback. In some embodiments, since the impact of large model switching in the question-answering system is global, the evaluation tuning link should give priority to solving the recall problem in the retrieval stage, such as supplementing documents, and then adjusting the large model configuration. After the adjustment is completed, the answer can be re-scored through the target evaluation model, and the optimization effect can be evaluated by monitoring the change in accuracy.

[0144] It should be noted that the big model mentioned in the present disclosure may correspond to the generation module in the question-answering system, and the retrieval stage may correspond to the retrieval module in the question-answering system.

[0145] In the disclosed embodiment, after the target event and the influencing factors of the target event are determined, the target event and / or the influencing factors of the target event can be classified and summarized to obtain the problem category and the optimization ideas for the corresponding problems. The question-answering system can be optimized based on the evaluation results, which can improve the accuracy of the answers output by the question-answering system.

[0146] In some embodiments, the method further comprises:

[0147] Processing the test object in the second data set to obtain a first query result of the test object;

[0148] For each test object, the candidate evaluation model determines a third target evaluation value of the test object based on the first query result and the reference result of the test object;

[0149] determining a target evaluation model from candidate evaluation models based on the third target evaluation value and the reference evaluation value of the test object;

[0150] Analyze the associated information of the target query object processed by the question-answering system to determine the target event that the first query result does not meet the preset indicator, including:

[0151] Based on the target evaluation model, the relevant information of the target query object processed by the question-answering system is analyzed to determine the target event.

[0152] It should be noted that the test object in the second data set can be processed based on the test question-answering system to obtain the first query result of the test object. The test object can be a query object used for testing, and the test question-answering system is a question-answering system used for testing.

[0153] Here, the second data set may also be referred to as a test set, wherein the second data set may be the same as the first data set or may be different from the second data set.

[0154] Exemplarily, the data structure in the second data set may be a test object Q (question), a reference result A (standard answer), a first query result A', a third target evaluation value P (including multiple evaluation model scores, namely P1, P2, P3, ...), and a reference evaluation value W. The reference evaluation value may be determined manually or obtained based on experiments.

[0155] Among them, the test object Q, the reference result A and the first query result A' are directly obtained by the question-and-answer system through a batch interface. The third target evaluation value P can be obtained based on the evaluation process using the evaluation model in any of the above embodiments. Exemplarily, the difference between the reference result A and the first query result A' can be determined based on the currently used evaluation model, and the third target evaluation value of the first query result can be determined based on the difference, and the first query result of the test object can be labeled based on the third target evaluation value. The process of determining the third target evaluation value is the same as that in the aforementioned embodiment, and can be determined based on the four indicators of accuracy, completeness, relevance and redundancy. The final score needs to be determined by multiple experts, which will not be repeated here.

[0156] In some embodiments, the sample size in the second data set can be set to more than 100, that is, the number of test objects can be greater than 100. Taking the number of test objects as 100 as an example, after all candidate evaluation models complete the processing of the first query results and reference results of the test objects, a data set consisting of multiple third target evaluation values ​​P and reference evaluation values ​​W can be obtained, and the third target evaluation value P and the reference evaluation value W each contain 100 data, and these data are all scored from 1 to 5.

[0157] In some embodiments, the degree of fit of the third target evaluation value P obtained by each candidate evaluation model relative to the reference evaluation value W can be determined by a statistical data analysis method, and the target evaluation model can be determined from each candidate evaluation model based on the degree of fit. Exemplarily, the candidate evaluation model with the best degree of fit can be determined as the target evaluation model. Exemplarily, the significance of the third target evaluation value P relative to the reference evaluation value W can be determined by SPSS multivariate regression analysis.

[0158] In some embodiments, through the scoring consistency analysis of a single first query result, the accuracy of the third target evaluation values ​​output by different candidate evaluation models compared to the reference evaluation value, that is, the fourth target evaluation value of the candidate evaluation model, can be compared.

[0159] In some embodiments, determining a target evaluation model from candidate evaluation models based on the third target evaluation value and the reference evaluation value of the test object includes:

[0160] determining a fourth target evaluation value of the candidate evaluation model based on the number of test subjects having the same reference evaluation value and the third target evaluation value, and the total number of test subjects;

[0161] Based on the respective fourth target evaluation values, a target evaluation model is determined from the candidate evaluation models.

[0162] In the embodiment of the present disclosure, the fourth target evaluation value of the candidate evaluation model can be determined based on the number of test objects with the same reference evaluation value and the third target evaluation value, and the total number of test objects. For example, the ratio of the number of test objects with the same reference evaluation value and the third target evaluation value to the total number of test objects can be determined, and the fourth target evaluation value can be determined based on the ratio. The calculation formula of the fourth target evaluation value is as follows:

[0163]

[0164] In formula (2), A2 represents the fourth target evaluation value; T represents the number of test objects whose reference evaluation value is the same as the third target evaluation value (consistent number); and R represents the total number of test objects. Take the automated scoring process as an example:

[0165] For example, in order to test the selection of evaluation models for the question-and-answer scenario of administrative personnel rules and regulations, there are candidate evaluation models A, B, and C; the number of test objects is 120, and the first query result is manually scored and labeled, as shown in Table 1.

[0166] Table 1

[0167]

[0168] Based on Table 1, we can see that the data performance of model A is better.

[0169] In the disclosed embodiment, a consistency analysis may be performed on the third target evaluation value obtained by the candidate evaluation model and the reference evaluation value of the test object, and a target evaluation model may be selected from the candidate evaluation models based on the analysis result.

[0170] Exemplarily, the candidate evaluation model with the highest fourth target evaluation value may be determined as the target evaluation model.

[0171] In some embodiments, since the semantic understanding and reasoning capabilities of the large model are associated with the training data, the target evaluation model determined for a certain business scenario is effective for all applications in that scenario. For example, in a question-and-answer scenario related to administrative personnel, the corpus is mostly some rules and regulations discourse, then the target evaluation model determined based on the above embodiment is applicable to similar scenarios, and there is no need to reselect the target evaluation model for the evaluation of a single application. Therefore, the selection of the target evaluation model can be determined by the development side without being exposed to the user side.

[0172] Figure 2 A data processing method according to an exemplary embodiment is shown in FIG. Figure 2 .like Figure 2 As shown, the method mainly includes the following steps:

[0173] In step 201, a first data set is obtained.

[0174] Here, the first data set may also be referred to as a test set. Since the evaluation is for a certain question-and-answer scenario, the same scenario can be evaluated multiple times. In order to ensure the comparability of the results before and after, a fixed first data set may be set. In some embodiments, the data in the first data set may be derived from a sampling of query objects (questions) in real scenarios that need to be evaluated to ensure the representativeness of the query objects for the real scenarios that need to be evaluated. The data structure in the first data set may be a query object Q (question) and a reference result A (standard answer), and the query object Q and the reference result A correspond one to one. Reference result A may be provided by personnel in real scenarios, and experts may be asked to review reference result A to ensure that reference result A is accurate, complete and easy to understand.

[0175] In step 202, a first query result is obtained.

[0176] Here, after obtaining the first data set, each query object in the first data set may be input into the question-answering system respectively, and the first query result is output based on the question-answering system.

[0177] In step 203, a first target evaluation value is obtained.

[0178] Here, after obtaining the first query result of the query object, for each query object, a first target evaluation value of the query object may be determined based on the first query result of the query object and the reference result of the query object.

[0179] In step 204, the target query object is analyzed.

[0180] Here, the associated information of the target query object processed by the question-and-answer system can be analyzed to determine the target event for which the first query result does not meet the preset indicators, as well as the influencing factors of the target event, and based on the target event and the influencing factors of the target event, obtain the evaluation results of the question-and-answer system.

[0181] In step 205, the question-answering system is adjusted.

[0182] Here, an adjustment strategy for the question-answering system may be determined based on the influencing factors of the target event and / or the type of the target event, and the question-answering system may be adjusted based on the adjustment strategy until the second target evaluation value is greater than or equal to the second threshold.

[0183] Figure 3 A data processing method according to an exemplary embodiment is shown in FIG. Figure 3 .like Figure 3 As shown, the target evaluation model 301 can determine a first evaluation value 302 based on the matching degree between the first query result of the query object and the reference result; determine a second evaluation value 303 based on the inclusion relationship between the first query result of the query object and the reference result; determine a third evaluation value 304 based on the relevance between the first query result of the query object and the query task indicated by the query object; determine a fourth evaluation value 305 based on the relevance between the interference data in the first query result of the query object and the query object; and then determine a first target evaluation value 306 based on at least one of the first evaluation value 302, the second evaluation value 303, the third evaluation value 304, and the fourth evaluation value 305. Then, the query object corresponding to the first query result whose first target evaluation value 306 is less than the first threshold is determined as the target query object (bad answer), and the cause analysis is performed based on the target evaluation model, and the second target evaluation value 307 of the question-answering system is determined based on the number of non-target query objects and the total number of query objects.

[0184] Figure 4 A data processing method according to an exemplary embodiment is shown in FIG. Figure 4 .like Figure 4 As shown, the method mainly includes the following steps:

[0185] In step 401, the interface automatically performs judgment.

[0186] Here, the interface can be used to automatically determine whether the network where the question-and-answer system is located is in a normal state. Taking the question-and-answer system as an example, the RAG system is a complex system with multiple links nested internally. For problems such as service response or error reporting in some links, the network status information (for example, gateway status code) automatically fed back by the interface can be used to determine whether the network where the question-and-answer system is located is in a normal state, that is, to determine whether the system service of the question-and-answer system is in a normal state. For example, the network status information can be used to indicate whether the gateway status code has been received. Exemplarily, if the gateway status code is not received, it can be determined that the system service of the question-and-answer system is in a normal state.

[0187] In the case where the system service of the question-answering system is in an abnormal state, the target event can be directly determined to be a system service problem, i.e., a large model problem, based on the gateway status code. Exemplarily, if the gateway status code is received, it is determined that the large model service for question-answering in the question-answering system is unstable, and the system service of the question-answering system is in an abnormal state.

[0188] In step 402, the accuracy determination is recalled.

[0189] Here, when the application interface of the question-and-answer system and the network in which the question-and-answer system is located are in normal status, the second query result generated by the question-and-answer system based on the target query object can be obtained. After determining the second query result of the target query object, the accuracy of the second query result can be determined based on the reference result of the target query object, that is, a recall accuracy judgment can be performed.

[0190] In step 403, an accuracy determination is generated.

[0191] Here, when the matching degree between the second query result of the target query object and the reference result is greater than the second matching degree threshold, all opinions and facts in the reference result A can be completely extracted in the second query result S, which can eliminate the recall problem in the retrieval stage, determine the recall accuracy, and make a generation accuracy judgment.

[0192] In step 404, the problem analysis is recalled.

[0193] Here, if the information in the reference result A cannot be extracted or inferred from the second query result S, it can be determined that the recall is inaccurate, that is, the target event is determined to be a recall problem, and a recall problem analysis can be performed.

[0194] Figure 5 A data processing method according to an exemplary embodiment is shown in FIG. Figure 5 .like Figure 5 As shown, the test objects in the second data set can be processed based on the test question and answer system to obtain the first query result of the test object; for each test object, the candidate evaluation model 501 determines the third target evaluation value 502 of the test object based on the first query result and the reference result of the test object; based on the third target evaluation value 502 and the reference evaluation value 503 of the test object, the target evaluation model 504 is determined from the candidate evaluation models.

[0195] It should be noted that the evaluation model may include an artificial intelligence model trained using large-scale data to evaluate the quality of questions and answers in the question-answering system and provide objective and consistent judgments. The evaluation model can analyze the factual consistency, logic, redundancy, etc. of the standard answer (reference result) and the AI-generated answer (first query result), and provide a reference for the effectiveness evaluation of the question-answering system. The evaluation model can also automatically analyze the specific reasons why the question-answering system is not effective based on the recalled fragment information (second query result).

[0196] This disclosure mentions two "big models", one is the question-answering big model, that is, the big model for the answer output of the entire question-answering system, and the other is the evaluation big model, that is, the big model for evaluating the entire question-answering system. The two models are independent of each other. It can be understood that one is the player (question-answering big model) and the other is the referee (referee big model).

[0197] The present disclosure realizes fully automated evaluation of the question-answering effect of the question-answering system by introducing a target evaluation model, helps application developers, deliverers, and users to realize the quality evaluation and attribution analysis of the question-answering effect of the question-answering system, optimizes the configuration of the question-answering system, enables the question-answering system to be quickly iterated and put into production, and can significantly reduce the labor cost and time cost of the evaluation method that originally relied on manual labeling.

[0198] Figure 6 FIG. 1 is a block diagram of a data processing device according to an exemplary embodiment. Figure 6 As shown, the device 600 mainly includes:

[0199] The acquisition module 601 is configured to process at least one query object in the first data set to obtain a first query result;

[0200] A first determination module 602 is configured to determine that the first query result does not meet a target query object of a preset indicator;

[0201] The second determination module 603 is configured to analyze the associated information of the target query object, determine the target event for which the first query result does not meet the preset index, and the influencing factors of the target event;

[0202] The evaluation module 604 is configured to obtain an evaluation result of the question-answering system based on the target event and the influencing factors of the target event.

[0203] In some embodiments, the first determining module 602 is configured to:

[0204] Determining a first target evaluation value of the first query result of the query object based on the first query result of the query object and the reference result of the query object;

[0205] The query object corresponding to the first query result whose first target evaluation value is less than the first threshold is determined as the target query object.

[0206] In some embodiments, the first determining module 602 is configured to:

[0207] Analyze the first query result of the query object and the reference result of the query object from at least two dimensions to obtain at least two evaluation values;

[0208] Based on at least one of the at least two evaluation values, the first target evaluation value is determined.

[0209] In some embodiments, the first determining module 602 is configured to:

[0210] Determining a first evaluation value based on a degree of match between the first query result of the query object and a reference result of the query object;

[0211] Determining a second evaluation value based on an inclusion relationship between the first query result of the query object and the reference result of the query object;

[0212] Determining a third evaluation value based on the first query result of the query object and the relevance of the query task indicated by the query object;

[0213] A fourth evaluation value is determined based on interference data in a first query result of the query object and a relevance of the query object; wherein the interference data is data in the first query result other than the reference result.

[0214] In some embodiments, the second determining module 603 is configured to:

[0215] Acquire a second query result generated based on the target query object; wherein the amount of information in the second query result is greater than the amount of information in the first query result;

[0216] The target event is determined based on a matching degree between the second query result of the target query object and the reference result of the target query object; wherein the target events determined by different matching degrees are different.

[0217] In some embodiments, the second determining module 603 is configured to:

[0218] When the matching degree of the second query result of the target query object and the reference result of the target query object is less than or equal to a first matching degree threshold, determining that a first target event exists; wherein the first target event indicates that a recall module of the question-answering system is abnormal;

[0219] When the degree of match between the second query result of the target query object and the reference result of the target query object is greater than the first matching degree threshold and less than the second matching degree threshold, it is determined that a second target event exists; wherein the second target event indicates that the corpus data of the question and answer system is abnormal.

[0220] In some embodiments, the second determining module 603 is configured to:

[0221] In the case where it is determined that the first target event exists, modifying the target query object based on the reference result of the target query object to obtain a third query object;

[0222] Processing the third query object based on the question-answering system to obtain a second query result of the third query object;

[0223] The second query result of the third query object is compared with the second query result of the target query object to obtain a comparison result, and an influencing factor of the first target event is determined based on the comparison result.

[0224] In some embodiments, the second determining module 603 is configured to:

[0225] When the matching degree between the second query result of the target query object and the reference result of the target query object is greater than the second matching degree threshold, determining the matching degree between the first query result of the target query object and the reference result of the target query object;

[0226] The target event is determined based on a matching degree between the first query result of the target query object and a reference result of the target query object.

[0227] In some embodiments, the apparatus 600 further includes:

[0228] A third determination module is configured to determine a second target evaluation value of the question-answering system based on the number of the target query objects and the total number of the query objects;

[0229] The second determining module 603 is configured to:

[0230] When the second target evaluation value is less than a second threshold, analyzing the associated information of the target query object processed by the question-answering system to determine the target event;

[0231] Based on the type of the target event, an influencing factor of the target event is determined; wherein the influencing factors of different types of target events are different.

[0232] In some embodiments, the apparatus 600 further includes:

[0233] a fourth determination module, configured to determine an adjustment strategy for the question-answering system based on the influencing factors of the target event and / or the type of the target event;

[0234] An adjustment module is configured to adjust the question-answering system based on the adjustment strategy until the second target evaluation value is greater than or equal to the second threshold.

[0235] In some embodiments, the apparatus 600 further includes:

[0236] a fifth determination module, configured to process the test object in the second data set based on the test question-answering system to obtain a first query result of the test object;

[0237] a sixth determination module configured to determine, for each of the test objects, a third target evaluation value of the test object based on the first query result of the test object and the test object reference result by the candidate evaluation model;

[0238] a seventh determination module configured to determine a target evaluation model from the candidate evaluation models based on the third target evaluation value and the reference evaluation value of the test object;

[0239] The second determining module is configured to:

[0240] Based on the target evaluation model, the associated information of the target query object processed by the question-answering system is analyzed to determine the target event.

[0241] In some embodiments, the seventh determining module is configured to:

[0242] determining a fourth target evaluation value of the candidate evaluation model based on the number of the test objects having the same reference evaluation value as the third target evaluation value and the total number of the test objects;

[0243] The target estimation model is determined from among the candidate estimation models based on each of the fourth target estimation values.

[0244] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0245] Based on the same inventive concept, an embodiment of the present disclosure further provides an electronic device, which may be a computer or a terminal in one or more of the above embodiments. Figure 7 FIG. 1 is a schematic diagram showing the structure of an electronic device according to an exemplary embodiment. Figure 7 As shown, the electronic device 700 adopts general computer hardware and includes a processor 701 , a memory 702 , a bus 703 , an input device 704 , and an output device 705 .

[0246] In some possible implementations, the memory 702 may include computer storage media in the form of volatile and / or non-volatile memory, such as read-only memory and / or random access memory. The memory 702 may store an operating system, application programs, other program modules, executable code, program data, user data, etc.

[0247] Input device 704 can be used to input commands and information to the electronic device, such as a keyboard or a pointing device, such as a mouse, trackball, touch pad, microphone, joystick, game pad, satellite TV antenna, scanner or similar device. Input device 704 can be connected to processor 701 via bus 703.

[0248] The output device 705 can be used to output information to the electronic device 700 . In addition to the monitor, the output device 705 can also be other peripheral output devices, such as a speaker and / or a printing device. The output device 705 can also be connected to the processor 701 via the bus 703 .

[0249] The electronic device 700 may be connected to a network, such as a local area network (LAN), via the antenna 706. In a networked environment, executable instructions may be stored in a remote storage device, rather than being limited to local storage.

[0250] When the processor 701 in the electronic device 700 executes the executable code or application stored in the memory 702, the electronic device 700 can implement the image processing method in the above embodiments. The specific execution process refers to the above embodiments and will not be repeated here.

[0251] The memory 702 may store information for implementing Figure 6 The executable instructions of the functions of the acquisition module 601, the first determination module 602, the second determination module 603 and the evaluation module 604 in the embodiment. Figure 6 The functions / implementation processes of the acquisition module 601, the first determination module 602, the second determination module 603 and the evaluation module 604 can be realized by Figure 7 The processor 701 in the embodiment calls the executable instructions stored in the memory 702 to implement the above-mentioned implementation process and functions.

[0252] Based on the same inventive concept, the embodiment of the present disclosure further provides a storage medium. The storage medium stores instructions. When the instructions are executed on a computer, the instructions are used to execute the data processing method in one or more of the above embodiments.

[0253] Based on the same inventive concept, the embodiments of the present disclosure further provide a computer program or a computer program product. When the computer program product is executed on a computer, the computer implements the data processing method in one or more of the above embodiments.

[0254] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

[0255] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A data processing method, characterized in that: include: Processing at least one query object in the first data set to obtain a first query result; Determine the target query object for which the first query result does not meet the preset indicator; Analyze the associated information of the target query object to determine the target event for which the first query result does not meet the preset index, and the influencing factors of the target event; Based on the target event and the influencing factors, an evaluation result is obtained.

2. The method according to claim 1, characterized in that The determining that the first query result does not meet the target query object of the preset index includes: Determining a first target evaluation value of the first query result of the query object based on the first query result of the query object and the reference result of the query object; The query object corresponding to the first query result whose first target evaluation value is less than the first threshold is determined as the target query object.

3. The method according to claim 2, characterized in that The determining, based on the first query result of the query object and the reference result of the query object, a first target evaluation value of the first query result of the query object comprises: Analyze the first query result of the query object and the reference result of the query object from at least two dimensions to obtain at least two evaluation values; Based on at least one of the at least two evaluation values, the first target evaluation value is determined.

4. The method according to claim 3, characterized in that The first query result of the query object and the reference result of the query object are analyzed from at least two dimensions to obtain at least two evaluation values, including: Determining a first evaluation value based on a degree of match between the first query result of the query object and a reference result of the query object; Determining a second evaluation value based on an inclusion relationship between the first query result of the query object and the reference result of the query object; Determining a third evaluation value based on the first query result of the query object and the relevance of the query task indicated by the query object; A fourth evaluation value is determined based on interference data in a first query result of the query object and a relevance of the query object; wherein the interference data is data in the first query result other than the reference result.

5. The method according to claim 1, characterized in that The analyzing the associated information of the target query object to determine that the first query result does not reach the target event of the preset indicator includes: Acquire a second query result generated based on the target query object; wherein the amount of information in the second query result is greater than the amount of information in the first query result; The target event is determined based on a matching degree between the second query result of the target query object and the reference result of the target query object; wherein the target events determined by different matching degrees are different.

6. The method according to claim 5, characterized in that The determining the target event based on the matching degree of the second query result of the target query object and the reference result of the target query object includes: When the matching degree of the second query result of the target query object and the reference result of the target query object is less than or equal to the first matching degree threshold, determining that a first target event exists; wherein the first target event indicates that a recall module of the question-answering system is abnormal; When the degree of match between the second query result of the target query object and the reference result of the target query object is greater than the first matching degree threshold and less than the second matching degree threshold, it is determined that a second target event exists; wherein the second target event indicates that the corpus data of the question and answer system is abnormal.

7. The method according to claim 6, characterized in that The determining of the influencing factors of the target event includes: In the case where it is determined that the first target event exists, modifying the target query object based on the reference result of the target query object to obtain a third query object; Processing the third query object to obtain a second query result of the third query object; The second query result of the third query object is compared with the second query result of the target query object to obtain a comparison result, and an influencing factor of the first target event is determined based on the comparison result.

8. The method according to claim 6, characterized in that The determining the target event based on the matching degree of the second query result of the target query object and the reference result of the target query object includes: When the matching degree between the second query result of the target query object and the reference result of the target query object is greater than the second matching degree threshold, determining the matching degree between the first query result of the target query object and the reference result; The target event is determined based on a matching degree between the first query result of the target query object and a reference result of the target query object.

9. The method according to claim 1, characterized in that: The method further comprises: Determining a second target evaluation value based on the number of the target query objects and the total number of the query objects; The analyzing the associated information of the target query object to determine the target event for which the first query result does not meet the preset index and the influencing factors of the target event includes: When the second target evaluation value is less than a second threshold, analyzing the associated information of the target query object to determine the target event; Based on the type of the target event, an influencing factor of the target event is determined; wherein the influencing factors of different types of target events are different.

10. The method according to claim 9, characterized in that The method further comprises: Determining an adjustment strategy for the question-answering system based on the influencing factors of the target event and / or determining the type of the target event; The question-answering system is adjusted based on the adjustment strategy until the second target evaluation value is greater than or equal to the second threshold.

11. The method according to claim 1, characterized in that: The method further comprises: Processing the test object in the second data set to obtain a first query result of the test object; For each of the test objects, the candidate evaluation model determines a third target evaluation value of the test object based on the first query result of the test object and the reference result of the test object; determining a target evaluation model from among the candidate evaluation models based on the third target evaluation value and the reference evaluation value of the test object; The analyzing the associated information of the target query object to determine that the first query result does not reach the target event of the preset indicator includes: The associated information of the target query object is analyzed based on the target evaluation model to determine the target event.

12. The method according to claim 11, characterized in that The step of determining a target evaluation model from the candidate evaluation models based on the third target evaluation value and the reference evaluation value of the test object comprises: determining a fourth target evaluation value of the candidate evaluation model based on the number of the test objects having the same reference evaluation value as the third target evaluation value and the total number of the test objects; The target estimation model is determined from among the candidate estimation models based on each of the fourth target estimation values.

13. A data processing device, characterized in that: include: An acquisition module is configured to acquire and process at least one query object in the first data set to obtain a first query result; A first determining module is configured to determine that the first query result does not meet a target query object of a preset indicator; A second determination module is configured to analyze the associated information of the target query object, determine the target event for which the first query result does not meet the preset index, and the influencing factors of the target event; The evaluation module is configured to obtain an evaluation result based on the target event and the influencing factors.

14. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 12 are implemented.

Images