Performance evaluation method of recall model and electronic equipment
By inputting the splicing of the target reply content and historical dialogue content into the recall model, generating a sample set and calculating accuracy evaluation indicators, the problem of inadequate performance evaluation of recall model in the existing technology is solved, and a more accurate and reliable performance evaluation of recall model is achieved. .
Patent Information
- Application Number
- CN202310632735.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-30
- Publication Date
- 2024-12-10
AI Technical Summary
The prior art is difficult to reliably evaluate the performance of the recall model, especially in the case of diverse chat content, where the candidate reply content of the recall model recalls both correct and accurate but incomplete reply, resulting in individual evaluation of whether the candidate reply content is The method of including the correct reply is not reliable enough.
By inputting the recall model the splicing of the target reply content and the preset historical dialogue content, a sample set of candidate reply content is generated, and the samples are input as negative samples into the pre-trained fine-scheduled model, and the accuracy evaluation indicators and weakened ability indicators are calculated to generate performance Evaluate the metrics to evaluate the performance of the recall model.
This method can more reliably evaluate the performance of the recall model, highlight the difference between accurate reply content and inaccurate reply content, and use the screening and sorting capabilities of the fine-scheduling model to quantify the performance of the recall model, which improves the reliability of performance evaluation and accuracy.
Smart Images
Figure CN119106270A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the technical field of natural language processing, and in particular to a performance evaluation method and electronic device for a recall model. Background Art
[0002] In customer service scenarios, the speech association technology can help agents intelligently recommend more appropriate and professional replies to agents during chats with customers based on the real-time chat content and the content the agents are typing. The speech association technology includes the recall stage and the fine ranking stage. In the recall stage, the recall model performs preliminary screening based on the context and keywords of the real-time chat, and uses information retrieval technology to quickly recall a batch of candidate replies related to the current problem from the huge reply library. In the fine ranking stage, the fine ranking model ranks and optimizes the candidate replies output by the recall model, and selects replies that meet the actual needs of the agents.
[0003] In order to evaluate the performance of the recall model, the tester constructs a test set, wherein the test set includes a correct reply content and multiple reply test contents, and inputs each reply content of the test set into the recall model so as to recall multiple candidate reply contents recalled by the model. The tester detects whether the multiple candidate reply contents include the above correct reply content in order to evaluate the performance of the recall model. However, due to the diversity of chat content, in the recall stage, among the multiple candidate reply contents recalled, not only the correct reply content exists that is suitable for the agent to reply, but also other candidate reply contents may exist that are suitable for the agent to reply. In other words, the multiple candidate reply contents may include not only the above correct reply content, but also the quasi-correct reply content that is suitable for the agent to reply. Alternatively, even if the multiple candidate reply contents do not include the above correct reply content, they can still include the quasi-correct reply content that is suitable for the agent to reply. If the performance of the recall model is evaluated solely according to the result of whether the multiple candidate reply contents include the above correct reply content, this approach is not conducive to reliably evaluating the performance of the recall model. Summary of the invention
[0004] One purpose of the embodiments of the present disclosure is to provide a performance evaluation method and electronic device for a recall model, aiming to solve the technical problem that related technologies are not easy to reliably evaluate the performance of a recall model.
[0005] In a first aspect, an embodiment of the present disclosure provides a performance evaluation method of a recall model, comprising:
[0006] Inputting the target reply content into the recall model, so that the recall model recalls a plurality of candidate reply contents according to the concatenation of the preset historical conversation content and the target reply content;
[0007] Generate a sample set according to the candidate reply content, wherein the sample set includes a plurality of samples;
[0008] Inputting samples of the sample set as negative samples into a pre-trained refined ranking model, so that the refined ranking model outputs a score for each of the samples;
[0009] Calculate the accuracy evaluation index according to the score of each of the samples;
[0010] A performance evaluation index is generated according to the accuracy evaluation index and the preset evaluation index of the refined ranking model, and the performance evaluation index is used to evaluate the performance of the recall model.
[0011] Optionally, the accuracy evaluation index includes a weakening capability upper limit index, and the accuracy evaluation index calculated according to the score of each sample includes:
[0012] When the target reply content is a complete reply content, the capability reduction upper limit indicator is calculated according to the score of each sample.
[0013] Optionally, the accuracy evaluation index includes a weakening capability upper limit index, and generating a performance evaluation index according to the accuracy evaluation index and a preset evaluation index of the refined ranking model includes:
[0014] Calculate the difference between the preset evaluation index of the refined ranking model and the upper limit index of the weakening ability to obtain an upper limit index of the weakening degree, wherein the upper limit index of the weakening degree is used to indicate the degree to which the accuracy of the refined ranking model is weakened when the samples of the sample set are used as negative samples;
[0015] A performance evaluation index is generated according to the upper limit index of the weakening degree.
[0016] Optionally, the accuracy evaluation index includes a weakening capability index, and the accuracy evaluation index calculated according to the score of each sample includes:
[0017] When the target reply content is an incomplete reply content, the weakening capability index is calculated according to the score of each of the samples.
[0018] Optionally, the accuracy evaluation index includes a weakening capability index, and generating a performance evaluation index according to the weakening degree upper limit index includes:
[0019] Calculating the difference between the preset evaluation index of the refined sorting model and the weakening capability index to obtain a weakening degree index;
[0020] The ratio of the attenuation degree index to the attenuation degree upper limit index is calculated to obtain an attenuation ratio, which is a performance evaluation index.
[0021] Optionally, when the attenuation ratio is larger, the performance of the recall model is better;
[0022] When the attenuation ratio is smaller, the performance of the recall model is worse.
[0023] Optionally, the method further comprises:
[0024] When the target reply content is an incomplete reply content, obtaining a positive sample;
[0025] Then: the step of inputting the samples of the sample set as negative samples into the pre-trained refined ranking model so that the refined ranking model outputs the scores of each of the samples comprises:
[0026] Setting the samples in the sample set as negative samples;
[0027] Each of the negative samples and the positive samples is input into a pre-trained refined ranking model, so that the refined ranking model outputs a score for each of the samples.
[0028] Optionally, inputting the target reply content into the recall model so that the recall model recalls a plurality of candidate reply contents according to the concatenation of the preset historical conversation content and the target reply content comprises:
[0029] The target reply content is input into the recall model so that the recall model searches for preset historical conversation content in a preset inverted index library according to the target reply content, and recalls multiple candidate reply contents according to the concatenation of the preset historical conversation content and the target reply content.
[0030] Optionally, the method further comprises:
[0031] Get the conversation log;
[0032] Analyze the historical conversation content and historical reply content according to the conversation log;
[0033] Constructing indexes for the historical conversation content and the historical reply content respectively;
[0034] Generate an inverted index library based on the indexed historical conversation content and the indexed historical reply content.
[0035] Optionally, generating a sample set according to the candidate reply content includes:
[0036] The historical conversation content and the candidate reply content are concatenated to obtain a sample, and samples corresponding to a plurality of the candidate reply contents form a sample set.
[0037] In a second aspect, an embodiment of the present disclosure provides a non-volatile readable storage medium storing computer executable instructions, wherein the computer executable instructions are used to enable an electronic device to execute the above-mentioned recall model performance evaluation method.
[0038] In a third aspect, an embodiment of the present disclosure provides an electronic device, including:
[0039] at least one processor; and,
[0040] a memory communicatively connected to the at least one processor; wherein,
[0041] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the performance evaluation method of the recall model mentioned above.
[0042] In the performance evaluation method of the recall model provided by the embodiment of the present disclosure, first, the embodiment inputs the target reply content into the recall model, so that the recall model recalls multiple candidate reply contents according to the spliced content of the preset historical conversation content and the target reply content, generates a sample set according to the candidate reply content, and the sample set includes multiple samples, and inputs the samples of the sample set as negative samples into the pre-trained fine ranking model, so that the fine ranking model outputs the score of each sample. In this embodiment, the samples of the sample set are all set as negative samples, so that the difference between the accurate reply content and the inaccurate reply content in the fine ranking model can be relatively highlighted, that is, the accuracy evaluation index with obvious difference can be obtained. Secondly, the embodiment calculates the accuracy evaluation index according to the score of each sample, generates the performance evaluation index according to the accuracy evaluation index and the preset evaluation index of the fine ranking model, and the performance evaluation index is used to evaluate the performance of the recall model. The embodiment can make use of the good screening ability and sorting ability of the fine ranking model, and quantitatively express the sorting result of the fine ranking model using the accuracy evaluation index, and then use the accuracy evaluation index and the preset evaluation index to generate the performance evaluation index, and evaluate the performance of the recall model in the form of negative feedback, so that the performance of the recall model can be relatively easy and relatively reliable. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.
[0044] Figure 1 A flowchart of a performance evaluation method for a recall model provided in an embodiment of the present disclosure;
[0045] Figure 2 A schematic diagram of a conversation scenario provided by an embodiment of the present disclosure;
[0046] Figure 3 A schematic diagram of a circuit structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION
[0047] In order to make the purpose, technical solutions and advantages of the present disclosure more clearly understood, the present disclosure is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present disclosure and are not intended to limit the present disclosure. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present disclosure.
[0048] It should be noted that, if there is no conflict, the various features in the embodiments of the present disclosure can be combined with each other, all within the scope of protection of the present disclosure. In addition, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flow chart, in some cases, the steps shown or described can be performed in a different order than the module division in the device or the flow chart. Furthermore, the words "first", "second", "third", etc. used in the present disclosure do not limit the data and execution order, but only distinguish the same items or similar items with basically the same functions and effects.
[0049] The present disclosure provides a method for evaluating the performance of a recall model. Figure 1 ,The performance evaluation method of the recall model includes the following steps:
[0050] S11: Input the target reply content into the recall model, so that the recall model recalls a plurality of candidate reply contents according to the concatenation of the preset historical conversation content and the target reply content.
[0051] In this step, the recall model is a model that can recall multiple candidate reply contents in a preset candidate reply library by combining historical conversation contents and target reply contents input by the agent. The recall model can support corresponding recall algorithms, including single-tower recall algorithms or dual-tower recall algorithms.
[0052] The target reply content is the content input by the agent. For example, customer A initiates a conversation, where the conversation is as follows:
[0053] Customer A: Found that students cannot log in
[0054] Seat B: Are you talking about seewo cloud class?
[0055] In the above conversation, the target's reply was "Are you talking about seewo Cloud Class?"
[0056] It is understandable that the target reply content may be a complete reply content, and the complete reply content is the complete content of the agent's reply to the customer's question. For example, in the above conversation, "Are you talking about seewo Cloud Class?" is the complete reply content.
[0057] It is also understandable that the target reply content may also be an incomplete reply content. The incomplete reply content is the existing content that the agent is currently typing but has not yet finished typing when answering a customer's question. For example, in the above conversation, "You mean Xi" is an incomplete reply content.
[0058] The preset historical conversation content is the historical content of the conversation between the agent and the customer. It can be understood that the present embodiment can intercept the historical conversation content of the specified round as the preset historical conversation content, and the specified round can be 6 rounds or 8 rounds, etc. For example, the agent B and the customer A have had 40 rounds of conversation. The present embodiment can intercept the historical conversation content of the most recent 6 rounds as the preset historical conversation content.
[0059] See also Figure 2 , the contents of the last six rounds of historical conversations between agent B and customer A are as follows:
[0060] Customer A: Found that students cannot log in
[0061] Seat B: Are you talking about seewo cloud class?
[0062] Customer A: Yes
[0063] Seat B: I found that the student cannot log in. Hi, could you provide a picture?
[0064] Agent B: Let me show you where you can't log in.
[0065] Customer A: Do you provide a login page image?
[0066] The concatenated content is obtained by concatenating the preset historical conversation content and the target reply content. It can be understood that the concatenation order can be the arrangement order of the preset historical conversation content and the target reply content, or the arrangement order of the target reply content and the preset historical conversation content. The candidate reply content is the content recalled by the recall model based on the concatenated content.
[0067] For example, when the target response content is "yes", the spliced content is "I found that students cannot log in. Are you talking about Seewo Cloud Class? Yes, I found that students cannot log in. Hello, could you provide a picture? I will show you specifically where you cannot log in. Can you provide a picture of the login page? Yes." The recall model recalls multiple candidate response contents based on the above spliced content.
[0068] In some embodiments, this embodiment inputs target reply content into the recall model, and the recall model searches for preset historical conversation content in a preset inverted index library based on the target reply content, and recalls multiple candidate reply contents based on the concatenation of the preset historical conversation content and the target reply content.
[0069] In some embodiments, the performance evaluation method of the recall model also includes: obtaining conversation logs, parsing historical conversation content and historical reply content based on the conversation logs, indexing the historical conversation content and the historical reply content respectively, and generating an inverted index library based on the indexed historical conversation content and the indexed historical reply content.
[0070] The conversation log is a log for recording preset historical conversation content. This embodiment uses the open source tool ElasticSearch to build an inverted index. A conversation log of a complete conversation will be cut into multiple logs containing historical conversation content and historical reply content. ElasticSearch builds indexes for historical conversation content and historical reply content respectively. The maximum length of historical conversation content is the conversation content of the most recent 6 rounds, and the historical reply content is the real reply of the agent.
[0071] This embodiment Figure 2 The index constructed by the historical conversation content and historical reply content is shown in Table 1:
[0072] Table 1
[0073]
[0074] This embodiment constructs an inverted index library, which is not only conducive to quickly retrieving preset historical conversation content, but also conducive to quickly generating target reply content of different lengths, and is conducive to providing rich and diverse target reply content so that the recall model can recall multiple candidate reply contents corresponding to different target reply contents, thereby improving the evaluation efficiency of the performance of the recall model.
[0075] S12: Generate a sample set according to the candidate reply content, where the sample set includes multiple samples.
[0076] In this step, the sample set is a collection of multiple samples, wherein, in some embodiments, generating a sample set based on candidate reply content includes: splicing historical conversation content with candidate reply content to obtain samples, and the samples corresponding to multiple candidate reply contents form a sample set. This embodiment splices and fuses historical conversation content with candidate reply content, so that the refined ranking model can more accurately and reliably screen candidate reply content.
[0077] For example, the sample set includes n samples, the historical conversation content is f0, and the candidate reply contents are: f1, f2, f3...fn, sample 1 is f0+f1, sample 2 is f0+f2, sample 3 is f0+f3, and sample n is f0+fn.
[0078] S13: Input the samples of the sample set as negative samples into the pre-trained refined ranking model, so that the refined ranking model outputs the score of each sample.
[0079] In this step, the refined ranking model is a model constructed by a preset refined ranking algorithm, wherein the preset refined ranking algorithm includes LR algorithm, FTRL algorithm, FM algorithm, Wide&Deep algorithm, DeepFM algorithm, etc. When training the refined ranking model, this embodiment can use a positive sample combined with a negative sample, or a positive sample combined with multiple negative samples, or multiple positive samples combined with multiple negative samples to train the refined ranking model, where the positive sample is a sample that meets the agent's reply requirements, and the negative sample is a sample selected from multiple candidate reply contents recalled by the recall model according to a random sampling method.
[0080] In this embodiment, the following evaluation indicators can be used to evaluate the performance of the refined ranking model, including the MAP indicator, the MRR indicator, the P@1 indicator, the R2 indicator and the R5 indicator. Among them, the MAP indicator is the average accuracy rate. In this embodiment, the accuracy rate is calculated based on the candidate reply contents ranked in the top 10 by score. The MRR indicator is the average value of the reciprocal of the ranking of the correct results. The P@1 indicator is the accuracy rate of the top 1 structure, the R2 indicator is the recall rate of the top 2 results, and the R5 indicator is the recall rate of the top 5 results.
[0081] The refined ranking model used in this embodiment can be a pre-trained BERT language model, with a total training data of 2 million samples. The evaluation indicators of the pre-trained refined ranking model are shown in Table 2:
[0082] Table 2
[0083] MAP MRR P@1 R2 R5 0.980 0.980 0.963 0.992 0.998
[0084] As shown in Table 2, the MAP index is 0.980, the MRR index is 0.980, the P@1 index is 0.963, the R2 index is 0.992, and the R5 index is 0.998.
[0085] In some embodiments, when the target reply content is a complete reply content, this embodiment directly uses all samples in the sample set as negative samples, and directly inputs all negative samples into the pre-trained refined ranking model so that the refined ranking model outputs the score of each sample.
[0086] It is understandable that when the target reply content is a complete reply content, the multiple candidate reply contents recalled by the recall model are likely to include candidate reply contents with a high similarity to the target reply content. When the candidate reply contents with a high similarity to the target reply content are input into the pre-trained fine ranking model as negative samples, this is equivalent to inputting the "worst sample" into the fine ranking model to minimize the precision evaluation index of the fine ranking model, that is, the precision evaluation index of the fine ranking model obtained at this time is obtained when the fine ranking model is in the "worst case". When the precision evaluation index is any one of the MAP index, MRR index, P@1 index, R2 index and R5 index, the MAP index, MRR index, P@1 index, R2 index and R5 index at this time are all obtained when the fine ranking model is in the "worst case".
[0087] In some embodiments, when the target reply content is an incomplete reply content, this embodiment obtains a positive sample. Then: inputting the samples of the sample set as negative samples into the pre-trained fine ranking model so that the fine ranking model outputs the score of each sample includes: setting the samples of the sample set as negative samples, inputting each negative sample and positive sample into the pre-trained fine ranking model so that the fine ranking model outputs the score of each sample.
[0088] It can be understood that when the target reply content is incomplete reply content, relative to the case where the target reply content is complete reply content, the multiple candidate reply contents recalled by the recall model are candidate reply contents with lower similarity to the target reply content. When the candidate reply contents with lower similarity to the target reply content are input into the pre-trained fine ranking model as negative samples, relatively speaking, this is equivalent to inputting "non-worst samples" into the fine ranking model. At this time, the accuracy evaluation index of the fine ranking model is obtained under the "non-worst case" of the fine ranking model.
[0089] S14: Calculate the accuracy evaluation index according to the score of each sample.
[0090] In this step, the accuracy evaluation index is an evaluation index calculated based on the scores of each sample. The accuracy evaluation index can be any one of the MAP index, the MRR index, the P@1 index, the R2 index and the R5 index, or the accuracy evaluation index includes any two or more indicators.
[0091] When the accuracy evaluation index includes the MAP index, this embodiment processes the score of each sample according to the MAP algorithm to obtain the MAP index. When the accuracy evaluation index includes the MRR index, this embodiment processes the score of each sample according to the MRR algorithm to obtain the MRR index. When the accuracy evaluation index includes the P@1 index, this embodiment processes the score of each sample according to the P@1 algorithm to obtain the P@1 index. When the accuracy evaluation index includes the R2 index, this embodiment processes the score of each sample according to the R2 algorithm to obtain the R2 index. When the accuracy evaluation index includes the R5 index, this embodiment processes the score of each sample according to the R5 algorithm to obtain the R5 index.
[0092] S15: Generate a performance evaluation index according to the accuracy evaluation index and the preset evaluation index of the refined ranking model, and the performance evaluation index is used to evaluate the performance of the recall model.
[0093] In this step, the preset evaluation index is the original evaluation index of the refined ranking model. For example, please refer to Table 2. The MAP index, MRR index, P@1 index, R2 index and R5 index shown in Table 2 are all preset evaluation indexes. The performance evaluation index is an index for evaluating the performance of the recall model. It is understandable that those skilled in the art can generate an index in a suitable form as a performance evaluation index based on the accuracy evaluation index and the preset evaluation index.
[0094] This embodiment sets all samples in the sample set as negative samples, so that the difference between accurate response content and inaccurate response content in the refined ranking model can be more prominent, that is, the precision evaluation index with obvious difference can be obtained. At the same time, this embodiment can make use of the good screening and sorting capabilities of the refined ranking model, and quantify the sorting results of the refined ranking model using the precision evaluation index, and then use the precision evaluation index and the preset evaluation index to generate the performance evaluation index, and evaluate the performance of the recall model in the form of negative feedback, so that the performance of the recall model can be evaluated relatively easily and reliably.
[0095] In some embodiments, the precision evaluation index includes an upper limit indicator of weakening capability, and the upper limit indicator of weakening capability is: when the agent inputs a complete reply content, the recalled candidate reply content is used as a negative sample to weaken the evaluation index of the precision ranking model.
[0096] The accuracy evaluation index calculated based on the score of each sample includes: when the target response content is a complete response content, the weakening ability upper limit index is calculated based on the score of each sample.
[0097] When the weakened capability upper limit indicator includes the MAP upper limit indicator, this embodiment processes the score of each sample according to the MAP algorithm to obtain the MAP upper limit indicator. When the weakened capability upper limit indicator includes the MRR upper limit indicator, this embodiment processes the score of each sample according to the MRR algorithm to obtain the MRR upper limit indicator. When the weakened capability upper limit indicator includes the P@1 upper limit indicator, this embodiment processes the score of each sample according to the P@1 algorithm to obtain the P@1 upper limit indicator. When the weakened capability upper limit indicator includes the R2 upper limit indicator, this embodiment processes the score of each sample according to the R2 algorithm to obtain the R2 upper limit indicator. When the weakened capability upper limit indicator includes the R5 upper limit indicator, this embodiment processes the score of each sample according to the R5 algorithm to obtain the R5 upper limit indicator.
[0098] See Table 3:
[0099] Table 3
[0100] MAP MRR P@1 R2 R5 0.514 0.514 0.326 0.475 0.783
[0101] As shown in Table 3, when the target reply content is a complete reply content, the recall model recalls multiple candidate reply contents based on the concatenation of the preset historical conversation content and the complete reply content, generates a sample set based on the candidate reply content, and inputs the samples of the sample set as negative samples into the fine ranking model. The fine ranking model outputs the score of each sample. This embodiment calculates the precision evaluation index shown in Table 3 based on the score of each sample. The precision evaluation index shown in Table 3 is the upper limit of the weakening capability.
[0102] As shown in Table 3, the accuracy evaluation index at this time is much lower than the preset evaluation index in Table 2. The accuracy evaluation index shown in Table 3 represents the evaluation index when the "worst sample" is input into the refined ranking model. Therefore, the accuracy evaluation index shown in Table 3 can be regarded as the original evaluation index of the refined ranking model after being weakened to the greatest extent.
[0103] In some embodiments, the accuracy evaluation index includes an upper limit indicator of weakening capability. Generating a performance evaluation index based on the accuracy evaluation index and a preset evaluation index of the refined ranking model includes the following steps: calculating the difference between the preset evaluation index of the refined ranking model and the upper limit indicator of weakening capability to obtain an upper limit indicator of weakening degree, the upper limit indicator of weakening degree is used to indicate the degree to which the accuracy of the refined ranking model is weakened when the samples in the sample set are used as negative samples, and generating a performance evaluation index based on the upper limit indicator of weakening degree.
[0104] It is understandable that as the number of words in the target reply content input by the agent increases, the candidate reply content given by the speech association will become more and more appropriate, that is, the recall model will have higher and higher retrieval performance under the premise that the number of words in the target reply content increases. In the related art, if the samples in the sample set are not all set as negative samples, and the fine ranking model is controlled to process each sample in a normal manner, then: when the performance of the recall model is higher, the recalled candidate reply content is more accurate. For the fine ranking model, the more accurate the candidate reply content will make it difficult for the fine ranking model to distinguish between the positive sample (the agent's real reply content) and the more accurate candidate reply content. In other words, the more accurate the candidate reply content is, the more "confusing" it is to the fine ranking model, and the less accurate the candidate reply content is, the less "confusing" it is to the fine ranking model, making it easier for the fine ranking model to distinguish between the agent's real reply content and the less accurate candidate reply content.
[0105] Since this embodiment is to provide a technical solution to evaluate the performance of the recall model, and utilize the good screening and sorting capabilities of the refined ranking model to evaluate the performance of the recall model in the form of negative feedback, this embodiment can first provide complete reply content as the target reply content, and set all samples as negative samples, so that the upper limit indicator of the weakening ability of the refined ranking model can be detected, and then the difference between the upper limit indicator of the weakening ability and the preset evaluation indicator of the refined ranking model is calculated to obtain the upper limit indicator of the weakening degree.
[0106] See Table 4:
[0107] Table 4
[0108]
[0109] As shown in Table 4, the upper limit indicators of the weakening degree include the upper limit indicator of MAP weakening degree, the upper limit indicator of MRR weakening degree, the upper limit indicator of P@1 weakening degree, the upper limit indicator of R2 weakening degree and the upper limit indicator of R5 weakening degree. When the target response content is the complete response content, the upper limit indicator of MAP weakening degree is 0.466, the upper limit indicator of MRR weakening degree is 0.466, the upper limit indicator of P@1 weakening degree is 0.634, the upper limit indicator of R2 weakening degree is 0.517, and the upper limit indicator of R5 weakening degree is 0.215.
[0110] Since the preset evaluation index is fixed, the precision evaluation index is the evaluation index when the target response content is the complete response content and the precision ranking model is input with the "worst sample". The "worst sample" is the sample that reverses the candidate response content output by the recall model. Therefore, when the upper limit of the weakening degree index is larger, it means that the "worst sample" is worse, the candidate response content is more accurate, and the recall performance is higher. When the upper limit of the weakening degree index is smaller, it means that the "worst sample" is worse, the candidate response content is more accurate, and the recall performance is lower.
[0111] In some embodiments, generating a performance evaluation indicator according to an upper limit indicator of the degree of weakening includes: using the upper limit indicator of the degree of weakening as a performance evaluation indicator.
[0112] In some embodiments, the precision evaluation index includes a weakening capability index, and the precision evaluation index calculated according to the score of each sample includes: when the target reply content is an incomplete reply content, the weakening capability index is calculated according to the score of each sample. The weakening capability index is: when the agent inputs different incomplete reply contents, the recalled candidate reply content as a negative sample weakens the evaluation index of the refined ranking model.
[0113] When the weakening capability indicator includes the MAP indicator, this embodiment processes the score of each sample according to the MAP algorithm to obtain the MAP indicator. When the weakening capability indicator includes the MRR indicator, this embodiment processes the score of each sample according to the MRR algorithm to obtain the MRR indicator. When the weakening capability indicator includes the P@1 indicator, this embodiment processes the score of each sample according to the P@1 algorithm to obtain the P@1 indicator. When the weakening capability indicator includes the R2 indicator, this embodiment processes the score of each sample according to the R2 algorithm to obtain the R2 indicator. When the weakening capability indicator includes the R5 indicator, this embodiment processes the score of each sample according to the R5 algorithm to obtain the R5 indicator.
[0114] See Table 5:
[0115] Table 5
[0116] Word count for incomplete responses MAP MRR P@1 R2 R5 0 0.780 0.792 0.773 0.834 0.936 1 0.619 0.620 0.511 0.602 0.901 3 0.559 0.559 0.347 0.562 0.884
[0117] As shown in Table 5, when the target reply content is an incomplete reply content, the number of words in the incomplete reply content can be 0, that is, the agent has not entered any reply content. In the scenario of speech association, the actual business pays more attention to the speech association effect when the agent enters a small amount of target reply content. If the performance of the recall model when the agent has no input is consistent with that when the input is complete, it means that the recall model performs better.
[0118] The number of words in the incomplete reply content can also be 1 or 3, such as "you" or "you said". The recall model recalls multiple candidate reply contents based on the concatenation of the preset historical conversation content and the incomplete reply content, generates a sample set based on the candidate reply content, uses the samples in the sample set as negative samples, and inputs each negative sample and positive sample into the refined ranking model. The refined ranking model outputs the score of each sample. This embodiment calculates the precision evaluation index shown in Table 5 based on the score of each sample. As shown in Table 5, the precision evaluation index at this time is much lower than the preset evaluation index in Table 2, but the precision evaluation index shown in Table 5 is higher than the precision evaluation index shown in Table 3. The precision evaluation index shown in Table 5 represents the evaluation index when "non-worst samples" are input into the refined ranking model.
[0119] In some embodiments, the accuracy evaluation index includes a weakening capability index, and generating a performance evaluation index based on the weakening degree upper limit index includes: calculating the difference between the preset evaluation index of the precision ranking model and the weakening capability index to obtain the weakening degree index, calculating the ratio of the weakening degree index to the weakening degree upper limit index to obtain the weakening ratio, and the weakening ratio is the performance evaluation index.
[0120] The weakening ratio is: when the agent inputs different incomplete reply contents, the ratio of the weakening degree index of the recalled candidate reply contents as negative samples to the weakening degree upper limit index of the evaluation index of the refined ranking model. When the weakening ratio is greater than 1, this embodiment still sets the weakening ratio at this time to 1, where the value range of the weakening ratio is 0-1.
[0121] Please refer to Table 6. The weakening degree index is shown in Table 6:
[0122] Table 6
[0123]
[0124] As shown in Table 6, the weakening degree indicators include MAP weakening degree indicator, MRR weakening degree indicator, P@1 weakening degree indicator, R2 weakening degree indicator and R5 weakening degree indicator. Taking the number of words in the target reply content as 1 as an example, the MAP weakening degree indicator is 0.361, the upper limit indicator of MRR weakening degree is 0.360, the upper limit indicator of P@1 weakening degree is 0.452, the upper limit indicator of R2 weakening degree is 0.390, and the upper limit indicator of R5 weakening degree is 0.097.
[0125] Please refer to Table 7. The weakening ratio is as shown in Table 7:
[0126] Table 7
[0127]
[0128] As shown in Table 7, the reduction ratios include MAP reduction ratio, MRR reduction ratio, P@1 reduction ratio, R2 reduction ratio and R5 reduction ratio. Taking the target reply content as 1 word as an example, the MAP reduction ratio is 0.775, the MRR reduction ratio is 0.773, the P@1 reduction ratio is 0.710, the R2 reduction ratio is 0.754, and the R5 reduction ratio is 0.451.
[0129] As can be seen from Table 7, as the number of words in the target reply content input by the agent increases, the weakening ratio tends to increase. Among them, when the weakening ratio is larger, the performance of the recall model is better, and when the weakening ratio is smaller, the performance of the recall model is worse. In this way, the performance of the recall model can be dynamically characterized, so that the performance of the recall model can be comprehensively evaluated from the range of the number of words in the target reply content from 0 to the full number of words.
[0130] In general, this embodiment sets all samples in the sample set as negative samples, so that the difference between accurate response content and inaccurate response content in the refined ranking model can be relatively more prominent, that is, more significantly different precision evaluation indicators can be obtained. In addition, this embodiment can make use of the good screening and sorting capabilities of the refined ranking model, and quantitatively express the sorting results of the refined ranking model using precision evaluation indicators, and then use the precision evaluation indicators and preset evaluation indicators to generate performance evaluation indicators, and evaluate the performance of the recall model in the form of negative feedback, so that the performance of the recall model can be relatively easy and relatively reliable.
[0131] It should be noted that, in each of the above-mentioned embodiments, there is not necessarily a certain order between the above-mentioned steps. A person skilled in the art can understand, based on the description of the embodiments of the present disclosure, that in different embodiments, the above-mentioned steps may have different execution orders, that is, they may be executed in parallel, may be executed interchangeably, and so on.
[0132] See also Figure 3 , Figure 3 The following is a schematic diagram of a circuit structure of an electronic device provided by an embodiment of the present disclosure. Figure 3 As shown, the electronic device 300 includes one or more processors 31 and a memory 32. Figure 3 A processor 31 is taken as an example.
[0133] The processor 31 and the memory 32 may be connected via a bus or other means. Figure 3 The example of connecting through bus is taken in the following.
[0134] The memory 32, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions / modules corresponding to the performance evaluation method of the recall model in the embodiment of the present disclosure. The processor 31 executes various functional applications and data processing of the performance evaluation device of the recall model by running the non-volatile software programs, instructions and modules stored in the memory 32, that is, realizes the performance evaluation method of the recall model provided by the above method embodiment and the functions of each module or unit of the above device embodiment.
[0135] The memory 32 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage devices. In some embodiments, the memory 32 may optionally include a memory remotely arranged relative to the processor 31, and these remote memories may be connected to the processor 31 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0136] The program instructions / modules are stored in the memory 32 , and when executed by the one or more processors 31 , the performance evaluation method of the recall model in any of the above method embodiments is executed.
[0137] The present disclosure also provides a non-volatile computer storage medium, wherein the computer storage medium stores computer executable instructions, and the computer executable instructions are executed by one or more processors, such as Figure 3 A processor 31 in the embodiment may enable the one or more processors to execute the performance evaluation method of the recall model in any of the above method embodiments.
[0138] An embodiment of the present disclosure also provides a computer program product, which includes a computer program stored on a non-volatile computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by an electronic device, the electronic device executes any one of the performance evaluation methods for the recall model.
[0139] The above described device or equipment embodiments are merely illustrative, wherein the unit modules described as separate components may or may not be physically separated, and the components displayed as module units may or may not be physical units, that is, they may be located in one place, or may be distributed on multiple network module units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.
[0140] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.
[0141] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present disclosure, rather than to limit them. Under the concept of the present disclosure, the technical features in the above embodiments or different embodiments may also be combined, the steps may be implemented in any order, and there are many other changes in different aspects of the present disclosure as described above, which are not provided in detail for the sake of simplicity. Although the present disclosure has been described in detail with reference to the aforementioned embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present disclosure.
Claims
1. A performance evaluation method for a recall model, characterized in that: include: Inputting the target reply content into the recall model, so that the recall model recalls a plurality of candidate reply contents according to the concatenation of the preset historical conversation content and the target reply content; Generate a sample set according to the candidate reply content, wherein the sample set includes a plurality of samples; Inputting samples of the sample set as negative samples into a pre-trained refined ranking model, so that the refined ranking model outputs a score for each of the samples; Calculate the accuracy evaluation index according to the score of each of the samples; A performance evaluation index is generated according to the accuracy evaluation index and the preset evaluation index of the refined ranking model, and the performance evaluation index is used to evaluate the performance of the recall model.
2. The method according to claim 1, characterized in that The accuracy evaluation index includes a weakening capability upper limit index, and the accuracy evaluation index calculated according to the score of each sample includes: When the target reply content is a complete reply content, the capability reduction upper limit indicator is calculated according to the score of each sample.
3. The method according to claim 1, characterized in that The precision evaluation index includes a weakening capability upper limit index, and the performance evaluation index generated according to the precision evaluation index and the preset evaluation index of the refined ranking model includes: Calculate the difference between the preset evaluation index of the refined ranking model and the upper limit index of the weakening ability to obtain an upper limit index of the weakening degree, wherein the upper limit index of the weakening degree is used to indicate the degree to which the accuracy of the refined ranking model is weakened when the samples of the sample set are used as negative samples; A performance evaluation index is generated according to the upper limit index of the weakening degree.
4. The method according to claim 3, characterized in that The accuracy evaluation index includes a weakening capability index, and the performance evaluation index generated according to the weakening degree upper limit index includes: Calculating the difference between the preset evaluation index of the refined sorting model and the weakening capability index to obtain a weakening degree index; The ratio of the attenuation degree index to the attenuation degree upper limit index is calculated to obtain an attenuation ratio, which is a performance evaluation index.
5. The method according to claim 4, characterized in that When the attenuation ratio is larger, the performance of the recall model is better; When the attenuation ratio is smaller, the performance of the recall model is worse.
6. The method according to claim 1, characterized in that The accuracy evaluation index includes a weakening capability index, and the accuracy evaluation index calculated according to the score of each sample includes: When the target reply content is an incomplete reply content, the weakening capability index is calculated according to the score of each of the samples.
7. The method according to claim 1, characterized in that Also includes: When the target reply content is an incomplete reply content, obtaining a positive sample; Then: the step of inputting the samples of the sample set as negative samples into the pre-trained refined ranking model so that the refined ranking model outputs the scores of each of the samples comprises: Setting the samples in the sample set as negative samples; Each of the negative samples and the positive samples is input into a pre-trained refined ranking model, so that the refined ranking model outputs a score for each of the samples.
8. The method according to claim 1, characterized in that The step of inputting the target reply content into the recall model so that the recall model recalls a plurality of candidate reply contents according to the concatenation of the preset historical conversation content and the target reply content includes: The target reply content is input into the recall model so that the recall model searches for preset historical conversation content in a preset inverted index library according to the target reply content, and recalls multiple candidate reply contents according to the concatenation of the preset historical conversation content and the target reply content.
9. The method according to claim 8, characterized in that Also includes: Get the conversation log; Analyze the historical conversation content and historical reply content according to the conversation log; Constructing indexes for the historical conversation content and the historical reply content respectively; Generate an inverted index library based on the indexed historical conversation content and the indexed historical reply content.
10. The method according to any one of claims 1 to 9, characterized in that: Generating a sample set according to the candidate reply content includes: The historical conversation content and the candidate reply content are concatenated to obtain a sample, and samples corresponding to a plurality of the candidate reply contents form a sample set.
11. A non-volatile readable storage medium, characterized in that: Computer executable instructions are stored, and the computer executable instructions are used to enable the electronic device to execute the performance evaluation method of the recall model as described in any one of claims 1 to 10.
12. An electronic device, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the performance evaluation method of the recall model as described in any one of claims 1 to 9.