Service scoring model evaluation method and device, storage medium and computer equipment

Abstract

According to the service scoring model evaluation method and device, the storage medium and the computer equipment provided by the invention, during model evaluation, the service dialogue set is firstly determined, the to-be-evaluated service scoring model and the plurality of verification scoring models are obtained, and the service dialogue set is respectively input into the service scoring model and each verification scoring model; obtaining a first scoring sequence output by the service scoring model and a second scoring sequence output by each verification scoring model; then, the Spearman similarity between the first scoring list and each second scoring sequence is determined, the scoring precision of the service scoring model is determined based on each Spearman similarity, then an evaluation result is generated, and the Spearman similarity pays attention to the relative position relation between data values instead of actual data values, so that the evaluation accuracy of the service scoring model is improved. Therefore, it can be avoided that the evaluation of the service scoring model is affected by the difference caused by the fact that the verification scoring model is not matched with the service scene due to different training conditions, and the evaluation precision of the service scoring model is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a service scoring model evaluation method, device, storage medium and computer equipment. Background Art

[0002] In today's digital age, the quality of service is directly related to the market competitiveness and user loyalty of enterprises. In order to accurately grasp user needs and continuously optimize shopping guide intelligent services, service scoring models have emerged and become an indispensable part of the optimization process of shopping guide intelligent services. This model uses big data analysis capabilities to extract key information from the service dialogue between shopping guide intelligent services and users, and conducts deep learning and reasoning on it to achieve accurate scoring of service quality.

[0003] With the rapid development of the market and the continuous increase in service scenarios, the diversification of user needs requires shopping guide intelligent services to have greater flexibility and adaptability, so as to provide better services for different types of service scenarios and user groups. In this context, the market's demand for the evaluation accuracy of the service scoring model that will be put into use has increased accordingly. However, the evaluation accuracy of the current service scoring model evaluation method still cannot meet market demand, resulting in poor optimization effects of shopping guide intelligent services. Summary of the invention

[0004] The purpose of this application is to solve at least one of the above-mentioned technical defects, especially the technical defect that the evaluation accuracy of the evaluation method of the service scoring model in the prior art still cannot meet the market demand, resulting in poor optimization effect of the shopping guide intelligent service.

[0005] This application provides a service scoring model evaluation method, the method comprising:

[0006] Determine the service dialogue set, and obtain the service scoring model to be evaluated and multiple different types of verification scoring models;

[0007] Inputting the service dialogue set into the service scoring model and each verification scoring model respectively, to obtain a first scoring sequence output by the service scoring model and a second scoring sequence output by each verification scoring model;

[0008] The Spearman similarity between the first scoring list and each second scoring sequence is determined, and the scoring accuracy of the service scoring model is determined based on each Spearman similarity, so as to determine the evaluation result of the service scoring model according to the scoring accuracy.

[0009] Optionally, determining the service dialogue set includes:

[0010] Identify multiple service scenarios and the user needs corresponding to each service scenario;

[0011] For each service scenario, text interaction is performed in a preset shopping guide service model based on the user needs of the service scenario to obtain a service dialogue corresponding to the service scenario;

[0012] The service dialogues of each service scenario are obtained to form a service dialogue set.

[0013] Optionally, the scoring process of the service scoring model is consistent with the scoring process of each verification scoring model; wherein the scoring process of the service scoring model includes:

[0014] Determine a scoring order for each service dialogue in the service dialogue set, and input each service dialogue in the service dialogue set into the service scoring model in sequence according to the scoring order to obtain service scores outputted in sequence by the service scoring model, thereby forming a first scoring sequence.

[0015] Optionally, determining the Spearman similarity between the first scoring list and each second scoring sequence comprises:

[0016] Respectively ranking each service score in the first scoring list and each second scoring sequence to generate a first ranking sequence corresponding to the first scoring list and a second ranking sequence corresponding to each second scoring sequence;

[0017] The similarity coefficient between the first level sequence and each second level sequence is calculated as the Spearman similarity.

[0018] Optionally, determining the scoring accuracy of the service scoring model based on each Spearman similarity includes:

[0019] The mean of each Spearman similarity is calculated, and the mean is used as the scoring accuracy of the service scoring model.

[0020] Optionally, determining the evaluation result of the service scoring model according to the scoring accuracy includes:

[0021] Determining whether the scoring accuracy is greater than a preset threshold;

[0022] If yes, then the evaluation result of the service scoring model is determined to be passed;

[0023] If not, the evaluation result of the service scoring model is determined to be failed.

[0024] The present application also provides a service scoring model evaluation device, including:

[0025] A model acquisition module is used to determine a service dialogue set and obtain a service scoring model to be evaluated and multiple different types of verification scoring models;

[0026] A model scoring module, used for inputting the service dialogue set into the service scoring model and each verification scoring model respectively, to obtain a first scoring sequence output by the service scoring model and a second scoring sequence output by each verification scoring model;

[0027] The model evaluation module is used to determine the Spearman similarity between the first scoring list and each second scoring sequence, and determine the scoring accuracy of the service scoring model based on each Spearman similarity, so as to determine the evaluation result of the service scoring model according to the scoring accuracy.

[0028] Optionally, the model acquisition module includes:

[0029] A scenario determination submodule is used to determine multiple service scenarios and the user needs corresponding to each service scenario;

[0030] The dialogue interaction submodule is used to perform text interaction in a preset shopping guide service model for each service scenario based on the user needs of the service scenario to obtain a service dialogue corresponding to the service scenario;

[0031] The dialogue set forming submodule is used to obtain the service dialogues of each service scenario and form a service dialogue set.

[0032] The present application also provides a storage medium, characterized in that: the storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the service scoring model evaluation method described in any of the above embodiments.

[0033] The present application also provides a computer device, characterized in that it includes: one or more processors, and a memory;

[0034] The memory stores computer-readable instructions, and when the computer-readable instructions are executed by the one or more processors, the steps of the service scoring model evaluation method described in any one of the above embodiments are performed.

[0035] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0036] The service scoring model evaluation method, device, storage medium and computer equipment provided in the present application can first determine the service dialogue set when evaluating the service scoring model, and obtain the service scoring model to be evaluated and multiple different types of verification scoring models, and then input the service dialogue set into the service scoring model and each verification scoring model respectively to obtain a first scoring sequence output by the service scoring model and a second scoring sequence output by each verification scoring model, so that each second scoring sequence can be used as a benchmark reference for the accuracy evaluation of the service scoring model; then the Spearman similarity between the first scoring list and each second scoring sequence can be determined, and the scoring accuracy of the service scoring model can be determined based on each Spearman similarity, and finally the evaluation result of the service scoring model can be determined according to the scoring accuracy, where the Spearman similarity focuses on the relative position relationship between data values ​​rather than the actual data values, so that the differences caused by different training conditions and incompatibility of the verification scoring model with the service scenario can be avoided to affect the evaluation of the service scoring model, thereby improving the evaluation accuracy of the service scoring model. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0038] Figure 1 A flowchart of a service scoring model evaluation method provided in an embodiment of the present application;

[0039] Figure 2 A flowchart of a service dialogue set determination process provided in an embodiment of the present application;

[0040] Figure 3 A flowchart of a process for determining an evaluation result provided in an embodiment of the present application;

[0041] Figure 4 A schematic diagram of the structure of a service scoring model evaluation device provided in an embodiment of the present application;

[0042] Figure 5 A schematic diagram of the internal structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0044] With the rapid development of the market and the continuous increase in service scenarios, the diversification of user needs requires shopping guide intelligent services to have greater flexibility and adaptability, so as to provide better services for different types of service scenarios and user groups. In this context, the market's demand for the evaluation accuracy of the service scoring model that will be put into use has increased accordingly. However, the evaluation accuracy of the current service scoring model evaluation method still cannot meet market demand, resulting in poor optimization effects of shopping guide intelligent services.

[0045] Based on this, this application proposes the following technical solutions, see below for details:

[0046] In one embodiment, Figure 1 As shown, Figure 1 A flow chart of a service scoring model evaluation method provided in an embodiment of the present application; the present application provides a service scoring model evaluation method, including:

[0047] S110: Determine a service dialogue set, and obtain a service scoring model to be evaluated and a plurality of different types of verification scoring models.

[0048] In this step, when the computer device performs accuracy evaluation on the service scoring model in the shopping guide scenario, it can first determine a service dialogue set containing multiple service dialogues as evaluation data for the service scoring model to be evaluated, and then obtain multiple scoring models trained by different types of pre-trained models from the model library as verification scoring models for the service scoring model, so that the service scoring model can be accurately evaluated using the service dialogue set and the verification scoring model.

[0049] Among them, service dialogue refers to the complete communication process between users and the shopping guide system or shopping guide personnel in the shopping guide scenario, which may include product consultation, price discussion, purchase intention confirmation, after-sales service request, etc. The verification scoring model is a reference model used to evaluate the accuracy of the service scoring model. These models are usually obtained based on different types of training data and algorithms, and are used to score service dialogues to evaluate the accuracy of the service scoring model.

[0050] It is understandable that service conversations are an important basis for evaluating the quality of shopping guide services. They directly reflect the user's experience and satisfaction during the purchase process. Therefore, the computer device can use the service conversations in the service conversation set as input data for each scoring model, so that the scoring model can perform quality analysis and scoring on the input service conversations. After the service scoring model scores the quality of the service conversation, the scoring accuracy of the service scoring model can also be reversely evaluated through the scoring results of the service scoring model.

[0051] To elaborate, during the evaluation process, the verification scoring model can verify and compare the scoring results of the service scoring model, and then evaluate the accuracy and reliability of the service scoring model. However, unlike the service scoring model, the verification scoring model is not trained using targeted scenarios, such as the scene refinement data in the shopping guide scenario corresponding to the service scoring model, which may result in some contingency and limitations when it scores the shopping guide dialogue. Therefore, this application can use multiple different types of verification scoring models to evaluate the service scoring model, and verify and validate the service scoring model from multiple angles, which can reduce the impact of some contingencies and limitations on the evaluation results, thereby improving the evaluation accuracy of the evaluation process.

[0052] S120: Input the service dialogue set into the service scoring model and each verification scoring model respectively, to obtain a first scoring sequence output by the service scoring model and a second scoring sequence output by each verification scoring model.

[0053] In this step, after obtaining the service conversation set and the verification scoring model through step S110, the computer device can input the service conversation set into the service scoring model and each verification scoring model respectively to obtain the first scoring sequence output by the service scoring model and the second scoring sequence output by each verification scoring model, so that each second scoring sequence can be verified and validated with the first scoring sequence from multiple angles as a benchmark reference for the accuracy evaluation of the service scoring model.

[0054] It is understandable that since the service scoring model is trained using service conversations as training samples and service scores corresponding to service conversations as training labels, after the computer device inputs the service conversation set into the service scoring model, it can directly obtain the first scoring sequence input into the service scoring model. Similarly, after the computer device inputs the same service conversation set into each verification scoring model, it can obtain the second scoring sequence output by each verification scoring model.

[0055] S130: Determine the Spearman similarity between the first scoring list and each second scoring sequence, and determine the scoring accuracy of the service scoring model based on each Spearman similarity, so as to determine the evaluation result of the service scoring model according to the scoring accuracy.

[0056] In this step, after obtaining the first scoring list of the service scoring model and the second scoring list of each verification scoring model through step S120, the computer device can first determine the Spearman similarity between the first scoring list and each second scoring sequence, and then determine the scoring accuracy of the service scoring model based on each Spearman similarity, and then determine the evaluation result of the service scoring model according to the scoring accuracy, so as to avoid the difference in the verification scoring model caused by different training conditions and incompatibility with the service scenario affecting the evaluation of the service scoring model, thereby improving the evaluation accuracy of the service scoring model.

[0057] It is understandable that Spearman similarity is also called Spearman rank correlation coefficient, which is used to measure the dependence between two variables. It is a method of realizing similarity judgment based on ranking value, and uses monotonic equation to evaluate the correlation between two statistical variables. Therefore, it focuses on the relative position relationship between data values ​​rather than the actual data values. Therefore, it can calculate and compare the ranking consistency of the prediction results of the service scoring model and each verification scoring model on the same service dialogue set, and then evaluate the performance of the service scoring model by quantifying this consistency, so as to avoid the differences in the verification scoring model caused by different training conditions and incompatibility with the service scenario, which affects the evaluation of the service scoring model, thereby improving the evaluation accuracy of the service scoring model.

[0058] In the above embodiment, when the service scoring model is evaluated, the service dialogue set can be determined first, and the service scoring model to be evaluated and multiple different types of verification scoring models can be obtained, and the service dialogue set can be input into the service scoring model and each verification scoring model respectively to obtain the first scoring sequence output by the service scoring model and the second scoring sequence output by each verification scoring model, so that each second scoring sequence can be used as a benchmark reference for the accuracy evaluation of the service scoring model; then the Spearman similarity between the first scoring list and each second scoring sequence can be determined, and the scoring accuracy of the service scoring model can be determined based on each Spearman similarity, and finally the evaluation result of the service scoring model can be determined according to the scoring accuracy. Here, the Spearman similarity focuses on the relative position relationship between the data values, rather than the actual data values, so it can avoid the difference in the verification scoring model caused by different training conditions and incompatibility with the service scenario affecting the evaluation of the service scoring model, thereby improving the evaluation accuracy of the service scoring model.

[0059] In one embodiment, Figure 2 As shown, Figure 2 A flowchart of a service dialogue set determination process provided in an embodiment of the present application; Figure 2 In step S110, determining the service dialogue set may include:

[0060] S111: Determine multiple service scenarios and user needs corresponding to each service scenario.

[0061] S112: For each service scenario, text interaction is performed in a preset shopping guide service model based on user needs of the service scenario to obtain a service dialogue corresponding to the service scenario.

[0062] S113: Obtain service dialogues for each service scenario to form a service dialogue set.

[0063] In this embodiment, when determining the service dialogue set, the computer device can first determine the user needs corresponding to each service scenario. Then, for each service scenario, the computer device can perform text interaction in a preset shopping guide service model based on the user needs of the service scenario to obtain the service dialogue corresponding to the service scenario, thereby obtaining the service dialogues of each service scenario and forming a service dialogue set.

[0064] Among them, the shopping guide service model of the present application refers to a model that responds to and guides the input inquiry text and outputs it as an interactive text. When the shopping guide service model is training the model, the inquiry text corresponding to the user needs in different service scenarios can be used as a training sample, and each training sample is labeled with a sample label, that is, the corresponding interactive text. When all the training samples are labeled, the training samples with sample labels can be input into the preset pre-training model for forward propagation to train the model, and the model parameters are tuned during the back propagation process of the model. When the model meets certain training conditions or parameter convergence conditions, such as the number of iterations reaches the set value, the training is considered to be completed. At this time, the trained model can be used as the final shopping guide service model. Therefore, the computer device can first determine the inquiry text corresponding to the user needs in different service scenarios, and then perform text interaction in the shopping guide service model based on the inquiry text, and finally output the corresponding service dialogue.

[0065] It is understandable that user needs are the core elements of each service scenario, reflecting the user's goals, expectations, and required service content in a specific situation. Therefore, in the process of generating service dialogues, the computer device needs to first determine the user needs corresponding to each service scenario, so as to better understand the differences in user needs in different service scenarios, and then generate more accurate service dialogues. The service dialogue set formed by the computer device from the service dialogue collection of each service scenario can cover user needs and interaction situations in different service scenarios, has a wide range of applicability, and can provide high-quality test data for subsequent dialogue evaluation and model verification.

[0066] For example, in a service scenario, the user needs to guide the purchase of sneakers. At this time, the user enters the corresponding query sentence "I want sneakers" in the shopping guide service model, and the shopping guide service model outputs the interactive sentence corresponding to the query sentence "No more, can ordinary shoes be used?" Then, the user continues to enter the query sentence "Why are there no sneakers?" in the model, and the shopping guide service model can continue to output the interactive text "Sorry, the sneakers are being restocked, do you want to see other shoes that are wear-resistant and comfortable?" Therefore, in this service scenario, its service dialogue is a complete dialogue consisting of these four sentences.

[0067] In one embodiment, the scoring process of the service scoring model in step S120 is consistent with the scoring process of each verification scoring model; wherein the scoring process of the service scoring model may include:

[0068] S121: Determine the scoring order of each service dialogue in the service dialogue set, and input each service dialogue in the service dialogue set into the service scoring model in sequence according to the scoring order to obtain the service scores outputted in sequence by the service scoring model, thereby forming a first scoring sequence.

[0069] In this embodiment, when the computer device uses the service scoring model for scoring, it can first determine the scoring order of each service conversation in the service conversation set, and then input each service conversation in the service conversation set into the service scoring model in sequence according to the scoring order, and obtain the service scores output in sequence by the service scoring model to form a first scoring sequence.

[0070] It can be understood that by determining the scoring order of each service conversation in the service conversation set, the computer device can ensure the orderly progress of the scoring process, and input each service conversation into the model for unified processing according to the predetermined scoring order, ensuring that each service conversation can be input and scored in the predetermined order; the scoring order here can be determined according to the time sequence of the service conversations, the scene priority or other specific sorting rules, and is not limited here.

[0071] It should be noted that the scoring process of each verification scoring model in the present application is consistent with the service scoring model, including the scoring order of each service dialogue in the service dialogue set. Therefore, for any service dialogue in the service dialogue set, its scoring result has the same output position in the first scoring sequence and each second scoring sequence, so that the first scoring sequence can be directly verified and compared with each second scoring sequence.

[0072] In one embodiment, determining the Spearman similarity between the first scoring list and each second scoring sequence in step S130 may include:

[0073] S131: ranking each service score in the first scoring list and each second scoring sequence respectively, to generate a first ranking sequence corresponding to the first scoring list and a second ranking sequence corresponding to each second scoring sequence.

[0074] S132: Calculate the similarity coefficient between the first level sequence and each second level sequence as the Spearman similarity.

[0075] In this embodiment, when determining the Spearman similarity between the first scoring list and each second scoring sequence, the computer device can rank each service score in the first scoring list and each second scoring sequence, respectively, to generate a first rank sequence corresponding to the first scoring list and a second rank sequence corresponding to each second scoring sequence. Then, the computer device can use the similarity coefficient between the first rank sequence and each second rank sequence as the Spearman similarity.

[0076] For example, in an evaluation scenario, the first scoring sequence of the service scoring model is S = [85, 70, 90, 60], and the second scoring sequences of each verification scoring model are: S1 = [80, 75, 95, 55], S2 = [70, 85, 90, 65]. Therefore, the first-level sequence R (S) corresponding to the first scoring sequence is R (S) = [2, 3, 1, 4], and the second-level sequence corresponding to the second scoring sequence is: R (S1) = [2, 3, 1, 4], R (S2) = [3, 2, 1, 4]. Then, the computer device can use the formula of the Spearman correlation coefficient to obtain the Spearman correlation coefficient of S and S1 as 1, indicating that the two are directly and completely positively correlated, and the ranking between S and S2 is not completely consistent, so the Spearman correlation coefficient obtained is a correlation coefficient value less than 1, and the value calculated here is 0.8.

[0077] In one embodiment, determining the scoring accuracy of the service scoring model based on each Spearman similarity in step S130 may include:

[0078] S133: Calculate the mean of each Spearman similarity and use the mean as the scoring accuracy of the service scoring model.

[0079] In this embodiment, after the computer device calculates the Spearman similarity value between the service scoring model and each verification scoring model, the calculated Spearman correlation coefficient can be added and then divided by the number of verification models to obtain the average Spearman similarity, that is, the computer device can calculate the mean of each Spearman similarity and then use the mean as the scoring accuracy of the service scoring model.

[0080] For example, in an evaluation scenario, after calculating the Spearman similarity, the correlation coefficient between the service scoring model and the verification scoring model S1 is 1, and the correlation coefficient with S2 is 0.8. At this time, the average Spearman similarity is (1 +0.8) / 2 = 0.9, that is, the scoring accuracy of the service scoring model is 0.9.

[0081] In one embodiment, Figure 3 As shown, Figure 3 A flowchart of a process for determining an evaluation result provided in an embodiment of the present application; Figure 3 In step S130, determining the evaluation result of the service scoring model according to the scoring accuracy may include:

[0082] S134: Determine whether the scoring accuracy is greater than a preset threshold;

[0083] S135: When the scoring accuracy is greater than a preset threshold, determining that the evaluation result of the service scoring model is passed;

[0084] S136: When the scoring accuracy is not greater than the preset threshold, determining that the evaluation result of the service scoring model is failed.

[0085] In this embodiment, after determining the scoring accuracy of the service scoring model, the computer device may compare the scoring accuracy with a preset threshold, and determine the evaluation result of the service scoring model according to the comparison result. When the scoring accuracy is greater than the preset threshold, the evaluation result of the service scoring model is determined to be passed; when the scoring accuracy is not greater than the preset threshold, the evaluation result of the service scoring model is determined to be failed.

[0086] For example, in an evaluation scenario, the preset threshold given in the computer device is 0.8, and the scoring accuracy of the service scoring model is 0.9, so the evaluation result of the service scoring model is passed. To elaborate, the scoring results of the service scoring model show a high consistency in ranking with the scoring results of each verification scoring model. Although the consistency with some of the verification scoring models is slightly lower, it still meets the set standards overall. Therefore, the evaluation result of the service scoring model is passed, and its accuracy performance is within the expected range.

[0087] The service scoring model evaluation device provided in an embodiment of the present application is described below. The service scoring model evaluation device described below and the service scoring model evaluation method described above can be referenced to each other.

[0088] In one embodiment, Figure 4 As shown, Figure 4A schematic diagram of the structure of a service scoring model evaluation device provided in an embodiment of the present application; the present application also provides a service scoring model evaluation device, including a model acquisition module 210, a model scoring module 220 and a model evaluation module 230, which specifically include the following:

[0089] The model acquisition module 210 is used to determine the service dialogue set and acquire the service scoring model to be evaluated and a plurality of different types of verification scoring models.

[0090] The model scoring module 220 is used to input the service dialogue set into the service scoring model and each verification scoring model respectively, to obtain a first scoring sequence output by the service scoring model and a second scoring sequence output by each verification scoring model.

[0091] The model evaluation module 230 is used to determine the Spearman similarity between the first scoring list and each second scoring sequence, and determine the scoring accuracy of the service scoring model based on each Spearman similarity, so as to determine the evaluation result of the service scoring model according to the scoring accuracy.

[0092] In the above embodiment, when the service scoring model is evaluated, the service dialogue set can be determined first, and the service scoring model to be evaluated and multiple different types of verification scoring models can be obtained, and the service dialogue set can be input into the service scoring model and each verification scoring model respectively to obtain the first scoring sequence output by the service scoring model and the second scoring sequence output by each verification scoring model, so that each second scoring sequence can be used as a benchmark reference for the accuracy evaluation of the service scoring model; then the Spearman similarity between the first scoring list and each second scoring sequence can be determined, and the scoring accuracy of the service scoring model can be determined based on each Spearman similarity, and finally the evaluation result of the service scoring model can be determined according to the scoring accuracy. Here, the Spearman similarity focuses on the relative position relationship between the data values, rather than the actual data values, so it can avoid the difference in the verification scoring model caused by different training conditions and incompatibility with the service scenario affecting the evaluation of the service scoring model, thereby improving the evaluation accuracy of the service scoring model.

[0093] In one embodiment, the model acquisition module 210 may include:

[0094] The scenario determination submodule is used to determine multiple service scenarios and the user needs corresponding to each service scenario.

[0095] The dialogue interaction submodule is used to perform text interaction in a preset shopping guide service model for each service scenario based on the user needs of the service scenario to obtain a service dialogue corresponding to the service scenario.

[0096] The dialogue set forming submodule is used to obtain the service dialogues of each service scenario and form a service dialogue set.

[0097] In one embodiment, the scoring process of the service scoring model is consistent with the scoring process of each verification scoring model; wherein the model scoring module 220 may include:

[0098] The scoring sequence output submodule is used to determine the scoring order of each service dialogue in the service dialogue set, and input each service dialogue in the service dialogue set into the service scoring model in sequence according to the scoring order to obtain the service scores output by the service scoring model in sequence to form a first scoring sequence.

[0099] In one embodiment, the model evaluation module 230 may include:

[0100] The ranking submodule is used to rank each service score in the first scoring list and each second scoring sequence respectively, and generate a first ranking sequence corresponding to the first scoring list and a second ranking sequence corresponding to each second scoring sequence.

[0101] The coefficient calculation submodule is used to calculate the similarity coefficient between the first level sequence and each second level sequence as the Spearman similarity.

[0102] In one embodiment, the model evaluation module 230 may further include:

[0103] The mean calculation submodule is used to calculate the mean of each Spearman similarity and use the mean as the scoring accuracy of the service scoring model.

[0104] In one embodiment, the model evaluation module 230 may further include:

[0105] The threshold judgment submodule is used to judge whether the scoring accuracy is greater than a preset threshold;

[0106] The first result determination submodule is used to determine that the evaluation result of the service scoring model is passed when the scoring accuracy is greater than a preset threshold;

[0107] The second result determination submodule is used to determine that the evaluation result of the service scoring model is failed when the scoring accuracy is not greater than a preset threshold.

[0108] In one embodiment, the present application also provides a storage medium, which stores computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the service scoring model evaluation method described in any of the above embodiments.

[0109] In one embodiment, the present application also provides a computer device having computer-readable instructions stored therein. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the service scoring model evaluation method as described in any one of the above embodiments.

[0110] Indicatively, Figure 5 As shown, Figure 5 This is a schematic diagram of the internal structure of a computer device provided in an embodiment of the present application. The computer device 300 may be provided as a server. Figure 5 The computer device 300 includes a processing component 302, which further includes one or more processors, and a memory resource represented by a memory 301, for storing instructions executable by the processing component 302, such as an application. The application stored in the memory 301 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 302 is configured to execute instructions to perform the service scoring model evaluation method of any of the above embodiments.

[0111] The computer device 300 may further include a power supply component 303 configured to perform power management of the computer device 300, a wired or wireless network interface 304 configured to connect the computer device 300 to a network, and an input / output (I / O) interface 305. The computer device 300 may operate based on an operating system stored in the memory 301, such as Windows Server TM, Mac OS X TM, Unix TM, Linux TM, Free BSD TM, or the like.

[0112] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0113] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0114] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can refer to each other.

[0115] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A service scoring model evaluation method, characterized in that: The method comprises: Determine the service dialogue set, and obtain the service scoring model to be evaluated and multiple different types of verification scoring models; Inputting the service dialogue set into the service scoring model and each verification scoring model respectively, to obtain a first scoring sequence output by the service scoring model and a second scoring sequence output by each verification scoring model; The Spearman similarity between the first scoring list and each second scoring sequence is determined, and the scoring accuracy of the service scoring model is determined based on each Spearman similarity, so as to determine the evaluation result of the service scoring model according to the scoring accuracy.

2. The service scoring model evaluation method according to claim 1, characterized in that: The determining of the service dialogue set includes: Identify multiple service scenarios and the user needs corresponding to each service scenario; For each service scenario, text interaction is performed in a preset shopping guide service model based on the user needs of the service scenario to obtain a service dialogue corresponding to the service scenario; The service dialogues of each service scenario are obtained to form a service dialogue set.

3. The service scoring model evaluation method according to claim 1, characterized in that: The scoring process of the service scoring model is consistent with the scoring process of each verification scoring model; wherein the scoring process of the service scoring model includes: Determine a scoring order for each service dialogue in the service dialogue set, and input each service dialogue in the service dialogue set into the service scoring model in sequence according to the scoring order to obtain service scores outputted in sequence by the service scoring model, thereby forming a first scoring sequence.

4. The service scoring model evaluation method according to claim 1, characterized in that: Determining the Spearman similarity between the first scoring list and each second scoring sequence includes: Respectively ranking each service score in the first scoring list and each second scoring sequence to generate a first ranking sequence corresponding to the first scoring list and a second ranking sequence corresponding to each second scoring sequence; The similarity coefficient between the first level sequence and each second level sequence is calculated as the Spearman similarity.

5. The service scoring model evaluation method according to claim 1, characterized in that: Determining the scoring accuracy of the service scoring model based on each Spearman similarity includes: The mean of each Spearman similarity is calculated, and the mean is used as the scoring accuracy of the service scoring model.

6. The service scoring model evaluation method according to claim 1, characterized in that: Determining the evaluation result of the service scoring model according to the scoring accuracy includes: Determining whether the scoring accuracy is greater than a preset threshold; If yes, then the evaluation result of the service scoring model is determined to be passed; If not, the evaluation result of the service scoring model is determined to be failed.

7. A service scoring model evaluation device, characterized in that: include: A model acquisition module is used to determine a service dialogue set and obtain a service scoring model to be evaluated and multiple different types of verification scoring models; A model scoring module, used for inputting the service dialogue set into the service scoring model and each verification scoring model respectively, to obtain a first scoring sequence output by the service scoring model and a second scoring sequence output by each verification scoring model; The model evaluation module is used to determine the Spearman similarity between the first scoring list and each second scoring sequence, and determine the scoring accuracy of the service scoring model based on each Spearman similarity, so as to determine the evaluation result of the service scoring model according to the scoring accuracy.

8. The service scoring model evaluation device according to claim 7, characterized in that: The model acquisition module includes: A scenario determination submodule is used to determine multiple service scenarios and the user needs corresponding to each service scenario; The dialogue interaction submodule is used to perform text interaction in a preset shopping guide service model for each service scenario based on the user needs of the service scenario to obtain a service dialogue corresponding to the service scenario; The dialogue set forming submodule is used to obtain the service dialogues of each service scenario and form a service dialogue set.

9. A storage medium, characterized in that: The storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the service scoring model evaluation method as described in any one of claims 1 to 6.

10. A computer device, characterized in that: include: one or more processors, and memory; The memory stores computer-readable instructions, and when the computer-readable instructions are executed by the one or more processors, the steps of the service scoring model evaluation method as described in any one of claims 1 to 6 are performed.

Images