Intelligent customer service large model quality evaluation method and device, medium and program product

Abstract

The embodiment of the invention relates to the field of large models, and provides an intelligent customer service large model quality evaluation method and device, a medium and a program product. The method comprises the steps that a target answer of an intelligent customer service large model responding to target input of a target user in a target time period is acquired, the target input comprises at least two times of messages of at least one mode, and the target answer comprises an answer based on the at least two times of messages; determining a target question and emotion data of the target user according to the modality of each message in the target input and the context relationship of each message; calling a quality inspection large model to analyze the quality of the target answer according to the target question, the emotion data, the target answer and the association degree of the target input; and setting a preset label for the answer of which the quality does not accord with a preset first condition. According to the scheme, the conversation content is accurately understood by carrying out quality inspection on the conversation, judgment is not limited to be carried out according to a preset rule of an e-commerce platform, misjudgment is avoided or reduced in time, and the quality inspection efficiency and the customer service reception quality are remarkably improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of large model technology, and in particular to a method, device, medium and program product for evaluating the quality of an intelligent customer service large model. Background Art

[0002] In the field of e-commerce customer service, customer service robots have become an indispensable tool. Robots will automatically answer users' questions, which can improve customer service efficiency and respond to users in a timely manner to improve user experience. Quality inspection is a very important link. Through quality inspection, problems in the customer service reception link can be discovered and corrected to improve the reception quality. Often, for large-scale customer service teams, quality inspection is a very time-consuming and labor-intensive task that requires manual quality inspection of each conversation, which is inefficient.

[0003] In the process of research and practice of the prior art, the inventors of the embodiments of the present application found that in the current quality inspection system, manual quality inspection is time-consuming, labor-intensive and inefficient; and ordinary machine quality inspection often cannot accurately understand the content of the conversation and can only make judgments based on rules, which is prone to misjudgment. Summary of the invention

[0004] The embodiments of the present application provide a method, device, medium and program product for evaluating the quality of a large intelligent customer service model, which can enable a customer service evaluation system running in a business server in a plug-in manner to perform offline or online real-time quality inspection on the conversation between the large intelligent customer service model and the user, thereby accurately understanding the content of the conversation, not limited to making judgments based on the preset rules of the e-commerce platform, timely avoiding or reducing misjudgments, and significantly improving quality inspection efficiency and customer service reception quality, saving labor costs.

[0005] In a first aspect, the present application provides a method for evaluating the quality of a large intelligent customer service model. The method is applied to a customer service evaluation system based on a multimodal large model, wherein the customer service evaluation system includes a quality inspection large model; the method includes: Display the target conversation interface between the intelligent customer service model and the target user; Obtaining, from the target conversation interface, a target answer of the intelligent customer service big model in response to a target input of a target user within a target period of time, wherein the target input includes at least two messages of at least one modality, and the target answer includes an answer based on the at least two messages; Determining the target question and sentiment data of the target user according to the modality of each message in the target input and the contextual relationship of each message; Calling the quality inspection big model to perform quality analysis on the target answer according to the target question, the sentiment data, and the correlation between the target answer and the target input; A preset label is set for an answer whose quality does not meet the preset first condition, wherein the preset label indicates that the score of the answer is lower than a preset threshold.

[0006] In some implementations, the calling of the quality inspection model to perform quality analysis on the target answer according to the target question, the sentiment data, and the relevance between the target answer and the target input includes: Obtain the semantic matching degree between the target answer and the target question; analyzing the sentiment data to determine whether the target response matches the expected sentiment; Determine whether the target response contains factual errors and necessary information to match the target input; and determining a logical coherence between a target answer and said target input based on a text generation quality assessment metric; The final quality score is calculated based on the above analysis results and corresponding weights.

[0007] In a second aspect, an embodiment of the present application provides a customer service evaluation system, which has the function of implementing the intelligent customer service large model quality evaluation method provided in the first aspect above. The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions, and the modules can be software and / or hardware.

[0008] In some implementations, the customer service evaluation system includes: A display module is used to display the target conversation interface between the intelligent customer service model and the target user; An input-output module, configured to obtain, from the target conversation interface, a target answer of the intelligent customer service model in response to a target input of a target user within a target period of time, wherein the target input includes at least two messages of at least one modality, and the target answer includes an answer based on the at least two messages; A processing module, used to determine the target question and sentiment data of the target user according to the modality of each message in the target input and the contextual relationship of each message; The processing module is also used to call the quality inspection model to perform quality analysis on the target answer based on the target question, the sentiment data, and the correlation between the target answer and the target input; and to set a preset label for the answer whose quality does not meet the preset first condition, wherein the preset label indicates that the score of the answer is lower than a preset threshold.

[0009] In some implementations, the processing module obtains the quality inspection large model by training through the following operations: Acquire a priori e-commerce customer service data source through the input-output module, wherein the priori e-commerce customer service data source includes multiple questions of at least one e-commerce platform, at least one positive answer corresponding to each question, and at least one negative answer corresponding to each question; the quality of the positive answer is higher than the quality of the negative answer; Building a mapping relationship between e-commerce questions, positive answers, and negative answers based on the prior e-commerce customer service data source; Dividing the mapping relationship into a training set and a test set; Iteratively training the large quality inspection model based on the training set; The accuracy of the large quality inspection model is verified based on the test set. If the scores of all positive answers are greater than the scores of negative answers, it is determined that the accuracy reaches the preset accuracy and converges.

[0010] In some implementations, the customer service evaluation system further includes an adversarial generation model, and the processing module is further configured to: Acquire a first prompt word input to the adversarial generation model through the input-output module; generating a first question set of the e-commerce platform based on the first prompt word, wherein the first question set includes a plurality of adversarial e-commerce questions generated based on adversarial technology; Inputting the first question set into the intelligent customer service big model through the input-output module to obtain a first answer set, where the first answer set is a candidate answer corresponding to each adversarial e-commerce question in the first question set; Calling the quality inspection big model to evaluate the first answer set to obtain a first target score set, wherein the first target score set includes a target score for each candidate answer in the first answer set; The first target score set is fed back to the adversarial generation model through the input-output module until the answer of the intelligent customer service big model meets the preset second condition, then the adversarial attack on the intelligent customer service big model is stopped. The second condition includes that the number of attacks of the adversarial generation model reaches a preset number, or the score of the answer of the intelligent customer service big model meets the second condition.

[0011] In some implementations, the customer service evaluation system further includes an adversarial generation model and a teacher model, and the processing module is further used to: Acquiring, through the input-output module, a second prompt word input to the adversarial generation model, and generating a second question set of the e-commerce platform based on the second prompt word, wherein the second question set includes a plurality of adversarial e-commerce questions generated based on adversarial technology; Inputting the second question set into the intelligent customer service big model through the input-output module to obtain a second answer set, where the second answer set is the candidate answers corresponding to each adversarial e-commerce question in the second question set; Calling the quality inspection big model to evaluate the second answer set to obtain a second target score set, where the second target score set includes a target score for each candidate answer in the second answer set; Feeding back the second target score set to the intelligent customer service model through the input-output module; Calling the teacher big model to generate an e-commerce adversarial sample and inputting it into the intelligent customer service big model to perform adversarial enhancement training on the intelligent customer service big model to obtain a target answer corresponding to the e-commerce adversarial sample; The verification result corresponding to the target answer is fed back to the intelligent customer service model through the input-output module until the intelligent customer service model converges; wherein the verification result is a positive attack, or a negative attack and a candidate optimization strategy corresponding to the negative attack.

[0012] In some implementation modes, after setting the first label for the answer that the quality does not meet the preset first condition, the processing module is further configured to: Generate a first prompt message, wherein the first prompt message is used to prompt that the emotion of the target user is of a negative type; In response to the first prompt message, performing context analysis on the target input and the target answer to determine the actual demands of the target user; A first strategy matching the actual demand is obtained and displayed on the target conversation interface through the display module, wherein the first strategy includes at least one of a soothing plan, an anthropomorphic speech, a solution matching the actual demand, and a manual customer service entrance.

[0013] In some implementation modes, after setting the first label for the answer that the quality does not meet the preset first condition, the processing module is further configured to: generating a second prompt message, wherein the second prompt message is used to prompt that the current answer is a negative answer; In response to the second prompt message, performing context analysis on the target input and the target answer to determine the actual demand of the target user; A target strategy matching the actual demand is obtained and displayed on the target session interface through the display module.

[0014] In some implementation modes, after performing quality analysis on the target answer, the processing module is further configured to: If the first result obtained by the analysis meets the first condition, it is determined that the target answer matches the question demand of the target user, and the target answer is updated to the first data set; If the first result obtained by the analysis does not meet the first condition, it is determined that the target answer does not match the question demand of the target user, and the target answer is updated to the second data set; Among them, the first data set is a positive answer data set of the e-commerce platform customer service, and the second data set is a negative answer data set of the e-commerce platform customer service. The first data set and the second data set are used to train and update the intelligent customer service big model.

[0015] In a third aspect, an embodiment of the present application provides a customer service big model quality evaluation device, the customer service big model quality evaluation device comprising: at least one processor and a memory; wherein the memory is used to store a computer program, and the processor is used to call the computer program stored in the memory to execute the above-mentioned first aspect, any one of the implementation methods of the first aspect, or any steps of the second aspect to implement the intelligent customer service big model quality evaluation method.

[0016] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a function of implementing the method for evaluating the quality of a large model of intelligent customer service provided in the first aspect above. The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions, and the modules can be software and / or hardware. Specifically, a computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for loading by a processor to execute the first aspect of the embodiment of the present application, any one of the implementation methods of the first aspect, or any step of implementing the method for evaluating the quality of a large model of intelligent customer service provided in the second aspect.

[0017] Compared with the prior art, in the solution provided by the embodiment of the present application, after calling the target conversation interface between the intelligent customer service big model and the target user, since the target answer is the answer of the intelligent customer service big model to at least two messages in at least one modality of the target input of the target user within the target time period, the intelligent customer service big model can understand multimodal messages, and clarify the target user's target question and emotional data from the multimodal messages and the contextual relationship of each message. Therefore, the obtained target question and emotional data can provide strong support for the subsequent quality inspection big model to evaluate the quality of the target answer.

[0018] Since the quality inspection model is a prophetic evaluation model, it can perform quality analysis on the target answer based on the above target question, sentiment data, and the correlation between the target answer and the target input, and set a preset label for the answer whose quality does not meet the preset first condition. The quality analysis can accurately understand the content of this conversation, rather than just judging based on simple preset machine rules. The quality inspection model is based on both data and knowledge drive, so the output of the quality inspection model is controllable, reliable, and explainable. It can be seen that, on the one hand, the use of this solution can significantly improve the efficiency of quality inspection; on the other hand, it can clarify user intentions, reduce misjudgments of user intentions and the answers of the intelligent customer service model, and provide strong data support for the subsequent optimization of the intelligent customer service model. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings: Figure 1a A schematic diagram of a scenario of a communication system for implementing a method for evaluating the quality of a large model of intelligent customer service provided in an embodiment of the present application; Figure 1b A schematic diagram of a scenario of a communication system for implementing a method for evaluating the quality of a large model of intelligent customer service provided in an embodiment of the present application; Figure 1c A schematic diagram of a scenario of a communication system for implementing a method for evaluating the quality of a large model of intelligent customer service provided in an embodiment of the present application; Figure 2 A schematic diagram of a training process of a large quality inspection model provided in an embodiment of the present application; Figure 3a A schematic diagram of the quality inspection model provided in the embodiment of the present application for identifying user intentions based on session records; Figure 3b A schematic diagram of a positive sample for training a large quality inspection model provided in an embodiment of the present application; Figure 3c A schematic diagram of a positive sample continuation page for training a large quality inspection model provided in an embodiment of the present application; Figure 3d A schematic diagram of negative samples for training a large quality inspection model provided in an embodiment of the present application; Figure 4 A flow chart of a method for evaluating the quality of a large model of intelligent customer service provided in an embodiment of the present application; Figure 5 A schematic diagram of a training framework for an intelligent customer service evaluation system provided in an embodiment of the present application; Figure 6A schematic diagram of a process for mining algorithm vulnerabilities of a large intelligent customer service model provided in an embodiment of the present application; Figure 7 A schematic diagram of a training framework for an intelligent customer service evaluation system provided in an embodiment of the present application; Figure 8 A flow chart of a large online optimization intelligent customer service model provided in an embodiment of the present application; Fig. 9 A schematic diagram comparing the answers before and after the optimized intelligent customer service model provided in the embodiment of the present application; Fig.10 A schematic diagram of the structure of a customer service large model quality evaluation device provided in an embodiment of the present application; Fig.11 A schematic diagram of a physical device structure for implementing a method for evaluating the quality of a large model of intelligent customer service provided in an embodiment of the present application; Fig.12 A schematic diagram of the structure of a terminal device for implementing a method for evaluating the quality of a large model of intelligent customer service provided in an embodiment of the present application; Fig.13 A structural diagram of a server for implementing a method for evaluating the quality of a large model of intelligent customer service provided in an embodiment of the present application. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0021] The terms "first", "second", etc. in the specification and claims of the embodiments of the present application and the above-mentioned drawings are used to distinguish similar objects (for example, the first prompt word and the second prompt word in the embodiments of the present application respectively represent different prompt words of the user input dialog box), and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments described here can be implemented in an order other than the content illustrated or described here. In some of the processes described in the embodiments of the present application and the drawings, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed or executed in parallel in the order in which they appear in the embodiments of the present application, and the sequence numbers of the operations, such as 101, 102, etc., are only used to distinguish between different operations, and the sequence numbers themselves do not represent any execution order. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or modules is not necessarily limited to those steps or modules that are clearly listed, but may include other steps or modules that are not clearly listed or inherent to these processes, methods, products or devices. The division of modules that appear in the embodiments of the present application is only a logical division. There may be other division methods when implemented in actual applications. For example, multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between each other shown or discussed can be through some interfaces, and the indirect coupling or communication connection between modules can be electrical or other similar forms, which are not limited in the embodiments of the present application. In addition, the modules or submodules described as separate components may or may not be physically separated, may or may not be physical modules, or may be distributed in multiple circuit modules, and some or all of the modules can be selected according to actual needs to achieve the purpose of the embodiment of the present application.

[0022] The embodiments of the present application provide a method, device, medium and program product for evaluating the quality of a large model of intelligent customer service, which can be used in a server or a terminal device. Specifically, it can be used in e-commerce platforms, social platforms, financial platforms, government platforms, retrieval systems and other platforms that require quality assessment or real-time detection of conversations between customer service and users. For example, it can be applied to a customer service evaluation system implanted in a business server, and the conversations between customer service (including large models of intelligent customer service and manual customer service) and users can be quality inspected offline or online in real time, so as to accurately understand the content of the conversation, not limited to judging according to the rules preset by the e-commerce platform, timely avoid or reduce misjudgments, and significantly improve the quality inspection efficiency and customer service reception quality, saving labor costs.

[0023] In some implementations, the present invention is applied to Figure 1aThe communication system shown in FIG. 1 may include at least one service server 10, a customer service system 20, and at least one user terminal 30 ( Figure 1a Only one user terminal is indicated). The customer service system 20 includes a customer service evaluation system 201 and at least one intelligent customer service model 202. The user terminal 30 can be a server or a terminal. When it is a terminal, it has a client installed. The client installed on each user terminal can be a client with the same or different version numbers, or a client with the same or different operating systems. The embodiment of the present application does not limit the version of the client and the type of operating system corresponding to the client. The following is a detailed description.

[0024] Business server 10: can be used to provide business services such as search, transaction, conversation, upload, download, etc. to users of each user terminal. For example, the server can provide users with resource transactions (such as purchasing things on the e-commerce platform), connect to customer service for real-time conversation, etc. The business server can be an e-commerce platform, social platform, financial platform, government platform, retrieval system, etc., which requires quality assessment or real-time detection of conversations between customer service and users. You can refer to the embodiments of this application about e-commerce platforms, which will not be repeated here, and this application does not limit this.

[0025] Customer service system 20: The customer service system 20 includes a customer service evaluation system 201 and at least one intelligent customer service robot 202. The customer service evaluation system 201 can be used to perform quality inspection on the responses of the intelligent customer service big model 202 (equivalent to the intelligent customer service robot, no distinction is made) or manual customer service to the input of user a of the user terminal 30, and give a quality score.

[0026] In some implementations, the customer service evaluation system 201 includes a quality inspection model 2011. Optionally, the customer service evaluation system may also include an adversarial generation model 2012 and a teacher model 2013. The customer service evaluation system 202 and the intelligent customer service model 202 may both be run in a business server in the form of a plug-in, such as in an e-commerce platform in the form of a plug-in, to perform quality inspection on the intelligent customer service model 202's answer to user a.

[0027] The quality inspection big model 2011 scores the answers fed back by the intelligent customer service big model based on the adversarial questions continuously input by the adversarial generation model. The quality inspection big model 2011 can be used only in a specific e-commerce transaction field, or it can cover any application field that implements conversations with users based on a multimodal big model. It can be selected based on the actual application direction of the intelligent customer service big model, and this application does not limit this. The quality inspection big model 2011 covers a variety of conversation scenarios and involves multiple evaluation dimensions. The specific conversation scenarios can refer to the various processing strategies, e-commerce platform regulations, and relevant laws and regulations in the e-commerce field for user consultation, pre-sale of resources, and after-sale of resources, etc., and this application does not limit this. The quality inspection big model, teacher big model, adversarial generation model, and intelligent customer service big model involved in the embodiments of this application are all large language models, which can specifically be multimodal large language models (MLLMs) based on LLM evolution. Since the quality inspection big model, teacher big model, adversarial generation model and intelligent customer service big model involved are all large language models, they bridge the gap between visual data and language context, enabling these big models to understand and generate content that combines text, images and other modalities, and are therefore more conducive to the quality evaluation of the intelligent customer service big model in this application.

[0028] User terminal 30: A client that matches the server deployed in the business server is installed, and a conversation can be conducted with the intelligent customer service big model 202 in the customer service system through the client.

[0029] When based on Figure 1a When the communication system shown implements the above-mentioned intelligent customer service big model quality evaluation method, the customer service evaluation system running in the business server in a plug-in manner performs offline or online real-time quality inspection on the conversation between the intelligent customer service big model and the user, thereby achieving accurate understanding of the content of the conversation. It is not limited to making judgments based on the preset rules of the e-commerce platform, and timely avoids or reduces misjudgments, and significantly improves quality inspection efficiency and customer service reception quality, saving labor costs.

[0030] Figure 1b It indicates that the adversarial attack training is performed on the intelligent customer service big model 202 based on the adversarial generation model 2012, and the quality of the answers of the intelligent customer service big model 202 is evaluated in combination with the quality inspection big model 2011 to dig out the algorithm loopholes of the intelligent customer service big model 202.

[0031] Figure 1c It means that after the algorithm loopholes of the intelligent customer service big model 202 are discovered based on the quality inspection big model 2011 against the generation model 2012, the intelligent customer service big model 202 is reinforced learning based on the transferability of the teacher big model 2013 to achieve rapid online optimization of the intelligent customer service big model 202.

[0032] It can be understood that the customer service system in this application can be applied to the service side of the e-commerce platform (i.e., the conversation quality inspection between the official customer service of the e-commerce platform and the user), and can also be used for the store service of a single merchant (i.e., the conversation quality inspection between the store customer service and the user). This application mainly takes the conversation quality inspection between the official customer service of the e-commerce platform and the user as an example. Other scenarios can be used as a reference and will not be elaborated on.

[0033] The solutions provided in the embodiments of the present application involve technologies such as artificial intelligence (AI), natural language processing (NLP), and machine learning (ML), which are specifically described by the following embodiments: Among them, AI is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0034] AI technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0035] NLP is an important direction in the fields of computer science and artificial intelligence. It studies various theories and methods that can achieve effective communication between people and computers using natural language. Natural language processing is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field will involve natural language, that is, the language people use in daily life, so it is closely related to the study of linguistics. Natural language processing technology usually includes text processing, semantic understanding, machine translation, robot question answering, knowledge graph and other technologies.

[0036] In terms of quality evaluation of the intelligent customer service big model in the field of artificial intelligence, this application can implant a multimodal quality inspection big model into the business server, conduct offline or online context analysis of the conversation between the intelligent customer service big model and the user, accurately identify user intentions and emotional changes, and conduct quality evaluation of the answers of the intelligent customer service big model, thereby improving the quality and efficiency of quality inspection. In the online quality inspection scenario, a solution that matches the user's intention can also be given in real time.

[0037] Among them, it should be particularly noted that the server involved in the embodiment of the present application (such as a business server, a search engine) can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The customer service evaluation system involved in the embodiment of the present application can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, a personal digital assistant, etc., but is not limited to this. The customer service evaluation system, the user terminal, and the server can be directly or indirectly connected via wired or wireless communication, and the embodiment of the present application is not limited here.

[0038] In view of the inefficiency of manual quality inspection and the misjudgment of machine quality inspection in the related technology, the embodiment of the present application mainly adopts the following technical solutions: 1. Accurately identify user intentions: Introduce a quality inspection model based on a multimodal big model. The customer service evaluation system running in the business server as a plug-in performs offline or online real-time quality inspection on the conversation between the intelligent customer service big model and the user, thereby accurately understanding the content of the conversation and user emotions. It is not limited to making judgments based on the preset rules of the e-commerce platform, and timely avoids or reduces misjudgments. It also significantly improves quality inspection efficiency and customer service reception quality, saving labor costs.

[0039] 2. Exploiting algorithmic vulnerabilities in the intelligent customer service big model: Based on the adversarial generation model, the intelligent customer service big model is trained for adversarial attacks, and the quality of the intelligent customer service big model's answers is evaluated in combination with the quality inspection big model to explore the algorithmic vulnerabilities of the intelligent customer service big model.

[0040] 3. Online optimization of the intelligent customer service big model: After the algorithm vulnerabilities of the intelligent customer service big model are discovered based on the adversarial generative model of the quality inspection big model, reinforcement learning is performed on the intelligent customer service big model based on the transferability of the teacher big model to achieve rapid online optimization of the intelligent customer service big model.

[0041] The following combination Figure 2-Figure 13 The technical solution of the embodiment of the present application is exemplified.

[0042] Since the intelligent customer service big model quality evaluation method of the embodiment of the present application processes the output (i.e., the response to the user input question) of the intelligent customer service big model (i.e., the customer service robot) based on the pre-trained quality inspection big model (also called the prophetic evaluation model), before introducing the intelligent customer service big model quality evaluation method, the quality inspection big model training process is first introduced. Figure 2 As shown, Figure 2 The following is a schematic diagram of a training process for a large quality inspection model. The training process includes: 101. Obtain prior e-commerce customer service data source.

[0043] The prior e-commerce customer service data source includes multiple positive samples and multiple negative samples. The multiple positive samples are multiple questions of at least one e-commerce platform and at least one positive answer corresponding to each question; the multiple negative samples are multiple questions of at least one e-commerce platform and at least one negative answer corresponding to each question.

[0044] The quality of the positive answer is higher than the quality of the negative answer. This application does not limit the quality assessment method of positive and negative answers. It can be dynamically adjusted according to the needs or accuracy in the actual business scenario, or it can be determined based on the experience of experts under each segment of the e-commerce platform or the publicly available prior knowledge to determine whether the answer is a positive answer or a negative answer. This application does not limit this.

[0045] Among them, the positive answer means that the user's intention can be understood based on the messages of various modes sent by the user, and the operation strategy or solution matching the user's intention can be provided based on the messages of various modes sent by the user, and it has certain advantages such as real-time, professionalism, legality and compliance. On the contrary, the negative answer means that the user's intention cannot be understood, the answer is irrelevant, the template words are repeated, and it is not real-time, professional, or even illegal and compliant.

[0046] For example, if Figure 3b and Figure 3c is an example of a positive sample, Figure 3c is an example of a negative sample.

[0047] 102. Construct a mapping relationship between e-commerce questions, positive answers, and negative answers based on the prior e-commerce customer service data source.

[0048] For example, the mapping relationship can be in the form of a triple sample: e-commerce question-positive answer-negative answer. For example, based on the preset prior knowledge, excellent answers and bad answers are determined among the multiple answers corresponding to each question. For example, as shown in Table 1 below: Table 1 103. Divide the mapping relationship into a training set and a test set.

[0049] 104. Perform iterative training on the large quality inspection model based on the training set.

[0050] 105. Verify the accuracy of the large quality inspection model based on the test set. If the scores of all positive answers are greater than the scores of negative answers, it is determined that the accuracy reaches a preset accuracy and converges.

[0051] For example, Table 2 below shows the scoring dimensions of the quality inspection model for the intelligent customer service model: Table 2 In some implementations, the quality inspection model may adopt a weighted scoring method, and assign corresponding weights according to the importance of each scoring dimension in Table 1 above. A calculation formula for calculating the score of the answer of the intelligent customer service model is as follows: Formula (1): Formula (1) Among them: Regarding dimension scores: the score in each dimension is calculated based on a full score of 100 (which can be obtained by taking the average of the scores of specific indicators).

[0052] About weights: The importance of dimensions is allocated according to the company's service standards, with a total weight of 100%.

[0053] For example, taking the e-commerce platform A that uses the quality inspection model and the intelligent customer service model in the embodiment of the present application as an example, it is assumed that the scores of each dimension of the intelligent customer service A of the e-commerce platform A in a session are as shown in Table 3 (the full score is 100): Table 3 Referring to the weights of each scoring dimension shown in Table 1 above, the total score R of the intelligent customer service A in this session is calculated according to the scores of each scoring dimension obtained in Table 3: R= It can be seen that the final total score is 85.25 points.

[0054] It should be noted that in addition to quality inspection of the answer quality of the intelligent customer service big model, the quality inspection big model in the embodiment of the present application can also perform quality inspection on the answers of the manual customer service to the user. The embodiment of the present application takes the quality inspection of the answers of the intelligent customer service big model based on the quality inspection big model as an example. Similar points can be referred to and will not be elaborated on.

[0055] It is understandable that before training the quality inspection model, in order to ensure the objectivity, legality and accuracy of the evaluation of the quality inspection model, various adjustments need to be made to the prior e-commerce customer service data source used for training. This can be done from four main dimensions: e-commerce platform rules, political sensitivity, laws and regulations of different countries or regions, and mental health, as well as from eight scoring dimensions: problem solving rate, response timeliness, communication professionalism, service attitude, customer experience satisfaction, compliance with process specifications, problem prediction and resolution capabilities, and complaint and traceback rates. Questions and answers on sensitive controversial topics are formed, and then the quality inspection model is trained based on supervised learning. The training of the intelligent customer service model is the same and will not be elaborated on.

[0056] It can be seen that after training the quality inspection big model, the trained quality inspection big model can score the answers output by the intelligent customer service big model, and then feed back the scoring results to the intelligent customer service big model, so that it can continuously strengthen the learning of the key points and differences between positive answers and negative answers, until the question-answering ability is gradually iterated to the optimal level.

[0057] As described in the above embodiment, after the above quality inspection model is obtained through model training, the answers output by the intelligent customer service model can be quality inspected based on the quality inspection model to obtain the answer quality score of each conversation between the intelligent customer service model and the user, so as to timely find out whether the service of the intelligent customer service model has defects, whether it solves the user's problem, etc. Specifically, see Figure 4 A method for evaluating the quality of a large model of intelligent customer service is provided. The method is applied to a customer service evaluation system based on a multimodal large model. The method for evaluating the quality of a large model of intelligent customer service is implemented based on a large quality inspection model of supervised learning. The method can be executed by a business server. The business server can be an e-commerce platform, a social platform, a financial platform, a government platform, a retrieval system, or other platform that requires quality assessment or real-time detection of conversations between customer service and users. The embodiments of this application are not limited to this. The subsequent embodiments only take the application to an e-commerce platform as an example. The embodiments of this application include: 201. Display a target conversation interface between the intelligent customer service model and the target user.

[0058] The target conversation interface refers to the display interface of the conversation (either real-time or historical) established between the user and the intelligent customer service model in the e-commerce platform when the user logs into the client of the e-commerce platform. Both the target user and the intelligent customer service model can send and receive messages of any mode in the target conversation interface.

[0059] 202. Obtain, from the target conversation interface, a target answer of the intelligent customer service model in response to a target input of a target user within a target time period.

[0060] The target period refers to the historical period in which the session records have occurred.

[0061] The target input includes at least two messages in at least one modality, and the target answer includes an answer based on the at least two messages.

[0062] It can be understood that the target input may include multimodal media data such as text, pictures, voice, video, expressions (dynamic or static), etc. The intelligent customer service model in the embodiment of the present application is trained based on multimodal data, so it can understand data of any modality in the conversation, and can also send data of any modality that matches user intentions or solves user demands in the conversation interface. The embodiment of the present application takes text conversation as an example and does not impose any limitation on this.

[0063] 203. Determine the target question and sentiment data of the target user according to the modality of each message in the target input and the contextual relationship of each message.

[0064] In some implementations, for the above target input, the target input includes many messages sent by users. If the intelligent customer service model is to quickly give an answer that matches the user's intention within a certain period of time, it is necessary to first infer the contextual relationship between each message in the target input.

[0065] The contextual relationship of each message can be realized based on in-context learning. The large quality inspection model is a large language model (LLM). When training, it can be based on pre-acquired training data (including various task description-corresponding answer pair structure data). The attention layers in LLM can realize contextual reasoning. For example Figure 3a As shown in the figure, the target user is asking about how to use the function of the e-commerce platform's delivery assistant. The intelligent customer service model quickly understands the target user's intention and gives a corresponding operation strategy. Figure 3a The two dotted boxes in the figure indicate that the intelligent customer service model understood the user's intention twice and gave corresponding operation strategies. The total time for effectively solving the user's demands in the conversation was about 8 minutes. It can be seen that the efficiency was high and the user's intention was accurately matched. The communication in the entire conversation was very smooth.

[0066] The above emotional data can also be realized based on the high-order perception capability of LLM, that is, by extracting the deep meaning of the target user's target input and the messages of each mode in the answer of the intelligent customer service big model, and inferring the implicit meaning from the visual and contextual relationship, so as to realize the emotional perception of the target user in the target period. This helps to evaluate whether the conversation between the intelligent customer service big model and the target user meets the indicators of each scoring dimension in Table 2 above. For example, if the current emotion of the target user is identified as anger, it is necessary to perform after-sales processing on the abnormal orders of the target user as soon as possible. For example Figure 3b As shown, the content of the dotted box (not fully marked) is a situation where the user is angry. It can be clearly identified that the current emotional data of the target user is angry and needs to be resolved as soon as possible.

[0067] 204. Call the quality inspection model to perform quality analysis on the target answer based on the target question, the sentiment data, and the correlation between the target answer and the target input.

[0068] In some implementations, the target answer may be quality analyzed in the following manner, including: Obtain the semantic matching degree between the target answer and the target question; analyzing the sentiment data to determine whether the target response matches the expected sentiment; Determine whether the target response contains factual errors and necessary information to match the target input; and determining a logical coherence between a target answer and said target input based on a text generation quality assessment metric; The final quality score is calculated based on the above analysis results and corresponding weights.

[0069] Specifically, the main indicators of quality analysis can be referred to in Table 4 below: Table 4 It can be understood that each quality analysis indicator in the above Table 1 is assigned a corresponding weight, and the basis for the weight value is not limited.

[0070] 205. Set a preset label for the answer whose quality does not meet the preset first condition, where the preset label indicates that the score of the answer is lower than a preset threshold.

[0071] For example, based on the context, we can analyze the user's question type and emotional changes, match the response strategy and embed the answer. Figure 3b , Figure 3c The positive sample shown in Example 1 (the entire conversation record between the user and the customer service) shows that during the communication between the user and the customer service, the user's emotions have repeatedly become angry and even to the point of complaining ( Figure 3b), and then the customer service responded with a series of responses ( Figure 3c ), from the beginning to the end, the customer service did not give any excessive responses that would further aggravate the situation due to the user's worsening emotions. Instead, they comforted the user without emotion, perfectly solving the user's demands and avoiding the situation where they did not understand the user's intentions and could not quickly solve the problem for the user, which ultimately led to user complaints or user loss.

[0072] For example, Figure 3d Among the negative samples shown, Examples 1, 2, and 3 did not understand the user's intention, but instead used template language to answer. The user's intention was not understood, and the user's demands were not resolved in a timely manner, which can be understood as invalid conversations.

[0073] In an embodiment of the present application, after calling the target conversation interface between the intelligent customer service big model and the target user, since the target answer is the answer of the intelligent customer service big model to at least two messages in at least one modality of the target input of the target user within the target time period, the intelligent customer service big model can understand the multimodal messages, and clarify the target user's target questions and emotional data from the multimodal messages and the contextual relationships of each message. Therefore, the obtained target questions and emotional data can provide strong support for the subsequent quality inspection big model to evaluate the quality of the target answer.

[0074] Since the quality inspection model is a prophetic evaluation model, it can perform quality analysis on the target answer based on the above target question, sentiment data, and the correlation between the target answer and the target input, and set a preset label for the answer whose quality does not meet the preset first condition. The quality analysis can accurately understand the content of this conversation, rather than just judging based on simple preset machine rules. The quality inspection model is based on both data and knowledge drive, so the output of the quality inspection model is controllable, reliable, and explainable. It can be seen that, on the one hand, the use of this solution can significantly improve the efficiency of quality inspection; on the other hand, it can clarify user intentions, reduce misjudgments of user intentions and the answers of the intelligent customer service model, and provide strong data support for the subsequent optimization of the intelligent customer service model.

[0075] Optionally, in some embodiments of the present application, in a real-time conversation scenario, the AI ​​customer service can contact the quality inspection model in the background when receiving users, conduct quality inspection on the questions being received online, and find problems in time. If the quality inspection model finds that the intelligent customer service answers randomly or does not solve the problem, it can respond in real time. Specifically, after setting the first label for the answer that does not meet the preset first condition, the method also includes: generating a second prompt message, wherein the second prompt message is used to prompt that the current answer is a negative answer; In response to the second prompt message, performing context analysis on the target input and the target answer to determine the actual demand of the target user; A target strategy matching the actual demand is obtained and displayed on the target session interface.

[0076] It can be seen that by introducing a quality inspection model to perform quality inspection on real-time conversations, problems with the answers of the intelligent customer service model can be discovered in a timely manner, facilitating timely response and correction, thus avoiding ineffective waiting for users and even worsening of the situation.

[0077] Optionally, in some embodiments of the present application, based on the historical feedback data of users, the answers of the customer service (intelligent customer service model or manual customer service) are reinforced and trained, and saved as high-quality data to further improve the answer quality of the intelligent customer service model. Specifically, after performing quality analysis on the target answer, the method further includes: If the first result obtained by the analysis meets the first condition, it is determined that the target answer matches the question demand of the target user, and the target answer is updated to the first data set; If the first result obtained by the analysis does not meet the first condition, it is determined that the target answer does not match the question demand of the target user, and the target answer is updated to the second data set; Among them, the first data set is a positive answer data set of the e-commerce platform customer service, and the second data set is a negative answer data set of the e-commerce platform customer service. The first data set and the second data set are used to train and update the intelligent customer service big model.

[0078] It can be seen that by strengthening the training of the customer service model based on user feedback on the answers, it is possible to distinguish between good and bad data and save them, further improving the quality of the response model.

[0079] Optionally, in some embodiments of the present application, in order to accurately and intelligently evaluate the quality of the intelligent customer service model's answers, the intelligent customer service model can be continuously attacked to find out in which risk dimensions the intelligent customer service model has algorithm vulnerabilities. Specifically, the customer service evaluation system also includes an adversarial generation model (the application scenario can be referred to Figure 5 A training framework diagram of the system can be used to quickly discover algorithm vulnerabilities in the intelligent customer service model based on the adversarial generation model. Figure 6 The embodiment shown comprises: 301. Obtain a first prompt word input into the adversarial generation model.

[0080] Among them, the first prompt word includes a prompt word associated with the e-commerce transaction platform, and its mode is not limited.

[0081] The adversarial generation model is used to generate a large number of adversarial e-commerce questions based on the prompt words associated with e-commerce transactions input by users. These adversarial e-commerce questions can be used to conduct adversarial attacks on the intelligent customer service big model, thereby exploring the algorithm vulnerabilities of the tested medical big model. That is, by continuously generating adversarial e-commerce questions and inputting them into the intelligent customer service big model, if the score of the answer output by the intelligent customer service big model is lower than the preset score, it means that the answer of the intelligent customer service big model to the adversarial e-commerce question is not a positive answer, but a negative answer. It can be seen that the intelligent customer service big model has obvious algorithm vulnerabilities in the adversarial e-commerce question, indicating that the robustness of the intelligent customer service big model is general and cannot fully cover the demands of e-commerce users.

[0082] 302. Generate a first set of questions for the e-commerce platform based on the first prompt word.

[0083] The first problem set includes multiple adversarial e-commerce problems generated based on adversarial technology.

[0084] 303. Input the first question set into the intelligent customer service big model to obtain a first answer set output by the intelligent customer service big model.

[0085] Among them, the first answer set is the candidate answers corresponding to each adversarial e-commerce question in the first question set.

[0086] Accordingly, the first answer set is sent to the quality inspection model to achieve quality scoring.

[0087] 304. Call the quality inspection big model to evaluate the first answer set to obtain a first target score set.

[0088] The first target score set includes the target score of each candidate answer in the first answer set.

[0089] 305. Feedback the first target score set to the adversarial generation model until the answer of the intelligent customer service big model meets a preset second condition, then stop the adversarial attack on the intelligent customer service big model. The second condition includes that the number of attacks of the adversarial generation model reaches a preset number, or the score of the answer of the intelligent customer service big model meets the second condition.

[0090] It can be seen that, on the one hand, the preset adversarial generative model generates multiple rounds of adversarial questions and inputs them into the intelligent customer service big model for multiple rounds of question-answer iterative training, and scores the answers of the intelligent customer service big model with the trained quality inspection big model, and then feeds the scores back to the intelligent customer service big model, so that the intelligent customer service big model can continuously learn the key points and differences between positive answers and negative answers, until the question-answering ability is gradually iterated to the optimal. On the other hand, since the adversarial generative model is based on a solid data foundation, adversarial e-commerce problems can be derived and expanded based solely on prompt words, thereby quickly and efficiently generating a massive, multi-dimensional, highly offensive first problem set for attacking the intelligent customer service big model in a very short time (no expert is required to construct but can pass the expert evaluation threshold), which can ensure that the data set used to test the algorithm vulnerabilities of the intelligent customer service big model is guaranteed in terms of testing breadth, diversity, accuracy, and objectivity, thereby ensuring that the adversarial training of the intelligent customer service big model is multi-dimensional and highly offensive. Compared with the traditional method of experts manually defining some questions to attack the trained intelligent customer service big model, this solution can generate a richer set of problems that can be used for attack in a shorter time, so it significantly improves the overall testing efficiency, that is, it can detect algorithm vulnerabilities in the intelligent customer service big model under more risk dimensions in a shorter time, and further improve the robustness of the intelligent customer service big model.

[0091] Optionally, in some embodiments of the present application, in order to accurately and intelligently evaluate the quality of the intelligent customer service model's answers, the intelligent customer service model is continuously attacked by the adversarial generative model. After finding the algorithm loopholes in the intelligent customer service model, the intelligent customer service model can also be optimized online to shorten the optimization cycle. Specifically, the intelligent customer service evaluation system also includes a teacher model (the application scenario can be referred to Figure 7 A training framework diagram of the intelligent customer service model can be used to quickly discover algorithm vulnerabilities of the intelligent customer service model based on the adversarial generative model, and then the quality of the intelligent customer service model's answers is scored based on the quality inspection model, and then the intelligent customer service model is strengthened based on the transferability of the teacher model. For details, please refer to Figure 8 The embodiment shown comprises: 401. Obtain a second prompt word input into the adversarial generation model.

[0092] 402. Generate a second set of questions for the e-commerce platform based on the second prompt word.

[0093] The second problem set includes multiple adversarial e-commerce problems generated based on adversarial technology.

[0094] 403. Input the second question set into the intelligent customer service model to obtain a second answer set.

[0095] Among them, the second answer set is the candidate answers corresponding to each adversarial e-commerce question in the second question set.

[0096] 404. Call the quality inspection big model to evaluate the second answer set to obtain a second target score set, where the second target score set includes a target score for each candidate answer in the second answer set.

[0097] 405. Feedback the second target score set to the intelligent customer service macro model.

[0098] 406. Call the teacher big model to generate an e-commerce adversarial sample and input it into the intelligent customer service big model to perform adversarial enhancement training on the intelligent customer service big model to obtain a target answer corresponding to the e-commerce adversarial sample.

[0099] 407. Feedback the verification result corresponding to the target answer to the intelligent customer service big model until the intelligent customer service big model converges.

[0100] The verification result is a positive attack, or a negative attack and a candidate optimization strategy corresponding to the negative attack.

[0101] For example, if Fig. 9 The following is a comparison diagram of the answers before and after the optimized intelligent customer service model (not fully listed). Fig. 9 Fig1 in the figure is the conversation record 1 between user a and the intelligent customer service model when user a asked the question "Why is it prompted that I cannot publish the product when I am trying to distribute it?" It can be seen that the intelligent customer service model did not understand its intention (specifically: the way of answering the question was not detailed enough and did not actually help the user solve the problem), and the answers were all irrelevant or invalid answers such as template words, which made user a very angry. Fig. 9 Fig. 2 in the figure shows the conversation record 2 between user a and the intelligent customer service model after the robustness of the intelligent customer service model is optimized. When user a asks the same question again, "Why is it prompted that I cannot publish the product when I am placing the product?", it can be seen that the intelligent customer service model can quickly understand its intention and guide the user to provide more detailed relevant information in the answer (for example, Fig. 9 The picture sent by user a in fig2), and the corresponding operation strategies and solutions are given based on these, and the user is satisfied.

[0102] It can be seen that the intelligent customer service big model optimized by this application has higher professionalism, accuracy, security, and better capabilities in platforms that conduct conversations based on multimodal big models (such as e-commerce platforms). For example, the intelligent customer service big model under test is attacked based on the adversarial generation model to explore the algorithm vulnerabilities of the intelligent customer service big model; training is performed based on the prophetic e-commerce customer service data source that is objective, non-discriminatory, and has correct values, so as to enhance the values, professionalism, service attitude, problem-solving ability and other dimensions of the intelligent customer service big model. After optimizing the intelligent customer service big model, the intelligent customer service big model can reduce invalid output, generation of inappropriate content, output of biased and discriminatory results, and avoid the occurrence of inappropriate content caused by attackers bypassing the security protection mechanism of the intelligent customer service big model through prompt injection or adversarial attack, thereby improving the professionalism and security of the intelligent customer service big model.

[0103] Specifically, based on the training framework of the quality inspection big model, the adversarial generation model and the teacher big model, on the one hand, after obtaining the second prompt word input by the user, a large number of adversarial questions are generated based on the adversarial generation model to attack the intelligent customer service big model, so as to quickly explore the algorithm vulnerabilities of the intelligent customer service big model; then the quality inspection big model is called to score the answers output by the intelligent customer service big model. Since the quality inspection big model is trained based on the objective, non-discriminatory, and correct value prophet e-commerce customer service data source, it is possible to enhance the values, professionalism, problem-solving ability and other dimensions of the intelligent customer service big model through continuous feedback, so that after the intelligent customer service big model is optimized through reinforcement learning based on the teacher big model, the intelligent customer service big model can reduce invalid output, inappropriate content generation, bias and discrimination results output, and avoid the attacker bypassing the security protection mechanism of the intelligent customer service big model through prompt injection or adversarial attack to generate inappropriate content, thereby improving the security of the intelligent customer service big model, and to a certain extent, it can also reduce or avoid the occurrence of user complaints or dissatisfaction.

[0104] On the other hand, the training framework of this solution does not require the owner of the intelligent customer service model to optimize the model offline based on the test results given by the platform, which significantly shortens the overall optimization cycle. After each round of optimization, the model security evaluation of the next process can be directly entered, and the cycle continues until the intelligent customer service model converges. It can be seen that the whole process is intelligent and no human participation is required, so an excellent intelligent customer service model can be obtained efficiently, which is especially suitable for massive industrial-level needs and also reduces the training technology threshold of the intelligent customer service model.

[0105] The customer service big model quality evaluation device in the embodiment of the present application has the function of implementing the intelligent customer service big model quality evaluation method provided in the above embodiment, the method for optimizing the intelligent customer service big model, and the training method of the quality inspection big model. The functions can be implemented by hardware, or by hardware executing the corresponding software. The hardware or software includes one or more modules corresponding to the above functions, and the modules can be software and / or hardware.

[0106] See also Fig.10 , Fig.10 The schematic diagram of the structure of the customer service big model quality evaluation device 50 provided in the embodiment of the present application, the customer service big model quality evaluation device 50 can be applied to computing devices in scenarios where model optimization is required, for example, it can be applied to a customer service evaluation system that evaluates an intelligent customer service big model. Specifically, the customer service big model quality evaluation device 50 can deploy an adversarial generation model 501 (optional), at least one quality inspection big model 502 and at least one teacher big model 503 (optional), an input and output module 504, a display module 505 and a processing module 506. Among them, the processing module 506 can be used to Figure 1a-Figure 9 In any embodiment, operations such as training the quality inspection model 502, determining the target questions and emotional data of the target user, and controlling the operations of the adversarial generation model 501, at least one quality inspection model 502 and at least one teacher model 503, the input and output module 504, and the display module 505 are performed. The same is true for other large models. The adversarial generation model 501, at least one quality inspection model 502 and at least one teacher model 503, the input and output module 504, the display module 505, and the processing module 506 can be deployed separately in a computing device or centrally in a computing device, and this application does not limit this.

[0107] In some implementations, the display module 505 is used to display a target conversation interface between the intelligent customer service model and the target user; An input-output module 504 is used to obtain, from the target conversation interface, a target answer of the intelligent customer service model in response to a target input of a target user within a target period of time, wherein the target input includes at least two messages of at least one modality, and the target answer includes an answer based on the at least two messages; A processing module 506 is used to determine the target question and sentiment data of the target user according to the modality of each message in the target input and the contextual relationship of each message; The processing module 506 is also used to call the quality inspection model 502 to perform quality analysis on the target answer based on the target question, the sentiment data, and the correlation between the target answer and the target input; and to set a preset label for the answer whose quality does not meet the preset first condition, wherein the preset label indicates that the score of the answer is lower than a preset threshold.

[0108] Other functions of the adversarial generation model 501, at least one quality inspection model 502, at least one teacher model 503, the input and output module 504, the display module 505, and the processing module 506 can be referred to in Figure 1a-Figure 9 The corresponding embodiments are not described in detail.

[0109] Compared with the prior art, in the embodiments of the present application, on the one hand, by performing quality inspection on the conversation, the content of the conversation can be accurately understood, and judgment is not limited to being made based on the preset rules of the e-commerce platform, so as to timely avoid or reduce misjudgments, thereby significantly improving the quality inspection efficiency and customer service reception quality.

[0110] On the other hand, it can quickly shorten the time for optimizing the intelligent customer service big model. After optimization, the intelligent customer service big model can reduce the output of invalid answers, the generation of inappropriate content, the output of biased and discriminatory results, and avoid the occurrence of inappropriate content by attackers bypassing the security protection mechanism of the intelligent customer service big model through prompt injection or counterattack, thereby improving the security of the intelligent customer service big model.

[0111] The above describes the customer service large model quality evaluation device in the embodiment of the present application from the perspective of modular functional entities. The following describes the large model optimization device in the embodiment of the present application from the perspective of hardware processing.

[0112] Fig.10 Each device shown can have Fig.11 The structure shown, when Fig.11 The large model optimization device shown has the following features: Fig.11 When the structure shown is Fig.11 The processor and transceiver in the device can realize the same or similar functions as the adversarial generation model 501, at least one quality inspection large model 502, at least one teacher large model 503, input and output module 504, display module 505 and processing module 506 provided in the device embodiment corresponding to the device. Fig.11 The memory storage processor in the computer program that needs to be called when executing the above-mentioned intelligent customer service big model quality evaluation method, optimization method based on intelligent customer service big model, and training method of quality inspection big model.

[0113] When the computing device in the embodiment of the present application is a terminal device, the embodiment of the present application also provides a terminal device, such as Fig.12 For the sake of convenience, only the parts related to the embodiments of the present application are shown. For specific technical details not disclosed, please refer to the method part of the embodiments of the present application. The terminal device can be any terminal device including a mobile phone, a tablet computer, a personal digital assistant (PDA), a point of sales (POS), a car computer, etc., taking the mobile phone as an example: Fig.12 FIG. 1 is a block diagram showing a partial structure of a mobile phone related to a terminal device provided in an embodiment of the present application. Fig.12 The mobile phone includes: a radio frequency (RF) circuit 710, a memory 720, an input unit 730, a display unit 740, a sensor 750, an audio circuit 760, a wireless fidelity (Wi-Fi) module 770, a processor 780, and a power supply 790. Those skilled in the art will understand that Fig.12 The mobile phone structure shown in the figure does not constitute a limitation on the mobile phone, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0114] Combine the following Fig.12 A detailed introduction to the various components of the mobile phone: The RF circuit 710 can be used for receiving and sending signals during the process of sending and receiving information or making calls. In particular, after receiving the downlink information of the base station, it is sent to the processor 780 for processing; in addition, the designed uplink data is sent to the base station. Generally, the RF circuit 710 includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (Low Noise Amplifier, LNA), a duplexer, etc. In addition, the RF circuit 710 can also communicate with the network and other devices through wireless communication. The above-mentioned wireless communication can use any communication standard or protocol, including but not limited to the Global System of Mobile communication (Global System of Mobile communication, GSM), General Packet Radio Service (General Packet Radio Service, GPRS), Code Division Multiple Access (Code Division Multiple Access, CDMA), Wideband Code Division Multiple Access (Wideband Code Division Multiple Access, WCDMA), Long Term Evolution (Long Term Evolution, LTE), email, Short Messaging Service (SMS), etc.

[0115] The memory 720 can be used to store software programs and modules. The processor 780 executes various functional applications and data processing of the mobile phone by running the software programs and modules stored in the memory 720. The memory 720 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory 720 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0116] The input unit 730 can be used to receive input digital or character information, and to generate key signal input related to the user settings and function control of the mobile phone. Specifically, the input unit 730 may include a touch panel 731 and other input devices 732. The touch panel 731, also known as a touch screen, can collect the user's touch operation on or near it (such as the user's operation on the touch panel 731 or near the touch panel 731 using any suitable object or accessory such as a finger, stylus, etc.), and drive the corresponding connection device according to a pre-set program. Optionally, the touch panel 731 may include two parts: a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into contact coordinates, and then sends it to the processor 780, and can receive and execute commands sent by the processor 780. In addition, the touch panel 731 can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic waves. In addition to the touch panel 731, the input unit 730 may further include other input devices 732. Specifically, the other input devices 732 may include but are not limited to one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, and the like.

[0117] The display unit 740 can be used to display information input by the user or information provided to the user and various menus of the mobile phone. The display unit 740 may include a display panel 741. Optionally, the display panel 741 may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), etc. Further, the touch panel 731 may cover the display panel 741. When the touch panel 731 detects a touch operation on or near it, it is transmitted to the processor 780 to determine the type of touch event. Subsequently, the processor 780 provides a corresponding visual output on the display panel 741 according to the type of touch event. Although in Fig.12 In the embodiment, the touch panel 731 and the display panel 741 are used as two independent components to realize the input and output functions of the mobile phone, but in some embodiments, the touch panel 731 and the display panel 741 can be integrated to realize the input and output functions of the mobile phone.

[0118] The mobile phone may also include at least one sensor 750, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel 741 according to the brightness of the ambient light, and the proximity sensor may turn off the display panel 741 and / or the backlight when the mobile phone is moved to the ear. As a type of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in all directions (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that identify the posture of the mobile phone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; as for other sensors that can be configured in the mobile phone, such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., they will not be repeated here.

[0119] The audio circuit 760, the speaker 761, and the microphone 762 can provide an audio interface between the user and the mobile phone. The audio circuit 760 can transmit the received audio data to the speaker 761 after converting the electrical signal, which is converted into a sound signal for output; on the other hand, the microphone 762 converts the collected sound signal into an electrical signal, which is received by the audio circuit 760 and converted into audio data, and then the audio data is output to the processor 780 for processing, and then sent to another mobile phone through the RF circuit 77, or the audio data is output to the memory 720 for further processing.

[0120] Wi-Fi is a short-range wireless transmission technology. The mobile phone can help users send and receive emails, browse web pages and access streaming media through the Wi-Fi module 770. It provides users with wireless broadband Internet access. Fig.12A Wi-Fi module 770 is shown, but it is understandable that it is not an essential component of the mobile phone and can be omitted as needed without changing the essence of the invention.

[0121] The processor 780 is the control center of the mobile phone. It uses various interfaces and lines to connect various parts of the entire mobile phone. By running or executing software programs and / or modules stored in the memory 720, and calling data stored in the memory 720, it executes various functions of the mobile phone and processes data, thereby monitoring the mobile phone as a whole. Optionally, the processor 780 may include one or more processing units; optionally, the processor 780 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 780.

[0122] The mobile phone also includes a power supply 790 (such as a battery) for supplying power to various components. Optionally, the power supply can be logically connected to the processor 780 through a power management system, thereby managing functions such as charging, discharging, and power consumption management through the power management system.

[0123] Although not shown, the mobile phone may also include a camera, a Bluetooth module, etc., which will not be described in detail here.

[0124] In an embodiment of the present application, the processor 780 included in the mobile phone also has the function of controlling the execution of any relevant processes in the intelligent customer service big model quality evaluation method performed by the customer service big model quality evaluation device, the optimization method based on the intelligent customer service big model, and the training method of the quality inspection big model.

[0125] The present application also provides a server. Fig.13 , Fig.13 : This is a schematic diagram of a server structure provided in an embodiment of the present application. The server 800 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (full name in English: central processing units, abbreviated in English: CPU) 822 (for example, one or more processors) and a memory 832, and one or more storage media 830 (for example, one or more mass storage devices) storing application programs 842 or data 844. Among them, the memory 832 and the storage medium 830 can be short-term storage or permanent storage. The program stored in the storage medium 830 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the server. Furthermore, the central processing unit 822 can be configured to communicate with the storage medium 830 to execute a series of instruction operations in the storage medium 830 on the server 800.

[0126] The server 800 may also include one or more power supplies 826, one or more wired or wireless network interfaces 850, one or more input and output interfaces 858, and / or one or more operating systems 841, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc.

[0127] Any step in the intelligent customer service big model quality evaluation method, the optimization method based on the intelligent customer service big model, and the training method of the quality inspection big model in the above embodiments can be based on this Fig.13 The structure of the server 800 is shown. For example, the central processor 822 performs the following operations by calling the instructions in the memory 832: Display the target conversation interface between the intelligent customer service model and the target user; Obtaining, from the target conversation interface through the input / output interface 858, a target answer of the intelligent customer service model in response to a target input of a target user within a target period of time, wherein the target input includes at least two messages of at least one modality, and the target answer includes an answer based on the at least two messages; Determining the target question and sentiment data of the target user according to the modality of each message in the target input and the contextual relationship of each message; Calling the quality inspection big model to perform quality analysis on the target answer according to the target question, the sentiment data, and the correlation between the target answer and the target input; A preset label is set for an answer whose quality does not meet the preset first condition, wherein the preset label indicates that the score of the answer is lower than a preset threshold.

[0128] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0129] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and modules described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0130] In the several embodiments provided in the embodiments of the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0131] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0132] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium.

[0133] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0134] The computer program product includes one or more computer instructions. When the computer program is loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a server, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means to another website site, computer, server, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or a data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive Solid State Disk (SSD)), etc.

[0135] The technical solutions provided in the embodiments of the present application are introduced in detail above. The principles and implementation methods of the embodiments of the present application are explained by using specific examples in the embodiments of the present application. The description of the above embodiments is only used to help understand the methods and core ideas of the embodiments of the present application. At the same time, for those skilled in the art, according to the ideas of the embodiments of the present application, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the embodiments of the present application.

Claims

1. A method for evaluating the quality of a large intelligent customer service model, characterized in that: The method is applied to a customer service evaluation system based on a multimodal large model, wherein the customer service evaluation system includes a quality inspection large model driven by both data and knowledge; the method includes: Display the target conversation interface between the intelligent customer service model and the target user; Obtaining, from the target conversation interface, a target answer of the intelligent customer service big model in response to a target input of a target user within a target period of time, wherein the target input includes at least two messages of at least one modality, and the target answer includes an answer based on the at least two messages; Determining the target question and sentiment data of the target user according to the modality of each message in the target input and the contextual relationship of each message; Calling the quality inspection big model to perform quality analysis on the target answer according to the target question, the sentiment data, and the correlation between the target answer and the target input; A preset label is set for an answer whose quality does not meet the preset first condition, wherein the preset label indicates that the score of the answer is lower than a preset threshold.

2. The method according to claim 1, characterized in that The quality inspection model is trained by the following operations: Acquire a priori e-commerce customer service data source, wherein the priori e-commerce customer service data source includes multiple questions of at least one e-commerce platform, at least one positive answer corresponding to each question, and at least one negative answer corresponding to each question; the quality of the positive answer is higher than the quality of the negative answer; Building a mapping relationship between e-commerce questions, positive answers, and negative answers based on the prior e-commerce customer service data source; Dividing the mapping relationship into a training set and a test set; Iteratively training the large quality inspection model based on the training set; The accuracy of the large quality inspection model is verified based on the test set. If the scores of all positive answers are greater than the scores of negative answers, it is determined that the accuracy reaches the preset accuracy and converges.

3. The method according to claim 1 or 2, characterized in that: The customer service evaluation system also includes an adversarial generation model, and the method also includes: Obtaining a first prompt word input into the adversarial generation model; generating a first question set of the e-commerce platform based on the first prompt word, wherein the first question set includes a plurality of adversarial e-commerce questions generated based on adversarial technology; Inputting the first question set into the intelligent customer service big model to obtain a first answer set, where the first answer set is a candidate answer corresponding to each adversarial e-commerce question in the first question set; Calling the quality inspection big model to evaluate the first answer set to obtain a first target score set, wherein the first target score set includes a target score for each candidate answer in the first answer set; The first target score set is fed back to the adversarial generation model until the answer of the intelligent customer service big model meets the preset second condition, then the adversarial attack on the intelligent customer service big model is stopped. The second condition includes that the number of attacks of the adversarial generation model reaches a preset number of times, or the score of the answer of the intelligent customer service big model meets the second condition.

4. The method according to claim 1 or 2, characterized in that: The customer service evaluation system further includes a generative adversarial model and a teacher model, and the method further includes: Obtaining a second prompt word input into the adversarial generation model, and generating a second question set of the e-commerce platform based on the second prompt word, wherein the second question set includes a plurality of adversarial e-commerce questions generated based on adversarial technology; Input the second question set into the intelligent customer service big model to obtain a second answer set, where the second answer set is the candidate answers corresponding to each adversarial e-commerce question in the second question set; Calling the quality inspection big model to evaluate the second answer set to obtain a second target score set, where the second target score set includes a target score for each candidate answer in the second answer set; Feeding back the second target score set to the intelligent customer service big model; Calling the teacher big model to generate an e-commerce adversarial sample and inputting it into the intelligent customer service big model to perform adversarial enhancement training on the intelligent customer service big model to obtain a target answer corresponding to the e-commerce adversarial sample; Feedback the verification result corresponding to the target answer to the intelligent customer service big model until the intelligent customer service big model converges; wherein the verification result is a positive attack, or a negative attack and a candidate optimization strategy corresponding to the negative attack.

5. The method according to claim 4, characterized in that After setting the first label for the answer that the quality does not meet the preset first condition, the method further includes: Generate a first prompt message, wherein the first prompt message is used to prompt that the emotion of the target user is of a negative type; In response to the first prompt message, performing context analysis on the target input and the target answer to determine the actual demands of the target user; A first strategy matching the actual demand is obtained and displayed on the target conversation interface, wherein the first strategy includes at least one of a soothing plan, an anthropomorphic speech, a solution matching the actual demand, and a manual customer service entrance.

6. The method according to claim 4, characterized in that After setting the first label for the answer that the quality does not meet the preset first condition, the method further includes: generating a second prompt message, wherein the second prompt message is used to prompt that the current answer is a negative answer; In response to the second prompt message, performing context analysis on the target input and the target answer to determine the actual demand of the target user; A target strategy matching the actual demand is obtained and displayed on the target session interface.

7. The method according to claim 1, characterized in that After performing quality analysis on the target answer, the method further includes: If the first result obtained by the analysis meets the first condition, it is determined that the target answer matches the question demand of the target user, and the target answer is updated to the first data set; If the first result obtained by the analysis does not meet the first condition, it is determined that the target answer does not match the question demand of the target user, and the target answer is updated to the second data set; Among them, the first data set is a positive answer data set of the e-commerce platform customer service, and the second data set is a negative answer data set of the e-commerce platform customer service. The first data set and the second data set are used to train and update the intelligent customer service big model.

8. A customer service large model quality evaluation device, characterized in that: The large model quality evaluation device comprises: at least one processor and memory; The memory is used to store a computer program, and the processor is used to call the computer program stored in the memory to execute the method according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that: The method comprises instructions which, when executed on a computer, cause the computer to execute the method according to any one of claims 1 to 7.

10. A computer program product comprising instructions, characterized in that The computer program product comprises program instructions, and when the program instructions are executed on a computer or a processor, the computer or the processor is caused to execute the method according to any one of claims 1 to 7.

Images