Online customer service response method, device, equipment, storage medium and product

Abstract

Embodiments of the present application provide an online customer service response method, device, equipment, storage medium and product. According to the embodiments of the present application, the user input information is obtained, the information type corresponding to the user input information is identified through a classification model, the user input information and the information type corresponding to the user input information are sent to a text extraction model, the user input information and the information type corresponding to the user input information are extracted by the text extraction model, the full-text information is obtained, the user input information is sent to an abstract extraction model, the full-text information is extracted by the abstract extraction model, the abstract information is obtained, the abstract information and the full-text information are sent to a large model, the large model obtains the reply text and the response template according to the abstract information and the full-text information, and the reply text is determined as the target reply according to the response template, so that the reply accuracy of the online customer service can be improved.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence, and in particular relates to an online customer service response method, apparatus, device, storage medium and product. Background Technology

[0002] Currently, online customer service on mobile applications can receive user questions and provide corresponding answers. For example, after receiving a user's question, it will search for the corresponding answer in a pre-set question-and-answer database and display it.

[0003] In existing technologies, online customer service relies on a pre-set question and answer database to answer users' questions. When a user asks a question to online customer service, the online customer service provides a response by matching the user's question with questions in the database.

[0004] However, the question and answer database has limited content and cannot cover all questions asked by users. Therefore, when users ask new questions, online customer service cannot provide accurate answers. Summary of the Invention

[0005] This invention provides an online customer service response method, apparatus, device, storage medium, and product that can improve the accuracy of online customer service responses.

[0006] In a first aspect, embodiments of the present invention provide an online customer service response method, the method comprising:

[0007] Obtain user input information, input the user input information into a classification model, and identify the information type of the user input information through the classification model;

[0008] The user input information and the information type are input into the text extraction model, and the text is extracted by the text extraction model to obtain the full text information.

[0009] The user input information is input into the summary extraction model, and the summary extraction model is used to extract the summary information.

[0010] The summary information and the full text information are input into the large model, and the large model generates the response text and response template.

[0011] Based on the response text and the response template, a target response is generated.

[0012] In one feasible implementation, the method further includes:

[0013] The summary information and the full text information are input into the large model, and the user intent is determined through the large model.

[0014] Based on an extended database and a question-and-answer database, the user intent is converted into the response text. The extended database includes extended information related to the field of communication, and the question-and-answer database includes pre-set question information and multiple response information corresponding to the question information.

[0015] Based on the response template library, the response template corresponding to the user intent is determined. The response template library has a pre-set correspondence between the user intent and the response template.

[0016] In one feasible implementation, the method further includes:

[0017] Based on the extended database, determine the extended information corresponding to the user intent;

[0018] Based on the question-and-answer database, determine the question information and the response information corresponding to the user's intent;

[0019] Based on the extended information, the question information, and the response information, a reply prompt message is generated;

[0020] The user intent and the response prompt information are input into the large model, and the user intent features and response prompt features are extracted through the large model;

[0021] The user intent features and the response prompt features are fused using a large model, and the response is generated based on the fused features to obtain the response text.

[0022] In one feasible implementation, the method further includes:

[0023] When the response template includes a reserved image space, the image corresponding to the response text is searched from the image library according to the response text;

[0024] According to the response template, the image and the response text are identified as the target response.

[0025] In one feasible implementation, the method further includes:

[0026] Acquire training set data and validation set data. The training set data includes a labeled dataset and an unlabeled dataset. Both the labeled dataset and the unlabeled dataset include: text input, voice input, and image input.

[0027] The labeled data is input into the initial multimodal multitask model for training to obtain the first multimodal multitask model;

[0028] The unlabeled data is input into the first multimodal multi-task model for pseudo-label prediction to obtain the pseudo-label corresponding to each unlabeled data in the unlabeled dataset;

[0029] Calculate the similarity between the pseudo-label of each unlabeled data and the label of each labeled data. If the similarity is higher than a first preset threshold, add the unlabeled data and the corresponding pseudo-label to the labeled dataset to obtain the first labeled dataset.

[0030] Based on the spectral clustering algorithm, the labeled data in the first labeled dataset are feature-enhanced to obtain the first labeled dataset with enhanced features.

[0031] The first multimodal multitask model is trained using the first labeled dataset with enhanced features to obtain the second multimodal multitask model.

[0032] The validation set data is input into the second multimodal multitask model to obtain a loss function value. If the loss function value is less than or equal to a second preset threshold, the second multimodal multitask model is a trained second multimodal multitask model. If the loss function value is greater than the second preset threshold, the network parameters of the second multimodal multitask model are adjusted according to the loss function value, and the process returns to the step of inputting the unlabeled dataset into the first multimodal multitask model for pseudo-label prediction to obtain the pseudo-label corresponding to each unlabeled data in the unlabeled dataset.

[0033] In one feasible implementation, the method further includes:

[0034] Based on the upper bound of the confidence interval, the weights corresponding to text input, voice input, and image input are determined in each round of training.

[0035] Based on the weights, the proportions of text input, voice input, and image input are determined in each round of training.

[0036] Secondly, embodiments of the present invention provide an online customer service response device, the device comprising:

[0037] The acquisition module is used to acquire user input information, input the user input information into a classification model, and identify the information type of the user input information through the classification model.

[0038] The first processing module is used to input the user input information and the information type into the text extraction model, and extract the text through the text extraction model to obtain the full text information.

[0039] The second processing module is used to input the user input information into the summary extraction model, and perform summary extraction through the summary extraction model to obtain summary information;

[0040] The third processing module is used to input the summary information and the full text information into the large model, and generate the response text and response template through the large model;

[0041] The fourth processing module is used to generate a target response based on the response text and the response template.

[0042] Thirdly, embodiments of the present invention provide an electronic device, the device comprising: a processor, and a memory storing computer program instructions; the processor reads and executes the computer program instructions to implement the online customer service response method as described in any one of the first aspects.

[0043] Fourthly, embodiments of the present invention provide a computer storage medium storing computer program instructions, which, when executed by a processor, implement the online customer service response method as described in any one of the first aspects.

[0044] Fifthly, embodiments of the present invention provide a computer program product, wherein instructions in the computer program product, when executed by a processor of an electronic device, cause the electronic device to perform the online customer service response method as described in any one of the first aspects.

[0045] This invention discloses an online customer service response method, apparatus, device, storage medium, and product. It acquires user input information, inputs this information into a classification model to identify the information type, and then inputs the user input information and its type into a text extraction model. The text extraction model extracts text based on the user input information and its corresponding information type to obtain full text information, enabling responses to image and voice inputs. Next, the user input information is input into a summary extraction model to extract a summary. This summary information, along with the full text information, is input into a large model. The large model then generates a response text and a response template based on the summary and full text information. Instead of relying on a pre-configured question-and-answer database, the large model can respond to any question, providing accurate answers regardless of whether the user's question is in the database. Furthermore, the large model responds based on both summary and full text information, using the summary information to guide it and focus on the core content and key points of the full text information, resulting in more accurate response text and improving the accuracy of online customer service responses. Attached Figure Description

[0046] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments of the present invention will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 A flowchart illustrating an embodiment of the online customer service response method provided by the present invention;

[0048] Figure 2 A flowchart illustrating a second embodiment of the online customer service response method provided by the present invention;

[0049] Figure 3 This is a flowchart illustrating an online customer service response method provided by the present invention;

[0050] Figure 4 This is a schematic diagram of the structure of an online customer service response device provided in an embodiment of the present invention;

[0051] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0052] The features and exemplary embodiments of various aspects of the present invention will now be described in detail. To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely intended to explain the present invention and not to limit it. For those skilled in the art, the present invention can be practiced without some of these specific details. The following description of the embodiments is merely to provide a better understanding of the present invention by illustrating examples of the invention.

[0053] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0054] In existing technologies, online customer service relies on a pre-configured question-and-answer database to answer users' questions. When a user's question does not exist in the database, the online customer service will determine an answer to a similar question and output it. However, the output answer is inaccurate at this time.

[0055] Based on this, the inventive concept of the present invention is to provide an online customer service response method, device, equipment, storage medium and product that is not limited by a pre-configured question and answer database, so that online customer service can still provide accurate answers regardless of whether the user's question is in the pre-configured question and answer database.

[0056] To address the problems of the prior art, embodiments of the present invention provide an online customer service response method, apparatus, device, storage medium, and product.

[0057] The online customer service response method provided in the embodiments of the present invention will be introduced first below.

[0058] Figure 1 This is a flowchart illustrating an embodiment of the online customer service response method provided by the present invention. Figure 1 As shown, the method may include the following steps:

[0059] S101: Obtain user input information, input the user input information into the classification model, and identify the information type of the user input information through the classification model.

[0060] S102: Input the user input information and information type into the text extraction model, and extract the text through the text extraction model to obtain the full text information.

[0061] S103: Input the user input information into the summary extraction model, and extract the summary information through the summary extraction model.

[0062] S104: Input the summary information and full text information into the large model, and generate the response text and response template through the large model.

[0063] S105: Generate the target response based on the response text and response template.

[0064] The specific implementation methods for each of the above steps are described below.

[0065] Step S101: Obtain user input information, input the user input information into the classification model, and identify the information type of the user input information through the classification model.

[0066] In this embodiment, user input information is acquired. In one example, the user input information includes text input, and / or voice input, and / or image input. A classification model is used to identify the information type corresponding to the user input information. In one example, the classification model pre-stores the correspondence between information types and file formats. When the classification model receives user input information, it identifies the corresponding information type based on the file format of the input information. For example, when the user input information is in JPG or PNG image format, the classification model classifies the user input information as image input. It is understood that the user input information type is not singular; that is, the user input information may include text input, voice input, or image input simultaneously.

[0067] Step S102: Input the user input information and information type into the text extraction model, and extract the text through the text extraction model to obtain the full text information.

[0068] In this embodiment, after the classification model identifies the information type corresponding to the user input information, it sends the user input information and its corresponding information type to the text extraction model. The text extraction model then extracts the text from the user input information to obtain the full text information. For example, the classification model includes an OCR (Optical Character Recognition) image recognition model and an ASR (Automatic Speech Recognition) speech recognition model. The OCR image recognition model can extract text information from images to obtain the full text information, while the ASR speech recognition model can extract text information from audio to obtain the full text information.

[0069] Understandably, when the user input information is not of a single type—that is, when the user input information includes text input, voice input, and image input—the classification model will send the image input portion of the user input information to the OCR image recognition model in the text extraction model. The OCR image recognition model will extract the text information from the image input. The classification model will send the voice input portion of the user input information to the ASR speech recognition model in the text extraction model. The ASR speech recognition model will extract the text information from the voice input. Finally, the obtained text information from the text input, the image input, and the voice input will be combined to obtain the complete text information.

[0070] In this embodiment, the classification model categorizes user input information to identify whether it is text input, image input, or voice input. Then, the text extraction model extracts the text to obtain the full text information. Therefore, the online customer service response method of the present invention can not only answer text questions, but also receive users' image questions and voice questions, thereby improving the user experience.

[0071] Step S103: Input the user input information into the summary extraction model, and extract the summary information through the summary extraction model.

[0072] In this embodiment, user input information can be input into the summary extraction model, and the summary extraction model can extract the user input information to obtain the summary information corresponding to the user input information.

[0073] In one example, the summary extraction model could be a standalone multimodal multitask learning model (UniT). Upon receiving user input, the input is sent to the multimodal multitask model for summary extraction, yielding the summary information corresponding to the user input. The multimodal multitask model consists of a shared Transformer encoder, a Transformer decoder for each modality, and classifiers corresponding to multiple heterogeneous detection tasks. Compared to previous Transformer-based multitask models, the multimodal multitask model eliminates the need for fine-tuning methods. Model parameters can be shared across different domains, and common knowledge can be shared across different tasks. For highly correlated task sets, the multimodal multitask model achieves stronger performance with fewer parameters.

[0074] In another example, the summary extraction model integrates a classification model, a text extraction model, and a multimodal multitasking model. When the summary extraction model receives user input, the classification model categorizes the user input, and inputs that are voice or image input are fed into the text extraction model. The text extraction model extracts the full text information based on the user input. Then, the multimodal multitasking model receives the full text information and extracts summary information based on it.

[0075] Step S104: Input the summary information and full text information into the large model, and generate the response text and response template through the large model.

[0076] In this embodiment, the full text information, along with the summary information extracted from the multimodal multi-task model, is sent together to the large model. This allows the large model to obtain the response text and response template based on the summary information and the full text information. The summary information serves as a guide, making the large model's response to the full text information more accurate. Therefore, the response text obtained by the large model based on the summary information and the full text information is more accurate.

[0077] In one example, the large model is an artificial intelligence model, which can be fine-tuned based on question and response texts from a historical database to obtain a large model capable of answering questions in the field of communication. Specifically, question and response text pairs are obtained from the historical database, and the question and response text pairs are cleaned to remove null and error values, resulting in cleaned question and response text pairs. Then, the pre-trained artificial intelligence model is fine-tuned using the cleaned question and response text pairs. In one example, the fine-tuning method includes full fine-tuning, partial fine-tuning, or efficient parameter fine-tuning.

[0078] To make the response text obtained by the large model based on the summary information and the full text information more accurate, as another implementation of the present invention, a specific implementation of S104 may further include the following steps:

[0079] The summary information and the full text information are input into the large model, and the user intent is determined by the large model.

[0080] In this embodiment, the large model determines the user intent based on the summary information and the full text information. The summary information is the information after the full text information is extracted and summarized, retaining the core content and key points of the full text information. The summary information guides the large model to focus on the core content and key points of the full text information, rather than secondary information. Therefore, the user intent obtained by the large model is more accurate.

[0081] For example, given the user's input text "I'm going to Europe for a week on business next month, how do I activate international roaming on my mobile data?", the corresponding summary information would be "Activate a European international roaming data plan". The large model encodes the full text information to obtain a feature vector corresponding to the full text, and encodes the summary information to obtain a feature vector corresponding to the summary information. A cross-attention mechanism is used to calculate weights for the feature vectors corresponding to the full text information and the feature vectors corresponding to the summary information, generating a fused feature vector. Finally, based on the fused feature vector, the user intent is output. Because redundant content in the full text information is filtered out with low weights, a more accurate user intent can be obtained.

[0082] Based on an extended database and a question-and-answer database, user intent is converted into response text. The extended database includes extended information related to the communication field, and the question-and-answer database includes pre-set question information and multiple response information corresponding to the question information.

[0083] In this embodiment, the large model combines an extended database and a question-and-answer database to convert user intent into response text. Based on the above-mentioned user intent obtained from summary information and full text information, the introduction of the extended database and question-and-answer database guides the process of the large model obtaining response text based on user intent, which can further improve the accuracy of the large model in generating response text.

[0084] The extended database includes information related to the communications field, such as data plans, voice plans, SIM card activation information, number selection information, or knowledge graphs corresponding to the communications field. The construction process for the knowledge graph corresponding to the communications field is as follows: unstructured raw communication data is obtained from communications-related websites or APIs; structured raw communication data is extracted from the operator's internal database; the raw communication data is cleaned to obtain target communication data; then, according to data modeling methodologies, such as defining entity-relationship-entity based on schema design, the target communication data is extracted according to entity-relationship-entity using triple extraction techniques to obtain multiple entity-relationship-entity triples; finally, the multiple triples are fused to obtain the knowledge graph corresponding to the communications field.

[0085] In addition, the question-and-answer database includes pre-set question information and multiple corresponding answer information for each question. The question and answer information in the question-and-answer database is constantly updated. For example, after the large model obtains the answer text based on the user input information, the large model will store the user input information and the answer text in the question-and-answer database to update the question-and-answer database. This allows the large model to refer to existing answers when it encounters similar questions in the future and generate more accurate and comprehensive answers. In one example, when storing multiple answer versions for the same question, the generation time and user satisfaction of each answer version are recorded, and the answer information with the later generation time and higher user satisfaction is selected first.

[0086] Based on the response template library, the response template corresponding to the user intent is determined. The response template library has a pre-set correspondence between user intent and response template.

[0087] In this embodiment, the response template library pre-stores the correspondence between user intents and response templates. Once the large model determines the user intent, it queries the response template library for the corresponding response template. In one example, the correspondence between user intents and response templates stored in the response template library is continuously updated. For instance, response templates are updated based on user satisfaction. If a user consistently gives the same response template a very low rating, manual review and intervention are triggered to modify the corresponding response template.

[0088] In this embodiment, user intent is determined based on summary information and full text information. The summary information guides the large model, enabling it to focus on the core content and key points of the full text information, thus obtaining a more accurate user intent. Then, based on the accurate user intent, the generated response text is made compliant with communication domain knowledge by expanding the database and reusing experience from the question-and-answer database. Furthermore, combining high-scoring historical question and response information improves the accuracy of the response text and user satisfaction. In addition, by combining a response template library to determine response templates corresponding to user intent, a consistent response style is achieved, eliminating the need for users to adapt to varying wording and allowing for a more accurate understanding of the online customer service's target response.

[0089] To further improve the accuracy of the response text generated by the large model, a specific implementation method for converting the user intent into response text, combining an extended database and a question-and-answer database, is as follows:

[0090] Based on the extended database, determine the extended information corresponding to the user's intent.

[0091] In this embodiment, the extended information corresponding to the user's intent is searched in the extended database. For example, if the user intent obtained from the large model is "European international roaming service", then the extended information related to "European international roaming service" is searched in the extended database according to the user intent. The extended information is the tariff standard and data package of "European international roaming service". In one example, the extended information can be obtained by using "Europe" and "international roaming service" as query conditions.

[0092] Based on the question-and-answer database, determine the question and answer information corresponding to the user's intent.

[0093] In this embodiment, the question information that is closest to the user's intent is determined in the question-and-answer database, and the response information is determined based on the question information. In one example, the user's intent can be encoded into a semantic vector, and then the question information with a similarity greater than a preset threshold to the user's intent is searched in the question-and-answer database. The response information corresponding to the question information is then queried based on the question information.

[0094] Based on the extended information, question information, and response information, a reply prompt message is generated.

[0095] In this embodiment, a response prompt is generated based on extended information, question information, and response information.

[0096] Input user intent and response prompts into a large model, and then extract user intent features and response prompt features from the large model.

[0097] In this embodiment, the user intent and response prompt information are sent to the large model. The large model extracts user intent features based on the user intent and response prompt features based on the prompt information.

[0098] The large model integrates user intent features and response prompt features, and then uses the fused features to generate a response text.

[0099] In this embodiment, the large model fuses the obtained user intent features and response prompt features to obtain fused features. Then, the large model responds based on the fused features to obtain the response text.

[0100] In this embodiment, extended information corresponding to the user's intent is determined based on an extended database, and question and response information corresponding to the user's intent are determined based on a question-and-answer database. The extended information, question information, and response information are then defined as prompt information. This prompt information combines information from both the extended and question-and-answer databases. The extended database includes extended information related to the communication domain, subjecting the large model to knowledge constraints during response text generation—meaning the response text must be relevant to the communication domain. The question-and-answer database includes pre-set question information and multiple response information corresponding to the question information, allowing the large model to output response text similar to historical question and response information. The large model then performs feature fusion based on user intent features and response prompt features to obtain fused features, and generates the response text based on these fused features. Thus, while ensuring the accuracy of the user intent, the introduction of prompt information guides the large model's response text generation process, transforming it from "free association" to "directed reasoning," thereby further improving the accuracy of the large model's response text generation.

[0101] Step S105: Generate the target response based on the response text and response template.

[0102] In this embodiment, after obtaining the response text and the response template, the response text is organized into the target response according to the format of the response template.

[0103] To improve user experience, when a user asks how to conduct a transaction, detailed steps for conducting the transaction are output and images related to the steps are displayed. As another implementation of the present invention, a specific implementation of S105 may further include the following steps:

[0104] When the response template includes a reserved image space, search the image library for an image that corresponds to the response text, based on the response text.

[0105] In this embodiment, when the obtained response template includes a reserved image location, the image corresponding to the response text is searched from the image library. In one example, the response template is an operation step type response template, so the screenshot corresponding to the business processing is searched from the image library, such as a screenshot of the login application interface or a screenshot of the package selection interface.

[0106] According to the response template, the image and response text are identified as the target response.

[0107] In this embodiment, according to the response template, the image and the response text are jointly determined as the target response. If the response text determined by the large model is: "Hello, you can change your plan by following these steps: 1. Log in to the application interface; 2. Go to the 'My Plans' page; 3. Select 'Change Plan'; 4. Choose your preferred plan and confirm the change.", then a screenshot of the application interface login will be attached next to "1. Log in to the application interface" and a screenshot of the plan selection interface will be attached next to "2. Go to the 'My Plans' page."

[0108] In this embodiment, by outputting detailed steps for handling business and displaying relevant images, users can more intuitively understand how to operate the system without having to search for or consult customer service, thus improving the user experience.

[0109] In this embodiment, user input information is acquired, and the information type corresponding to the user input information is identified through a classification model. The user input information and its corresponding information type are then sent to a text extraction model. The text extraction model extracts text based on the user input information and its corresponding information type to obtain the full text information, enabling responses to image and voice inputs. The user input information is then sent to a multimodal multi-task model, which extracts a summary from the full text information. This summary information, along with the full text information, is sent to a large model. The large model then generates a response text and a response template based on the summary information and the full text information. Instead of relying on a pre-configured question-and-answer database, the large model responds to user input information based on any question information. This ensures that the large model can provide an accurate response regardless of whether the user's question is in the question-and-answer database. Furthermore, the large model responds based on both the summary information and the full text information. The summary information guides the large model, allowing it to focus on the core content and key points of the full text information, resulting in a more accurate response text and improving the accuracy of online customer service responses.

[0110] Figure 2 This is a flowchart illustrating Embodiment 2 of the online customer service response method provided by the present invention, as follows: Figure 2 As shown, when the abstract extraction model is a multimodal multi-task model, before step S103, the following steps are also included:

[0111] S201: Obtain training set data and validation set data. The training set data includes labeled datasets and unlabeled datasets. Both labeled and unlabeled datasets include: text input, voice input, and image input.

[0112] In this embodiment, since the input types include text input, voice input and image input, the cost of annotating data is high, resulting in high training cost of multimodal multitask models. Therefore, in order to reduce the model training cost, this invention uses a consistent semi-supervised learning method to train multimodal multitask models.

[0113] Specifically, the acquired training set data includes both labeled and unlabeled datasets, with the labeled dataset comprising a smaller proportion than the unlabeled dataset. For example, 20% of the labeled data is used for consistency-based semi-supervised learning training of a multimodal, multi-task model. It should be noted that the labels on the labeled data are annotation summary information; for example, "Insufficient data allowance, any data plans recommended for this month?" would correspond to the summary information "Recommended monthly data plan". Both labeled and unlabeled data include text input, voice input, and image input.

[0114] S202: Input labeled data into the initial multimodal multitask model for training to obtain the first multimodal multitask model.

[0115] In this embodiment, the initial multimodal multitasking model is trained using a labeled dataset to obtain the first multimodal multitasking model.

[0116] S203: Input the unlabeled data into the first multimodal multi-task model to perform pseudo-label prediction and obtain the pseudo-label corresponding to each unlabeled data in the unlabeled dataset.

[0117] In this embodiment, unlabeled data is input into the first multimodal multi-task model, which generates corresponding pseudo-labels based on the unlabeled data, and finally obtains the pseudo-labels corresponding to each unlabeled data in the unlabeled dataset.

[0118] S204: Calculate the similarity between the pseudo-label of each unlabeled data and the label of each labeled data. If the similarity is higher than the first preset threshold, add the unlabeled data and the corresponding pseudo-label to the labeled dataset to obtain the first labeled dataset.

[0119] In this embodiment, the similarity between the pseudo-label of each unlabeled data and the label of each labeled data is calculated. In one example, the similarity between the pseudo-label and the label of the labeled data is calculated based on the JS divergence algorithm. If the similarity between the two is higher than a first preset threshold, the unlabeled data and the corresponding pseudo-label are added to the labeled dataset to expand the labeled dataset and obtain the first labeled dataset.

[0120] S205: Based on the spectral clustering algorithm, feature enhancement is performed on the labeled data in the first labeled dataset to obtain the first labeled dataset with enhanced features.

[0121] In this embodiment, feature enhancement is performed on the labeled data in the first labeled dataset based on the spectral clustering algorithm to obtain a feature-enhanced first labeled dataset. In one example, if there are m labeled data in the first labeled dataset, where m is a positive integer, then feature extraction is performed on the m labeled data to obtain m first features. Spectral clustering is then performed on the m first features to obtain multiple category prototype features and first feature groups corresponding to the multiple category prototype features. Each first feature group includes multiple first features. Among the multiple category prototype features, the target category prototype feature that is closest to each first feature is determined. For each first feature, feature enhancement is performed using the target category prototype feature to obtain the feature-enhanced first labeled dataset.

[0122] S206: Train the first multimodal multitask model using the first labeled dataset with enhanced features to obtain the second multimodal multitask model.

[0123] In this embodiment, a first multimodal multitask model is trained on a first labeled dataset with enhanced features to obtain a second multimodal multitask model. In one example, the first multimodal multitask model is used to summarize the first labeled dataset with enhanced features to obtain predicted labels. The consistency loss function value and cross-entropy loss function value corresponding to the predicted labels are determined. The consistency loss function value is used to measure the consistency between the predicted labels and the pseudo-labels, promoting unsupervised learning. The cross-entropy loss function value is used to measure the difference between the predicted labels and the true labels, ensuring the accuracy of supervised learning. The consistency loss and cross-entropy loss are weighted and combined to determine the target loss function value. Based on the target loss function value, the training parameters of the first multimodal multitask model are adjusted. The above steps are repeated until the target loss function value converges to obtain the second multimodal multitask model.

[0124] To prevent overfitting or underfitting during the training of multimodal multitask models, the training of the first-modal multitask model includes:

[0125] Based on the upper bound of the confidence interval, the weights corresponding to text input, voice input, and image input are determined in each round of training.

[0126] In this embodiment, since the multimodal multitask model fits different tasks (text input, image input, and voice input) at different speeds, to prevent overfitting or underfitting of some tasks, a task selector is designed based on the confidence interval upper bound algorithm to automatically adjust the proportion of different tasks in the next training round. The task selector is driven by the model validation results on the validation set, and continuously and dynamically adjusts the proportion of task iterations to ensure that the model has a relatively balanced recognition ability for different tasks.

[0127] In one example, after each round of training, the multimodal multitask model calculates the recognition accuracy of each task on the validation set, and then calculates the upper bound of the confidence interval:

[0128]

[0129] Among them, UCB i μ represents the upper bound of the confidence interval for task i. i N represents the historical average accuracy of task i. i Let represent the number of training rounds for task i, T represent the total number of training rounds, lnT represent the logarithm of the total number of training rounds, and c represent the exploration coefficient.

[0130] Based on the weights, the proportions of text input, voice input, and image input are determined in each round of training.

[0131] After normalizing the UCB values ​​corresponding to text input, image input, and voice input, the training weights for each task in the next round are determined. Based on the weights, the proportion of text input, voice input, and image input data in the training batch is adjusted, prioritizing training opportunities for tasks with low UCB.

[0132] In this embodiment, based on the confidence interval upper bound algorithm, the proportion of different input type tasks in each training process is dynamically adjusted to balance the recognition ability of the first multimodal multi-task model for different input type tasks. This ensures that even when the input type samples are unbalanced, such as more text input and fewer image input, the multimodal multi-task model can still have a balanced recognition ability.

[0133] S207: Input the validation set data into the second multimodal multitask model to obtain the loss function value. If the loss function value is less than or equal to the second preset threshold, the second multimodal multitask model is a trained second multimodal multitask model. If the loss function value is greater than the second preset threshold, adjust the network parameters of the second multimodal multitask model according to the loss function value and return to S203.

[0134] In this embodiment, the second multimodal multitask model is validated using validation set data to obtain a loss function value. If the loss function value is greater than a second preset threshold (in one example, the second preset threshold is a value revised from the first preset threshold based on data characteristics and scenario requirements), the hyperparameters of the model are adjusted according to the loss function value, and the process returns to step S203 to continue predicting pseudo-labels for unlabeled data. That is, the pseudo-labels for unlabeled data are predicted using the second multimodal multitask model, and the similarity between the pseudo-label of each unlabeled data and the labeled data is calculated. Unlabeled data with similarity higher than the first threshold are added back to the labeled dataset to continuously partition the data and obtain an expanded labeled dataset until the loss function value of the multimodal multitask model converges, thus obtaining a trained multimodal multitask model.

[0135] In this embodiment, labeled and unlabeled datasets are acquired. The labeled dataset is used to train an initial multimodal multitasking model, resulting in a first multimodal multitasking model. Based on this first model, pseudo-label prediction is performed on the unlabeled dataset. Unlabeled data with similarity to labeled data exceeding a preset threshold are selected and added to the labeled dataset, forming the first labeled dataset. Label prediction on the unlabeled dataset expands the labeled dataset, reducing the cost of manual data annotation and thus lowering model training costs. Furthermore, spectral clustering is used to enhance the features of the labeled data in the first labeled dataset, resulting in a feature-enhanced first labeled dataset. The first multimodal multitasking model is then trained on this enhanced dataset until the loss function converges, yielding a trained multimodal multitasking model. Using spectral clustering to enhance the features of the training data during model training improves the training accuracy of the multimodal multitasking model.

[0136] Figure 3 This is a flowchart illustrating an online customer service response method provided by the present invention, as shown below. Figure 3 As shown, the terminal is application 301, and the cloud includes: preprocessing module 302, response module 303, and postprocessing module 304. The preprocessing module 302 includes: classification model 3021, text extraction submodule 3022, and multimodal multitasking model 3023.

[0137] Application 301 receives user input information and sends it to classification model 3021. Classification model 3021 then sends the user input information to multimodal multitasking model 3023. Simultaneously, classification model 3021 identifies the user input type, sending image and voice inputs to text extraction submodule 3022 to extract full text information, and then sending the corresponding full text information to response module 303. Upon receiving voice and image inputs, text extraction submodule 3022 extracts the corresponding full text information and sends it to response module 303. Upon receiving user input information, multimodal multitasking model 3023 extracts summary information based on the user input and sends the summary information to response module 303. Response module 303 generates a response template and reply text based on the summary information and full text information, and sends them to post-processing module 304. Post-processing module 304 determines the target reply based on the response template and reply text and sends the target reply to application 301.

[0138] Figure 4 This is a schematic diagram of the structure of an online customer service response device provided in an embodiment of the present invention, as shown below. Figure 4As shown, an online customer service response device 400 includes an acquisition module 401, a first processing module 402, a second processing module 403, a third processing module 404, and a fourth processing module 405.

[0139] The acquisition module 401 acquires user input information, inputs the user input information into a classification model, and identifies the information type of the user input information through the classification model.

[0140] The first processing module 402 is used to input the user input information and the information type into the text extraction model, and extract the text through the text extraction model to obtain the full text information.

[0141] The second processing module 403 is used to input the user input information into the summary extraction model, and perform summary extraction through the summary extraction model to obtain summary information.

[0142] The third processing module 404 is used to input the summary information and the full text information into the large model, and generate the response text and response template through the large model.

[0143] The fourth processing module 405 is used to generate a target response based on the response text and the response template.

[0144] The online customer service response device provided by the present invention further includes:

[0145] The third processing module 404 is also used to input the summary information and full text information into the large model, and to determine the user intent through the large model.

[0146] The third processing module 404 is also used to convert user intent into response text based on an extended database and a question-and-answer database. The extended database includes extended information related to the communication field, and the question-and-answer database includes pre-set question information and multiple response information corresponding to the question information.

[0147] The third processing module 404 is also used to determine the response template corresponding to the user intent based on the response template library, which has a pre-set correspondence between user intent and response template.

[0148] The online customer service response device provided by the present invention further includes:

[0149] The third processing module 404 is also used to determine the extended information corresponding to the user's intent based on the extended database;

[0150] The third processing module 404 is also used to determine the question information and response information corresponding to the user's intent based on the question and answer database;

[0151] The third processing module 404 is also used to generate a reply prompt message based on the extended information, question information and response information;

[0152] The third processing module 404 is also used to input user intent and response prompt information into the large model, and extract user intent features and response prompt features through the large model.

[0153] The third processing module 404 is also used to fuse user intent features and response prompt features through a large model, and to respond based on the fused features to obtain the response text.

[0154] The online customer service response device provided by the present invention further includes:

[0155] The fourth processing module 405 is also used to search for an image corresponding to the response text from the image library when the response template includes a reserved image location;

[0156] The fourth processing module 405 is also used to determine the image and response text as the target response according to the response template.

[0157] The online customer service response device provided by the present invention further includes:

[0158] The acquisition module is used to acquire training set data and validation set data. The training set data includes a labeled dataset and an unlabeled dataset. Both the labeled dataset and the unlabeled dataset include text input, voice input, and image input.

[0159] The first training module is used to input the labeled data into the initial multimodal multitask model for training, thereby obtaining the first multimodal multitask model.

[0160] The prediction module is used to input the unlabeled data into the first multimodal multi-task model to perform pseudo-label prediction and obtain the pseudo-label corresponding to each unlabeled data in the unlabeled dataset.

[0161] The calculation module is used to calculate the similarity between the pseudo-label of each unlabeled data and the label of each labeled data. If the similarity is higher than a first preset threshold, the unlabeled data and the corresponding pseudo-label are added to the labeled dataset to obtain the first labeled dataset.

[0162] The feature enhancement module is used to enhance the features of the labeled data in the first labeled dataset based on the spectral clustering algorithm, so as to obtain the first labeled dataset with enhanced features.

[0163] The second training module is used to train the first multimodal multitask model based on the first labeled dataset with enhanced features to obtain the second multimodal multitask model.

[0164] The third training module is used to input the validation set data into the second multimodal multitask model to obtain a loss function value. If the loss function value is less than or equal to a second preset threshold, the second multimodal multitask model is a trained second multimodal multitask model. If the loss function value is greater than the second preset threshold, the network parameters of the second multimodal multitask model are adjusted according to the loss function value, and the process returns to the step of inputting the unlabeled dataset into the first multimodal multitask model for pseudo-label prediction to obtain the pseudo-label corresponding to each unlabeled data in the unlabeled dataset.

[0165] The online customer service response device provided by the present invention further includes:

[0166] The first determination module is used to determine the weights of text input, voice input, and image input in each round of training based on the upper bound of the confidence interval.

[0167] The second determination module is used to determine the proportion of text input, voice input, and image input data in each round of training based on weights.

[0168] Figure 5 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention is shown.

[0169] An electronic device may include a processor 501 and a memory 502 storing computer program instructions.

[0170] Specifically, the processor 501 may include a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of the present invention.

[0171] Memory 502 may include mass storage for data or instructions. For example, and not limitingly, memory 502 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. In one instance, memory 502 may include removable or non-removable (or fixed) media, or memory 502 may be non-volatile solid-state storage. Memory 502 may be internal or external to the integrated gateway disaster recovery device.

[0172] In one instance, memory 502 may be read-only memory (ROM). In one instance, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or flash memory, or a combination of two or more of these.

[0173] Memory 502 may include read-only memory (ROM), random access memory (RAM), disk storage media device, optical storage media device, flash memory device, electrical, optical, or other physical / tangible memory storage device. Therefore, generally, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to one aspect of this disclosure.

[0174] The processor 501 reads and executes computer program instructions stored in the memory 502 to achieve... Figure 1 The online customer service response method in the illustrated embodiment.

[0175] Furthermore, in conjunction with the online customer service response methods in the above embodiments, this invention can be implemented using a computer storage medium. This computer storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement any of the online customer service response methods described in the above embodiments.

[0176] This invention also provides a computer program product, including a computer program, which, when executed, implements any of the online customer service response methods described in the above embodiments.

[0177] It should be clarified that the present invention is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of the present invention.

[0178] The functional blocks shown in the above-described structural diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this invention are programs or code segments used to perform the required tasks. The programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried in a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, read-only memory (ROM), flash memory, erasable read-only memory (EROM), floppy disks, compact disc read-only memory (CD-ROM), optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0179] It should also be noted that the exemplary embodiments mentioned in this invention describe methods or systems based on a series of steps or apparatus. However, this invention is not limited to the order of the steps described above; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0180] The aspects of this disclosure have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by special-purpose hardware performing the specified functions or actions, or can be implemented by a combination of special-purpose hardware and computer instructions.

[0181] The above description is merely a specific embodiment of the present invention. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the protection scope of the present invention.

Claims

1. An online customer service response method, characterized in that, include: Obtain user input information, input the user input information into a classification model, and identify the information type of the user input information through the classification model; The user input information and the information type are input into the text extraction model, and the text is extracted by the text extraction model to obtain the full text information. The user input information is input into the summary extraction model, and the summary extraction model is used to extract the summary information. The summary information and the full text information are input into the large model, and the large model generates the response text and response template. The step of inputting the summary information and the full text information into the large model, and generating the response text and response template through the large model, includes: The summary information and the full text information are input into the large model, and the user intent is determined through the large model. Based on an extended database and a question-and-answer database, the user intent is converted into the response text. The extended database includes extended information related to the field of communication, and the question-and-answer database includes pre-set question information and multiple response information corresponding to the question information. The process of converting the user intent into the response text based on the extended database and the question-and-answer database includes: Based on the extended database, determine the extended information corresponding to the user intent; Based on the question-and-answer database, determine the question information and the response information corresponding to the user's intent; Based on the extended information, the question information, and the response information, a reply prompt message is generated; The user intent and the response prompt information are input into the large model, and the user intent features and response prompt features are extracted through the large model; The user intent features and the response prompt features are fused using a large model, and the response is generated based on the fused features to obtain the response text. Based on the response template library, the response template corresponding to the user intent is determined. The response template library has a pre-set correspondence between the user intent and the response template. Based on the response text and the response template, a target response is generated.

2. The method according to claim 1, characterized in that, The step of generating a target response based on the response text and the response template includes: When the response template includes a reserved image space, the image corresponding to the response text is searched from the image library according to the response text; According to the response template, the image and the response text are identified as the target response.

3. The method according to claim 1, characterized in that, The summary extraction model is a multimodal, multi-task model. Therefore, before inputting the user input information into the summary extraction model and performing summary extraction, the following steps are also included: Acquire training set data and validation set data. The training set data includes a labeled dataset and an unlabeled dataset. Both the labeled dataset and the unlabeled dataset include: text input, voice input, and image input. The labeled data is input into the initial multimodal multitask model for training to obtain the first multimodal multitask model; The unlabeled data is input into the first multimodal multi-task model for pseudo-label prediction to obtain the pseudo-label corresponding to each unlabeled data in the unlabeled dataset; Calculate the similarity between the pseudo-label of each unlabeled data and the label of each labeled data. If the similarity is higher than a first preset threshold, add the unlabeled data and the corresponding pseudo-label to the labeled dataset to obtain the first labeled dataset. Based on the spectral clustering algorithm, the labeled data in the first labeled dataset are feature-enhanced to obtain the first labeled dataset with enhanced features. The first multimodal multitask model is trained using the first labeled dataset with enhanced features to obtain the second multimodal multitask model. The validation set data is input into the second multimodal multitask model to obtain a loss function value. If the loss function value is less than or equal to a second preset threshold, the second multimodal multitask model is a trained second multimodal multitask model. If the loss function value is greater than the second preset threshold, the network parameters of the second multimodal multitask model are adjusted according to the loss function value, and the process returns to the step of inputting the unlabeled dataset into the first multimodal multitask model for pseudo-label prediction to obtain the pseudo-label corresponding to each unlabeled data in the unlabeled dataset.

4. The method according to claim 3, characterized in that, The step of training the first multimodal multitask model based on the first labeled dataset enhanced by the aforementioned features further includes: Based on the upper bound of the confidence interval, the weights corresponding to text input, voice input, and image input are determined in each round of training. Based on the weights, the proportions of text input, voice input, and image input are determined in each round of training.

5. An online customer service response device, characterized in that, include: The acquisition module is used to acquire user input information, input the user input information into a classification model, and identify the information type of the user input information through the classification model. The first processing module is used to input the user input information and the information type into the text extraction model, and extract the text through the text extraction model to obtain the full text information. The second processing module is used to input the user input information into the summary extraction model, and perform summary extraction through the summary extraction model to obtain summary information; The third processing module is used to input the summary information and the full text information into the large model, and generate the response text and response template through the large model; The third processing module is further configured to: input the summary information and the full text information into the large model, and determine the user intent through the large model; Based on an extended database and a question-and-answer database, the user intent is converted into the response text. The extended database includes extended information related to the communication field, and the question-and-answer database includes pre-set question information and multiple response information corresponding to the question information. Based on a response template library, a response template corresponding to the user intent is determined. The response template library has a pre-set correspondence between the user intent and the response template. The third processing module is further configured to: determine the extended information corresponding to the user intent based on the extended database; determine the question information and the response information corresponding to the user intent based on the question-and-answer database; generate response prompt information based on the extended information, the question information, and the response information; input the user intent and the response prompt information into the large model, and extract user intent features and response prompt features through the large model; perform feature fusion of the user intent features and the response prompt features through the large model, and respond based on the fused features to obtain the response text; The fourth processing module is used to generate a target response based on the response text and the response template.

6. An electronic device, characterized in that, The device includes: a processor and a memory storing computer program instructions; the processor reads and executes the computer program instructions to implement the online customer service response method as described in any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, The computer storage medium stores computer program instructions, which, when executed by a processor, implement the online customer service response method as described in any one of claims 1-4.

8. A computer program product, characterized in that, When the instructions in the computer program product are executed by the processor of the electronic device, the electronic device performs the online customer service response method as described in any one of claims 1-4.

Images