User opinion analysis method, apparatus and medium
By performing double distillation on user feedback text and using Generative Language Model (LLM) to extract and summarize user sentiment, intent, and opinion subject, the problem of difficulty in multi-dimensional attribution summarization in existing technologies is solved, thereby improving the efficiency and accuracy of service quality analysis.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHINA UNITED NETWORK COMM GRP CO LTD
- Filing Date
- 2024-02-02
- Publication Date
- 2026-07-21
AI Technical Summary
Existing technologies lack effective methods for attributing user opinions, particularly in obtaining multi-dimensional attribution summaries of user emotions, user intentions, and opinion subjects. Manual analysis is inefficient, while machine learning methods are labor-intensive and lack sustainability.
Two generative language large model (LLM) are used to distill user opinion text twice. The first distillation obtains metadata of user sentiment, intent and opinion subject. The second distillation summarizes and analyzes the metadata and outputs a global perspective attribution summary.
It enables efficient and stable extraction of attribution summaries of emotions, intentions, and opinion subjects from a large number of user opinions, thereby improving the analytical capabilities for service quality.
Smart Images
Figure CN117952081B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates at least to the field of data analysis technology, and in particular to a user opinion analysis method, a user opinion analysis device, and a computer-readable storage medium. Background Technology
[0002] After providing services to users, service providers receive a large amount of user feedback. Attributing and summarizing this feedback helps service providers improve service quality. In some cases, manual sampling may be used to analyze user feedback, which is inefficient and depends on the expert knowledge level of the analysts, potentially leading to the loss of valuable information. Alternatively, machine learning clustering analysis may be used to analyze user feedback, but this requires extensive labeling by professional customer service personnel to obtain training data for machine learning, which is labor-intensive and lacks sustainability.
[0003] It is evident that existing technologies lack effective solutions for attributing and summarizing user opinions, particularly in obtaining attribution summaries from multiple dimensions, including user emotions, user intent, and the subject of user opinions. Summary of the Invention
[0004] The technical problem to be solved by this disclosure is to provide a user opinion analysis method, a user opinion analysis device, and a computer-readable storage medium to address the above-mentioned shortcomings, so as to solve the problem of how to obtain the user's emotions, intentions, and attribution summaries of the opinion subject based on user opinions.
[0005] Firstly, this disclosure provides a method for analyzing user opinions, the method comprising:
[0006] Obtain user feedback text that includes multiple user comments;
[0007] The first generative language model (LLM) is used to perform the first distillation of user opinion text to obtain metadata including user sentiment, user intent, and user opinion subject from multiple user opinions;
[0008] The second set of generative language large model (LLM) is used to perform a second distillation of metadata to obtain attribution summaries of different user emotions, user intentions and / or user opinions.
[0009] Secondly, this disclosure provides a user opinion analysis device, the device comprising:
[0010] The text module is used to retrieve user feedback text, which includes feedback from multiple users.
[0011] The first distillation module, connected to the text module, is used to perform the first distillation of the user opinion text using the first set of generative language large model LLM, in order to obtain metadata including user sentiment, user intent and user opinion subject of multiple user opinions;
[0012] The second distillation module, connected to the first distillation module, is used to perform a second distillation of metadata using a second set of generative language large models (LLM) to obtain attribution summaries of different user sentiments, user intentions, and / or user opinions.
[0013] Thirdly, this disclosure provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the user opinion analysis method described above.
[0014] This disclosure provides a user opinion analysis method, a user opinion analysis device, and a computer-readable storage medium. It employs two sets of LLMs to perform two distillations on user opinion text. The user opinion text includes multiple user opinions. The first distillation obtains stable metadata, which includes user sentiment, user intent, and user opinion subject for multiple user opinions. The second distillation summarizes and analyzes the metadata, ultimately deriving a global perspective attribution summary of user opinions. This can obtain attribution summaries for different user sentiments, user intents, and / or user opinion subjects, and can output analysis results for a large number of user opinions, helping service providers improve service quality. Attached Figure Description
[0015] Figure 1 This is a flowchart of a user opinion analysis method according to an embodiment of this disclosure;
[0016] Figure 2 This is a flowchart of another user opinion analysis method according to an embodiment of this disclosure;
[0017] Figure 3 This is a flowchart of another user opinion analysis method according to an embodiment of this disclosure;
[0018] Figure 4 This is a schematic diagram of the structure of a user opinion analysis device according to an embodiment of the present disclosure. Detailed Implementation
[0019] To enable those skilled in the art to better understand the technical solutions of this disclosure, the embodiments of this disclosure will be further described in detail below with reference to the accompanying drawings.
[0020] It is understood that the specific embodiments and accompanying drawings described herein are for illustrative purposes only and are not intended to limit the scope of this disclosure.
[0021] It is understood that, without conflict, the various embodiments and features in the embodiments of this disclosure can be combined with each other.
[0022] It is understood that, for ease of description, only the parts relevant to this disclosure are shown in the accompanying drawings, while parts unrelated to this disclosure are not shown in the drawings.
[0023] It is understood that each unit or module involved in the embodiments of this disclosure may correspond to only one entity structure, or may be composed of multiple entity structures, or multiple units or modules may be integrated into one entity structure.
[0024] It is understood that, without conflict, the functions and steps marked in the flowcharts and block diagrams of this disclosure may occur in a different order than that marked in the accompanying drawings.
[0025] It is understood that the flowcharts and block diagrams of this disclosure illustrate the architecture, functions, and operations of possible implementations of systems, apparatuses, devices, and methods according to various embodiments of this disclosure. Each block in a flowchart or block diagram may represent a unit, module, program segment, or code, containing executable instructions for implementing the specified function. Furthermore, each block or combination of blocks in the block diagrams and flowcharts may be implemented using a hardware-based system to implement the specified function, or using a combination of hardware and computer instructions.
[0026] It is understood that the units and modules involved in the embodiments of this disclosure can be implemented by software or by hardware, for example, the units and modules can be located in a processor.
[0027] Example 1:
[0028] like Figure 1 As shown, this disclosure provides a method for analyzing user opinions.
[0029] include:
[0030] S1. Obtain user feedback text including multiple user comments;
[0031] S2. The first set of generative language large model LLM is used to perform the first distillation on the user opinion text to obtain metadata including user sentiment, user intent and user opinion subject of multiple user opinions;
[0032] S3. The second set of generative language large model (LLM) is used to perform a second distillation of the metadata to obtain attribution summaries of different user emotions, user intentions and / or user opinions.
[0033] In this embodiment, the method can be more specifically described as follows: Figure 2As shown, by employing two sets of LLM (Large Language Model) to perform two distillations (first distillation and second distillation) on user opinion text, the user opinion text includes multiple user opinions (i.e., the analysis object is a set of user opinions). The first distillation obtains stable metadata, which includes user sentiment, user intent, and user opinion subject of multiple user opinions. The second distillation summarizes and analyzes based on the metadata, and finally obtains a global perspective summary of user opinion attribution. It can obtain attribution summaries of different user sentiments, user intents, and / or user opinion subjects, and can output analysis results of a large number of user opinions, which helps service providers improve service quality.
[0034] Taking the analysis of customer service complaints from telecom operators as an example, each province receives millions of customer service complaint calls per month, along with a large amount of online customer service data. Furthermore, due to regional differences in pronunciation, user feedback is often a mixture of voice and text formats, making analysis extremely difficult. This embodiment uses intelligent identification of user complaints, assembling a large volume of user complaints into user feedback text. Utilizing the semantic understanding capabilities of LLM (Local Language Management), it quickly extracts user emotions (e.g., satisfaction or dissatisfaction), user intentions (e.g., changing plans, providing fee suggestions), and the subject of the user feedback (e.g., the product or service involved in the complaint) by identifying the translated dialogue text of all user calls. This initial extraction of user emotions, intentions, and the subject of the feedback forms stable metadata. LLM is then used to analyze this metadata again to obtain attribution summaries of different user emotions, intentions, and / or the subject of the feedback, such as analyzing the reasons for users changing plans, and the reasons and proportions of user satisfaction or dissatisfaction with products or services. This helps operators understand users' experiences with the products or services they provide.
[0035] In one embodiment, obtaining user comment text that includes multiple user comments specifically includes:
[0036] Obtain multiple user comments received by the customer service system;
[0037] In response to the presence of several voice user opinions among multiple user opinions, the several voice user opinions are converted into text using Automatic Speech Recognition (ASR) technology;
[0038] Multiple user comments in text format are concatenated to form the user comment text.
[0039] In this embodiment, before performing the analysis, the customer service call text is preprocessed into user feedback text according to the overall customer service call text processing flow established in the early stage. Specifically, the multi-turn dialogues of multiple users are translated, and the multiple text dialogues generated are spliced into an article using the general technology of Automatic Speech Recognition (ASR).
[0040] In one embodiment, a first set of LLMs is used to perform a first distillation on the user comment text to obtain metadata including user sentiment, user intent, and the subject of the user comment, specifically including:
[0041] The first part of the first group of LLM is used to extract dialogue summaries of multiple user opinions based on user opinion text;
[0042] The second part of the first set of LLM and the vector knowledge base are used to obtain user sentiment, user intent and user opinion subject of multiple user opinions based on dialogue summary;
[0043] The metadata is formed by combining user emotions, user intentions, and user opinions.
[0044] In this embodiment, a large model (LLM) understands the dialogue and outputs a dialogue summary, intent, and intent classification. This is then matched with a vector knowledge base for appropriate output, improving the LLM's compliance and ensuring stable output of the desired classification results. Currently mainstream generative language large models and vector knowledge bases can be used. For example, the large model could be a 13B (13 billion parameters) open-source large model, and the vector knowledge base could be an existing domain-specific knowledge base corresponding to user opinions. For example, in the customer service domain, a knowledge base such as... Figure 3 As shown, product and service documents are created for the products and services provided by service providers to users. Customer service experts summarize several user intentions from historical user feedback to form corresponding formatted tags and establish user intention clusters. Based on the experience and knowledge of customer service experts, dialogue summary samples of user feedback are created to build a knowledge vector library. The output of the large model is compared with the knowledge library, and unstable intentions and subjects are dynamically clustered in a short time. The output of the large model is then fine-tuned to continuously output stable intention and subject tags, solving the problem that the output of large models for open-ended questions is not fixed and cannot be applied industrially. This approach can be applied to customer service understanding in various industries and questioning by operators' maintenance personnel.
[0045] In one implementation, the first part of the first set of LLMs is used to extract dialogue summaries of multiple user opinions based on the user opinion text, specifically including:
[0046] The user comment text is analyzed using the first LLM and the first prompt word Prompt, so that the first LLM outputs a first dialogue summary of multiple user comments;
[0047] The first dialogue summary is processed using a second LLM and a second prompt word Prompt format, so that the second LLM outputs a second dialogue summary of multiple user comments in a first format;
[0048] The metadata also includes a second dialogue summary.
[0049] In this embodiment, specifically as follows: Figure 2 As shown, the user feedback set is input into LLM1 based on Prompt1 to obtain the first dialogue summary. For example, Prompt1 is "Please summarize the following user speech in the speech-to-text. The summary should include the user's emotions, intentions, and the product names mentioned by the user." The first dialogue summary is input into LLM2 based on Prompt2 to obtain the second dialogue summary. That is, the formatted output capability of LLM is used to summarize the content of the dialogue. Finally, the summary result is output. For example, the LangChain framework is used to compose the result into a sentence using a template. For example, the following is a summary of a user speech, where the speech summary is: {Summary}, the user's dialogue emotion is: {Emotion}, the user's possible intention is: {Intention}, and the user's main complaint objects are {Product} {Service} {Quality}.
[0050] In one implementation, the second part of the first set of LLMs and a vector knowledge base are used to obtain user sentiment, user intent, and user opinion subject of multiple user opinions based on dialogue summaries, specifically including:
[0051] The dialogue summary is processed using third-party LLM and tagged third-party prompt analysis and formatting to enable the third-party LLM to output user sentiment of multiple user opinions in a second format.
[0052] The dialogue summary is analyzed using the fourth LLM and the fourth prompt word Prompt, so that the fourth LLM outputs the first user intent with multiple user opinions, and the first user intent is formatted into a second user intent with multiple user opinions in a third format using the first vector knowledge base;
[0053] The fifth prompt word, labeled, is obtained using a second vector knowledge base. The dialogue summary is then processed and formatted using the fifth LLM and the fifth prompt, so that the fifth LLM outputs the user opinion subject of multiple user opinions in the fourth format.
[0054] In this embodiment, specifically as follows: Figure 2As shown, the first dialogue summary includes information on user sentiment, user intent, and the main body of the user's opinion. The user sentiment is obtained by inputting the first dialogue summary into LLM3 based on Prompt3, the user intent is obtained by inputting the first dialogue summary into LLM4 based on Prompt4, and the main body of the user's opinion is obtained by inputting the first dialogue summary into LLM5 based on Prompt5. Specifically... Figure 3 As shown, in order to obtain stable output, the first distillation is performed by combining the vector knowledge base. The user intent clustering in the vector knowledge base is used to stabilize the user intent output, and the user opinion subject output is stabilized by using expert knowledge and product and service documents. User emotions can be obtained stably based on pre-established labels, such as satisfied and dissatisfied, positive, negative and neutral.
[0055] In one embodiment, a second user intent, which uses a first vector knowledge base to format a first user intent into multiple user opinions in a third format, specifically includes:
[0056] The first user intent is vectorized using text embedding to obtain the first user intent data.
[0057] Calculate the first similarity between each first user intent data and the anchor point of the user intent cluster in the first vector knowledge base;
[0058] In response to a first similarity greater than a first threshold, the corresponding user opinion is assigned to the corresponding user intent cluster in the first vector knowledge base;
[0059] The corresponding user intent cluster labels are obtained from the first vector knowledge base to serve as the second user intent of the corresponding user opinion in the third format.
[0060] In this embodiment, specifically as follows: Figure 2 and 3As shown, similarity calculation is performed using a vector knowledge base. For user intent output, the similarity calculation corrects the unstable labels output by LLM4. For example, the first user intent includes two types: "want to subscribe to data traffic" and "need to subscribe to data traffic." Through similarity calculation, these can be grouped into the same user intent, thus requiring the same user intent label. Specifically, the method for obtaining user intent clusters in the vector knowledge base involves collecting all user intents within a fixed time period, using sentence vectors as mappings, and employing the DBSCAN (Density-Based Spatial Clustering of Applications with Noise) algorithm for clustering. The centroid of all vectors within a cluster (the method for finding the centroid of a single cluster generally involves summing the weights and dividing by the number of vectors) is used as the anchor point of the cluster. When a new intent arrives, if its cosine similarity with an existing cluster anchor point is higher than 0.707 (here, cosine similarity is used to calculate similarity; a 0° angle between the data point and the anchor point (cos0 = 1) indicates high overlap, and a 90° angle (cos90 = 0) indicates orthogonality and no correlation; here, 45° is taken as the median value), it is classified as the same intent. If its cosine similarity with all known classes is lower than 0.707, then the new intent is used as the new class anchor point.
[0061] In one implementation, a second vector knowledge base is used to obtain the tagged fifth prompt word, Prompt, specifically including:
[0062] Each dialogue summary is vectorized using text embedding to obtain the first dialogue summary data;
[0063] Calculate the second similarity between each first dialogue summary data and the second dialogue summary data in the second vector knowledge base;
[0064] Obtain the subject tags in the second vector knowledge base corresponding to the top N second similarity values that are greater than the second threshold;
[0065] The fifth prompt is obtained based on the top N main tags.
[0066] In this embodiment, specifically as follows: Figure 2 and 3As shown, for the user opinion subject (such as the product or service complained about by the user), the similarity calculation result is used to limit Prompt5. Specifically, the Similarity algorithm is used to calculate the similarity between the first dialogue summary and the user opinion sample set defined by expert knowledge, and the top N (e.g., N=3) product or service tag sets that match are matched. For example, after the dialogue data is embedded, all candidate products above a certain similarity threshold are found from the vector database using an ANN (Approximate Nearest Neighbor) algorithm. Generally, Euclidean similarity or cosine similarity is used. Here, candidate products with a cosine similarity greater than 0.707 are selected, and the Prompt template is designed using the candidate product: "Please select the product names involved in the following dialogue from the above product candidates (e.g., "1. Product A 2. Product B 3. Product C")", input into the large model LLM5, and output the actual complaint subject involved in this dialogue.
[0067] In one implementation, a second set of LLMs is used to perform a second distillation of the metadata to obtain attribution summaries of different user sentiments, user intentions, and / or user opinions, specifically including:
[0068] The sixth prompt word is obtained based on different user emotions, user intentions, and / or user opinions. The sixth LLM and the sixth prompt are used to analyze metadata so that the sixth LLM can output attribution summaries for different user emotions, user intentions, and / or user opinions.
[0069] In this embodiment, specifically as follows: Figure 2 and 3 As shown, the first distillation ultimately yields stable output metadata, such as archived and stored in JSON format as {"Products Involved":"Product A","User Intent":"Package Change"...}. The metadata is then input into LLM6 based on Prompt6 to obtain the attribution summary output of the second distillation. For example, to investigate the attribution of all users wanting to change their packages, all package changes and attribution analyses can be summarized through the user intent field of the metadata.
[0070] like Figure 2 and Figure 3 The more detailed and complete user feedback analysis process is shown below:
[0071] (1) System Development Phase: This phase specifically involves embedding the user feedback analysis function disclosed herein into the original customer service system, including:
[0072] The system acquires multi-turn dialogues from the customer service interaction interface for analysis, and displays the analysis report through the customer service interaction interface after analysis. It designs a vector knowledge base and LLM for the customer service domain.
[0073] The vector knowledge base vectorizes the list of tags such as product, service, and quality of user intent and user complaint subject into text embedding. For example, it uses the CLS output vector of the BERT (Bidirectional Encoder Representations from Transformers) model trained on supervised text similarity samples based on open source data as the embedding vector of the whole sentence text, and encodes it into multiple vectors and stores them in the knowledge base.
[0074] LLM includes a model library, which is used to adjust parameters and select models from LLM1 to LLM6 based on the application. The prompt word library creates prompt word templates for LLM1 to LLM6. Through an open-source large model framework, tests are conducted for different application scenarios. At the same time, prompt words are written, an overall customer service call text processing flow is established, a corresponding processing agent flow is established, and a prompt based on expert-written fault handling process descriptions is written for LLM, forming a prompt library. The best-performing LLM model is selected, and the most suitable large model parameters, framework, and prompt word templates are found and stored in the prompt template library of the LangChain framework.
[0075] LangChain is a large language model development framework and an important part of the LLM application architecture. LangChain can integrate LLM models, vector databases, interaction layer Prompt, external knowledge, and external tools, and thus can freely build LLM applications. LangChain has features such as multi-turn dialogue memory, chain, agent, text vectors, prompt words, large models, and few-shot prompts.
[0076] (2) User opinion analysis stage: This includes two stages: preprocessing and distillation. The preprocessing stage mainly obtains user opinion texts containing multiple user opinions. Distillation is a process of summarizing and abstracting the transcribed text to address the instability of speech recognition results. It filters out information that is not meaningful to the business and extracts meaningful words as tags for storage. Specifically, distillation is performed through LLM1-6. LLM1-6 can be the same large model, or they can be different or multiple different large models. Different large models refer to LLMs that have been fine-tuned with different specially labeled data. They can be different open-source frameworks or different parameters, depending on the actual project. Distillation is divided into first distillation and second distillation.
[0077] The first distillation yields stable metadata, including: inputting articles into a large model, using the LangChain framework and prompt word templates to output summaries, sentiments, stable user intents, and stable corresponding product services; specifically, this includes: putting the integrated dialogue articles into the large model LLM1, asking questions using prompt words to output the first dialogue summary, then using an agent to call LLM2-5 for each step, adding prompt words to ask questions, calling the LLM's formatted output capabilities, summarizing the output results, and using the LangChain framework to assemble the results into a sentence using templates, such as the following is a summary of a user conversation, where the conversation summary is: {Summary}, the user's dialogue sentiment is: {Sentence}, the user's possible intent is: {Intent}, and the user's main complaint targets are {Product} {Service} {Quality};
[0078] The model is divided into three parts: LLM2, LLM3, LLM4, and LLM5. LLM2 is used to summarize the content of the dialogue and complete the summary; LLM3 is used to judge user emotions; LLM4 is used to summarize user thoughts; and LLM5 is used to summarize the products, services, or quality of the complaints. The work done by LLM4-5 generally requires providing a list of prior knowledge tags summarized by expert experience, and then matching based on the list to meet the requirements of stable output applications in industry. However, the large model is generative and cannot stably output the required product subject tag list (LLM5), and the intent is not summarized by prior knowledge (LLM4). Therefore, it needs to be combined with a vector knowledge base to output stable tags. After inputting the dialogue into LLM4, the large model outputs the user intent. Through LangChain's text vector function, the unstable tags (such as "want to apply for data traffic" or "need to apply for data traffic") are vectorized into text embeddings, and then matched against the vector knowledge base. The user complaint subject is matched against the Top N products in the vector knowledge base. These Top N products are used as prompt words input into the large model LLM5 to output the real complaint subject involved in this dialogue. Through the above steps, enumerable and non-repeating user intents and specified complaint subjects are obtained. This stable structured data is called metadata.
[0079] The second distillation yields an attribution analysis summary. The results of the first distillation are used to generate a sentence using a template, such as: The summary of the user's message is as follows: xxxx The intent of the user's message is as follows: xxxx The product information involved by the user is as follows: xxxx. Each user call is represented as one line. Using LangChain's prompt word template, LLM6 generates attribution summaries for all users' calls. For example, "Given a user call, the user's intent is: {Package change}, and the product involved is: {Product A}... Please summarize the user call's attribution based on the above information." LLM6 outputs results such as {"Products Involved": "Product A", "User Intent": "Package Change"...} and the attribution summary. Formatted data is then stored and statistically analyzed. For example, embedding is vectorized and placed in a vector knowledge base, using metadata as the index. Attribution analysis and overall data are stored in the knowledge base for later retrieval. For instance, to investigate the attributions for all users wanting to change their packages, all package change and attribution analyses can be summarized through the user intent domain of the metadata. Business experts provide feedback based on their post-hoc experience, continuously judging whether unstable outputs such as intent, product, service, and quality meet business requirements. Modifications are made in the vector knowledge base according to business requirements to ensure stable labels in subsequent outputs.
[0080] The above process addresses the issue of large models generating inconsistent and unsuitable data for industrial applications in open-ended problems. The distillation process employs two distillations: the first distillation yields stable metadata, while the second distillation is a summary analysis based on the first, conducted in isolation. The second distillation provides a global perspective analysis. The processing of unstable intent dynamic clustering output addresses the challenge that large generative language models typically generate random utterances, failing to consistently output fixed labels—a key difficulty in industrial applications. By combining this with a vector knowledge base and accumulating posterior experience, unstable intent labels can be summarized into stable intent labels. Then, clustering or similarity matching yields stable results that meet industrial requirements.
[0081] Example 2:
[0082] like Figure 3 As shown, this disclosure provides a user opinion analysis device, the device comprising:
[0083] Text module 1 is used to obtain user opinion text that includes multiple user opinions;
[0084] The first distillation module 2, connected to the text module 1, is used to perform the first distillation of the user opinion text using the first set of generative language large model LLM, in order to obtain metadata including user sentiment, user intent and user opinion subject of multiple user opinions;
[0085] The second distillation module 3, connected to the first distillation module 2, is used to perform a second distillation of metadata using a second set of generative language large models (LLM) to obtain attribution summaries of different user emotions, user intentions, and / or user opinions.
[0086] In one embodiment, text module 1 specifically includes:
[0087] The customer service interaction unit is used to obtain multiple user opinions received by the customer service system;
[0088] The text conversion unit, connected to the customer service interaction unit, is used to convert several voice user opinions into text using automatic speech recognition (ASR) technology in response to multiple user opinions.
[0089] A text splicing unit, connected to a text conversion unit, is used to splice multiple user comments in text format to form the user comment text.
[0090] In one embodiment, the second distillation module 3 is specifically used for:
[0091] The sixth prompt word is obtained based on different user emotions, user intentions, and / or user opinions. The sixth LLM and the sixth prompt are used to analyze metadata so that the sixth LLM can output attribution summaries for different user emotions, user intentions, and / or user opinions.
[0092] In one embodiment, the first distillation module 2 specifically includes:
[0093] The summary extraction unit is used to extract dialogue summaries of multiple user opinions based on the user opinion text using the first part of the first group of LLMs.
[0094] The summary analysis unit, connected to the summary extraction unit, is used to obtain user sentiment, user intent, and user opinion subject of multiple user opinions based on the dialogue summary using the second part of the first set of LLM and the vector knowledge base;
[0095] The metadata acquisition unit, connected to the summary analysis unit, is used to combine user sentiment, user intent, and user opinion to form the metadata.
[0096] In one embodiment, the summary extraction unit specifically includes:
[0097] A text analysis unit is used to analyze user comment text using a first LLM and a first prompt word Prompt, so that the first LLM outputs a first dialogue summary of multiple user comments;
[0098] The first formatting unit, connected to the text analysis unit, is used to process the first dialogue summary using the second LLM and the second prompt word Prompt format, so that the second LLM outputs a second dialogue summary of multiple user comments in the first format;
[0099] The metadata acquisition unit is also connected to the first formatting unit for combining the second dialogue summary.
[0100] In one embodiment, the summary analysis unit specifically includes:
[0101] The emotion acquisition unit, connected to the summary extraction unit, is used to perform dialogue summary analysis and formatting using a third LLM and labeled third prompt words, so that the third LLM outputs user emotions of multiple user opinions in a second format.
[0102] The intent acquisition unit, connected to the summary extraction unit, is used to analyze the dialogue summary using the fourth LLM and the fourth prompt word Prompt, so that the fourth LLM outputs a first user intent with multiple user opinions, and uses a first vector knowledge base to format the first user intent into a second user intent with multiple user opinions in a third format.
[0103] The subject acquisition unit, connected to the summary extraction unit, is used to acquire the tagged fifth prompt word Prompt using the second vector knowledge base, and to perform dialogue summary processing and formatting using the fifth LLM and the fifth Prompt, so that the fifth LLM outputs the user opinion subject of multiple user opinions in the fourth format.
[0104] In one embodiment, the intent acquisition unit specifically includes a third formatting unit, used for:
[0105] The first user intent is vectorized using text embedding to obtain the first user intent data.
[0106] Calculate the first similarity between each first user intent data and the anchor point of the user intent cluster in the first vector knowledge base;
[0107] In response to a first similarity greater than a first threshold, the corresponding user opinion is assigned to the corresponding user intent cluster in the first vector knowledge base;
[0108] The corresponding user intent cluster labels are obtained from the first vector knowledge base to serve as the second user intent of the corresponding user opinion in the third format.
[0109] In one embodiment, the subject acquisition unit specifically includes a fourth formatting unit, used for:
[0110] Each dialogue summary is vectorized using text embedding to obtain the first dialogue summary data;
[0111] Calculate the second similarity between each first dialogue summary data and the second dialogue summary data in the second vector knowledge base;
[0112] Obtain the subject tags in the second vector knowledge base corresponding to the top N second similarity values that are greater than the second threshold;
[0113] The fifth prompt is obtained based on the top N main tags.
[0114] Example 3:
[0115] Embodiment 3 of this disclosure provides a computer-readable storage medium storing a computer program. When the computer program is run by a processor, it implements the user opinion analysis method as described in Embodiment 1, or the user opinion analysis device as described in Embodiment 2.
[0116] The computer-readable storage medium includes volatile or non-volatile, removable or non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, computer program modules, or other data). Computer-readable storage media include, but are not limited to, RAM (Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory or other memory technologies, CD-ROM (Compact Disc Read-Only Memory), DVD or other optical disc storage, cartridges, magnetic tapes, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer.
[0117] In addition, this disclosure may also provide a computer device including a memory and a processor, wherein the memory stores a computer program, and when the processor runs the computer program stored in the memory, the processor executes the user opinion analysis method as described in Embodiment 1, and the computer device may be the user opinion analysis device as described in Embodiment 2.
[0118] The memory is connected to the processor. The memory can be flash memory, read-only memory or other types of memory. The processor can be a central processing unit or a microcontroller.
[0119] Embodiments 1-3 of this disclosure provide a user opinion analysis method, a user opinion analysis device, and a computer-readable storage medium. By employing two sets of LLMs to perform two distillations on multiple user opinions, the first distillation obtains stable metadata, and the second distillation summarizes and analyzes the metadata to finally derive a global perspective attribution summary of user opinions.
[0120] It is understood that the above embodiments are merely exemplary embodiments used to illustrate the principles of this disclosure, and this disclosure is not limited thereto. For those skilled in the art, various modifications and improvements can be made without departing from the spirit and substance of this disclosure, and these modifications and improvements are also considered to be within the scope of protection of this disclosure.
Claims
1. A method for analyzing user opinions, characterized in that, The method includes: Obtain user feedback text that includes multiple user comments; The first generative language model (LLM) is used to perform the first distillation on the user opinion text to obtain metadata including user sentiment, user intent, and user opinion subject from multiple user opinions, specifically including: The first part of the first group of LLM is used to extract dialogue summaries of multiple user opinions based on user opinion text. Using the second part of the first set of LLM and a vector knowledge base, user sentiment, user intent, and user opinion subject of multiple user opinions are obtained based on dialogue summaries. The metadata is formed by combining user emotions, user intentions, and user opinions. A second set of generative language models (LLMs) is used to perform a second distillation of metadata to obtain attribution summaries of different user sentiments, user intentions, and / or user opinions, specifically including: The sixth prompt word is obtained based on different user emotions, user intentions, and / or user opinions. The sixth LLM and the sixth prompt are used to analyze metadata so that the sixth LLM can output attribution summaries for different user emotions, user intentions, and / or user opinions.
2. The method according to claim 1, characterized in that, Obtain user feedback text that includes multiple user comments, specifically: Obtain multiple user comments received by the customer service system; In response to the presence of several voice user opinions among multiple user opinions, the several voice user opinions are converted into text using Automatic Speech Recognition (ASR) technology; Multiple user comments in text format are concatenated to form the user comment text.
3. The method according to claim 1 or 2, characterized in that, The first part of the first set of LLM uses dialogue summaries extracted from multiple user comments based on the user comment text, specifically including: The user comment text is analyzed using the first LLM and the first prompt word Prompt, so that the first LLM outputs a first dialogue summary of multiple user comments; The first dialogue summary is processed using a second LLM and a second prompt word Prompt format, so that the second LLM outputs a second dialogue summary of multiple user comments in a first format; The metadata also includes a second dialogue summary.
4. The method according to claim 1 or 2, characterized in that, Using the second part of the first set of LLM and a vector knowledge base, user sentiment, user intent, and user opinion subject of multiple user opinions are obtained based on dialogue summaries, specifically including: The dialogue summary is processed using third-party LLM and tagged third-party prompt analysis and formatting to enable the third-party LLM to output user sentiment of multiple user opinions in a second format. The dialogue summary is analyzed using the fourth LLM and the fourth prompt word Prompt, so that the fourth LLM outputs the first user intent with multiple user opinions, and the first user intent is formatted into a second user intent with multiple user opinions in a third format using the first vector knowledge base; The fifth prompt word, labeled, is obtained using a second vector knowledge base. The dialogue summary is then processed and formatted using the fifth LLM and the fifth prompt, so that the fifth LLM outputs the user opinion subject of multiple user opinions in the fourth format.
5. The method according to claim 4, characterized in that, The second user intent, which uses a first vector knowledge base to format the first user intent into multiple user opinions in a third format, specifically includes: The first user intent is vectorized using text embedding to obtain the first user intent data. Calculate the first similarity between each first user intent data and the anchor point of the user intent cluster in the first vector knowledge base; In response to a first similarity greater than a first threshold, the corresponding user opinion is assigned to the corresponding user intent cluster in the first vector knowledge base; The corresponding user intent cluster labels are obtained from the first vector knowledge base to serve as the second user intent of the corresponding user opinion in the third format.
6. The method according to claim 4, characterized in that, The fifth prompt word, Prompt, is obtained using a second vector knowledge base, specifically including: Each dialogue summary is vectorized using text embedding to obtain the first dialogue summary data; Calculate the second similarity between each first dialogue summary data and the second dialogue summary data in the second vector knowledge base; Obtain the subject tags in the second vector knowledge base corresponding to the top N second similarity values that are greater than the second threshold; The fifth prompt is obtained based on the top N main tags.
7. A user opinion analysis device, characterized in that, The device includes: The text module is used to retrieve user feedback text, which includes feedback from multiple users. The first distillation module, connected to the text module, is used to perform the first distillation of user opinion text using the first set of generative language large-scale models (LLMs) to obtain metadata including user sentiment, user intent, and the user opinion subject from multiple user opinions. Specifically, this includes: The first part of the first group of LLM is used to extract dialogue summaries of multiple user opinions based on user opinion text. Using the second part of the first set of LLM and a vector knowledge base, user sentiment, user intent, and user opinion subject of multiple user opinions are obtained based on dialogue summaries. The metadata is formed by combining user emotions, user intentions, and user opinions. The second distillation module, connected to the first distillation module, is used to perform a second distillation of metadata using a second set of generative language large-scale models (LLMs) to obtain attribution summaries of different user sentiments, user intentions, and / or user opinions. Specifically, this includes: The sixth prompt word is obtained based on different user emotions, user intentions, and / or user opinions. The sixth LLM and the sixth prompt are used to analyze metadata so that the sixth LLM can output attribution summaries for different user emotions, user intentions, and / or user opinions.
8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the user opinion analysis method as described in any one of claims 1-6.