Product satisfaction evaluation method and device and readable storage medium

Abstract

The invention provides a product satisfaction evaluation method and device and a readable storage medium. The method comprises the steps of obtaining a plurality of customer service work orders related to complaints; inputting complaint contents in the plurality of customer service work orders into a trained work order complaint classification model to obtain complaint reasons to which the complaint contents in each customer service work order belong; for each customer service work order in the plurality of customer service work orders, obtaining a complaint product corresponding to the customer service work order according to the complaint content in the customer service work order, the corresponding complaint reason and the product ordering information of the user corresponding to the customer service work order; and obtaining the complaint amount of the target product according to the complaint products corresponding to all the customer service work orders, and evaluating the product satisfaction degree of the target product according to the complaint amount and the order amount of the target product. According to the method, the device and the readable storage medium, the problem of low accuracy caused by single algorithm used by an existing product satisfaction evaluation method can be solved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a product satisfaction evaluation method, device and readable storage medium. Background Art

[0002] In today's highly competitive and rapidly iterating communications industry, user satisfaction is one of the key indicators to measure the success of a company. With the advancement of technology and changes in consumer preferences, communications companies not only need to provide high-quality communications services, but also need to continuously innovate and optimize product portfolios to meet the increasingly diverse needs of users. Therefore, establishing a scientific and comprehensive product satisfaction analysis and evaluation model plays a vital role in guiding corporate decision-making and improving market competitiveness.

[0003] However, the existing product satisfaction evaluation methods use a single algorithm, resulting in low accuracy. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a product satisfaction evaluation method, device and readable storage medium in view of the above-mentioned deficiencies in the prior art, so as to solve the problem that the existing product satisfaction evaluation method uses a single algorithm, resulting in low accuracy.

[0005] In a first aspect, the present invention provides a method for evaluating product satisfaction, comprising:

[0006] Get multiple customer service tickets involving complaints;

[0007] Input the complaint contents in the plurality of customer service work orders into the trained work order complaint classification model to obtain the complaint reasons to which the complaint contents in each customer service work order belong.

[0008] The trained work order complaint classification model is obtained based on the complaint contents and corresponding complaint reasons in multiple historical customer service work orders;

[0009] For each of the multiple customer service work orders, obtaining a complaint product corresponding to the customer service work order according to the complaint content and the corresponding complaint reason in the customer service work order and the product order information of the user corresponding to the customer service work order;

[0010] The number of complaints about the target product is obtained based on the complaint products corresponding to all the customer service work orders, and the product satisfaction of the target product is evaluated based on the number of complaints and the order volume of the target product.

[0011] Furthermore, before inputting the complaint contents in the plurality of customer service work orders into the trained work order complaint classification model to obtain the complaint reasons to which the complaint contents in each customer service work order belong, the method further comprises:

[0012] Get multiple historical customer service tickets;

[0013] Marking the complaint reasons corresponding to the complaint contents in the multiple historical customer service tickets;

[0014] The BERT-TextCNN model is used as the work order complaint classification model, and the work order complaint classification model is trained using the complaint contents in the multiple historical customer service work orders after labeling to obtain the trained work order complaint classification model.

[0015] Furthermore, the reasons for complaints include: traffic disputes, voice disputes, fee disputes, account opening for unknown reasons, binding for unknown reasons, subscription for unknown reasons, discount and contract disputes, cancellation of subscription, disputes over package resources and non-product categories.

[0016] Further, obtaining the complaint product corresponding to the customer service work order according to the complaint content in the customer service work order and the corresponding complaint reason and the product order information of the user corresponding to the customer service work order specifically includes:

[0017] Perform product inspection on the complaint content in the customer service ticket to further refine the complaint reason and obtain specific complaint reason subcategories;

[0018] According to the complaint content in the customer service work order and the corresponding complaint reason subcategory, the complaint product corresponding to the customer service work order is located from multiple products in the product order information of the user corresponding to the customer service work order within a preset time period.

[0019] Furthermore, before locating the complaint product corresponding to the customer service work order from multiple products in the product order information of the user corresponding to the customer service work order within a preset time period based on the complaint content and the corresponding complaint reason subcategory in the customer service work order, the method further includes:

[0020] The product ordering information of the user corresponding to the customer service work order within a preset time period is obtained from a preset user ordering database.

[0021] Furthermore, the products include: ordered products and non-ordered products, and locating the complaint product corresponding to the customer service work order from multiple products in the product ordering information of the user corresponding to the customer service work order within a preset time period according to the complaint content in the customer service work order and the corresponding complaint reason subcategory specifically includes:

[0022] For subscription products, the complaint reason subcategory corresponding to the customer service ticket is associated with multiple subscription products in the product ordering information to obtain multiple subscription products related to the complaint reason subcategory, and the complaint content in the customer service ticket, multiple subscription products related to the complaint reason subcategory and the user complaint account period corresponding to the customer service ticket are matched according to a preset string fuzzy matching rule, and the subscription product with the highest matching degree is selected as the complaint product;

[0023] For non-subscription products, the complaint reason subcategory corresponding to the customer service ticket is associated with multiple non-subscription products in the product ordering information to obtain multiple non-subscription products related to the complaint reason subcategory, and the complaint content in the customer service ticket, multiple non-subscription products related to the complaint reason subcategory and the user complaint account period corresponding to the customer service ticket are matched according to a preset string fuzzy matching rule, and the non-subscription product with the highest matching degree is selected as the complaint product initially located; and,

[0024] The expense items in the bill of the user's complaint period corresponding to the customer service work order are matched with the complaint content in the customer service work order to obtain the expense item with the highest matching degree, and based on the expense item with the highest matching degree, the user's value-added details sheet corresponding to the customer service work order is used to identify the initially located complaint product to obtain the complaint product subcategory in the finally located complaint product.

[0025] Furthermore, the evaluating the product satisfaction of the target product according to the number of complaints and the number of orders for the target product specifically includes:

[0026] The product satisfaction of the target product is obtained according to the ratio of the complaint quantity to the order quantity of the target product.

[0027] In a second aspect, the present invention provides a device for evaluating product satisfaction, comprising:

[0028] The first acquisition module is used to obtain multiple customer service tickets involving complaints;

[0029] A first obtaining module, connected to the first obtaining module, is used to input the complaint contents in the plurality of customer service work orders into the trained work order complaint classification model to obtain the complaint reasons to which the complaint contents in each customer service work order belong, wherein the trained work order complaint classification model is trained based on the complaint contents in a plurality of historical customer service work orders and the corresponding complaint reasons;

[0030] A second obtaining module, connected to the first obtaining module, is used to obtain, for each of the multiple customer service work orders, a complaint product corresponding to the customer service work order according to the complaint content and the corresponding complaint reason in the customer service work order and the product order information of the user corresponding to the customer service work order;

[0031] An evaluation module is obtained, which is connected to the second obtaining module, and is used to obtain the complaint volume of the target product based on the complaint products corresponding to all the customer service work orders, and to evaluate the product satisfaction of the target product based on the complaint volume and order volume of the target product.

[0032] In a third aspect, the present invention provides a device for evaluating product satisfaction, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to implement the product satisfaction evaluation method described in the first aspect.

[0033] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method for evaluating product satisfaction as described in the first aspect above is implemented.

[0034] The product satisfaction evaluation method, device and readable storage medium provided by the present invention first obtain a plurality of customer service work orders involving complaints; then input the complaint contents in the plurality of customer service work orders into a trained work order complaint classification model to obtain the complaint reasons to which the complaint contents in each of the customer service work orders belong, wherein the trained work order complaint classification model is trained based on the complaint contents and the corresponding complaint reasons in a plurality of historical customer service work orders; then for each of the plurality of customer service work orders, according to the complaint contents in the customer service work order and the corresponding complaint reasons and the product ordering information of the user corresponding to the customer service work order, obtain the complaint product corresponding to the customer service work order; finally, according to the complaint products corresponding to all the customer service work orders, obtain the complaint quantity of the target product, and according to the complaint quantity and ordering quantity of the target product, evaluate the product satisfaction of the target product. The present invention classifies the complaint content in the customer service work order through a trained work order complaint classification model, and determines the complaint product corresponding to the customer service work order based on the complaint content in the customer service work order and the corresponding complaint reason and the product ordering information of the corresponding user, further obtains the complaint volume of the target product based on the complaint products corresponding to all customer service work orders, and evaluates the product satisfaction of the target product based on the complaint volume and order volume of the target product, thereby achieving accurate judgment on the user complaint product, greatly improving the accuracy of product satisfaction evaluation, and solving the problem that the existing product satisfaction evaluation method uses a single algorithm, resulting in low accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 This is a flow chart of a method for evaluating product satisfaction according to Embodiment 1 of the present invention;

[0036] Figure 2 A schematic diagram of a method for evaluating product satisfaction according to an embodiment of the present invention;

[0037] Figure 3 A flowchart of a work order complaint classification model based on BERT-TextCNN according to an embodiment of the present invention;

[0038] Figure 4 A flowchart of a user complaint product intelligent positioning model according to an embodiment of the present invention;

[0039] Figure 5 This is a schematic diagram of the structure of a device for evaluating product satisfaction according to Embodiment 2 of the present invention;

[0040] Figure 6 This is a schematic diagram of the structure of a device for evaluating product satisfaction according to Embodiment 3 of the present invention. DETAILED DESCRIPTION

[0041] In order to enable those skilled in the art to better understand the technical solution of the present invention, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0042] It should be understood that the specific embodiments and drawings described herein are only used to explain the present invention rather than to limit the present invention.

[0043] It can be understood that, in the absence of conflict, the various embodiments of the present invention and the various features in the embodiments can be combined with each other.

[0044] It can be understood that, for the convenience of description, the drawings of the present invention only show the parts related to the present invention, while the parts irrelevant to the present invention are not shown in the drawings.

[0045] It can be understood that each unit and module involved in the embodiments of the present invention may correspond to only one physical structure, or may be composed of multiple physical structures, or multiple units and modules may be integrated into one physical structure.

[0046] It can be understood that, without conflict, the functions and steps marked in the flowcharts and block diagrams of the present invention may occur in an order different from that marked in the drawings.

[0047] It is understood that the flowcharts and block diagrams of the present invention illustrate the possible architectures, functions, and operations of the systems, devices, equipment, and methods according to the various embodiments of the present invention. Each box in the flowchart or block diagram may represent a unit, module, program segment, or code, which contains executable instructions for implementing the specified functions. Moreover, each box or combination of boxes in the block diagram and flowchart may be implemented by a hardware-based system that implements the specified functions, or may be implemented by a combination of hardware and computer instructions.

[0048] It can be understood that the units and modules involved in the embodiments of the present invention can be implemented by software or hardware. For example, the units and modules can be located in a processor.

[0049] Application Overview

[0050] In the evaluation of product satisfaction in the communications industry, existing methods mainly include questionnaire surveys, user interviews, data analysis and modeling, as follows:

[0051] Questionnaire survey method: Questionnaire survey is a common method to collect user satisfaction data on products or services. By designing a questionnaire containing a series of questions, companies can systematically understand users' overall impression of communication products, usage experience, service quality evaluation and other aspects.

[0052] User interview method: Through face-to-face communication or telephone interviews with users, companies can obtain more specific and detailed user feedback.

[0053] Data analysis method: Use a single machine learning or deep learning algorithm to process and analyze the collected user data to reveal the patterns and trends behind the data and provide a scientific basis for corporate decision-making.

[0054] The current methods used to evaluate product satisfaction in the communications industry have the following shortcomings:

[0055] Questionnaire survey method: Although the questionnaire survey method can collect user feedback on a large scale, the authenticity and integrity of the data are challenged. Users may fill out the questionnaire casually for various reasons (such as time constraints, lack of interest or privacy concerns), resulting in biased data collected. In addition, the questionnaire design itself may have limitations, such as unclear question statements and unreasonable option settings, which may affect the authenticity and accuracy of user answers.

[0056] User interview method: User interview method can deeply understand the specific needs and feelings of users, but its sample size is usually small and difficult to represent the overall user group. In addition, the interview results may be affected by multiple factors such as the interviewer's skills, the interviewee's subjective emotions, and the interview environment, which leads to doubts about the objectivity and reliability of the results. At the same time, the interview process is time-consuming and costly, and is not suitable for large-scale application.

[0057] Data analysis method: Data analysis method is key in the evaluation of communication product satisfaction, but it depends on data quality, has high technical barriers, and the results are abstract and may ignore non-quantitative factors. It is also limited by timeliness. The common problem of a single machine learning algorithm is that it overemphasizes the algorithm and ignores the business process, and attempts to build a complex product evaluation system through a single algorithm, which is not ideal in accuracy.

[0058] In response to the above technical problems, the present application provides a product satisfaction evaluation method, device and readable storage medium, which classifies the complaint content in the customer service work order through a trained work order complaint classification model, and determines the complaint product corresponding to the customer service work order based on the complaint content in the customer service work order and the corresponding complaint reason and the product order information of the corresponding user, and further obtains the complaint volume of the target product based on the complaint products corresponding to all customer service work orders, and evaluates the product satisfaction of the target product based on the complaint volume and order volume of the target product, thereby achieving accurate judgment of the user complaint product and greatly improving the accuracy of the product satisfaction evaluation, so as to at least solve the problem that the existing product satisfaction evaluation method uses a single algorithm, resulting in low accuracy.

[0059] After introducing the basic principles of the present application, various non-limiting embodiments of the present application will be described in detail with reference to the accompanying drawings.

[0060] Embodiment 1:

[0061] This embodiment provides a method for evaluating product satisfaction. Figure 1 As shown, the method includes:

[0062] Step S101: Obtain multiple customer service tickets involving complaints.

[0063] It should be noted that customer service tickets involving complaints include: complaint tickets, consultation tickets, processing tickets and other types of tickets. Among them, each type of ticket contains user information, complaint content, customer service replies, etc.

[0064] Step S102: Input the complaint contents in the multiple customer service work orders into the trained work order complaint classification model to obtain the complaint reasons belonging to the complaint contents in each customer service work order, wherein the trained work order complaint classification model is trained based on the complaint contents and corresponding complaint reasons in multiple historical customer service work orders.

[0065] In this embodiment, a work order complaint classification model is trained based on the complaint contents and corresponding complaint reasons in multiple historical customer service work orders, so that the trained work order complaint classification model can classify the complaint reasons corresponding to the complaint contents in the customer service work orders.

[0066] It should be noted that the reasons for complaints include: traffic disputes, voice disputes, fee disputes, account opening for unknown reasons, binding for unknown reasons, subscription for unknown reasons, discount and contract disputes, cancellation of subscriptions, disputes over package resources and non-product categories.

[0067] In an optional embodiment, before inputting the complaint contents in the plurality of customer service work orders into the trained work order complaint classification model to obtain the complaint reason to which the complaint contents in each customer service work order belong, the method further comprises:

[0068] Get multiple historical customer service tickets;

[0069] Marking the complaint reasons corresponding to the complaint contents in the multiple historical customer service tickets;

[0070] The BERT-TextCNN model is used as the work order complaint classification model, and the work order complaint classification model is trained using the complaint contents in the multiple historical customer service work orders after labeling to obtain the trained work order complaint classification model.

[0071] Specifically, we collected multiple historical customer service tickets from the customer service side, and verified the specific reasons for complaints by referring to customer service staff’s labeling + manual analysis. We also labeled the reasons for complaints and deleted data irrelevant to the complaint content, such as customer personal information, timestamps, etc. We sorted out the manual analysis results and used them as data for model training. We used the BERT-TextCNN natural language classification model as the work order complaint classification model, imported the manually labeled training data, and trained the complaint reason classification model to obtain a trained work order complaint classification model.

[0072] Step S103: For each customer service work order among the multiple customer service work orders, obtain the complaint product corresponding to the customer service work order according to the complaint content in the customer service work order and the corresponding complaint reason and the product ordering information of the user corresponding to the customer service work order.

[0073] It should be noted that the complaint product corresponding to the customer service ticket refers to the product that the user complained about through the customer service ticket.

[0074] In an optional embodiment, obtaining the complaint product corresponding to the customer service work order according to the complaint content in the customer service work order and the corresponding complaint reason and the product order information of the user corresponding to the customer service work order specifically includes:

[0075] Perform product inspection on the complaint content in the customer service ticket to further refine the complaint reason and obtain specific complaint reason subcategories;

[0076] According to the complaint content in the customer service work order and the corresponding complaint reason subcategory, the complaint product corresponding to the customer service work order is located from multiple products in the product order information of the user corresponding to the customer service work order within a preset time period.

[0077] Specifically, the product verification is performed according to the complaint reasons of the complaint content in the customer service work order, which are divided into directional, community, capping, speed limit, and contract, so as to further refine the complaint reasons, obtain specific complaint reason subcategories, and obtain the product ordering information of the user corresponding to the customer service work order within a preset time period from the preset user ordering database. And based on the complaint content in the customer service work order and the corresponding complaint reason subcategories, the complaint product corresponding to the customer service work order is located from multiple products in the product ordering information.

[0078] In an optional embodiment, the product includes: ordered products and non-ordered products, and locating the complaint product corresponding to the customer service work order from multiple products in the product ordering information of the user corresponding to the customer service work order within a preset time period according to the complaint content in the customer service work order and the corresponding complaint reason subcategory specifically includes:

[0079] For subscription products, the complaint reason subcategory corresponding to the customer service ticket is associated with multiple subscription products in the product ordering information to obtain multiple subscription products related to the complaint reason subcategory, and the complaint content in the customer service ticket, multiple subscription products related to the complaint reason subcategory and the user complaint account period corresponding to the customer service ticket are matched according to a preset string fuzzy matching rule, and the subscription product with the highest matching degree is selected as the complaint product;

[0080] For non-subscription products, the complaint reason subcategory corresponding to the customer service ticket is associated with multiple non-subscription products in the product ordering information to obtain multiple non-subscription products related to the complaint reason subcategory, and the complaint content in the customer service ticket, multiple non-subscription products related to the complaint reason subcategory and the user complaint account period corresponding to the customer service ticket are matched according to a preset string fuzzy matching rule, and the non-subscription product with the highest matching degree is selected as the complaint product initially located; and,

[0081] The expense items in the bill of the user's complaint period corresponding to the customer service work order are matched with the complaint content in the customer service work order to obtain the expense item with the highest matching degree, and based on the expense item with the highest matching degree, the user's value-added details sheet corresponding to the customer service work order is used to identify the initially located complaint product to obtain the complaint product subcategory in the finally located complaint product.

[0082] Specifically, for subscription products, the complaint reason subcategory corresponding to the customer service ticket is associated with multiple subscription products in the product ordering information of the user corresponding to the customer service ticket, and associated with multiple subscription products related to the complaint reason subcategory. FuzzyWuzzy technology is used to match the complaint content in the customer service ticket, multiple subscription products related to the complaint reason subcategory, and the complaint account period of the user corresponding to the customer service ticket according to the string fuzzy matching rules, and the subscription product with the highest matching degree is selected as the complaint product.

[0083] It should be noted that non-subscription products may refer to products ordered by users on third-party platforms, among which non-subscription products can be further subdivided. For example, TV value-added products belong to non-subscription products. TV value-added products may include TV on demand-children's membership, TV interactive entertainment-e-sports, TV interactive entertainment-life and other TV value-added product subcategories.

[0084] Specifically, for non-subscription products, the complaint reason subcategory corresponding to the customer service ticket is associated with multiple non-subscription products in the product ordering information of the user corresponding to the customer service ticket, and associated with multiple non-subscription products related to the complaint reason subcategory. The FuzzyWuzzy technology is used to match the complaint content in the customer service ticket, multiple non-subscription products related to the complaint reason subcategory, and the complaint account period of the user corresponding to the customer service ticket according to the string fuzzy matching rules. The non-subscription product with the highest matching degree is selected as the complaint product initially located, and the various expense items in the complaint account period bill of the user corresponding to the customer service ticket are matched with the complaint content in the customer service ticket to identify the expense item with the highest matching degree. The complaint product subcategory in the initially located complaint product is identified based on the expense item with the highest matching degree in the value-added detailed bill of the user corresponding to the customer service ticket, so as to obtain the complaint product subcategory in the finally located complaint product.

[0085] Step S104: Obtain the number of complaints about the target product based on the complaint products corresponding to all the customer service work orders, and evaluate the product satisfaction of the target product based on the number of complaints and the order volume of the target product.

[0086] Specifically, the number of complaints about the target product is calculated based on the complaint products corresponding to all customer service tickets, and the product satisfaction of the target product is obtained based on the ratio of the number of complaints about the target product to the number of orders. Specifically, the complaint / order ratio is used as the main indicator, where the numerator is the number of complaints about the target product and the denominator is the number of orders for the target product. The larger the value of this indicator, the higher the complaint rate of the product, that is, the lower the product satisfaction.

[0087] In a specific embodiment, Figure 2 As shown, the implementation of the present invention needs to be divided into two layers of architecture: the capability building layer and the capability application layer. First, in the capability building layer, the service ticket (that is, the customer service ticket) is associated and analyzed through the combination of "big model + workflow + small model" to achieve rapid perception of sensitive business. This part is divided into three modules, namely: building an intelligent classification model for user complaint tickets based on the big model, associating user orders based on workflows, and accurately locating sensitive businesses based on the small model of similarity analysis. The capability application layer is to implement the evaluation of product satisfaction based on the capabilities of these three modules. The implementation process of these four modules is introduced below:

[0088] 1. Generate an intelligent classification model for user complaint tickets (also known as a ticket complaint classification model)

[0089] It should be noted that if Figure 3 As shown, the core idea of ​​the user complaint ticket intelligent classification model is to deeply comb through and automatically classify customer service tickets (including complaints, consultations, handling, etc.) based on the current BERT-TextCNN language model. Especially for the content of complaints, this method combines BERT's deep semantic understanding ability with TextCNN's text feature extraction expertise to automatically analyze the specific details and core issues of the complaint content to achieve accurate classification of complaint issues. Customer service tickets are divided into 10 categories, namely: traffic disputes, voice disputes, fee disputes, account opening for unknown reasons, binding for unknown reasons, subscription for unknown reasons, discount and contract disputes, cancellation of subscriptions, disputes over in-package resources, and non-product categories, that is, the output result is one of the 10 categories. The following are the main steps for generating the model:

[0090] 1. Complaint data collection

[0091] Specifically, complaint-related data was collected from the customer service side, and about 10,000 complaint, consultation and processing work orders were extracted from some customer service work orders, including user information, complaint content, and customer service replies. Among them: when processing complaints made in the form of voice, speech recognition (Speech-to-Text, STT) technology was used to efficiently convert voice content into text.

[0092] 2. Complaint data analysis

[0093] Specifically, the data extracted in the first step is first verified by referring to the customer service staff's labeling + manual analysis method to verify the specific reasons for the complaints, and the reasons for the complaints are classified and labeled (traffic disputes, voice disputes, fee disputes, account opening for unknown reasons, binding for unknown reasons, subscription for unknown reasons, discount and contract disputes, cancellation of subscriptions, disputes over in-package resources, and non-product categories).

[0094] 3. Data preprocessing

[0095] Specifically, remove irrelevant information: delete data parts that are irrelevant to the complaint content, such as customer personal information, timestamps, etc. Data labeling: organize the manual analysis results (complaint content-complaint reason) as data for model training.

[0096] 4. Data Splitting

[0097] (1) Modeling data splitting: The data is divided into M categories according to the user's location. Taking Shandong as an example, it is divided into 18 categories, 17 cities and the whole province.

[0098] (2) Training and testing data split: The dataset is divided into a training set and a test set, using the “2:8 ratio”, that is, 80% of the data is used for training and 20% is used to test the performance of the model.

[0099] 5. Model selection and training

[0100] Specifically, the BERT-TextCNN natural language classification model is used to import manually labeled training data (optimized complaint content text-complaint root cause) to train the complaint cause classification model.

[0101] Specifically, first use keras-bert to load the pre-trained BERT, input the constructed TextCNN network with multiple kernel sizes, obtain features through TextCNN, encapsulate the model according to the input and output, and tune the parameters.

[0102] Parameter tuning: By adjusting the model training parameters to achieve the optimal accuracy, the batch_size parameter involved in model training is used to adjust the number of items processed in each batch, which is adjusted according to the CPU usage; max_len is the text length of the processed text, which is calculated based on the length of the work order and the maximum length supported by the model; learning_rate is adjusted according to the task and data set. A larger learning rate is used at the beginning of training for rapid convergence, and the learning rate is gradually reduced during training to avoid skipping the optimal solution due to excessive learning rate when approaching the optimal solution; epochs is the number of iterations, which is continuously tuned according to the size and complexity of the data set, the model's learning rate strategy, the convergence speed, and the available training resources (such as GPU usage time).

[0103] 6. Model Evaluation

[0104] Specifically, the trained BERT-TextCNN model is used to make predictions on the test set. Indicators such as mean square error (MSE) and mean absolute error (MAE) are used to evaluate the model performance.

[0105] 7. Model deployment and use

[0106] Specifically, the multi-instance cluster concurrent calling mode can support the rapid location and analysis of more than 100,000 complaints per day; it supports customized pre-trained models and parameters to adapt to different needs, and can quickly adjust the training deployment scale to the production environment according to business data of different customer service data volumes for prediction implementation.

[0107] 2. Intelligent Positioning Model for User Complaint Products

[0108] It should be noted that if Figure 4 As shown in the figure, the core idea of ​​the user complaint product intelligent positioning model is to associate the results of the user complaint ticket intelligent classification model with the user's orders in the past three months (it is necessary to sort out the user's order-related data), and use FuzzyWuzzy technology to achieve real-time comparison of complaint issues (including user descriptions, customer service replies) and user-ordered products based on string fuzzy matching rules, and select the product with the highest similarity as the complaint product:

[0109] 1. Generate user ordering database

[0110] Construct a user subscription database, as shown in Table 1, capture the key fields: user id, number, belonging, subscription, and form a user subscription database. Among them: subscription includes user subscription products, group subscription products, and user monthly value-added subscription.

[0111] Table 1: User ordering database

[0112]

[0113] 2. Product verification

[0114] Specifically, product verification is carried out based on the results of the intelligent classification model of user complaint work orders, which are divided into five categories: directional, community, capped, speed limit, and contract. For example: traffic dispute - directional traffic is not exempted, traffic dispute - community traffic is not exempted, traffic dispute - speed limit, etc.

[0115] 3. Intelligent positioning of complaint products

[0116] It should be noted that since there are differences between subscription products and non-subscription products, this section introduces subscription products and non-subscription products separately:

[0117] (1) Order products

[0118] According to the detailed classification of the product verification link + the user's own ordering information, it is associated with the specific tariffs, products, and orders that are not satisfactory. FuzzyWuzzy technology is used to realize real-time comparison of complaint issues and user-ordered products according to string fuzzy matching rules, and the product with the highest similarity is selected as the complaint product. Refer to Example 1 below. Among them, FuzzyWuzzy technology has fuzzy string matching function, which can compare the similarity between two strings, even if they are slightly different in spelling or format. By using various algorithms and methods, such as character-based, word-based matching, partial matching and sorting matching techniques, FuzzyWuzzy can effectively handle spelling errors, synonyms and duplicates in text data. The application of this library can achieve more accurate positioning of user complaint products.

[0119] Example 1:

[0120] User complaint: The customer called to say that he did not agree with the traffic he activated but did not use (TikTok directional traffic 5G refueling package 20GB), and asked for a refund. Please help to process the call back, thank you.

[0121] Processing flow: Based on the content of the user complaint, use the complaint classification model to identify the complaint as a traffic dispute - directional traffic free flow scenario, analyze the user's orders and the products ordered by the group and group members, filter out the directional products ordered by the user according to the complaint classification scenario, and then match them according to the products ordered by the user, the complaint content, and the complaint account period, and identify the product with the highest match as the complained product.

[0122] (2) Non-order products

[0123] ① Preliminary positioning: products of the same order category. Based on the detailed classification of the product verification link + the user's own order information, it is associated with the specific tariff, product, and order that the user is dissatisfied with. FuzzyWuzzy technology is used to achieve real-time comparison of the complaint problem and the user's ordered product based on the string fuzzy matching rules, and the product with the highest similarity is selected as the complaint product for preliminary positioning.

[0124] ②Precise positioning: Obtain the expense items in the user’s complaint bill and match them with the complaint content, identify the expense items with the highest matching degree, and then use the user’s value-added detailed bill to identify and accurately locate the subcategory of the user’s complaint product.

[0125] Specifically, the TV value-added category belongs to the non-subscription category. Taking this as an example, refer to Example 2 below.

[0126] Example 2:

[0127] Disputed month: 2024.07, Business name: TV on demand-children's membership, TV interactive entertainment-e-sports, TV interactive entertainment-life.

[0128] Customer request and additional questions: The user called and stated that he was unaware of the TV on demand-children's membership, TV interactive entertainment-esports, and TV interactive entertainment-life services. He requested a refund for the fees incurred, requested expedited verification, and requested a cash refund. Please assist in processing the matter. Thank you.

[0129] Processing flow: Based on the content of the user's complaint, the complaint classification model is used to identify that the complaint is a fee dispute - TV value-added scenario. This scenario requires product positioning from two aspects. First: match the products ordered by the user on the third-party platform, the complaint content and the complaint account period, and identify the product with the highest match as the preliminary complaint product; second: use the fee items in the user's complaint account period bill and the complaint content to match, identify the fee items with the highest match, and then use the user's value-added detailed bill for identification, and accurately obtain the subcategory of the user's complaint product.

[0130] 3. Product Satisfaction Evaluation

[0131] The complaint / order ratio is the main indicator, where the numerator is the number of complaints about a product in a certain period of time (day, month, year...), and the denominator is the number of orders for the product in the same period of time. The larger the value of this indicator, the higher the complaint rate of the product, that is, the lower the product satisfaction.

[0132] It is worth mentioning that the present invention proposes a new evaluation method for product satisfaction in the communications industry that combines business and technology. This method uses a combination of "big model + workflow + small model" to perform correlation analysis on service tickets to achieve rapid perception of sensitive services. This part extracts complaints, consultations and handling tickets from some customer service tickets, performs manual labeling analysis, applies the BERT-TextCNN model to classify user complaints, and uses the workflow to adjust the user's 3-month order, and applies the similarity analysis small model to achieve accurate positioning of sensitive services. This method achieves accurate judgment of user complaint products. Obtain the user's actual order, and determine the product with the highest matching degree based on the user's description (that is, the complaint content). At present, this mode is changed from a fill-in-the-blank question to a multiple-choice question compared to the direct application of the model to output high-complaint risk products, which greatly reduces the computing power threshold of production support. To achieve accurate classification of all complaint ticket issues. This combination method shows good recognition accuracy in a variety of semantic analysis and recognition tasks, and the classification and recognition accuracy of complaint issues can reach more than 90%; it supports rapid analysis and positioning of a single line of up to 512 characters at a time, meeting almost all daily call problem recognition scenarios. This method uses a multi-instance cluster concurrent call mode, which can support the rapid location analysis of more than 100,000 complaints per day; it supports custom pre-trained models and parameters to adapt to different needs, and can quickly adjust the training deployment scale for business data with different data volumes of customer service. This method clearly defines the complaint / order ratio as the final evaluation indicator of product satisfaction, and locates products with low satisfaction based on this indicator.

[0133] The product satisfaction evaluation method provided in the embodiment of the present invention first obtains multiple customer service work orders involving complaints; then the complaint contents in the multiple customer service work orders are input into a trained work order complaint classification model to obtain the complaint reasons to which the complaint contents in each of the customer service work orders belong, wherein the trained work order complaint classification model is trained based on the complaint contents and the corresponding complaint reasons in multiple historical customer service work orders; then for each of the multiple customer service work orders, the complaint product corresponding to the customer service work order is obtained based on the complaint contents in the customer service work order and the corresponding complaint reasons and the product ordering information of the user corresponding to the customer service work order; finally, based on the complaint products corresponding to all the customer service work orders, the complaint volume of the target product is obtained, and based on the complaint volume and ordering volume of the target product, the product satisfaction of the target product is evaluated. The present invention classifies the complaint content in the customer service work order through a trained work order complaint classification model, and determines the complaint product corresponding to the customer service work order based on the complaint content in the customer service work order and the corresponding complaint reason and the product ordering information of the corresponding user, further obtains the complaint volume of the target product based on the complaint products corresponding to all customer service work orders, and evaluates the product satisfaction of the target product based on the complaint volume and order volume of the target product, thereby achieving accurate judgment on the user complaint product, greatly improving the accuracy of product satisfaction evaluation, and solving the problem that the existing product satisfaction evaluation method uses a single algorithm, resulting in low accuracy.

[0134] Embodiment 2:

[0135] like Figure 5 As shown, this embodiment provides a product satisfaction evaluation device, which is used to execute the above-mentioned product satisfaction evaluation method, including:

[0136] The first acquisition module 11 is used to acquire multiple customer service tickets involving complaints;

[0137] A first obtaining module 12, connected to the first obtaining module 11, is used to input the complaint contents in the plurality of customer service work orders into the trained work order complaint classification model, and obtain the complaint reasons to which the complaint contents in each customer service work order belong, wherein the trained work order complaint classification model is trained based on the complaint contents in a plurality of historical customer service work orders and the corresponding complaint reasons;

[0138] A second obtaining module 13, connected to the first obtaining module 12, is used to obtain, for each of the multiple customer service work orders, a complaint product corresponding to the customer service work order according to the complaint content and the corresponding complaint reason in the customer service work order and the product order information of the user corresponding to the customer service work order;

[0139] The evaluation module 14 is connected to the second obtaining module 13 and is used to obtain the complaint volume of the target product according to the complaint products corresponding to all the customer service work orders, and evaluate the product satisfaction of the target product according to the complaint volume and order volume of the target product.

[0140] Furthermore, the device also includes:

[0141] The second acquisition module is used to obtain multiple historical customer service work orders;

[0142] A marking module, used to mark the complaint reasons to which the complaint contents in the plurality of historical customer service tickets belong;

[0143] A training module is used to use the BERT-TextCNN model as the work order complaint classification model, and use the complaint content in the multiple historical customer service work orders after labeling to train the work order complaint classification model to obtain the trained work order complaint classification model.

[0144] Furthermore, the reasons for complaints include: traffic disputes, voice disputes, fee disputes, account opening for unknown reasons, binding for unknown reasons, subscription for unknown reasons, discount and contract disputes, cancellation of subscription, disputes over package resources and non-product categories.

[0145] Furthermore, the second obtaining module 13 specifically includes:

[0146] A product inspection unit, used to perform product inspection on the complaint content in the customer service ticket to further refine the complaint cause and obtain specific complaint cause subcategories;

[0147] A positioning unit is used to locate the complaint product corresponding to the customer service work order from multiple products in the product order information of the user corresponding to the customer service work order within a preset time period based on the complaint content and the corresponding complaint reason subcategory in the customer service work order.

[0148] Furthermore, the second obtaining module 13 further includes:

[0149] The acquisition unit is used to acquire the product ordering information of the user corresponding to the customer service work order within a preset time period from a preset user ordering database.

[0150] Furthermore, the products include: ordered products and non-ordered products, and the positioning unit specifically includes:

[0151] The subscription product positioning unit is used to associate the complaint reason subcategory corresponding to the customer service work order with the multiple subscription products in the product ordering information for subscription products, obtain multiple subscription products related to the complaint reason subcategory, and match the complaint content in the customer service work order, the multiple subscription products related to the complaint reason subcategory, and the user complaint account period corresponding to the customer service work order according to a preset string fuzzy matching rule, and select the subscription product with the highest matching degree as the complaint product;

[0152] a non-subscription product positioning unit, for non-subscription products, to associate the complaint reason subcategory corresponding to the customer service ticket with multiple non-subscription products in the product ordering information, to obtain multiple non-subscription products related to the complaint reason subcategory, and to match the complaint content in the customer service ticket, multiple non-subscription products related to the complaint reason subcategory, and the user complaint account period corresponding to the customer service ticket according to a preset string fuzzy matching rule, and select the non-subscription product with the highest matching degree as the complaint product initially positioned; and,

[0153] The expense items in the bill of the user's complaint period corresponding to the customer service work order are matched with the complaint content in the customer service work order to obtain the expense item with the highest matching degree, and based on the expense item with the highest matching degree, the user's value-added details sheet corresponding to the customer service work order is used to identify the initially located complaint product to obtain the complaint product subcategory in the finally located complaint product.

[0154] Further, the obtaining evaluation module 14 includes:

[0155] The obtaining unit is used to obtain the product satisfaction of the target product according to the ratio of the complaint quantity to the order quantity of the target product.

[0156] Embodiment 3:

[0157] refer to Figure 6 This embodiment provides a product satisfaction evaluation device, including a memory 21 and a processor 22, wherein the memory 21 stores a computer program, and the processor 22 is configured to run the computer program to execute the product satisfaction evaluation method in Example 1.

[0158] The memory 21 is connected to the processor 22. The memory 21 may be a flash memory, a read-only memory or other memory. The processor 22 may be a central processing unit or a single-chip microcomputer.

[0159] Embodiment 4:

[0160] This embodiment provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method for evaluating product satisfaction in the above-mentioned embodiment 1 is implemented.

[0161] The computer-readable storage medium includes volatile or non-volatile, removable or non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, computer program modules or other data). Computer-readable storage media include, but are not limited to, RAM (Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable read only memory), flash memory or other memory technology, CD-ROM (Compact Disc Read-Only Memory), digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer.

[0162] In summary, the product satisfaction evaluation method, device and readable storage medium provided in the embodiments of the present invention first obtain multiple customer service work orders involving complaints; then input the complaint contents in the multiple customer service work orders into the trained work order complaint classification model to obtain the complaint reasons to which the complaint contents in each of the customer service work orders belong, wherein the trained work order complaint classification model is trained based on the complaint contents and the corresponding complaint reasons in multiple historical customer service work orders; then for each of the multiple customer service work orders, according to the complaint contents in the customer service work order and the corresponding complaint reasons and the product ordering information of the user corresponding to the customer service work order, the complaint product corresponding to the customer service work order is obtained; finally, according to the complaint products corresponding to all the customer service work orders, the complaint volume of the target product is obtained, and according to the complaint volume and the ordering volume of the target product, the product satisfaction of the target product is evaluated. The present invention classifies the complaint content in the customer service work order through a trained work order complaint classification model, and determines the complaint product corresponding to the customer service work order based on the complaint content in the customer service work order and the corresponding complaint reason and the product ordering information of the corresponding user, further obtains the complaint volume of the target product based on the complaint products corresponding to all customer service work orders, and evaluates the product satisfaction of the target product based on the complaint volume and order volume of the target product, thereby achieving accurate judgment on the user complaint product, greatly improving the accuracy of product satisfaction evaluation, and solving the problem that the existing product satisfaction evaluation method uses a single algorithm, resulting in low accuracy.

[0163] It is to be understood that the above embodiments are merely exemplary embodiments used to illustrate the principles of the present invention, but the present invention is not limited thereto. For those of ordinary skill in the art, various modifications and improvements can be made without departing from the spirit and essence of the present invention, and these modifications and improvements are also considered to be within the scope of protection of the present invention.

Claims

1. A method for evaluating product satisfaction, characterized in that: The method comprises: Get multiple customer service tickets involving complaints; Inputting the complaint contents in the plurality of customer service work orders into the trained work order complaint classification model to obtain the complaint reasons to which the complaint contents in each customer service work order belong, wherein the trained work order complaint classification model is trained based on the complaint contents in the plurality of historical customer service work orders and the corresponding complaint reasons; For each of the multiple customer service work orders, obtaining a complaint product corresponding to the customer service work order according to the complaint content and the corresponding complaint reason in the customer service work order and the product order information of the user corresponding to the customer service work order; The number of complaints about the target product is obtained based on the complaint products corresponding to all the customer service work orders, and the product satisfaction of the target product is evaluated based on the number of complaints and the order volume of the target product.

2. The method according to claim 1, characterized in that Before inputting the complaint contents in the plurality of customer service work orders into the trained work order complaint classification model to obtain the complaint reasons to which the complaint contents in each customer service work order belong, the method further comprises: Get multiple historical customer service tickets; Marking the complaint reasons corresponding to the complaint contents in the multiple historical customer service tickets; The BERT-TextCNN model is used as the work order complaint classification model, and the work order complaint classification model is trained using the complaint contents in the multiple historical customer service work orders after labeling to obtain the trained work order complaint classification model.

3. The method according to claim 1, characterized in that The reasons for complaints include: traffic disputes, voice disputes, fee disputes, account opening for unknown reasons, binding for unknown reasons, subscription for unknown reasons, discount and contract disputes, cancellation of subscription, disputes over package resources and non-product categories.

4. The method according to claim 1, characterized in that: The obtaining, according to the complaint content in the customer service work order and the corresponding complaint reason and the product order information of the user corresponding to the customer service work order, the complaint product corresponding to the customer service work order specifically includes: Perform product inspection on the complaint content in the customer service ticket to further refine the complaint reason and obtain specific complaint reason subcategories; According to the complaint content in the customer service work order and the corresponding complaint reason subcategory, the complaint product corresponding to the customer service work order is located from multiple products in the product order information of the user corresponding to the customer service work order within a preset time period.

5. The method according to claim 4, characterized in that Before locating the complaint product corresponding to the customer service work order from multiple products in the product order information of the user corresponding to the customer service work order within a preset time period based on the complaint content and the corresponding complaint reason subclass in the customer service work order, the method further includes: The product ordering information of the user corresponding to the customer service work order within a preset time period is obtained from a preset user ordering database.

6. The method according to claim 4, characterized in that The products include: ordered products and non-ordered products. The locating of the complaint product corresponding to the customer service work order from multiple products in the product order information of the user corresponding to the customer service work order within a preset time period according to the complaint content in the customer service work order and the corresponding complaint reason subcategory specifically includes: For subscription products, the complaint reason subcategory corresponding to the customer service ticket is associated with multiple subscription products in the product ordering information to obtain multiple subscription products related to the complaint reason subcategory, and the complaint content in the customer service ticket, multiple subscription products related to the complaint reason subcategory and the user complaint account period corresponding to the customer service ticket are matched according to a preset string fuzzy matching rule, and the subscription product with the highest matching degree is selected as the complaint product; For non-subscription products, the complaint reason subcategory corresponding to the customer service ticket is associated with multiple non-subscription products in the product ordering information to obtain multiple non-subscription products related to the complaint reason subcategory, and the complaint content in the customer service ticket, multiple non-subscription products related to the complaint reason subcategory and the user complaint account period corresponding to the customer service ticket are matched according to a preset string fuzzy matching rule, and the non-subscription product with the highest matching degree is selected as the complaint product initially located; and, The expense items in the bill of the user's complaint period corresponding to the customer service work order are matched with the complaint content in the customer service work order to obtain the expense item with the highest matching degree, and based on the expense item with the highest matching degree, the user's value-added details sheet corresponding to the customer service work order is used to identify the initially located complaint product to obtain the complaint product subcategory in the finally located complaint product.

7. The method according to claim 1, characterized in that The evaluating the product satisfaction of the target product according to the number of complaints and the number of orders for the target product specifically includes: The product satisfaction of the target product is obtained according to the ratio of the complaint quantity to the order quantity of the target product.

8. A device for evaluating product satisfaction, characterized in that: include: The first acquisition module is used to obtain multiple customer service tickets involving complaints; A first obtaining module, connected to the first obtaining module, is used to input the complaint contents in the plurality of customer service work orders into the trained work order complaint classification model to obtain the complaint reasons to which the complaint contents in each customer service work order belong, wherein the trained work order complaint classification model is trained based on the complaint contents in a plurality of historical customer service work orders and the corresponding complaint reasons; A second obtaining module, connected to the first obtaining module, is used to obtain, for each of the multiple customer service work orders, a complaint product corresponding to the customer service work order according to the complaint content and the corresponding complaint reason in the customer service work order and the product order information of the user corresponding to the customer service work order; An evaluation module is obtained, which is connected to the second obtaining module, and is used to obtain the complaint volume of the target product based on the complaint products corresponding to all the customer service work orders, and to evaluate the product satisfaction of the target product based on the complaint volume and order volume of the target product.

9. A device for evaluating product satisfaction, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to implement the product satisfaction evaluation method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for evaluating product satisfaction as described in any one of claims 1 to 7 is implemented.

Images