User evaluation classification method, system and device and storage medium

Abstract

The invention relates to the technical field of text classification, in particular to a user evaluation classification method, system and device and a storage medium. The user evaluation classification method comprises the following steps: exporting evaluation information of a user as an Excel table, labeling category labels, carrying out word segmentation processing and data statistics, and indexing, cutting or filling segmented words of evaluation contents into one-dimensional vectors; a Mish activation function, a global attention module (GAM) and a weighted cross entropy loss function are introduced into an encoder structure of the Transform model; and classifying user evaluation information collected in real time based on the trained model, and outputting a classification result. The user evaluation classification method, system and device and the storage medium are high in feasibility and robustness, can accurately classify the user evaluation information, improve the value of the evaluation information, and promote a product optimization mode to tend to informatization and intellectualization.

Description

Technical Field

[0001] The present invention relates to the technical field of text classification, and particularly relates to a user evaluation classification method, system, device, and storage medium. Background Art

[0002] Users are the core of a product and the ultimate judges of its value. Only by paying attention to the user experience and listening to the voices of users can a successful and lasting product be created. With the improvement of people's consumption levels and the transformation of consumption concepts, consumers not only focus on the functionality of products but also pay more attention to the experience during the purchase and use processes. Consumers are more inclined to buy products with high service quality and emotional value. Therefore, constantly paying attention to user evaluation feedback, polishing and improving product quality, and meeting user needs have become the key to enhancing the market competitiveness of products.

[0003] However, in the face of a vast amount of user evaluation feedback information, how to efficiently reply and summarize product suggestions is a major challenge.

[0004] With the continuous development of scientific and technological innovation and information technology, informatization and intelligence have gradually become important trends and currents in social development. Massive information processing solutions led by artificial intelligence and deep learning have greatly reduced labor costs and time costs, efficiently empowered the extraction of the value of massive information resources, and promoted the product management mode to tend towards informatization and intelligent development.

[0005] In order to effectively classify user evaluations and feedback information on products, efficiently process different types of user needs, and improve user service quality, the present invention proposes a user evaluation classification method, system, device, and storage medium. Summary of the Invention

[0006] The present invention provides a simple and efficient user evaluation classification method, system, device, and storage medium to make up for the deficiencies of the prior art.

[0007] The present invention is implemented through the following technical solutions: A user evaluation classification method includes the following steps: Step S1: Preprocess user evaluation data; First, export the user evaluation information from the database to an Excel table and label category tags for the evaluation information through manual annotation; Then, perform word segmentation processing and data statistics on the evaluation content, index the word segmentation of the evaluation content, and crop or pad the word segmentation quantity of the sentence into a vector of a custom length; Finally, divide the training set and the test set according to a custom ratio; In step S1, according to the user evaluation content, the user evaluation is divided into eight categories: product quality, service attitude, logistics feedback, price feedback, after-sales feedback, suggestions and opinions, environmental feedback and brand feedback, and then the category to which each evaluation content belongs is manually determined, and the corresponding category label is respectively marked for each evaluation content; After the sample data is labeled, the number of samples in each category is counted.

[0008] In the step S1, the evaluation content is segmented by using the jieba tool library of Python to remove punctuation marks and function words in the evaluation content, including "了", "的", "也", "就" and "很", and the complete sentence is segmented into several words.

[0009] After word segmentation is completed, the number of words contained in the sentence content is counted to determine the feature vector step size of the neural network, which is also the dimension of the attention mechanism; All user reviews are indexed through the Tokenizer class of the keras library, and an index dictionary is established for 20,000 words to achieve the mapping relationship between words and index numbers.

[0010] In the step S1, the maximum length of the vocabulary is set to 32, and the indexed one-dimensional vector is fully filled or trimmed into a one-dimensional tensor with a length of 32; Finally, the dataset is divided into training set and test set in a ratio of 9:1.

[0011] Step S2, improving the Transform model under the condition of uneven sample distribution; Firstly, in order to solve the problem of uneven distribution of samples of different categories in real sample data and avoid gradient disappearance during model training, a self-regular non-monotonic Mish activation function is introduced into the encoder structure of the Transformer model. At the same time, in order to solve the problem of semantic continuity of user evaluation information after word segmentation, a global attention module GAM (Global Attention Mechanism) is introduced to strengthen the model's semantic understanding of the overall user evaluation information, correctly understand the user evaluation information, and improve the accuracy of user evaluation information classification.

[0012] Then, the weighted cross entropy loss function is introduced to improve the feature attention of small category samples; In step S2, the global attention module GAM is composed of a channel attention submodule and a spatial attention submodule; The channel attention submodule retains information in three dimensions through three-dimensional rearrangement, where the three dimensions are the width W, height H and number of channels C of the feature map; Then, a multi-layer perceptron network MLP, i.e., a fully connected layer network structure, is adopted to amplify the correlation of multi-dimensional features; and then the feature map is restored to the original dimension through a reverse three-dimensional rearrangement method; Finally, the feature map is output through the sigmoid activation layer; The spatial attention sub-module uses two convolutional layers for spatial information fusion: First, a convolutional layer with a 7×7 convolutional kernel reduces the dimension according to the reduction ratio coefficient r of the dimension (number of channels C), and the set reduction ratio coefficient r is 4; then, it is restored to the previous dimension through a convolutional layer with a 7×7 convolutional kernel; finally, the feature map is output through the sigmoid activation layer.

[0013] In the step S2, the calculation formula of the weighted cross-entropy loss function is: ; In the above formula, N is the total number of samples; C is the total number of categories, and the user evaluation classification is 8 categories, corresponding to the value 8; is the weight of category q; is an indicator variable (0 or 1), which takes the value of 1 if the model predicts that the category of sample i is the same as category q, otherwise 0; is the probability that the model predicts that sample i belongs to category q, which is the prediction result of the model; Customize to assign different weights to different categories , the larger the proportion of the sample category, the smaller the assigned weight value; During the iterative training process, gradually improve the prediction accuracy of small categories, so that the loss function value gradually converges to the minimum value, thereby guiding the improved Transform model to focus on the sample features of small categories.

[0014] Step S3: Classify the user evaluation information based on the improved Transform model; Train the improved Transform model based on the training set, and verify the classification accuracy of the improved Transform model through the test set; Classify the user evaluation information collected in real time based on the trained model, and output the classification result.

[0015] In the step S3, during the training process, the early stopping training strategy is adopted, and the tolerance for the weighted cross-entropy loss function value not to decrease is set to 5 epochs, that is, if the loss function decreases by less than the custom threshold in 5 consecutive epochs, the model training will be terminated in advance to improve the training efficiency.

[0016] The hardware environment for network training is to use the Ubuntu operating system and use the NVIDIA GeForce GTX1080Ti graphics card for platform safety area segmentation model training with 11GB of video memory.

[0017] Set up an ablation experiment to evaluate the effect of improving the Transform model A user evaluation classification system, used to implement the above method, comprises: The data preprocessing module is responsible for exporting the user's evaluation information from the database into an Excel table and labeling the evaluation information with category labels through manual labeling. Then, the evaluation content is segmented and data statistics are performed, the segmented evaluation content is indexed, and the segmented amount of the sentence is trimmed or filled to a vector of a custom length. Finally, the training set and test set are divided according to the custom ratio. The Transform model improvement module is responsible for introducing the self-regular non-monotonic Mish activation function into the encoder structure of the Transformer model to solve the problem of uneven distribution of samples of different categories in real sample data and avoid gradient disappearance during model training; the global attention module GAM (Global Attention Mechanism) is introduced to strengthen the model's semantic understanding of the overall user evaluation information, correctly understand user evaluation information, and improve the accuracy of user evaluation information classification; the weighted cross entropy loss function is introduced to increase the feature attention of small category samples; The improved Transform model application module is responsible for training the improved Transform model based on the training set, verifying the classification accuracy of the improved Transform model through the test set; classifying the user evaluation information collected in real time based on the trained model, and outputting the classification results.

[0018] A user evaluation classification device comprises a memory and a processor; the memory is used to store a computer program, and the processor is used to implement the method when executing the computer program.

[0019] A readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described is implemented.

[0020] The beneficial effects of the present invention are: the user evaluation classification method, system, device and storage medium provide a highly feasible and robust solution to the classification problem of uneven sample distribution, can accurately classify user evaluation information, enhance the value of evaluation information, and promptly guide processing personnel to reply to comments and summarize product optimization, thereby promoting the product optimization model to be more information-based and intelligent. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0022] Appendix Figure 1 It is a schematic diagram of the design architecture of the global attention module GAM of the present invention.

[0023] Appendix Figure 2 It is a schematic diagram of the improved Transformer model structure of the present invention.

[0024] Appendix Figure 3 It is a schematic diagram of the user evaluation classification method of the present invention.

[0025] Appendix Figure 4 It is a schematic diagram of the statistical results of the number of samples in each category of the present invention.

[0026] Appendix Figure 5 It is a schematic diagram of the comparison of the function images and derivative images of the self-regulated non-monotonic Mish activation function and the rectified linear unit ReLU activation function of the present invention.

[0027] Appendix Figure 6 It is a schematic diagram of the comparison of the training results of the traditional Transformer model and the improved Transformer model of the present invention.

[0028] Appendix Figure 7 It is a schematic diagram of the comparison of the prediction confusion matrices of the traditional Transformer model and the improved Transformer model of the present invention. Detailed implementation manners

[0029] In order to enable those skilled in the art of the present technology to better understand the technical solutions in the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0030] The user evaluation classification method includes the following steps: Step S1, preprocessing of user evaluation data; First, export the evaluation information of users from the database to an Excel table, and label the category labels for the evaluation information by manual annotation; Then, perform word segmentation processing and data statistics on the evaluation content, index the word segmentation of the evaluation content, and crop or pad the word segmentation quantity of the sentence into a vector of a custom length; Finally, divide the training set and the test set according to a custom ratio; In the step S1, according to the user evaluation content, the user evaluation is divided into eight categories: product quality, service attitude, logistics feedback, price feedback, after-sales feedback, suggestions and opinions, environment feedback, and brand feedback. Then, manually judge the category to which each evaluation content belongs, and label the corresponding category label for each evaluation content respectively; After the sample data annotation is completed, count the number of samples in each category.

[0031] In the step S1, use the jieba tool library in Python to perform word segmentation processing on the evaluation content, remove punctuation marks and function words in the evaluation content. The function words include "le", "de", "ye", "jiu", and "hen", and cut the complete sentence into several words.

[0032] After the word segmentation is completed, count the number of words contained in the sentence content to determine the feature vector step length of the neural network, which is also the dimension of the attention mechanism; Index the word segmentation of all user evaluations through the Tokenizer class in the keras library, establish an index dictionary for 20,000 words, and implement the mapping relationship between words and index numbers.

[0033] In the step S1, set the maximum word length to 32, and pad or crop all the indexed one-dimensional vectors into one-dimensional tensors with a length of 32; Finally, divide it into a training set and a test set according to a ratio of 9:1.

[0034] Step S2: Improve the Transformer model in the case of unbalanced sample distribution; First, to solve the problem of unbalanced sample distribution of each category in the real sample data and avoid gradient disappearance during the model training process, introduce the self-regular non-monotonic Mish activation function in the encoder Encoder structure of the Transformer model. At the same time, to solve the problem of semantic continuity after word segmentation of user evaluation information, introduce the global attention module GAM (Global Attention Mechanism) to strengthen the model's semantic understanding of the overall user evaluation information, correctly understand the user evaluation information, and improve the accuracy of user evaluation information classification.

[0035] Then, introduce the weighted cross-entropy loss function to enhance the feature attention of small-category samples; In the step S2, the global attention module GAM is composed of a channel attention sub-module and a spatial attention sub-module; The channel attention sub-module preserves information in three dimensions through three-dimensional rearrangement. The three dimensions are the width W, height H, and number of channels C of the feature map; Then, a multi-layer perceptron network MLP, that is, a fully connected layer network structure, is used to amplify the correlation of multi-dimensional features; and then the feature map is restored to the original dimension through the reverse three-dimensional rearrangement method; Finally, the feature map is output through the sigmoid activation layer; The spatial attention sub-module uses two convolutional layers for spatial information fusion: First, a convolutional layer with a 7×7 convolutional kernel reduces the dimension according to the reduction ratio coefficient r in the dimension (number of channels C), and the set reduction ratio coefficient r is 4; then, it is restored to the previous dimension through a convolutional layer with a 7×7 convolutional kernel; finally, the feature map is output through the sigmoid activation layer.

[0036] In the step S2, the calculation formula of the weighted cross-entropy loss function is: ; In the above formula, N is the total number of samples; C is the total number of categories. The user evaluation is classified into 8 categories, and the corresponding value is 8; is the weight of category q; is an indicator variable (0 or 1), which takes the value of 1 if the category predicted by the model for sample i is the same as category q, otherwise it is 0; is the probability that the model predicts that sample i belongs to category q, which is the prediction result of the model; is customized to assign different weights to different categories The larger the proportion of the sample category, the smaller the assigned weight value; During the iterative training process, the prediction accuracy of small categories is gradually improved, so that the loss function value gradually converges to the minimum value, thereby guiding the improved Transform model to focus on the sample features of small categories.

[0037] Step S3: Classify the user evaluation information based on the improved Transform model; Train the improved Transform model based on the training set, and verify the classification accuracy of the improved Transform model through the test set; Classify the user evaluation information collected in real time based on the trained model, and output the classification result.

[0038] In the step S3, during the training process, the early stopping training strategy is adopted, and the tolerance for the weighted cross-entropy loss function value not to decrease is set to 5 epochs, that is, if the loss function decreases by less than the customized threshold in 5 consecutive epochs, the model training is terminated in advance to improve the training efficiency.

[0039] The hardware environment for network training uses the Ubuntu operating system and the NVIDIA GeForce GTX1080Ti graphics card to train the platform security area segmentation model, with a video memory of 11GB.

[0040] Set up ablation experiments to evaluate the improvement effect of the improved Transformer model. This user evaluation classification system is used to implement the above method and includes: A data preprocessing module responsible for exporting the user's evaluation information from the database to an Excel table, and manually annotating category labels for the evaluation information; then, performing word segmentation processing and data statistics on the evaluation content, indexing the word segmentation of the evaluation content, and cropping or padding the word segmentation amount of the sentence into a vector of a custom length; finally, dividing the training set and the test set according to a custom ratio. An improved Transformer model module responsible for introducing the self-regular non-monotonic Mish activation function into the encoder Encoder structure of the Transformer model to solve the problem of uneven distribution of various category samples in real sample data and avoid gradient disappearance during model training; introducing the global attention module GAM (Global Attention Mechanism) to strengthen the model's semantic understanding of the overall user evaluation information, correctly understand the user evaluation information, and improve the accuracy of user evaluation information classification; introducing a weighted cross-entropy loss function to enhance the feature attention of small-category samples. An improved Transformer model application module responsible for training the improved Transformer model based on the training set and verifying the classification accuracy of the improved Transformer model through the test set; classifying the real-time collected user evaluation information based on the trained model and outputting the classification results.

[0041] A user evaluation classification device includes a memory and a processor; the memory is used to store a computer program, and the processor is used to implement the method when executing the computer program.

[0042] A readable storage medium stores a computer program, and the method is implemented when the computer program is executed by a processor.

[0043] Example: The user evaluation data preprocessing method includes the following steps: (1) Export the real user evaluation data from the database to an Excel table. (2) According to the real data, it is found that there are 76,661 pieces of data exported to Excel. Organize the Excel table, remove other table columns, and only keep the column of user evaluation content, and name this column "appraise". (3)According to the user evaluation content, the user evaluations are divided into eight categories: "Product Quality", "Service Attitude", "Logistics Feedback", "Price Feedback", "After-sales Feedback", "Suggestions and Opinions", "Environment Feedback", and "Brand Feedback". Then, manually judge which category the evaluation content belongs to and label this evaluation as the category it belongs to; (4)After the sample data annotation is completed, count the number of samples in each category. The statistical results are as Figure 4 shown. As can be seen from the appendix Figure 4 It can be seen that the "Product Quality" category corresponds to 58,321 user evaluations, while the "Environment Feedback" category and the "Brand Feedback" category only correspond to 204 and 406 user evaluations respectively. Therefore, there is an uneven distribution among the sample categories. This problem should be addressed emphatically during the classification model design process to avoid affecting the classification accuracy; (5)Use the jieba toolkit in Python to perform word segmentation on the user evaluation content, remove punctuation marks in the evaluation content, or function words such as "le", "de", "ye", "jiu", "hen", etc., and split the complete sentence into multiple words for representation. For example, if the user evaluation content is "The battery life is much lower than the advertised duration.", after word segmentation, it becomes "battery life much lower than advertised duration"; (6)Count the number of vocabulary words contained in the sentence content after word segmentation; among the user evaluations, the proportion of evaluation content composed of 8 segmented words is the highest, corresponding to 9,354 user evaluations; the maximum number of segmented words in the evaluation content is 81, but there are only two user evaluations.

[0044] Counting the number of segmented words aims to determine the feature vector step size of the neural network and also the dimension of the attention mechanism. Since the vocabulary lengths of each user evaluation are different, and the model requires tensors of the same dimension during training (padding or truncating), it is necessary to determine the maximum length of the vocabulary. If the maximum length is too small, a lot of information in many sentences will be lost, while if the maximum length is too large, the data matrix will be too sparse and the computational complexity will increase sharply.

[0045] Through experimental tests, it is found that when the maximum vocabulary length is set to 32 in the current dataset, the classification accuracy is relatively high; (7)Index all the segmented words of the user evaluations through the Tokenizer class in the keras library, establish an index dictionary for 20,000 vocabulary words, and implement the mapping relationship between the vocabulary and the index numbers; (8)Represent the segmented words "battery life much lower than advertised duration" in step (5) as a one-dimensional tensor [18, 478, 71, 698, 380, 123, 699], that is, the vocabulary "battery" corresponds to the index number 18, the vocabulary "life" corresponds to the index number 478, the vocabulary "time" corresponds to the index number 71, and so on; (9) According to the maximum vocabulary length of 32 set in step (6), pad or truncate the indexed one-dimensional tensor to a length of 32, then the one-dimensional tensor in step (7) is transformed into [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,18, 478,71,698,380,123,699], that is, a one-dimensional tensor with a length of 32. (10) Divide the processed sample data into a training set and a test set according to a ratio of 9:1.

[0046] Optimization and improvement methods for the Transform model include the following steps: Step 1): Construct a classification model based on the Transform model The Transformer model is a deep learning model based on the self-attention mechanism, which is widely used in natural language processing (NLP) tasks such as machine translation, text generation, text classification, etc. The Transformer model consists of two parts: an encoder and a decoder. The encoder is composed of multiple identical layers stacked together, and each layer contains a multi-head self-attention mechanism and a feed-forward neural network, which is mainly used to transform the source language sequence into a tensor-encoded representation and is suitable for text classification tasks. Therefore, it is necessary to reconstruct the encoder structure of the Transformer model to construct a text classification model suitable for the scenario of unbalanced sample distribution.

[0047] To solve the problem of unbalanced distribution of various types of samples in real sample data, a self-regular non-monotonic Mish activation function is introduced into the encoder structure of the Transformer model. At the same time, to solve the problem of semantic continuity after word segmentation of user evaluation information, a global attention module GAM (Global Attention Mechanism) is introduced to strengthen the model's semantic understanding of the overall user evaluation information, correctly understand the user evaluation information, and improve the accuracy of user evaluation information classification.

[0048] In traditional Transformer models, the rectified linear unit ReLU activation function is adopted. During the forward propagation process of this activation function, if the weight parameter is less than zero, the derivative of the rectified linear unit ReLU activation function is zero, that is, the gradient value is zero, resulting in the neuron remaining in an inactive state and causing gradient vanishing during backpropagation, making the weight parameter unable to be iteratively updated and the network unable to learn. If the rectified linear unit ReLU activation function is applied to samples with unbalanced distributions, it is easier to ignore the features of samples with a smaller class proportion during the model training process, that is, the weight parameter gradually tends to 0, while focusing on the features of samples with a larger class proportion, further deepening the problem of unbalanced sample distribution. Therefore, the self-regulated non-monotonic Mish activation function is introduced to replace the rectified linear unit ReLU activation function.

[0049] The function images and derivative images of the self-regulated non-monotonic Mish activation function and the rectified linear unit ReLU activation function are compared as attached. Figure 5 Shown as follows.

[0050] The self-regulated non-monotonic Mish activation function integrates the advantages of the softplus activation function and the tanh activation function, and optimizes the performance of the rectified linear unit ReLU activation function. Its characteristics include being unbounded above, having a lower bound, and being non-monotonic. Among them, being unbounded above can avoid gradient saturation, thus accelerating the training process; having a lower bound helps to achieve a strong regularization effect. Moreover, as can be seen from the attached Figure 5 derivative image, the self-regulated non-monotonic Mish activation function will not have a situation where the gradient value is zero, effectively avoiding the problem of feature loss of samples with a smaller class proportion. These characteristics help to improve the stability and robustness of the network model.

[0051] The calculation formula of the self-regulated non-monotonic Mish activation function is as follows, where x represents the feature variable: ; ; ; The calculation formula of the derivative of the self-regulated non-monotonic Mish activation function is as follows, where x represents the feature variable: ; ; ; Introducing the GAM global attention mechanism can amplify global features while reducing information diffusion. The global attention module GAM is composed of a channel attention sub-module and a spatial attention sub-module. The design architecture of the global attention module GAM is as attached Figure 1 Shown as follows.

[0052] The channel attention sub-module preserves information in three dimensions through a three-dimensional rearrangement method. For example, in the permutation structure in the appendix, the input feature map is transformed from the C×H×W dimension to the W×H×C dimension, where W, H, and C represent the width, height, and number of channels of the feature map respectively. Then, a multi-layer perceptron network MLP, that is, a fully connected layer network structure, is used to amplify the correlation of multi-dimensional features. Then, the feature map is restored to the C×H×W dimension through an inverse three-dimensional rearrangement method. Finally, the output feature map is obtained through a sigmoid activation layer. Figure 1 The spatial attention sub-module performs spatial information fusion by using two convolutional layers. First, a convolutional layer with a 7×7 convolutional kernel reduces the dimension according to the reduction ratio coefficient r of the dimension (number of channels C). In this embodiment, the set reduction ratio coefficient is 4. Then, it is restored to the previous dimension through a convolutional layer with a 7×7 convolutional kernel. Finally, the output feature map is obtained through a sigmoid activation layer.

[0053] The spatial attention sub-module performs spatial information fusion by using two convolutional layers. First, a convolutional layer with a 7×7 convolutional kernel reduces the dimension according to the reduction ratio coefficient r of the dimension (number of channels C). In this embodiment, the set reduction ratio coefficient is 4. Then, it is restored to the previous dimension through a convolutional layer with a 7×7 convolutional kernel. Finally, the output feature map is obtained through a sigmoid activation layer.

[0054] The spatial attention sub-module performs spatial information fusion by using two convolutional layers. First, a convolutional layer with a 7×7 convolutional kernel reduces the dimension according to the reduction ratio coefficient r of the dimension (number of channels C). In this embodiment, the set reduction ratio coefficient is 4. Then, it is restored to the previous dimension through a convolutional layer with a 7×7 convolutional kernel. Finally, the output feature map is obtained through a sigmoid activation layer.

[0055] Therefore, the design formula of the global attention module GAM is: ; ; In the above formula, is the input feature map; is the convolution operation; is the operation process of the channel attention sub-module; is the output feature map of the channel attention sub-module; is the operation process of the spatial attention sub-module; is the input feature map.

[0056] In summary, the global attention module GAM extends the feature correlation in the global range and improves the semantic understanding ability of the overall user evaluation by performing dimension transformation and spatial information fusion on the feature map.

[0057] Step 2): Design of the weighted cross-entropy loss function To solve the problem of uneven distribution of various types of samples in the real sample data and improve the classification accuracy of text categories with a relatively small proportion, a weighted cross-entropy loss function is introduced, and its calculation formula is: ; In the above formula, N is the total number of samples; C is the total number of categories. The user evaluation classification is 8 categories, and the corresponding value is 8; is the weight of category q; is an indicator variable (0 or 1), which takes the value of 1 if the model predicts that the category of sample i is the same as category q, otherwise it is 0; is the probability that the model predicts that sample i belongs to category q, which is the prediction result of the model.

[0058] The weighted cross-entropy loss function adjusts the calculation result of the loss function by assigning different weights to different categories (corresponding to in the formula), that is, assigns a larger weight value to the sample category with a smaller proportion and a smaller weight value to the sample category with a larger proportion, guiding the model to focus on the sample features with a smaller proportion.

[0059] Specifically, in the training stage of the model, the classification error cost of the samples with a smaller proportion is increased. When the model predicts the sample category with a smaller proportion incorrectly, since the weight value ( ) assigned to the sample category with a smaller proportion is larger, as can be seen from the above formula, the increase in the weight value will increase the loss function value, and the goal of model training is that the loss function value gradually decreases and approaches 0 infinitely. Therefore, in the subsequent iterative training process, the model will focus on improving the prediction accuracy of this category to ensure that the loss function value is minimized, so that the loss function value can gradually converge to the minimum value.

[0060] By the above method, it is equivalent to adding a penalty mechanism in the model training process. Once the sample category with a smaller proportion is predicted incorrectly, the loss function value will be greatly increased.

[0061] The user evaluation classification method based on the improved Transform model includes the following steps: Step 1): Model construction According to the above improvement scheme, construct an improved Transform user evaluation classification model, and the model structure is as shown in the appendix Figure 2 ; Step 2): Model training The hardware environment for network training in this embodiment is to apply the Ubuntu operating system and use the NVIDIA GeForce GTX1080Ti graphics card to train the platform safety area segmentation model, and the video memory is 11GB. During the training process, the early stopping training strategy is adopted, and the tolerance for the loss function value not to change (decrease) is set to 5 epochs, that is, if the loss function does not decrease significantly in 5 consecutive epochs, the model training will be terminated in advance to improve the training efficiency.

[0062] Step 3: Ablation experiment To verify the reliability of the improved Transform model solution, this embodiment designs an ablation experiment to compare the classification accuracy of the traditional Transform model and the improved Transform model in the unbalanced sample classification problem. Among them, the training results of the two models are as attached Figure 6 as shown. The model training results show that the improved Transform model achieves a higher classification accuracy in the early stage of training, and during the subsequent iterative training process, its classification accuracy converges faster, indicating that the improved Transform model can quickly and accurately extract the key features of user evaluations.

[0063] In this embodiment, among the 7,663 user evaluations in the test set, the classification situations of the traditional Transform model and the improved Transform model are statistically analyzed, and visual evaluation is carried out through the confusion matrix. The prediction confusion matrices of the two models are as attached Figure 7 as shown. Among them, the confusion matrix is a square matrix of size (n_classes, n_classes), where n_classes represents the number of categories, corresponding to 8 categories in this embodiment. Each row of this matrix represents the instances in the true class, and each column represents the instances in the predicted class.

[0064] Taking the second row "Price feedback" of the confusion matrix of the improved Transform model as an example, the data is [0, 186, 0, 1, 1, 0, 0, 0]. Summing this one-dimensional array indicates that there are a total of 188 "Price feedback" test data. Through model prediction, a total of 186 data are predicted as the "Price feedback" category by the model, and the predicted category is consistent with the vertical axis category "Price feedback", that is, the prediction is correct; 1 data is predicted as the "After-sales feedback" category by the model, and the predicted category is inconsistent with the vertical axis category "Price feedback", that is, the prediction is incorrect; 1 data is predicted as the "Suggestions and opinions" category by the model, and the predicted category is inconsistent with the vertical axis category "Price feedback", that is, the prediction is incorrect; Therefore, among the 188 user evaluation data of the "Price feedback" category, a total of 186 can be correctly classified, and 2 are misclassified. It can be found through the confusion matrix that the improved Transform model has a higher classification accuracy than the traditional Transform encoder model in each classification category.

[0065] In summary, in the scenario of unbalanced sample distribution, the improved Transform model still has a high classification accuracy, verifying the effectiveness and reliability of the Transform model optimization solution proposed in this embodiment.

[0066] Summarizing the above algorithm process, its working process is as attached Figure 3As shown, taking specific user evaluation data as input samples, the data processing flow of the algorithm and the model prediction classification results are demonstrated.

[0067] With the help of an improved Transformer encoder model, massive user evaluation information is classified. For the user needs of each category, it guides the staff to reply and formulate product improvement plans, promoting the product optimization mode to be more inclined to informatization and intelligence.

[0068] Those of ordinary skill in the art can realize that the units and algorithm steps of the examples described in conjunction with this embodiment can be implemented by electronic hardware or a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.

[0069] The above has introduced in detail a user evaluation classification method, system, device, and storage medium in the embodiments of the present invention. This part uses specific examples to elaborate on the principle and implementation manner of the invention. The above examples are only used to help understand the core idea of the present invention. Without departing from the principle of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the scope of protection of the present invention.

Claims

1. A user evaluation classification method, characterized by: It includes the following steps: Step S1: Preprocess user evaluation data; First, export the user evaluation information from the database to an Excel table, and label the category tags for the evaluation information through manual annotation; Then, perform word segmentation processing and data statistics on the evaluation content, index the word segmentation of the evaluation content, and crop or pad the word segmentation quantity of the sentence to a vector of a custom length; Finally, divide the training set and the test set according to a custom ratio; Step S2: Improve the Transformer model in the case of unbalanced sample distribution; First, to solve the problem of unbalanced sample distribution of each category in the real sample data and avoid gradient disappearance during model training, introduce the self-regularized non-monotonic Mish activation function in the encoder Encoder structure of the Transformer model. At the same time, to solve the problem of semantic continuity after word segmentation of user evaluation information, introduce the global attention module GAM to strengthen the model's semantic understanding of the overall user evaluation information, correctly understand the user evaluation information, and improve the accuracy of user evaluation information classification; Then, introduce the weighted cross-entropy loss function to enhance the feature attention of small-category samples; Step S3: Classify user evaluation information based on the improved Transformer model; Train the improved Transformer model based on the training set, and verify the classification accuracy of the improved Transformer model through the test set; Classify the user evaluation information collected in real time based on the trained model and output the classification result.

2. The user evaluation classification method according to claim 1, characterized in that: In the step S1, according to the user evaluation content, the user evaluation is divided into eight categories: product quality, service attitude, logistics feedback, price feedback, after-sales feedback, suggestions and opinions, environment feedback, and brand feedback. Then, manually judge the category to which each evaluation content belongs, and label the corresponding category tags for each evaluation content respectively; After the sample data annotation is completed, count the number of samples in each category.

3. The user evaluation classification method according to claim 1, characterized in that: In the step S1, use the jieba tool library in Python to perform word segmentation processing on the evaluation content, remove punctuation marks and function words from the evaluation content. The function words include "le", "de", "ye", "jiu", and "hen", and cut the complete sentence into several words; After word segmentation is completed, count the number of words contained in the sentence content to determine the feature vector step size of the neural network, which is also the dimension of the attention mechanism; Index the word segmentation of all user evaluations through the Tokenizer class in the keras library, establish an index dictionary for the words, and implement the mapping relationship between the words and the index numbers.

4. The user evaluation classification method according to claim 3, characterized in that: In the step S1, set the maximum length of the words to 32, and pad or crop all the indexed one-dimensional vectors to one-dimensional tensors with a length of 32; Finally, divide it into a training set and a test set according to a ratio of 9:

1.

5. The user evaluation classification method according to claim 1, characterized in that: In the step S2, the global attention module GAM is composed of a channel attention sub-module and a spatial attention sub-module; The channel attention sub-module retains information in three dimensions through a three-dimensional rearrangement method. The three dimensions are the width W, height H, and number of channels C of the feature map; Then, a multi-layer perception network (MLP), i.e. a fully connected layer network structure, is used to amplify the correlation of multi-dimensional features; and the feature map is restored to its original dimension through a reverse three-dimensional rearrangement method; Finally, the feature map is output through the sigmoid activation layer; The spatial attention submodule uses two convolutional layers to fuse spatial information: First, a convolution layer with a 7×7 convolution kernel is used to reduce the dimension according to the dimension reduction ratio factor r; then, a convolution layer with a 7×7 convolution kernel is used to restore it to the previous dimension; finally, a sigmoid activation layer is used to output the feature map.

6. The user evaluation classification method according to claim 1, characterized in that: In step S2, the weighted cross entropy loss function calculation formula is: ； In the above formula, N is the total number of samples; C is the total number of categories; is the weight of category q; is an indicator variable, which takes the value of 1 if the model predicts that the category of sample i is the same as category q, otherwise it takes the value of 0; is the probability that the model predicts that sample i belongs to category q, and is the prediction result of the model; Customize and assign different weights to different categories , the larger the proportion of sample categories, the smaller the assigned weight value; During the iterative training process, the prediction accuracy of small categories is gradually improved, so that the loss function value gradually converges to the minimum value, thereby guiding the improved Transform model to focus on the sample characteristics of small categories.

7. The user evaluation classification method according to claim 1, characterized in that: In step S3, during the training process, an early stopping training strategy is adopted, and the tolerance for the weighted cross entropy loss function value not decreasing is set to 5 cycles, that is, if the loss function decreases below the custom threshold within 5 consecutive cycles, the model training is terminated early to improve the training efficiency.

8. A user evaluation classification system, characterized by: Used to implement the method according to any one of claims 1 to 7, comprising: The data preprocessing module is responsible for exporting the user's evaluation information from the database into an Excel table and labeling the evaluation information with category labels through manual labeling. Then, the evaluation content is segmented and data statistics are performed, the segmented evaluation content is indexed, and the segmented amount of the sentence is trimmed or filled to a vector of a custom length. Finally, the training set and test set are divided according to the custom ratio. The Transform model improvement module is responsible for introducing the self-regular non-monotonic Mish activation function into the encoder structure of the Transformer model to solve the problem of uneven distribution of samples of different categories in real sample data and avoid gradient disappearance during model training; the global attention module GAM is introduced to strengthen the model's semantic understanding of the overall user evaluation information, correctly understand user evaluation information, and improve the accuracy of user evaluation information classification; the weighted cross entropy loss function is introduced to increase the feature attention of small category samples; The improved Transform model application module is responsible for training the improved Transform model based on the training set, verifying the classification accuracy of the improved Transform model through the test set; classifying the user evaluation information collected in real time based on the trained model, and outputting the classification results.

9. A user evaluation classification device, characterized in that: The method comprises a memory and a processor; the memory is used to store a computer program, and the processor is used to implement the method according to any one of claims 1 to 7 when executing the computer program.

10. A readable storage medium, characterized in that: The readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images