Emotion analysis method based on convolutional neural network and YOLO algorithm

Abstract

The invention discloses an emotion analysis method based on a convolutional neural network and a YOLO algorithm, and the method comprises the steps: collecting data, and carrying out the preprocessing of the data; constructing a convolutional neural network model; training and adjusting the model; training and optimizing the model; identifying and reasoning the emotion; and analyzing and fusing the emotion. Through the convolutional neural network, different data modes and problems can be automatically learned and adapted, features can be automatically extracted from input data, complex feature engineering does not need to be manually carried out, the YOLO algorithm is relatively simple in structure and easy to understand and implement, meanwhile, the training process is simple, and emotion analysis can be comprehensively and accurately carried out.

Description

Technical Field

[0001] The present invention belongs to the technical field of sentiment analysis, and specifically is a sentiment analysis method based on convolutional neural network and YOLO algorithm. Background Art

[0002] Sentiment analysis, also known as sentiment analysis or opinion mining, is an important field in natural language processing. Its background technology mainly involves the following aspects:

[0003] Natural language processing is the basis of sentiment analysis. With the continuous advancement of computer technology, natural language processing technology has made remarkable progress. From the early rule-based methods, to the later statistical methods and machine learning methods, to today's deep learning methods, the accuracy and efficiency of natural language processing have been continuously improved. For example, technologies such as lexical analysis, syntactic analysis, and semantic analysis can help computers better understand the meaning of text. Part-of-speech tagging can determine the part of speech of each word, such as noun, verb, adjective, etc. Named entity recognition can identify entities such as names of people, places, and organizational names in the text. These technologies provide important support for sentiment analysis.

[0004] Machine learning and deep learning algorithms play a key role in sentiment analysis. Through training with a large amount of annotated data, machine learning models can learn the sentiment features in text and classify new texts by sentiment. Common machine learning algorithms include support vector machines (SVM), naive Bayes, decision trees, etc. These algorithms have certain effects in sentiment analysis, but their processing capabilities for complex texts and large-scale data are limited. Deep learning algorithms, such as convolutional neural networks (CNN), recurrent neural networks (RNN), and long short-term memory networks (LSTM), have stronger feature extraction and generalization capabilities. They can automatically learn the semantic features and sentiment information in texts and improve the accuracy of sentiment analysis.

[0005] The popularity of social media and the rise of big data provide a rich data source for sentiment analysis. The texts, comments, likes and other behavioral data posted by users on social media contain a lot of emotional information. By analyzing these big data, we can understand the public's emotional tendencies, opinions and attitudes. For example, companies can understand the advantages and disadvantages of products and user needs by analyzing consumers' comments on social media, thereby improving products and services. The government can understand public opinion and formulate more scientific and reasonable policies by analyzing the public's response to policies.

[0006] In addition to text data, multimodal data such as images, audio, and video can also provide rich emotional information. Multimodal sentiment analysis combines data from different modalities to comprehensively analyze the user's emotional state. For example, by analyzing facial expressions, voice intonation, language content, and other information in the video, the user's emotions can be judged more accurately. Multimodal sentiment analysis has broad application prospects in human-computer interaction, mental health monitoring, intelligent customer service, and other fields.

[0007] In the current field of sentiment analysis and recognition technology, direct analysis of the target object is the core solution strategy. However, a large number of related studies seem to focus on the behavior aggregation scale of the target object, or direct text feature extraction and other fixed emotion features.

[0008] Obviously, the influence of external factors on the object (the characteristic information contained in the offset information of the target) is generally ignored, but in fact, this part of the data is of decisive significance in certain specific situations. CN113326723B A method, device, equipment and system for emotion recognition, when multiple objects are detected to be gathered, identifies the emotion of the object group based on the target emotion information of the multiple objects that meet the preset emotion conditions and the non-emotion information of the object group composed of multiple objects. Only focusing on the target object, the extracted features are not comprehensive, and the influence of external factors on emotion analysis is ignored. Furthermore, considering the high complexity of emotion analysis.

[0009] Many current studies have failed to fully explore the impact of micro-expressions on the interference of individual identity information attributes, which makes accurate identification difficult. CN116306677B discloses a sentiment analysis method, system and device based on neural topic model enhancement. The bag-of-words model representation of the text is obtained after preprocessing the text, and the word vector representation of the words in the text. However, it does not take into account that the text information will interfere with the emotional characteristics during collection and processing. In addition, the instantaneous and disturbed characteristics of sentiment analysis make the feature data not only sparse, but also the essential characteristics are not prominent. In this case, only using the target object analysis model will lead to the problem of single analysis and low accuracy. Summary of the invention

[0010] In view of the above problems, the present invention is proposed to provide a sentiment analysis method based on a convolutional neural network and a YOLO algorithm to overcome the above problems or at least partially solve the above problems.

[0011] To achieve the above object, the present invention adopts the following technical solutions:

[0012] A sentiment analysis method based on a convolutional neural network and a YOLO algorithm, the method comprising:

[0013] Collect data and pre-process the data;

[0014] Build a convolutional neural network model;

[0015] Train and adjust the model;

[0016] Train and optimize the model;

[0017] Identify and reason about emotions;

[0018] Analyze and integrate emotions.

[0019] Optionally, collect data and pre-process the data including:

[0020] Use FER2013 as the basis and CK+ as the auxiliary to download the dataset;

[0021] The openCV tool is used for feature description, and the feature points in the image are used to generate descriptors for feature matching.

[0022] Optionally, building a convolutional neural network model includes:

[0023] For image data, convolutional neural networks are used to automatically extract features of facial expressions;

[0024] Determine the number of layers, number of neurons, activation function, and loss function of the neural network;

[0025] Add a deep residual network and introduce a residual module.

[0026] Optionally, training and tuning the model includes:

[0027] Input the preprocessed multimodal data into the convolutional neural network model for training;

[0028] The YOLO model is trained using the collected preprocessed image dataset.

[0029] Optionally, the preprocessed multimodal data is input into a convolutional neural network model for training including:

[0030] The weights and biases of the neural network are continuously adjusted through the back-propagation algorithm to minimize the loss function, so that the model can learn the complex mapping relationship between different modal data and emotions;

[0031] Use stochastic gradient descent and its variant optimizers to update model parameters, use early stopping to prevent overfitting, and use regularization techniques to constrain the complexity of the model;

[0032] Divide the model into training sets, validation sets, and test sets to evaluate the performance of the model, and adjust hyperparameters and improve the model based on the results of the validation set.

[0033] Optionally, training the YOLO model using the collected preprocessed image dataset includes:

[0034] The face region in the image and the corresponding emotion category label are used as input and output, allowing the YOLO model to learn the association between facial features and emotions;

[0035] Adjust and optimize the parameters of the YOLO model to improve the accuracy of emotion recognition and the generalization ability of the model;

[0036] Evaluate the performance of the model at different stages by dividing it into training sets, validation sets, and test sets, and adjust the parameters based on the results of the validation set;

[0037] Data enhancement technology is used to improve the adaptability of the YOLO model to changes in lighting, angles, and expression intensity.

[0038] Optionally, training and optimizing the model includes:

[0039] Adjust the hyperparameters of the convolutional neural network to improve the performance of the model, and use grid search and random search to adjust the hyperparameters;

[0040] The YOLO algorithm is specifically manifested in multi-scale training, which uses images of different sizes for training to allow the model to adapt to targets of different scales;

[0041] Use the model weights pre-trained on large-scale datasets to fine-tune on your own dataset;

[0042] Use distributed training to speed up the model training process, distribute data and computing tasks to multiple computing nodes, and perform training in parallel to shorten the training time.

[0043] Optionally, emotion recognition reasoning includes:

[0044] After the model training is completed, for new input images, face detection technology is used to determine the location of the face in the image;

[0045] Input the detected face area into the trained YOLO model, and the YOLO model will output the predicted emotion category and the corresponding confidence level;

[0046] The final emotion analysis result is determined according to the confidence threshold. If the confidence is higher than the set threshold, the emotion prediction result is considered reliable and the corresponding emotion category is output; if the confidence is lower than the threshold, the result is marked as uncertain and further analysis and processing is performed.

[0047] Optionally, analyzing and integrating emotions includes:

[0048] For data of different modalities, after being processed by their respective convolutional neural networks, the corresponding emotional feature representation is obtained;

[0049] A fusion strategy is used to fuse multimodal emotion features;

[0050] The output is the emotion classification result after comprehensively considering multimodal information, or a continuous emotion intensity value.

[0051] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0052] 1. The present invention can automatically learn and adapt to different data patterns and problems through convolutional neural networks. It can automatically extract features from input data without the need for manual complex feature engineering. In image recognition tasks, neural networks can automatically learn various features in images, such as edges, textures, shapes, etc., without the need to manually design feature extractors; and it has high fault tolerance. Neural networks have a certain tolerance for noise and errors in input data. Even if there is a certain degree of noise or error in the input data, the neural network can still give a relatively accurate output result.

[0053] 2. The present invention has strong parallel processing capabilities, and the calculations of the neural network can be performed in parallel, which makes it very efficient when processing large-scale data and complex tasks; it has strong generalization capabilities, and a well-trained neural network has strong generalization capabilities and can accurately predict unseen data. The neural network can learn general rules and patterns from existing data and apply them to new data.

[0054] 3. The present invention can simultaneously predict bounding boxes and determine categories in a single network through the YOLO algorithm. It only needs to perform one forward propagation of the image to obtain all object positions and category information, which makes it possible to achieve real-time object detection. Unlike some methods that can only detect a single object in an image, the YOLO algorithm can simultaneously detect multiple objects of different categories in an image, and has good detection capabilities for objects of different sizes, shapes and positions.

[0055] 4. The present invention processes images through the YOLO algorithm, which takes into account the contextual information of the entire image rather than just the local information. When dealing with some complex situations that require understanding of contextual information, the occlusion problem can be better handled, and the category and position of the occluded object can be more accurately determined by combining the surrounding environmental information. Moreover, the structure of the YOLO algorithm is relatively simple, easy to understand and implement, and the training process is simple. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1A flow chart of a sentiment analysis method based on a convolutional neural network and the YOLO algorithm provided in an embodiment of the present application. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0058] See also Figure 1 This embodiment provides a sentiment analysis method based on a convolutional neural network and a YOLO algorithm, the method comprising the following steps:

[0059] S1. Collect data and preprocess the data.

[0060] Collecting data and preprocessing the data include:

[0061] Download the dataset using FER2013 as the basis and CK+ as the auxiliary. FER2013 was launched at the 2013 International Conference on Machine Learning (ICML). It was one of the more important facial expression recognition datasets at the time and became one of the benchmarks for performance comparison of subsequent expression recognition models. It was also used as the data for the 2013 Kaggle face recognition competition. FER2013 contains about 30,000 facial images. Among them, the training set has 28,709 images, and the public test set and private test set each have 3589 images. The images are 48×48 pixel grayscale images. This dataset has a wide range of applications in the field of facial expression recognition research. It can be used to train and evaluate various facial expression recognition algorithms and models, and is of great significance to promoting the development of expression recognition technology. Researchers can develop, optimize, and compare algorithms based on this dataset. The CK+ dataset is an extension of the Cohn-Kanade (CK) dataset. It was compiled and released by researchers from the University of South Florida and other institutions. The CK+ dataset contains 593 image sequences from 123 different individuals. Each image sequence usually starts with a neutral expression and gradually transitions to a specific expression peak. The images are in color and have a high resolution, which can clearly show facial details. Each image sequence is annotated in detail, including information such as expression category, intensity, start and end frames, etc. In addition, facial key points such as the location of eyes, mouth, eyebrows, etc. are also annotated, providing a basis for further facial analysis.

[0062] The openCV tool is used for feature description, and descriptors are generated for feature points in the image for feature matching. At the same time, the openCV tool is used (OpenCV is a cross-platform computer vision library released under the BSD license (open source). OpenCV provides algorithms such as Harris corner detection and Shi-Tomasi corner detection. Feature description: Descriptors are generated for feature points in the image for feature matching. For example, algorithms such as SIFT (Scale Invariant Feature Transform), SURF (Speeded Robust Features) and ORB (Oriented FAST and Rotated BRIEF) can generate feature descriptors with rotation and scale invariance.

[0063] S2. Build a convolutional neural network model.

[0064] Building a convolutional neural network model includes:

[0065] Select the appropriate neural network architecture based on the characteristics of the data and the analysis requirements. For image data, use a convolutional neural network to automatically extract the features of facial expressions.

[0066] Determine the number of neural network layers, number of neurons, activation function, and loss function; the choice of loss function should match the task objective of sentiment analysis. For example, the cross entropy loss function can be used for classification tasks.

[0067] Added deep residual network and introduced residual module. Added ResNet (deep residual network, Residual Neural Network). Before ResNet, increasing the number of layers of the neural network would lead to increased training error and network degradation. ResNet solved this problem well by introducing residual modules. In theory, as the network deepens, even if the newly added layers do not learn useful features, the residual connection can allow the network to at least maintain the same performance as the shallow network, so that the network performance can be improved by simply increasing the depth of the network. The core component of ResNet is the residual module. The residual module contains a "shortcut connection" that adds the input directly to the output after the convolution operation. This structure makes it easier for the network to learn the residual between the input and output, that is, the "difference" part, instead of directly learning the complete mapping relationship.

[0068] S3. Train and adjust the model.

[0069] Training and tuning the model includes:

[0070] The preprocessed multimodal data is input into the convolutional neural network model for training.

[0071] The YOLO model is trained using the collected preprocessed image dataset.

[0072] Inputting the preprocessed multimodal data into the convolutional neural network model for training includes:

[0073] The weights and biases of the neural network are continuously adjusted through the back-propagation algorithm to minimize the loss function, enabling the model to learn the complex mapping relationship between different modal data and emotions.

[0074] A variety of training techniques are used to improve the performance and stability of the model, such as using stochastic gradient descent and its variants (such as Adagrad, Adadelta, RMSProp, Adam, etc.) optimizers to update model parameters, using early stopping to prevent overfitting, and using regularization techniques (such as L1 regularization, L2 regularization) to constrain the complexity of the model.

[0075] Divide the model into training sets, validation sets, and test sets to evaluate the performance of the model, and adjust hyperparameters and improve the model based on the results of the validation set.

[0076] Training the YOLO model using the collected preprocessed image dataset includes:

[0077] During the training process, the face area in the image and the corresponding emotion category label are used as input and output, allowing the YOLO model to learn the association between facial features and emotions.

[0078] Adjust and optimize the parameters of the YOLO model, such as learning rate, batch size, network structure, etc., to improve the accuracy of emotion recognition and the generalization ability of the model.

[0079] The performance of the model at different stages is evaluated by dividing the model into training sets, validation sets, and test sets, and the parameters are adjusted according to the results of the validation set.

[0080] Data enhancement technology is used to improve the adaptability of the YOLO model to changes in lighting, angles, and expression intensity, such as random rotation, flipping, cropping, adding noise, etc., to increase the diversity of training data.

[0081] S4. Train and optimize the model.

[0082] Training and optimizing the model includes:

[0083] Adjust the hyperparameters of the convolutional neural network, such as learning rate, batch size, number of network layers, number of neurons, etc., to improve the performance of the model, and use grid search and random search to adjust the hyperparameters; Data enhancement: If the amount of data is small, data enhancement technology can be used to expand the data set.

[0084] The YOLO algorithm is specifically manifested in multi-scale training. During the training process, images of different sizes are used for training to allow the model to adapt to targets of different scales. For example, the size of the image can be randomly selected and adjusted within a certain range, and then input into the model for training.

[0085] Use the model weights pre-trained on large-scale datasets to fine-tune on your own dataset; this can speed up the convergence of the model and improve performance. The pre-trained model can be a general image classification model trained on datasets such as ImageNet.

[0086] If there are sufficient computing resources, use distributed training to speed up the model training process, distribute data and computing tasks to multiple computing nodes, and perform training in parallel to shorten the training time.

[0087] Similarly, the training set, validation set, and test set are divided to evaluate the performance of the model, and hyperparameter adjustment and model improvement are performed based on the results of the validation set.

[0088] S5. Identify and reason about emotions.

[0089] Recognition and reasoning about emotions include:

[0090] After the model training is completed, for new input images, face detection technology is used to determine the location of the face in the image.

[0091] The detected face area is input into the trained YOLO model, and the YOLO model will output the predicted emotion category and the corresponding confidence level. For example, the model may predict that the emotion of a face is happy, with a confidence level of 0.8, indicating that there is an 80% certainty that the face is showing a happy emotion.

[0092] The final emotion analysis result is determined according to the confidence threshold. If the confidence is higher than the set threshold (such as 0.7), the emotion prediction result is considered reliable and the corresponding emotion category is output; if the confidence is lower than the threshold, the result is marked as uncertain and further analysis and processing is performed.

[0093] S6. Analyze and integrate emotions.

[0094] Analysis and integration of emotions include:

[0095] For data of different modalities, after being processed by their respective convolutional neural networks, the corresponding emotional feature representations are obtained; for example, the feature vector extracted from the facial expression image represents the emotional tendency of the facial expression, and the feature vector extracted from the speech signal represents the emotional information in the speech, etc.

[0096] A fusion strategy is used to fuse multimodal emotional features; the fusion strategy can be a simple concatenation, weighted summation, or a more complex fusion method based on the attention mechanism, allowing the model to comprehensively consider information from different modalities, thereby performing emotional analysis more accurately.

[0097] The final output is the emotion classification result after comprehensively considering multimodal information, or a continuous emotion intensity value.

[0098] This embodiment can automatically learn and adapt to different data patterns and problems through convolutional neural networks. It can automatically extract features from input data without the need for complex feature engineering. In image recognition tasks, neural networks can automatically learn various features in images, such as edges, textures, shapes, etc., without the need to manually design feature extractors; and they have high fault tolerance. Neural networks have a certain tolerance for noise and errors in input data. Even if there is a certain degree of noise or error in the input data, the neural network can still give a relatively accurate output result.

[0099] This embodiment has strong parallel processing capabilities, and the calculations of the neural network can be performed in parallel, which makes it very efficient when processing large-scale data and complex tasks; it has strong generalization capabilities, and a well-trained neural network has strong generalization capabilities and can accurately predict unseen data. The neural network can learn general rules and patterns from existing data and apply them to new data.

[0100] This embodiment can simultaneously predict bounding boxes and determine categories in a single network through the YOLO algorithm. Only one forward propagation of the image is required to obtain all object positions and category information, which enables real-time object detection. Unlike some methods that can only detect a single object in an image, the YOLO algorithm can simultaneously detect multiple objects of different categories in an image, and has good detection capabilities for objects of different sizes, shapes, and positions.

[0101] This embodiment processes images through the YOLO algorithm, which takes into account the context information of the entire image rather than just the local information. When dealing with some complex situations that require understanding of context information, it can better handle occlusion problems and more accurately determine the category and position of the occluded object by combining the surrounding environmental information. Moreover, the structure of the YOLO algorithm is relatively simple, easy to understand and implement, and the training process is simple.

[0102] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. A sentiment analysis method based on convolutional neural network and YOLO algorithm, characterized in that: The method comprises: Collect data and pre-process the data; Build a convolutional neural network model; Train and adjust the model; Train and optimize the model; Identify and reason about emotions; Analyze and integrate emotions.

2. The sentiment analysis method based on convolutional neural network and YOLO algorithm as claimed in claim 1, characterized in that: Collecting data and preprocessing the data include: Use FER2013 as the basis and CK+ as the auxiliary to download the dataset; The openCV tool is used for feature description, and the feature points in the image are used to generate descriptors for feature matching.

3. The sentiment analysis method based on convolutional neural network and YOLO algorithm as claimed in claim 1, characterized in that: Building a convolutional neural network model includes: For image data, convolutional neural networks are used to automatically extract features of facial expressions; Determine the number of layers, number of neurons, activation function, and loss function of the neural network; Add a deep residual network and introduce a residual module.

4. The sentiment analysis method based on convolutional neural network and YOLO algorithm as claimed in claim 1, characterized in that: Training and tuning the model includes: Input the preprocessed multimodal data into the convolutional neural network model for training; The YOLO model is trained using the collected preprocessed image dataset.

5. The sentiment analysis method based on convolutional neural network and YOLO algorithm as claimed in claim 4, characterized in that: Inputting the preprocessed multimodal data into the convolutional neural network model for training includes: The weights and biases of the neural network are continuously adjusted through the back-propagation algorithm to minimize the loss function, so that the model can learn the complex mapping relationship between different modal data and emotions; Use stochastic gradient descent and its variant optimizers to update model parameters, use early stopping to prevent overfitting, and use regularization techniques to constrain the complexity of the model; Divide the model into training sets, validation sets, and test sets to evaluate the performance of the model, and adjust hyperparameters and improve the model based on the results of the validation set.

6. The sentiment analysis method based on convolutional neural network and YOLO algorithm as claimed in claim 4, characterized in that: Training the YOLO model using the collected preprocessed image dataset includes: The face region in the image and the corresponding emotion category label are used as input and output, allowing the YOLO model to learn the association between facial features and emotions; Adjust and optimize the parameters of the YOLO model to improve the accuracy of emotion recognition and the generalization ability of the model; Evaluate the performance of the model at different stages by dividing it into training sets, validation sets, and test sets, and adjust the parameters based on the results of the validation set; Data enhancement technology is used to improve the adaptability of the YOLO model to changes in lighting, angles, and expression intensity.

7. The sentiment analysis method based on convolutional neural network and YOLO algorithm as claimed in claim 1, characterized in that: Training and optimizing the model includes: Adjust the hyperparameters of the convolutional neural network to improve the performance of the model, and use grid search and random search to adjust the hyperparameters; The YOLO algorithm is specifically manifested in multi-scale training, which uses images of different sizes for training to allow the model to adapt to targets of different scales; Use the model weights pre-trained on large-scale datasets to fine-tune on your own dataset; Use distributed training to speed up the model training process, distribute data and computing tasks to multiple computing nodes, and perform training in parallel to shorten the training time.

8. The sentiment analysis method based on convolutional neural network and YOLO algorithm as claimed in claim 1, characterized in that: Recognition and reasoning about emotions include: After the model training is completed, for new input images, face detection technology is used to determine the location of the face in the image; Input the detected face area into the trained YOLO model, and the YOLO model will output the predicted emotion category and the corresponding confidence level; The final emotion analysis result is determined according to the confidence threshold. If the confidence is higher than the set threshold, the emotion prediction result is considered reliable and the corresponding emotion category is output; if the confidence is lower than the threshold, the result is marked as uncertain and further analysis and processing is performed.

9. The sentiment analysis method based on convolutional neural network and YOLO algorithm as claimed in claim 1, characterized in that: Analysis and integration of emotions include: For data of different modalities, after being processed by their respective convolutional neural networks, the corresponding emotional feature representation is obtained; A fusion strategy is used to fuse multimodal emotion features; The output is the emotion classification result after comprehensively considering multimodal information, or a continuous emotion intensity value.

Images