Image emotion recognition method based on convolutional neural networks and computer-readable media

Abstract

This invention provides an image emotion recognition method based on a convolutional neural network and a computer-readable medium. [Solution] The method includes the steps of: acquiring sample feature image data, preprocessing the image, and extracting image features using a convolutional neural network; inputting the extracted features into a fully connected layer for classification and learning the relationship between different feature combinations and emotion categories using an emotion model; optimizing the emotion model using a cross-entropy loss function, adjusting network parameters by backpropagation, and improving the classification accuracy of the training set; inputting a client image into the emotion model, outputting the emotion category of this image, and substituting values ​​into this emotion category; and comprehensively generating music based on the substituted values ​​of the emotion category using an RNN and an LSTM, and pushing this music segment to the client.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and specifically to an image emotion recognition method and a computer-readable medium based on a convolutional neural network.

Background Art

[0002] As social network platforms are booming, the way of network information exchange is no longer limited to the conventional forms such as characters, photos and videos. These diverse content forms bring rich and colorful emotional interactive experiences to users. Specifically, when a user posts an image or a video on a social platform, these contents often convey rich emotional information. However, when other users view these images or videos, they have subjective emotional reactions. Such emotional interactions not only include viewing, but also further involve emotional communication and interaction with the sender in ways such as commenting, liking, and sharing.

[0003] Therefore, in order to better understand and utilize these emotional information, it is necessary to recognize and analyze the emotions attached to images and videos. This not only helps to improve the user experience, but also can promote the development of many related technologies, such as video image classification, question-and-answer systems based on images or videos, and search and recommendation of images or videos. These technologies have broad application needs in the current market, but at the same time face many challenges.

[0004] This specification proposes an innovative solution, that is, the uploaded photo is set to music ​I developed an app mini-program that can perform conversions. This mini-program is for users This not only meets the demand for emotional interaction, but also addresses several issues in the current market. The problem can be solved. By converting photos into music, users can access a completely new way. The law allows us to experience and share their emotions. Such innovative applications are for users. This not only provides a new way of expressing emotions, but also introduces new concepts for emotional analysis of image and video content. And they provided a method. [Overview of the project] [Problems that the invention aims to solve]

[0005] In response to the shortcomings of conventional technology, the present invention provides a convolutional neural network to solve the above problems. A network-based image emotion recognition method and a computer-readable medium are disclosed. [Means for solving the problem]

[0006] This invention is realized by the following technical proposal.

[0007] According to a first aspect, the present invention relates to image emotion recognition based on a convolutional neural network. This provides an image emotion recognition method, The sample features image data is obtained, the images are preprocessed, and a convolutional neural network is created. The steps include extracting image features using and The extracted features are input into a fully connected layer for classification, and different combinations of features are used depending on the emotion model. Steps to learn the relationship between sighs and emotion categories, We optimize the emotion model using the cross-entropy loss function and backpropagation The network parameters are adjusted by the `-n` to improve the classification accuracy of the training set. The steps, The client's image is input into the emotion model, and the emotion category of this image is output. The steps involve assigning values ​​to the emotion categories, RNN and LSTM comprehensively generate music based on the substituted values ​​of emotion categories. This includes the step of completing the process and pushing this music segment to the client.

[0008] Furthermore, in the aforementioned method, the emotion model uses image features extracted by the forward GRU, and The features extracted by GRU and the features extracted by CNN branching are fused together. These three By stitching (concat) the features together, the final feature representation of the model is achieved. It is formed. Then, the features after fusion are input into a Softmax classifier to determine the final model. Obtain a classification output.

[0009] Furthermore, the above method uses cross-entropy loss in the Softmax classifier. The function is, JPEG0007863711000001.jpg1859 And, In the formula, M is the classification result of the image, di is the category probability of the image, and n is the number of categories, and pi is the probability of occurrence at time i.

[0010] Furthermore, in the above method, different methods are used in the convolutional layer of the convolutional neural network. By performing a convolution operation on an image using a convolution kernel of a certain size, the image Edges, corner points, and texture features can be extracted. At the same time, Poo The role of the ring layer is to reduce the dimensionality of features, thereby reducing computational complexity and preserving important feature information. It is about holding on.

[0011] Furthermore, in the above method, the operation procedure of the convolutional neural network is as follows: The backbone network of the convolutional neural network obtains shallow-layer features and deep-layer features regarding an image. Based on the features obtained by the convolutional neural network branch network, processing is performed. Using the features output from the convolutional neural network branch network as the input of the BiGRU network, deeper-level features regarding the image are obtained. The features obtained from the convolutional neural network branch network and the BiGRU network are connected and fused respectively to obtain the final fused features. Using the image features as the input of the Softmax classifier, the classification result of the model is obtained.

[0012] Furthermore, in the above method, based on the values substituted with emotion categories by the RNN and LSTM, music is comprehensively generated, analyzed using the music software CarMusic, and these music segments are pushed to the client.

[0013] Furthermore, in the above method, the emotion categories represented by images cover three types: positive, negative, and neutral. In these categories, for positive emotions, label A is assigned, for negative emotions, label B is assigned, and for neutral emotions, label C is assigned.

[0014] Furthermore, in the above method, variable A is analyzed using the music software CarMusic and matched with fast-tempo songs, variable B is analyzed using the music software CarMusic and matched with slow-tempo songs, and variable C is analyzed using the music software CarMusic and matched with medium-tempo songs. ​

[0015] Furthermore, the above method performs grayscale and normalization on the input sample feature image data. Perform preprocessing steps and utilize data augmentation techniques such as rotation, scaling, and trimming. This is used to improve the diversity of training samples.

[0016] According to a second aspect, the present invention provides a computer-readable medium, which is computer-readable The reading medium comprises a processor, memory, and data stored in the memory and processed by the processor. The memory includes a computer program configured to be executed by the The processor is coupled to a processor, and when the computer program is executed, , realizing the image emotion recognition method based on the convolutional neural network described in the first embodiment. do. [Effects of the Invention]

[0017] The beneficial effects of the present invention are as follows:

[0018] 1. Improved the accuracy and efficiency of image emotion recognition. - Learning capabilities allow for more accurate extraction and optimization of emotional features in images. High-precision emotion classification can be achieved using similar devices.

[0019] 2. Enriched the user's emotional interactive experience. Converted images into music segments. By doing so, users can experience and share their emotions in a completely new way. This allows us to enhance the interactivity and entertainment value on social platforms. ru.

[0020] 3. It has promoted the development of related technologies. This invention provides new concepts and methods for image sentiment analysis. In addition to this, video image classification, image or video-based question and answer systems, and images Alternatively, it can provide new application scenarios and research directions for technologies such as video search and recommendation. did.

[0021] 4. Lowered the technical hurdles. By developing user-friendly app mini-programs... By easily experiencing the convenience provided by image emotion recognition technology, This can promote the dissemination and application of the technology.

[0022] 5. Improved the user experience. Users can upload photos and select appropriate emotional music. By acquiring segments and expressing and sharing one's emotions in a unique way, it becomes more —It becomes personalized, allowing for a rich social experience.

[0023] As described above, the present invention not only represents a technical breakthrough but also has a wide range of practical applications. It has prospects and commercial value. By combining image emotion recognition with music segments, this project Akira provides users with a completely new emotional interactive method, as well as related technologies It is providing a new driving force for the development of the technique. [Brief explanation of the drawing]

[0024] In order to more clearly explain the embodiments of the present invention or the technical methods in the prior art, the following is an example. This section briefly introduces the drawings that need to be used in the description of the examples or prior art, and states that it is obvious. The drawings in the following description are merely some embodiments of the present invention and will be understood by those skilled in the art. Assuming no creative effort is required, other drawings can be obtained based on these drawings. [Figure 1] This is a step-by-step diagram of an image emotion recognition method based on a convolutional neural network. [Figure 2] This is a flowchart illustrating the operation procedure of a convolutional neural network. [Modes for carrying out the invention]

[0025] To further clarify the purpose, technical proposal and advantages of the embodiments of the present invention, the following describes the embodiments of the present invention. The technical concept in the embodiment of the present invention will be described clearly and completely, with accompanying drawings in the example. Clearly, the described embodiments are only some embodiments of the present invention, and not all embodiments. i. Based on the embodiments of the present invention, all obtained without requiring any creative effort by a person skilled in the art All other embodiments described herein fall within the scope of protection of the present invention.

[0026] In one embodiment, referring to Figure 1, the present invention is based on a convolutional neural network. We provide an image emotion recognition method, and this image emotion recognition method is The sample features image data is obtained, the images are preprocessed, and a convolutional neural network is created. The steps include extracting image features using and The extracted features are input into a fully connected layer for classification, and different combinations of features are used depending on the emotion model. Steps to learn the relationship between sighs and emotion categories, We optimize the emotion model using the cross-entropy loss function and backpropagation The network parameters are adjusted by the `-n` to improve the classification accuracy of the training set. The steps, The client's image is input into the emotion model, and the emotion category of this image is output. The steps involve assigning values ​​to the emotion categories, RNN and LSTM comprehensively generate music based on the substituted values ​​of emotion categories. This includes the step of completing the process and pushing this music segment to the client.

[0027] In one embodiment, preprocessing including operations such as grayscale conversion and normalization is performed on the input image. Then, ensure that the image data satisfies the model input requirements. Next, convolutional neural Image features are extracted using a network (CNN). A CNN consists of multiple convolutional layers. It consists of pooling layers and fully connected layers, and extracts local features and higher-level lexical information from the image. It can learn automatically.

[0028] In one embodiment, the convolutional layer uses convolutional kernels of different sizes to process the image. By performing a convolution operation, the edges, corner points, textures, and other basic elements of the image are transformed. Extract fundamental features. The pooling layer reduces feature dimension and computational complexity, It is used to retain important feature information. A combination of multiple convolutions and pooling layers. This allows the network to extract more abstract and complex features in a step-by-step manner. .

[0029] The extracted features are then sent to a fully connected layer for classification. The fully connected layer then processes the different features. By learning the relationship between combinations and emotion categories, we can improve our ability to recognize emotions in images. To achieve this, to improve the generalization ability of the model, one or more layers are typically added after the fully connected layer. Add a Dropout layer to prevent overfitting.

[0030] In one embodiment, during the training process, a cross-entropy loss function is used to model The network parameters are optimized using a backpropagation algorithm. By constantly adjusting the settings, the classification accuracy in the model's training set can be continuously improved. It is improving. To further improve model performance, data augmentation techniques are being used, such as rotation and scaling. By employing techniques such as casing and trimming, the diversity of training samples can be increased. Cut.

[0031] Finally, the trained model is applied to an actual image emotion recognition task. After inputting an image, preprocessing, and feature extraction, the model determines the sentiment category of this image. For example, it outputs positive, negative, neutral, etc. This method allows for a large amount of data. It can effectively perform sentiment analysis on images, and is used for social media monitoring and city It is widely used in fields such as field analysis and psychological health assessment.

[0032] In this embodiment, the emotion model improves its performance by combining multiple feature extraction methods. Specifically, the model uses a forward GRU (Gating Cycle Unit). By using this method, time-series features are extracted from the image, and at the same time, the background GRU is used to extract the background time from the image. It captures sequential features. Furthermore, the model uses a convolutional neural network (CNN). The spatial features of the image are extracted using branching. These three feature extraction methods are each... By capturing different aspects of the image, a more comprehensive representation of its characteristics was achieved.

[0033] To make full use of these features, the model extracts features from the forward GRU. The features extracted by the backgru and the features extracted by the CNN branch are stitched together (c (oncat) merges. Through this merging strategy, the model extracts feature information from different branches. By integrating the information, it is possible to form a single comprehensive characteristic expression. The expression can better capture the complex modes and emotional information within an image.

[0034] The model then inputs the fused features into a Softmax classifier. The classifier is a commonly used multi-classifier that maps the input feature vectors to a probability distribution. By using ping, it becomes possible to classify emotions into different categories. Using this method, the model generates the final classification output based on the fused features. This allows for the accurate recognition and classification of emotional content within images.

[0035] In one embodiment, the cross-entropy loss function used in the Softmax classifier is: JPEG0007863711000002.jpg1959 And, In the formula, M is the classification result of the image, di is the category probability of the image, and n is the number of categories, and pi is the probability of occurrence at time i.

[0036] In one embodiment, in the convolutional layer of a convolutional neural network, different sizes By performing a convolution operation on the image using the convolution kernel, the edges of the image are processed. Corner points and texture features can be extracted. Simultaneously, pooling layers Its role is to reduce the computational complexity by reducing the dimensionality of features, while preserving important feature information. And so it is.

[0037] In one embodiment, the present invention provides an emotional state classification algorithm based on deep learning. This algorithm includes the following steps: First, in the data collection phase, multiple This system collects text, audio, or multimodal data containing user emotional expressions from different sources. Collect this data. This data includes social media, online chat records, and voice call records. The data may be obtained from multiple channels, such as recordings. In the next preprocessing step, these raw data will be processed In contrast, a series of preprocessing operations such as cleaning, splitting, and feature extraction are performed to improve the quality of the data. Ensure that the format is applicable to subsequent model training.

[0038] In this embodiment, in the emotional feature extraction stage, natural language processing technology or voice-based feature extraction technology is used. This is used to extract emotion-related features from preprocessed data. These features include text Emotional vocabulary, tone, accent, and pitch, volume, and speaking speed during speech. These may include:

[0039] Then, in the stage of building the emotional state classification model, a deep learning neural network A model is constructed, and this model places the extracted emotional features into a predefined emotional state space. This can be mapped to: joy, sadness, anger, surprise. It can include multiple emotion categories such as "ki".

[0040] During the model training phase, a large, expressive sentiment dataset is used to train the classification model. The model is trained, and the model parameters are adjusted using a backpropagation algorithm. Optimize to improve the model's accuracy and generalization ability. Finally, in the emotional state recognition stage, The data entered by the user is fed into a trained model to analyze the user's emotional state. To enable accurate recognition and classification of [the subject].

[0041] This invention enables positive interaction between art and emotion through such algorithms. Instead of promoting negativity and using technology to amplify negative emotions, we should promote positive and healthy It encourages users to express their emotions in a natural way. Such algorithms are user-centric. Not only does it help improve experiences, but it also provides strong technical support in the field of mental health, and helps people to... It can also help you better understand and manage your emotional state.

[0042] Referring to Figure 2, in this embodiment, the operation procedure of the convolutional neural network is as follows: The backbone network of the convolutional neural network provides shallow analysis of images. Layer features and deep features are obtained, The convolutional neural network processes based on the features obtained from the branched network. Features output from convolutional neural networks and branched networks are processed by BiGRU. As input to the network, we obtain deeper level features related to the image, From convolutional neural networks, branched networks, and BiGRU networks The obtained features are connected and merged to obtain the final merged features, The goal is to use image features as input to a Softmax classifier and obtain the model's classification result.

[0043] In this example, the inventors demonstrate the relationship between a cycle neural network (RNN) and long-term short-term memory. It employs network (LSTM) technology and associates it with the substituted values ​​of emotion categories. It comprehensively generates music. This method allows the music to be generated according to the user's emotional state. It can generate corresponding music segments. To achieve this goal, CarMu I am using music software called sic, and this software generates By analyzing and processing the music segments and pushing them to the client, the user It can provide a personalized music experience.

[0044] In this example, the emotion categories represented by the images are positive, negative, and It includes three types, including neutral. To facilitate distinction and processing, it categorizes emotions by type. Specific labels were assigned. Specifically, positive emotions were assigned label A, and negative emotions Positive emotions are assigned the marker B, while neutral emotions are assigned the marker C. Next These markers are analyzed using the music software CarMusic to determine which songs have different tempos. Matching.

[0045] Specifically, variable A (representing positive emotions) is analyzed using CarMusic, and the tempo is determined. By matching it to fast-paced songs, it evokes positive emotions in users. Next, we analyze variable B (representing negative emotions) using CarMusic, and then select slow-tempo songs. It matches and helps the user alleviate negative emotions. Finally, the variable C( CarMusic analyzes (representing neutral emotions) and matches it to songs with a medium tempo. This approach aims to provide users with a peaceful and comfortable music experience. By generating music segments that correspond to the user's state, the user's music experience is enhanced. It can improve.

[0046] In one embodiment, the present invention provides a computer-readable medium, and this computer-readable medium It comprises a processor, memory, and a function stored in the memory and executed by the processor. The computer program is configured to include the memory and the processor The processor is coupled to the computer program and, when it executes the computer program, This method implements an image emotion recognition method based on a complex neural network.

[0047] In summary, the present invention improves the accuracy and efficiency of image emotion recognition. The deep learning capabilities of the multi-network allow for more accurate recognition of emotional features in images. By extracting and optimizing the data, highly accurate sentiment classification can be achieved. The emotional interactive experience was enriched by converting images into music segments. By experiencing and sharing their emotions in a completely new way, users can connect with the community. This can enhance the interactivity and enjoyment of the platform.

[0048] This invention has spurred the development of related technologies. This invention provides new concepts and methods for image sentiment analysis. In addition to providing video image classification, image or video-based question and answer systems, New application scenarios and research directions for technologies such as image or video search and recommendation. Provided.

[0049] This invention lowered the technical hurdles, enabling the development of user-friendly app mini-programs. By doing so, general users can easily experience the convenience of image emotion recognition technology. This will promote the widespread adoption and application of this technology.

[0050] This invention improves the user experience. Users can upload photos and express the corresponding emotions. By acquiring music segments and expressing and sharing your emotions in a unique way, It can be personalized and provide a rich social experience.

[0051] This invention not only represents a technological breakthrough, but also has broad application prospects and commercial viability for practical applications. It is valuable. By linking image emotion recognition with music segments, the present invention allows users to It offers a completely new emotional interactive method, and contributes to the development of related technologies. It provides the driving force.

[0052] The above examples are merely for illustrating the technical concept of the present invention and are not intended to limit the invention to them. Although the present invention has been described in detail with reference to the above examples, those skilled in the art will understand that It is still not possible to modify the technical proposals described in each of the above embodiments, or to replace some of the technical features with equivalent ones. These can be replaced or altered, but these modifications or replacements may alter the essence of the corresponding technical proposal. It is understood that this does not deviate from the spirit and scope of the technical concept of each embodiment of the present invention. It should be possible.

Claims

1. A method for recognizing image emotions based on a convolutional neural network, The sample features image data is obtained, the images are preprocessed, and a convolutional neural network is created. The steps include extracting image features using and The extracted features are input into a fully connected layer for classification, and different combinations of features are used depending on the emotion model. Steps to learn the relationship between sighs and emotion categories, We optimize the emotion model using the cross-entropy loss function and backpropagation The network parameters are adjusted by the `-n` to improve the classification accuracy of the training set. The steps, The client's image is input into the emotion model, and the emotion category of this image is output. The steps involve assigning values ​​to the emotion categories, RNN and LSTM comprehensively generate music based on the assigned values ​​for emotion categories. This includes the step of completing and pushing this music segment to the client, The emotion model uses image features extracted by the forward GRU and features extracted by the back GRU. The features include those extracted by CNN branching, and these three features are concatenated. This process results in the final fused features of the model, and the fused features are then classified using Softmax. A convolutional neural network characterized by inputting data into a device and obtaining the final classification result of the model. A network-based image emotion recognition method.

2. The cross-entropy loss function used in the Softmax classifier is: And, In the formula, M is the image classification result, di is the category probability of the image, and n The characteristics are that is the number of categories, and pi is the probability of occurrence at time i. Image emotion recognition method based on the convolutional neural network described in 1.

Images