AI dialogue system based on emotion recognition

Abstract

The invention provides an AI dialogue system based on emotion recognition, relates to the field of man-machine interaction, and adopts a data processing and coding technology based on artificial intelligence to perform semantic coding on a consultation text input by a user and perform facial feature extraction on a facial image when the user inputs the consultation text. Performing image-text fine-grained cross-modal analysis on the basis of semantic coding features and facial feature representation features of the consultation text to obtain feature representation to intelligently obtain an emotion type identification result; and finally, generating a reply text according to the emotion type identification result and the consultation text. Therefore, the emotion of the user can be better understood, so that the system can generate more personalized and appropriate replies, and the user experience can be improved.

Description

Technical Field

[0001] The present application relates to the field of human-computer interaction, and more specifically, to an AI dialogue system based on emotion recognition. Background Art

[0002] AI dialogue system, also known as chatbot, is a computer program based on artificial intelligence technology to simulate human conversation. Such systems can understand the user's input (usually text or voice) and generate corresponding responses based on the input. It is widely used in various scenarios, such as customer service, personal assistant, educational counseling, medical consultation, etc.

[0003] Traditional AI dialogue systems usually only understand and generate responses based on user input, ignoring the user's emotional state. This may cause the system's response content to not match the user's emotions, thus affecting the user experience.

[0004] Therefore, we look forward to an AI dialogue system based on emotion recognition. Summary of the invention

[0005] In order to solve the above technical problems, this application is proposed. The embodiment of this application provides an AI dialogue system based on emotion recognition.

[0006] According to one aspect of the present application, there is provided an AI dialogue system based on emotion recognition, which includes: a data acquisition module for acquiring a consultation text input by a user and a facial image of the user when inputting the consultation text; a consultation text semantic encoding module for semantically encoding the consultation text to obtain consultation text semantic encoding features; a facial feature extraction module for extracting facial features from the facial image to obtain facial feature representation features; a cross-modal feature data fusion module for performing image-text fine-grained cross-modal analysis on the consultation text semantic encoding features and the facial feature representation features to obtain facial image-consultation text cross-modal fine-grained aggregation features; a reply data generation module for obtaining an emotion type identification result based on the facial image-consultation text cross-modal fine-grained aggregation features, and generating a reply text according to the consultation text and the emotion type identification result.

[0007] Compared with the prior art, the AI ​​dialogue system based on emotion recognition provided by the present application uses artificial intelligence-based data processing and coding technology to semantically encode the consultation text input by the user and extract facial features from the facial image of the user when inputting the consultation text, and then intelligently obtains the emotion type identification result based on the feature representation obtained by image-text fine-grained cross-modal analysis based on the semantic coding features of the consultation text and the facial feature representation features, and finally generates a reply text based on the emotion type identification result and the consultation text. In this way, the user's emotions can be better understood, so that the system can generate more personalized and appropriate replies, which in turn helps to improve the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0009] In the attached picture: Figure 1 This is a system block diagram of an AI dialogue system based on emotion recognition according to an embodiment of the present application.

[0010] Figure 2 Schematic diagram of data flow of an AI dialogue system based on emotion recognition according to an embodiment of the present application.

[0011] Figure 3 It is a block diagram of a cross-modal feature data fusion module in an AI dialogue system based on emotion recognition according to an embodiment of the present application.

[0012] Figure 4 This is a block diagram of a reply data generation module in an AI dialogue system based on emotion recognition according to an embodiment of the present application. DETAILED DESCRIPTION

[0013] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0014] An AI dialogue system, also known as a chatbot, is a computer program that uses artificial intelligence technology to simulate human communication. It can interpret user input (usually text or voice) and generate corresponding responses based on it. The system is widely used in many fields, including customer service, personal assistants, educational counseling, medical consultation, etc.

[0015] Traditional AI dialogue systems mainly understand and generate responses based on user input, but often ignore the user's emotional state. This may cause the answers provided by the system to be inconsistent with the user's emotions, thereby affecting the user experience.

[0016] In response to the above technical problems, the technical concept of this application is to first obtain the consultation text input by the user and the facial image of the user when entering the consultation text, then use artificial intelligence-based data processing and coding technology to semantically encode the consultation text and extract facial features from the facial image, and then perform image-text fine-grained cross-modal analysis based on the semantic coding features of the consultation text and the facial feature representation features to obtain the feature representation to intelligently obtain the emotion type identification result, and finally generate a reply text based on the emotion type identification result and the consultation text. In this way, the user's emotions can be better understood, so that the system can generate more personalized and appropriate replies, which in turn helps to improve the user experience.

[0017] Figure 1 This is a system block diagram of an AI dialogue system based on emotion recognition according to an embodiment of the present application. Figure 2 Schematic diagram of data flow of an AI dialogue system based on emotion recognition according to an embodiment of the present application. Figure 1 and Figure 2 As shown, in the AI ​​dialogue system 100 based on emotion recognition, it includes: a data acquisition module 110, which is used to obtain the consultation text input by the user and the facial image of the user when inputting the consultation text; a consultation text semantic encoding module 120, which is used to semantically encode the consultation text to obtain the consultation text semantic encoding features; a facial feature extraction module 130, which is used to extract facial features from the facial image to obtain facial feature representation features; a cross-modal feature data fusion module 140, which is used to perform image-text fine-grained cross-modal analysis on the consultation text semantic encoding features and the facial feature representation features to obtain facial image-consultation text cross-modal fine-grained aggregation features; a reply data generation module 150, which is used to obtain the emotion type identification result based on the facial image-consultation text cross-modal fine-grained aggregation features, and generate a reply text according to the consultation text and the emotion type identification result.

[0018] In the embodiment of the present application, the data acquisition module 110 is used to obtain the consultation text input by the user and the facial image of the user when inputting the consultation text. It should be understood that the consultation text input by the user contains the questions, requests or expressions that the user wants to convey. For example, the user hopes to get a solution, suggestion or understanding of a topic. In particular, the words used in the consultation text input by the user imply the user's emotional state (such as formal, casual, eager, etc.). The facial image of the user when inputting the consultation text is a visual data source, which can capture the user's expression and some non-verbal signals. Specifically, it contains expression feature information. Facial expressions such as smiling, frowning, and wide eyes can directly reflect the user's emotions. It also contains facial muscle activity information. Although minor muscle changes may not be obvious, they can be analyzed in depth to understand more subtle emotional changes. In general, the consultation text input by the user and the user's facial image provide different perspectives for understanding the user's emotions. By comprehensively analyzing these two pieces of information, a more accurate user emotional state analysis result can be obtained, and then a more personalized and appropriate response can be provided to improve the user experience.

[0019] The following is a detailed description of the specific implementation process of "obtaining the consultation text input by the user and the facial image of the user when entering the consultation text": The first is the acquisition of consultation text data. On the web platform, the development team uses HTML and CSS to build the text input area. The design of the input box focuses on user experience. It is not only visually concise and clear, but also equipped with eye-catching prompt information to guide users to accurately enter the consultation content. At the same time, JavaScript is used to give the input box dynamic interactive functions. In terms of mobile applications, with the help of powerful development frameworks such as React Native or Flutter, an input interface that is highly compatible with the operating system of mobile devices is created.

[0020] When the user completes the consultation text input, the text data needs to be transmitted. In web application scenarios, HTTP or HTTPS protocols are usually used for data transmission. These protocols are based on the reliable TCP / IP protocol and can ensure the integrity and security of data during network transmission. After the user clicks the send button, the browser will encapsulate the input text data into an HTTP request message in a predetermined format and send it to the server side of the AI ​​dialogue system through the network. The server side runs efficient network application frameworks such as the Express framework of Node.js or the Flask framework of Python. These frameworks can quickly receive and parse HTTP requests from the client, extract the consultation text data from it, and properly store it in the system's database to prepare for subsequent processing.

[0021] When it comes to facial image acquisition, camera devices are key. On the desktop, the computer's built-in camera or an external USB camera is used to acquire relevant data. These cameras follow the USB Video Class (UVC) standard, which specifies the communication protocol and data transmission format between the camera and the computer, ensuring that the camera can work stably with the computer system. When the user uses the AI ​​dialogue system, the camera can accurately capture the user's facial image. On the mobile side, the device's built-in camera can be used to collect facial image data. With the continuous advancement of smartphone technology, the imaging quality of the camera is improving, which can meet the AI ​​dialogue system's fine requirements for facial images.

[0022] It should be noted that obtaining facial images requires legal permissions. In the Android system, applications must explicitly declare camera access permissions in the AndroidManifest.xml file during development. When a user installs and launches an application for the first time, the system automatically detects this permission declaration and pops up a detailed permission request dialog box to the user, clearly explaining why and for what the application needs to use the camera. Only after the user explicitly agrees can the application call the camera for image acquisition through the API provided by the system at runtime. On the web side, HTML5's getUserMedia API provides technical support for accessing the camera, but it also needs to follow the browser's security policy. When an application tries to access the camera, the browser will display a clear prompt message to the user, informing the user that the application is requesting camera access permissions and waiting for the user's authorization operation.

[0023] After the facial image is collected, the relevant data is processed and transmitted. First, in order to reduce the amount of data and improve transmission efficiency, the image will be compressed and encoded. The JPEG format is a common choice. It uses an efficient lossy compression algorithm to significantly reduce the file size while ensuring image quality. During the encoding process, the pixel information of the image is converted and compressed according to the JPEG standard to generate a data stream suitable for network transmission. Subsequently, the image data is sent to the server through the network, and the transmission process also relies on a reliable network protocol to ensure data security. After receiving the image data, the server will use a special image processing library to perform decoding operations to restore the compressed image data to the original image format. Then, the image is further converted into a digital matrix form so that the computer vision algorithm can be used for subsequent operations such as facial feature extraction.

[0024] Through the above steps, it can be ensured that the AI ​​dialogue system effectively collects the user's consultation text data and the user's facial image data when entering the consultation text.

[0025] In an embodiment of the present application, the consultation text semantic encoding module 120 is used to semantically encode the consultation text to obtain the semantic encoding features of the consultation text. Specifically, in an embodiment of the present application, the consultation text semantic encoding module is used to: use a semantic encoder based on a Transformer structure to semantically encode the consultation text to obtain a consultation text semantic encoding feature vector as the consultation text semantic encoding feature. It should be understood that when it comes to sentiment analysis, in order to help the system identify the relationship between words in the consultation text, the structure of sentences, and the semantic information in the context, and then more deeply understand the implicit emotional color in the user's consultation text, so that the system can more accurately infer the user's emotional state, it is necessary to use a semantic encoder based on a Transformer structure in the present application to semantically encode the consultation text. Those of ordinary skill in the art should know that the main features of the Transformer structure include self-attention mechanism and position encoding, which make it very suitable for processing consultation texts. Specifically, the self-attention mechanism allows the model to pay attention to all other words in the consulting text while processing each word, which can well understand the relationship between words and help capture complex emotional patterns; position encoding can well preserve the order information of elements in the sequence, which enables the semantic encoder to maintain the integrity of the sentence structure semantics of the original consulting text during processing, thereby facilitating the fine capture of the emotional details implied in the consulting text.

[0026] In an embodiment of the present application, the facial feature extraction module 130 is used to extract facial features from the facial image to obtain facial feature characterization features. Specifically, in an embodiment of the present application, the facial feature extraction module is used to: use a feature extractor based on a dilated convolutional neural network model to extract facial features from the facial image to obtain a facial feature characterization feature map as the facial feature characterization feature. It should be understood that facial expression is a very direct and important way for humans to convey emotions. For example, a smile usually indicates happiness or satisfaction, while a frown may imply confusion or dissatisfaction. In order to obtain feature information that can reflect the emotional state of the user during interaction from the facial image, in the present application, it is necessary to use a feature extractor based on a dilated convolutional neural network model to extract facial features from the facial image to obtain a facial feature characterization feature map containing rich feature information of the user's face. Those of ordinary skill in the art should know that dilated convolution is a special convolution operation that expands the receptive field without increasing the number of parameters. For facial feature extraction from facial images, the atrous convolutional neural network allows the network to view a wider range of input images at the same computational cost, which can capture the overall structure of the face and expression changes in a larger area, while expanding the receptive field without reducing the image resolution, which helps retain more detail information and better recognize subtle expressions, such as tiny muscle movements around the eyes.

[0027] The following is a detailed description of the specific implementation process of "using a feature extractor based on a hole convolutional neural network model to extract facial features from the facial image to obtain a facial feature representation feature map": First, before facial feature extraction, a series of preprocessing operations need to be performed on the input facial image. Since the collected facial images may have various differences, such as inconsistent lighting intensity, different angles, and different resolutions, these factors will interfere with the accuracy of subsequent feature extraction. Therefore, it is necessary to normalize the image and map the pixel values ​​of the image to a unified interval, such as the common [0,1] or [-1,1] interval, through a specific algorithm. This step enables the image data to maintain relative stability and consistency in the subsequent calculation process, avoiding the effect of feature extraction being affected by excessive differences in pixel values.

[0028] At the same time, in order to highlight the key facial areas and reduce the interference of irrelevant information such as background, the image will be cropped and scaled. During the cropping process, the position and range of the face will be determined based on the face detection algorithm, and the image will be cropped to only contain the face area, so as to focus on the facial part that is closely related to emotional expression. The scaling operation is to adjust the cropped facial image to a suitable size to meet the input requirements of the hollow convolutional neural network model. For example, the image is uniformly scaled to a specific resolution, such as 224x224 pixels, to ensure that the image data can be effectively processed in subsequent convolution operations.

[0029] Next, we build a feature extractor architecture based on a dilated convolutional neural network. A dilated convolutional neural network is usually composed of multiple convolutional layers, pooling layers, and activation function layers in an orderly manner. In the initial convolutional layer, a relatively small dilation rate, such as 1 or 2, is generally set. For a simple example, when the dilation rate is 1, the convolution operation is similar to the traditional convolution, mainly extracting local features of the facial image, such as the edge and texture information of the eyes, nose, mouth, etc. These basic features are the cornerstone for the subsequent construction of more complex emotional features. As the number of network layers gradually increases, the dilation rate will gradually increase, for example, to 4, 8, etc. In this process, each convolutional layer is constantly further abstracting and combining the features extracted by the previous layer. When the dilation rate is 4, the sampling interval of the convolution kernel on the image increases, which can cover a wider area, thereby extracting the associated features between different parts of the face, such as the relative position relationship between the eyes and eyebrows, the linkage features of the mouth and cheek muscles, etc. These features are crucial for recognizing subtle changes in facial expressions. The pooling layer plays a key role in downsampling in the network. Common pooling operations include maximum pooling and average pooling. For example, in the maximum pooling operation, the maximum value in a small area (such as a 2x2 or 3x3 pixel area) is selected as the representative value of the area, which can effectively reduce the size of the feature map while retaining the key feature information in the area, greatly improving the calculation efficiency. Through the processing of the pooling layer, the network can reduce the dimension of the data without losing important information, making subsequent calculations more efficient. The activation function layer introduces nonlinear factors to the network, among which the ReLU function is one of the commonly used activation functions. When the input of the ReLU function is greater than 0, the output is equal to the input; when the input is less than or equal to 0, the output is 0. This simple and effective activation function can enhance the network's expressive ability and enable the network to learn complex nonlinear relationships between facial features. For example, it can help the network distinguish different degrees of expression changes, such as the nonlinear change characteristics of facial muscles when smiling and laughing.

[0030] After the preprocessed facial image is input into the constructed dilated convolutional neural network, the core process of feature extraction begins. During the forward propagation of the network, the facial image data passes through each convolution layer, pooling layer, and activation function layer in turn according to the established network structure. In the shallow convolution layer, as mentioned earlier, the basic features of the face are mainly extracted, which are the basis for the subsequent construction of advanced features. As the data is continuously transmitted in the network, the deep convolution layers begin to play a role. They can integrate shallow features and extract high-level semantic features that are closely related to facial expressions. For example, it can accurately identify the degree of squinting of the eyes, the angle of the corners of the mouth, the changes in forehead wrinkles, etc., which are important indicators reflecting the user's emotional state. After multiple convolution and pooling operations of the network, the facial feature representation feature map is finally obtained.

[0031] In the embodiment of the present application, the cross-modal feature data fusion module 140 is used to perform image-text fine-grained cross-modal analysis on the consultation text semantic encoding features and the facial feature representation features to obtain facial image-consultation text cross-modal fine-grained aggregated features. Specifically, Figure 3 FIG. 1 is a block diagram of a cross-modal feature data fusion module in an AI dialogue system based on emotion recognition according to an embodiment of the present application. Figure 3 As shown, the cross-modal feature data fusion module 140 includes: a facial feature representation local feature generation unit 141, which is used to perform feature decoupling and feature flattening on the facial feature representation feature map to obtain a set of facial feature representation local feature coding vectors; a facial kernel feature extraction unit 142, which is used to perform kernel feature extraction on the set of facial feature representation local feature coding vectors to obtain a facial feature representation kernel feature coding vector; a priori clustering center determination unit 143, which is used to determine the facial image-consulting text priori clustering center coding vector based on the consulting text semantic coding feature vector and the facial feature representation kernel feature coding vector; a cross-modal fine-grained aggregation feature generation unit 144, which is used to perform cross-domain fine-grained aggregation based on the cluster center on the facial image-consulting text prior clustering center coding vector and the set of facial feature representation local feature coding vectors to obtain a facial image-consulting text cross-modal fine-grained aggregation coding vector as the facial image-consulting text cross-modal fine-grained aggregation feature.

[0032] It should be understood that the semantic encoding feature vector of the consulting text is obtained by extracting semantic features from the consulting text, and the facial feature representation feature map is obtained by extracting features from the facial image. The consulting text and the facial image belong to different modal data. In order to fully utilize the advantages of these two modalities and thus more comprehensively understand the user's emotional state, it is necessary in this application to perform image-text fine-grained cross-modal analysis on the consulting text semantic encoding features and the facial feature representation features to obtain facial image-consulting text cross-modal fine-grained aggregation features. By integrating the feature data of text and image modalities, fine-grained information fusion and aggregation can be performed at different feature levels, thereby generating a comprehensive representation that can accurately reflect the user's emotional state, which helps capture more subtle and implicit emotional changes, thereby improving the accuracy of emotion recognition.

[0033] Specifically, the facial feature representation feature map is first feature decoupled and feature flattened to obtain a set of facial feature representation local feature encoding vectors. The above process can be expressed as: ;in, represents the facial feature characterization feature map, For Perform feature decoupling and feature flattening operations. They represent the first, second and third facial feature encoding vectors in the set of local feature encoding vectors. facial features represent local feature encoding vectors, Represents a collection of local feature encoding vectors representing facial features.

[0034] It should be understood that by performing feature decoupling on the facial feature representation feature map, different components can be separated from the original feature map, so that each facial feature representation local feature encoding vector only carries a single type of semantic information. For example, one facial feature representation local feature encoding vector only focuses on capturing the facial expression features of the user's smile, while the other focuses on other expression features such as frowning. This helps to improve the model's understanding accuracy of specific emotional expressions. Feature flattening is to convert the original multi-dimensional feature map into a one-dimensional vector form, which makes it easier for the facial feature representation features to be processed by downstream tasks. In general, by performing feature decoupling and feature flattening, the redundant information of the original facial feature representation feature map can be reduced, and the generalization ability of the model can be enhanced, while also ensuring that the facial feature representation local features can be uniformly processed while maintaining their respective features. In particular, here, the facial feature representation feature map is first feature decoupled along the channel dimension to obtain a set of facial feature representation feature matrices, and then, each facial feature representation feature matrix is ​​feature flattened to obtain a set of facial feature representation local feature encoding vectors.

[0035] Specifically, in the embodiment of the present application, the facial kernel feature extraction unit is used to: calculate the feature difference coefficient of each facial feature characterization local feature coding vector in the set of facial feature characterization local feature coding vectors relative to other facial feature characterization local feature coding vectors to obtain a set of facial feature characterization feature difference coefficients; normalize the set of facial feature characterization feature difference coefficients to obtain a set of normalized facial feature characterization feature difference coefficients; use each normalized facial feature characterization feature difference coefficient in the set of normalized facial feature characterization feature difference coefficients as a weight, calculate the position-weighted sum of the set of facial feature characterization local feature coding vectors to obtain the facial feature characterization kernel feature coding vector. The above process can be expressed as: ;in, express The number of vectors in the equation is reduced by one. and Respectively represent the facial feature representation of the local feature encoding vector set and facial features represent local feature encoding vectors, represents the one-norm of a vector, express The corresponding characteristic difference coefficient, represents the exponential function value with the natural constant e as the base, Express Perform kernel feature extraction operations, Represents the facial feature characterization kernel feature encoding vector.

[0036] It should be understandable that although the original facial feature representation local feature encoding vector already carries certain facial information, it contains a large amount of redundant data. Therefore, through kernel feature extraction, we can use the powerful mapping ability of the kernel function to map these high-dimensional, complex and mixed data into a more discriminative low-dimensional space, dig out the core facial features hidden therein, remove irrelevant interference, and make the features more pure and stable.

[0037] Specifically, in the embodiment of the present application, the prior cluster center determination unit is used to: cascade the facial feature characterization kernel feature encoding vector and the consulting text semantic encoding feature vector and multiply them by a weight matrix to obtain a facial kernel feature-consulting text semantic interaction vector; add the facial kernel feature-consulting text semantic interaction vector and the bias vector by position to obtain the facial image-consulting text prior cluster center encoding vector. The above process can be expressed as: ;in, represents the facial feature characterization kernel feature encoding vector, represents the semantic encoding feature vector of the consulting text, Indicates cascade operation, represents the weight matrix, represents the bias vector, Represents the facial image-consulting text prior clustering center encoding vector.

[0038] It should be understandable that by determining the facial image-consultation text prior cluster center encoding vector, we are actually creating an anchor bridge connecting the two modal data (consultation text and facial image). This anchor bridge allows the model to understand and associate feature information from different modalities, so that when analyzing the user's emotional state, not only the user's facial expression information is considered, but also the detailed information hidden in the consultation text.

[0039] Finally, the facial image-consulting text prior cluster center encoding vector and the facial feature characterization local feature encoding vector are subjected to cluster center-based cross-domain fine-grained aggregation to obtain a facial image-consulting text cross-modal fine-grained aggregated encoding vector. The above process can be expressed as: ;in, Respectively represent the facial feature representation of the local feature encoding vector set facial features represent local feature encoding vectors, represents the square of the vector length, represents the inverse hyperbolic cosine function, represents the facial image-advisory text prior clustering center encoding vector, Representation calculation and The Poincare distance between Indicates the preset threshold value, Express The coefficient obtained after clustering judgment is Not a cluster center , , on the contrary , express The number of vectors in express Middle The eigenvalues ​​at the positions, express Middle The eigenvalues ​​at the positions, Represents the face image-consultation text cross-modal fine-grained aggregated encoding vector.

[0040] Here, the facial image-consulting text prior cluster center encoding vector is used as an anchor point for fine-grained interaction, which can ensure that the feature data from the facial image and consulting text are reasonably compared and combined in a common space, which helps to capture the subtle connection between the two modalities and improve the accuracy of the final emotion recognition and understanding. In actual application scenarios, the data often contains noise or outliers, which may affect the performance of the model. By performing cross-domain fine-grained aggregation of feature data from different modalities based on cluster centers, unnecessary interference factors can be filtered out and focus on the part that best reflects the user's emotional characteristics. The generated facial image-consulting text cross-modal fine-grained aggregated encoding vector not only effectively fuses the information of the original two modalities, but also realizes fine-grained level representation, which is conducive to more comprehensive and accurate reflection of the user's emotional state.

[0041] In the embodiment of the present application, the reply data generation module 150 is used to obtain an emotion type identification result based on the cross-modal fine-grained aggregation features of the facial image-consultation text, and generate a reply text according to the consultation text and the emotion type identification result. Specifically, Figure 4 FIG. 1 is a block diagram of a response data generation module in an AI dialogue system based on emotion recognition according to an embodiment of the present application. Figure 4 As shown, the reply data generation module 150 includes: an emotion type identification result generation unit 151, which is used to input the facial image-consultation text cross-modal fine-grained aggregate coding vector into a classifier-based emotion identifier to obtain the emotion type identification result; a reply text generation unit 152, which is used to add the emotion type identification result to the end of the consultation text, and then input it into a large model-based dialogue module to obtain the reply text.

[0042] In the embodiment of the present application, the emotion type identification result generating unit 151 is used to input the facial image-consulting text cross-modal fine-grained aggregated coding vector into the emotion identifier based on the classifier to obtain the emotion type identification result. Specifically, in the embodiment of the present application, the emotion type identification result generating unit is used to: use the fully connected layer of the classifier to fully connect the facial image-consulting text cross-modal fine-grained aggregated coding vector to obtain the facial image-consulting text cross-modal fine-grained aggregated fully connected coding classification feature vector; input the facial image-consulting text cross-modal fine-grained aggregated fully connected coding classification feature vector into the Softmax classification function of the classifier to obtain the emotion type identification result. It should be understood that the facial image-consulting text cross-modal fine-grained aggregated coding vector integrates the complex information of the user's emotions from two different modalities (facial image and consultation text). In order to specifically understand the user's emotion type based on the emotional information contained in this coding vector, it is necessary to input the facial image-consulting text cross-modal fine-grained aggregated coding vector into the emotion identifier based on the classifier. A classifier is a machine learning model whose main function is to classify input data into different categories based on its features. Specifically, the classifier contains a series of algorithms and parameters, which are obtained by training on a large amount of training data with labeled emotion types. By parsing the newly input facial image-consultation text cross-modal fine-grained aggregate encoding vector, the user's emotion type can be classified. The emotion type identification result is usually a predefined set of discrete emotion labels, such as basic emotions such as happiness, sadness, anger, surprise, fear, disgust, etc., or a more refined emotional state, such as anxiety, depression, satisfaction, etc. In addition, it can also include neutral or mixed emotional states. The generated response strategy directly affects the dialogue system. For example, if the user is identified as "happy" emotion, the dialogue system can reply with a more positive and cheerful tone; if it is "angry" emotion, the system can adopt a soothing tone. Such a reply can better match the user's emotion and improve the user experience.

[0043] In particular, since the consultation text semantic encoding feature vector and the facial feature representation feature map respectively represent the semantic encoding features of the consultation text and the image semantic features of the facial image, when performing cross-domain fine-grained aggregation analysis based on modal prior, the semantic feature population attributes corresponding to different modal data will have differences in cross-domain fine-grained aggregation fairness based on the modal prior level, thereby affecting the inclusiveness of the aggregated feature distribution of the facial image-consultation text cross-modal fine-grained aggregation encoding vector, and reducing the accuracy of the emotion type identification result obtained by inputting the classifier-based emotion identifier.

[0044] Preferably, when the facial image-consulting text cross-modal fine-grained aggregated coding vector is input into a classifier-based emotion identifier to obtain an emotion type identification result, the facial image-consulting text cross-modal fine-grained aggregated coding vector is optimized, specifically comprising: optimizing the norm of the facial image-consulting text cross-modal fine-grained aggregated coding vector Divided by the length of the face image-advisory text cross-modal fine-grained aggregate encoding vector To obtain the first facial image-consultation text cross-modal fine-grained aggregate representation balance value .

[0045] The bi-norm of the face image-consultation text cross-modal fine-grained aggregation encoding vector Divide by the square root of the length of the face image-consulting text cross-modal fine-grained aggregation encoding vector to obtain a second face image-consulting text cross-modal fine-grained aggregation representation equilibrium value .

[0046] The facial image-consultation text cross-modal fine-grained aggregation encoding vector The point subtraction vector of the first facial image-consulting text cross-modal fine-grained aggregation representation equalization value is point-multiplied with the second facial image-consulting text cross-modal fine-grained aggregation representation equalization value to obtain the first facial image-consulting text cross-modal fine-grained aggregation encoding fairness indication vector .

[0047] The facial image-consulting text cross-modal fine-grained aggregate coding vector and the second facial image-consulting text cross-modal fine-grained aggregate representation equalization value are point-multiplied with the first facial image-consulting text cross-modal fine-grained aggregate representation equalization value to obtain the second facial image-consulting text cross-modal fine-grained aggregate coding fairness indication vector .

[0048] The second facial image-consulting text cross-modal fine-grained aggregate coding fairness indication vector is taken as the base 2 logarithm of the bit-by-bit, and then multiplied by the first facial image-consulting text cross-modal fine-grained aggregate coding fairness indication vector to obtain the first facial image-consulting text cross-modal fine-grained aggregate coding information correction vector .

[0049] The first facial image-consulting text cross-modal fine-grained aggregate coding fairness indication vector is taken as the base 2 logarithm of the bit-by-bit, and then multiplied by the second facial image-consulting text cross-modal fine-grained aggregate coding fairness indication vector to obtain the second facial image-consulting text cross-modal fine-grained aggregate coding information correction vector .

[0050] Calculate the weighted sum of the first facial image-consulting text cross-modal fine-grained aggregated coding information correction vector and the second facial image-consulting text cross-modal fine-grained aggregated coding information correction vector to obtain an optimized facial image-consulting text cross-modal fine-grained aggregated coding vector .

[0051] Here, it should be understood that in the above formula, represents dot product, Indicates point addition, It represents point subtraction, that is, the eigenvalue at each position of the vector is multiplied by / added to / subtracted from the corresponding value.

[0052] That is, in order to improve the diversity of feature distribution under the condition of multimodal representation tolerance, in response to the representation balance deviation problem of cross-modal data sets existing in the facial image-consulting text cross-modal fine-grained aggregated coding vector, an interactive fairness index is constructed by adopting a mixed low-rank constraint adjustment mechanism of the facial image-consulting text cross-modal fine-grained aggregated coding vector, and constraint correction is implemented on the group feature dynamic coupling optimization process of the facial image-consulting text cross-modal fine-grained aggregated coding vector. At the same time, the joint feature information response mechanism of the facial image-consulting text cross-modal fine-grained aggregated coding vector is introduced as a balance parameter for multi-level fairness optimization to construct an anti-disturbance fairness representation model, thereby realizing the establishment of a collaborative decision-making architecture under the feature distribution framework of the facial image-consulting text cross-modal fine-grained aggregated coding vector, and finally forming a fairness-enhanced multimodal feature fusion system with dynamic adaptability for the facial image-consulting text cross-modal fine-grained aggregated coding vector, thereby improving the accuracy of the emotion type identification result obtained by inputting the classifier-based emotion identifier.

[0053] In the embodiment of the present application, the reply text generation unit 152 is used to add the emotion type identification result to the end of the consultation text, and then input it into the dialogue module based on the large model to obtain the reply text. It should be understood that the large model generally refers to those deep learning models with a large number of parameters (such as hundreds of millions or even billions), which are trained through large-scale data sets and have strong representation learning and generalization capabilities. In detail, the large model is generally pre-trained on a large amount of text data without supervision to learn the general knowledge of natural language, such as the grammar, semantics, common sense, etc. of the language. After that, the model can be fine-tuned according to the specific task to better adapt to the specific task requirements. Here, by attaching the emotion type identification result to the end of the consultation text and passing it to the dialogue module as additional input information, the large model can directly obtain the current emotional state of the user, thereby adjusting the way and tone of generating the reply, making the response more appropriate and humane. That is, with the support of emotional information, the dialogue module can adjust the answer strategy according to the user's emotions. This makes the reply text generated by the dialogue system not only limited to the accuracy of the content, but also better meet the psychological needs of the user, and improve the overall interaction quality and user experience.

[0054] The following is a detailed description of the specific implementation process of "after adding the emotion type identification result at the end of the consultation text, input it into the dialogue module based on the large model to obtain the reply text": First, in the integration of the emotion type identification result and the consultation text, it is necessary to ensure the accuracy and consistency of the data. When the emotion type identification result is generated, the system will add the corresponding identification at the end of the consultation text according to strict programming logic and data processing rules. This process is not a simple text splicing, but it must fully consider the convenience and effectiveness of subsequent processing. For example, when designing an algorithm for adding identification, the emotion type identification result will be standardized and converted into a specific format so that the dialogue module can quickly identify and parse it. At the same time, the consultation text will be preprocessed to check the integrity and standardization of the text to avoid affecting the subsequent analysis and generation process due to problems with the text itself. This may involve operations such as removing noise information in the text, correcting spelling errors, and unifying the encoding format of the text.

[0055] Next, for the dialogue module based on the large model, it is necessary to efficiently load the pre-trained model parameters from large-scale storage devices. These parameters are usually obtained through training with a large amount of text data, and contain rich language knowledge and dialogue patterns. The loading process needs to ensure the integrity and accuracy of the parameters. Once errors or omissions occur, the performance of the model may be greatly reduced. After loading is completed, the dialogue module should be finely adapted according to the specific application scenario. This includes setting appropriate hyperparameters according to business needs, such as parameters that control the diversity and logic of the generated text.

[0056] When the consultation text with the result of sentiment type identification is input into the dialogue module, the model will immediately start a series of complex processing mechanisms. The first is the deep understanding process of the input text, which relies on the powerful natural language processing ability of the model. The model will use advanced word segmentation technology to divide the text into meaningful units. For Chinese text, it will use word segmentation tools such as Jieba to convert continuous Chinese character sequences into word sequences; for English text, it will be segmented according to spaces and punctuation marks. After completing the word segmentation, each word will be converted into a low-dimensional vector representation through word embedding technology. These vectors can capture the semantic and grammatical information of the words, allowing the model to efficiently process and analyze the text in mathematical space.

[0057] During the generation phase of the reply text, the model will perform complex reasoning and calculations based on its internal neural network structure. It will select appropriate words from the pre-trained vocabulary and gradually build the sentence structure of the reply text according to language rules and semantic logic. This process requires comprehensive consideration of the information of the input text, the sentiment type identifier, and the language knowledge and conversation experience learned by the model. In order to improve the quality of the reply text, the model will also conduct multiple rounds of self-evaluation and optimization. For example, it will check whether the generated reply text conforms to grammatical rules, whether it is logically coherent, whether it can accurately respond to the questions in the consultation text, and whether it matches the sentiment type identifier. If problems are found, the model will make adjustments and improvements to try to generate more appropriate replies.

[0058] In actual applications, such as in an intelligent customer service system, when a user asks the system about a product problem and provides a facial image, the system goes through a series of processing steps to obtain the emotion type identification result and generates a reply text according to the above steps. This processing method enables the customer service system to better understand the user's needs and emotional state, thereby providing more personalized and considerate services. For example, if the user shows anxiety during the consultation process, the system will speed up the reply after identifying the emotion type, and provide solutions in concise and clear language, while expressing understanding and concern for the user in the reply, thereby effectively improving user satisfaction and interactive experience.

[0059] In short, the implementation process of replying to text is a highly integrated and intelligent process. By integrating emotional information and consultation text information and utilizing powerful big model technology, it can provide strong support for achieving more efficient, intelligent and humane conversational interactions.

[0060] In summary, the AI ​​dialogue system 100 based on emotion recognition according to the embodiment of the present application is explained, which uses artificial intelligence-based data processing and coding technology to semantically encode the consultation text input by the user and extract facial features from the facial image of the user when entering the consultation text, and then intelligently obtains the emotion type identification result by performing image-text fine-grained cross-modal analysis based on the semantic coding features of the consultation text and the facial feature representation features, and finally generates a reply text based on the emotion type identification result and the consultation text. In this way, the user's emotions can be better understood, so that the system can generate more personalized and appropriate replies, which in turn helps to improve the user experience.

Claims

1. An AI dialogue system based on emotion recognition, characterized in that: include: A data acquisition module for acquiring a consultation text input by a user and a facial image of the user when inputting the consultation text; a consultation text semantic encoding module for semantically encoding the consultation text to obtain consultation text semantic encoding features; a facial feature extraction module for extracting facial features from the facial image to obtain facial feature representation features; a cross-modal feature data fusion module for performing image-text fine-grained cross-modal analysis on the consultation text semantic encoding features and the facial feature representation features to obtain facial image-consultation text cross-modal fine-grained aggregated features; The reply data generation module is used to obtain an emotion type identification result based on the cross-modal fine-grained aggregation features of the facial image-consultation text, and to generate a reply text according to the consultation text and the emotion type identification result.

2. The AI ​​dialogue system based on emotion recognition according to claim 1, characterized in that: The consultation text semantic encoding module is used to: use a semantic encoder based on a Transformer structure to semantically encode the consultation text to obtain a consultation text semantic encoding feature vector as the consultation text semantic encoding feature.

3. The AI ​​dialogue system based on emotion recognition according to claim 2, characterized in that: The facial feature extraction module is used to: use a feature extractor based on a hole convolutional neural network model to extract facial features from the facial image to obtain a facial feature characterization feature map as the facial feature characterization feature.

4. The AI ​​dialogue system based on emotion recognition according to claim 3 is characterized in that: The cross-modal feature data fusion module includes: a facial feature representation local feature generation unit, which is used to perform feature decoupling and feature flattening on the facial feature representation feature map to obtain a set of facial feature representation local feature coding vectors; a facial kernel feature extraction unit, which is used to perform kernel feature extraction on the set of facial feature representation local feature coding vectors to obtain a facial feature representation kernel feature coding vector; a priori clustering center determination unit, which is used to determine a facial image-consulting text priori clustering center coding vector based on the consulting text semantic coding feature vector and the facial feature representation kernel feature coding vector; a cross-modal fine-grained aggregation feature generation unit, which is used to perform cross-domain fine-grained aggregation based on cluster centers on the facial image-consulting text prior clustering center coding vector and the set of facial feature representation local feature coding vectors to obtain a facial image-consulting text cross-modal fine-grained aggregation coding vector as the facial image-consulting text cross-modal fine-grained aggregation feature.

5. The AI ​​dialogue system based on emotion recognition according to claim 4, characterized in that: The facial kernel feature extraction unit is used to: calculate the feature difference coefficient of each facial feature representation local feature coding vector in the set of facial feature representation local feature coding vectors relative to other facial feature representation local feature coding vectors to obtain a set of facial feature representation feature difference coefficients; Normalizing the set of facial feature characterization feature difference coefficients to obtain a set of normalized facial feature characterization feature difference coefficients; Using each normalized facial feature representation feature difference coefficient in the set of normalized facial feature representation feature difference coefficients as a weight, a position-weighted sum of the set of facial feature representation local feature coding vectors is calculated to obtain the facial feature representation kernel feature coding vector.

6. The AI ​​dialogue system based on emotion recognition according to claim 5, characterized in that: The prior clustering center determination unit is used to: cascade the facial feature characterization kernel feature coding vector and the consulting text semantic coding feature vector and multiply them by a weight matrix to obtain a facial kernel feature-consulting text semantic interaction vector; add the facial kernel feature-consulting text semantic interaction vector and the bias vector by position to obtain the facial image-consulting text prior clustering center coding vector.

7. The AI ​​dialogue system based on emotion recognition according to claim 6, characterized in that: The reply data generation module includes: an emotion type identification result generation unit, which is used to input the facial image-consultation text cross-modal fine-grained aggregate coding vector into a classifier-based emotion identifier to obtain the emotion type identification result; a reply text generation unit, which is used to add the emotion type identification result to the end of the consultation text and then input it into a large model-based dialogue module to obtain the reply text.

8. The AI ​​dialogue system based on emotion recognition according to claim 7, characterized in that: The emotion type identification result generating unit is used to: use the fully connected layer of the classifier to perform fully connected encoding on the facial image-consulting text cross-modal fine-grained aggregated encoding vector to obtain a facial image-consulting text cross-modal fine-grained aggregated fully connected encoding classification feature vector; The facial image-consulting text cross-modal fine-grained aggregated fully connected encoded classification feature vector is input into the Softmax classification function of the classifier to obtain the emotion type identification result.

Images