Psychological counseling robot with multi-modal emotion recognition capability and counseling method of psychological counseling robot

Abstract

The invention relates to the technical field of robots, in particular to a psychological counseling robot with a multi-mode emotion recognition capability and a counseling method thereof. Which comprises a robot main body and is characterized in that a multi-mode sensing module is installed on the head of the robot main body, a data processing and analyzing module is installed in the robot main body, and a real-time feedback generating module is installed in the robot main body and located on one side in the robot main body. According to the psychological counseling robot with the multi-mode emotion recognition capability and the counseling method of the psychological counseling robot, through cooperative work of multiple modules, the intelligent psychological counseling robot can recognize user emotion in real time at multiple angles, personalized psychological counseling feedback is generated, and the interactive experience and the emotion adjustment effect of the user are improved; meanwhile, through reasonable position layout and electrical connection among the modules, the system is stable in operation and has strong expansibility, and more intelligent psychological health service can be realized.

Description

Technical Field

[0001] The present invention relates to the field of robotics technology, and in particular to a psychological counseling robot with multimodal emotion recognition capabilities and a counseling method thereof. Background Art

[0002] Intelligent psychological counseling robots are the product of the combination of artificial intelligence and mental health in recent years, and are designed to provide users with psychological support and consulting services. Because intelligent psychological counseling robots can solve many inconveniences in traditional treatment of psychological problems, such as: the number of psychologists is far from enough, the professional level of psychologists varies, the cost of psychological treatment is high, and personal privacy issues are difficult to talk about. At present, in the technical field of intelligent psychological counseling robots, the psychological state of the consultant is basically judged through voice, text, or expression recognition. At the same time, these intelligent psychological counseling robots are mostly based on single-modal interaction systems or simple multi-modal fusion systems and primary virtual reality interaction systems when interacting.

[0003] With the continuous development of science and technology, especially the rapid progress of artificial intelligence and machine deep learning technology, artificial intelligence psychological counseling robots will inevitably develop in a more intelligent direction. In the future, intelligent psychological counseling robots will pay more attention to the accuracy and efficiency of identifying people's emotions and psychological states. At the same time, more advanced technologies will inevitably be used in interaction to enhance the user experience and immersion;

[0004] At present, intelligent psychological counseling robots have the following problems:

[0005] 1. Single interaction method: There are limited ways or channels to interact with the system, device or application.

[0006] 2. Feedback is not timely and lacks immersion: Some systems are very complex and consist of multiple modules. Interactive feedback takes a long time and has high hardware requirements. Many systems mainly answer based on data in the database, lacking personalization. At the same time, some answer through voice, and some through screen output, which has a low sense of immersion.

[0007] 3. Low recognition accuracy and efficiency: Most systems only use a single method or a simple multimodal method for recognition analysis, which is not accurate and efficient enough.

[0008] 4. Personal privacy leakage: Some systems collect personal voice and expression data for analysis during recognition, and some are uploaded to large cloud databases for comparison and analysis, which can easily lead to personal privacy issues.

[0009] As social pressure increases, the number of people with psychological problems increases. A cheap and fast psychological diagnosis and treatment method is needed. At the same time, the complexity of people's psychological problems is also increasing. More advanced technology is needed for more accurate psychological diagnosis, and more excellent interactive systems are needed to improve the user's interactive experience and immersion.

[0010] To this end, a psychological counseling robot with multimodal emotion recognition capabilities is designed to provide another technical solution to the above technical problems. Summary of the invention

[0011] Based on this, it is necessary to provide a psychological counseling robot and a counseling method with multimodal emotion recognition capabilities to solve the technical problems raised in the above background technology.

[0012] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0013] A psychological counseling robot with multimodal emotion recognition capabilities includes a robot body, and is characterized in that a multimodal perception module is installed on the head of the robot body, a data processing and analysis module is installed inside the robot body, a real-time feedback generation module is installed inside the robot body and on one side of the robot body, an adaptive emotion regulation module is arranged inside the data processing and analysis module, a human-machine interaction module is installed on the front of the robot body, a storage and learning module is installed inside the robot body, a power and control module is installed on the bottom of the robot body, and a power supply module is installed on the back of the robot body.

[0014] As a preferred embodiment of the psychological counseling robot with multimodal emotion recognition capability provided by the present invention, the multimodal perception module is used to collect emotion data such as voice, facial expression and text input of the user, and obtain the user's emotion expression information in multiple modes;

[0015] The data processing and analysis module identifies the user's emotional state by comprehensively processing and analyzing the voice, image and text collected by the multimodal perception module;

[0016] The real-time feedback generation module generates instant psychological consultation feedback through data processing results, and transmits it to the user through language or display screen;

[0017] The adaptive emotion regulation module is responsible for adjusting the feedback mode according to the identified emotion state;

[0018] The human-computer interaction module is used to interact with the user and convey the sentiment analysis results and psychological counseling feedback;

[0019] The storage and learning module is used to store historical user data and feedback content, and continuously optimize emotion recognition and feedback effects through learning algorithms;

[0020] The power and control module is responsible for driving the robot to move and perform various actions;

[0021] The power module is used to provide power required by the device.

[0022] As a preferred implementation of the psychological counseling robot with multimodal emotion recognition capability provided by the present invention, the multimodal perception module also includes a physiological signal module for acquiring the user's physiological signals through the physiological signal module.

[0023] As a preferred embodiment of the psychological consultation robot with multimodal emotion recognition capability provided by the present invention, the real-time feedback generation module includes a feedback generation unit and a language synthesizer;

[0024] The feedback generation unit is used to generate psychological consultation suggestions according to the sentiment analysis results;

[0025] The speech synthesizer is used to convert the information of the feedback generation unit into voice feedback, or to display it visually through the display screen of the human-computer interaction module.

[0026] As a preferred embodiment of the psychological consultation robot with multimodal emotion recognition capability provided by the present invention, the storage and learning module also includes a privacy protection module for ensuring the security of the data during user communication;

[0027] The privacy protection module includes a data encryption module and an access control module;

[0028] The data encryption module is responsible for encrypting user data before transmission;

[0029] The access control module is used to implement data usage only within the scope of user authorization.

[0030] The consulting method of the psychological consulting robot with multimodal emotion recognition capability is used for any of the above items, and the steps are as follows:

[0031] The collected multimodal data of users is processed through the emotion recognition model to identify different emotional information;

[0032] Generate accurate sentiment analysis results by extracting features, calculating similarities, and fusing different input modalities;

[0033] Through virtual reality technology, the virtual scene is linked to the user's emotional state in real time;

[0034] Through cloud-edge collaborative computing, emotion recognition tasks are distributed on the cloud and edge devices.

[0035] As a preferred implementation of the consultation method of the psychological consultation robot with multimodal emotion recognition capability provided by the present invention, the steps of processing the collected user multimodal data through the emotion recognition big model to identify different emotion information are as follows:

[0036] Using the Transformer architecture, the multi-head self-attention mechanism is used to improve the ability of multimodal emotion recognition;

[0037] The feature extraction module extracts features from the information collected by the multimodal perception module and generates different high-dimensional vectors;

[0038] The data is further processed using a feedforward neural network to generate a unified sentiment representation, which is expressed as follows:

[0039] FFN(x)=max(0,xW1+b1)W2+b2

[0040] Among them, x is the data input to the feedforward neural network, which can be a vector representation obtained after feature extraction of multimodal data such as speech, images, text, etc. W1 is the weight matrix of the first fully connected layer, which is used to map the input data x from the original feature space to the hidden layer feature space. b1 is the bias term of the first fully connected layer, which is used to adjust the offset of the output signal. max(0,xW1+b1) is the ReLU (Rectified LinearUnit, ReLU) activation function, which is used to increase the nonlinearity of the network and introduce sparsity to help the network learn more complex feature representations. It ensures that all negative values ​​are set to 0 and only positive values ​​are retained. W2 is the weight matrix of the second fully connected layer, which is used to map the hidden layer features to the output layer. b2 is the bias term of the second fully connected layer, which is used to further adjust the output signal;

[0041] Based on the user emotional feedback in the training data, the weight distribution of the multi-head attention mechanism is continuously adjusted to form a large emotion recognition model.

[0042] As a preferred implementation of the consultation method of the psychological consultation robot with multi-modal emotion recognition capability provided by the present invention, the steps of generating accurate emotion analysis results by extracting features, calculating similarities and fusing different input modalities are as follows:

[0043] Through the pyramid pooling technology, multi-scale features are extracted according to pooling operations at different scales. The expression is as follows:

[0044] y=W·concat(p1,p1,...p n )+b

[0045] Where y is the fused feature vector obtained after pyramid pooling technology processing. W is the weight matrix used to map the spliced ​​multi-scale feature vector to the target feature space. p1,p1,...p n They represent the feature representations extracted at different pooling scales, which are concatenated and output through linear projection. concat(p1,p1,...p n ) is a concatenation operation, which combines the feature vectors p1, p1, ... p extracted at different pooling scales n concatenate into a longer vector. b is the bias term, which is used to adjust the offset of the fused feature vector;

[0046] The data of different modes are fused by weighted averaging, and the expression is as follows:

[0047]

[0048] Among them, α i represents the weight of each modality, and these weights are dynamically adjusted according to the similarity and importance of the modalities. i is the sentiment representation of the ith modality. Each modality (such as text, audio, video, etc.) has a corresponding sentiment representation, which can be numerical, vector or other forms, used to quantify the sentiment information contained in the modality. fused To form a fused emotion;

[0049] The weights are adjusted according to the different proportions of user voice, facial expressions, and text input collected by the multimodal perception module.

[0050] As a preferred implementation of the consultation method of the psychological consultation robot with multimodal emotion recognition capability provided by the present invention, the steps of real-time linkage between the virtual scene and the user's emotional state through virtual reality technology are as follows:

[0051] The linkage mechanism between virtual scenes and emotional states is expressed as follows:

[0052] s=W e ·e+b s

[0053] Among them, e is the user's emotion vector, W e is the mapping matrix, b s is a bias term, and the output s is used to control the changes of elements in the virtual scene;

[0054] Make real-time responses based on the user's current emotional state and change the output data of s.

[0055] As a preferred implementation of the consultation method of the psychological consultation robot with multimodal emotion recognition capability provided by the present invention, the emotion recognition task is distributed on the cloud and edge devices through cloud-edge collaborative computing, and the steps are as follows:

[0056] The formula processed by the edge device is as follows:

[0057] D edge =f edge (x1,x2,x3);

[0058] Among them, D edge Data or results processed by edge devices. It is a preprocessed data set or a data structure containing preliminary analysis results. edge It will be uploaded to the cloud for further processing, analysis or storage. x1, x2, x3 are the inputs of multimodal data. They represent different types of data, such as x1 is video data, x2 is audio data, x3 is text data or other types of sensor data. edge It is a preliminary processing function executed on the edge device. Its role may be to pre-process the original multimodal data, such as data cleaning, feature extraction, compression, encryption, etc., so that it can be uploaded to the cloud for more complex processing or analysis.

[0059] It can be seen without a doubt that the above-mentioned technical solution of the present application can definitely solve the technical problem to be solved by the present application.

[0060] At the same time, through the above technical solutions, the present invention has at least the following beneficial effects:

[0061] 1. The psychological counseling robot with multimodal emotion recognition capability and the counseling method thereof provided by the present invention, through the coordinated work of the multimodal perception module, the data processing and analysis module, the real-time feedback generation module, the adaptive emotion regulation module, the human-computer interaction module, the storage and learning module, the power and control module and the power supply module, realizes that the intelligent psychological counseling robot can recognize the user's emotions from multiple angles and in real time, and generate personalized psychological counseling feedback, thereby improving the user's interactive experience and emotion regulation effect; at the same time, through the reasonable position layout and electrical connection between modules, the system runs stably and has strong scalability, which helps to realize more intelligent mental health services.

[0062] 2. The present invention not only relies on traditional text input analysis, but also combines subtle changes in facial expressions captured by high-definition cameras, voice intonation information collected by high-sensitivity microphones, and possible physiological signals (such as heart rate, skin conductance, etc.) to achieve multi-dimensional and in-depth analysis of emotional states; through deep learning algorithms and neural network models, these multi-source heterogeneous data are fused and processed to significantly improve the accuracy and robustness of emotion recognition.

[0063] 3. The present invention adopts an architecture that combines edge computing and cloud computing; deploys lightweight models on local devices for preliminary processing to quickly respond to user needs; at the same time, uses the powerful computing power of the cloud to conduct deeper analysis and model training to ensure the real-time and accuracy of recognition; in addition, through algorithm optimization and hardware acceleration technology, it further shortens processing delays and improves user experience.

[0064] 4. The present invention has designed a strict privacy protection mechanism; all user data is encrypted before transmission and is only used within the scope of user authorization; at the same time, advanced technologies such as differential privacy and federated learning are used to ensure that personal sensitive information is not exposed during model training, thereby maintaining user data sovereignty and security.

[0065] 5. The present invention can intelligently adjust communication strategies and content, and provide personalized psychological counseling suggestions; through natural language processing technology and emotion generation models, the robot can communicate with users in a more natural and empathetic way, enhance users' sense of trust and immersion, and promote the maximization of psychological counseling effects; this can greatly improve the timeliness and pertinence of the psychological counseling process, and improve the psychological relief and support effects of users.

[0066] 6. The robot of the present invention can dynamically adjust the way of interaction with the user and respond with more appropriate language, tone or expression, thereby effectively regulating the user's emotions and helping the user to restore a stable emotional state when under great psychological pressure. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0068] Figure 1 is a flow chart of the present invention;

[0069] Figure 2 It is a schematic diagram of the structure of the robot of the present invention. DETAILED DESCRIPTION

[0070] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0071] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings.

[0072] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features and technical solutions in the embodiments may be combined with each other.

[0073] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0074] Embodiment 1

[0075] Reference Figure 1 , a consultation method of a psychological consultation robot with multimodal emotion recognition capabilities.

[0076] By introducing large models, multimodal fusion and virtual reality technology, and ensuring real-time processing through cloud-edge collaborative computing, the accuracy of emotion recognition and the naturalness of interaction have been improved.

[0077] 1. Emotion recognition system based on large model

[0078] The core of the big model is a deep learning model generated through a large amount of data training, which can process complex and diverse emotional input signals. The system uses the Transformer architecture and a multi-head self-attention mechanism to improve the ability of multimodal emotion recognition.

[0079] 1.1 Multi-head self-attention mechanism

[0080] In the emotion recognition task, the system needs to comprehensively process multiple inputs from the user (such as voice, expression, body posture, etc.). These input data are interrelated and their meaning needs to be understood based on the context. The Multi-head Self-Attention mechanism achieves this goal through the following steps:

[0081] 1.1.1 Input matrix processing: The input data of each modality (such as speech or expression) is linearly transformed to generate query (Q), key (K) and value (V) matrices respectively. These matrices calculate the relevance and find the importance of each input data to the current emotional state.

[0082] 1.2.1 Attention score calculation: The attention mechanism determines the relationship between the inputs by calculating the similarity between Q and K. The formula is:

[0083]

[0084] Among them, Attention (Q, K, V) is a single attention score. The calculation formula of a single attention score is usually based on dot product attention or scaled dot product attention. Here it is the scaled dot product attention score. Q is the query matrix, which is used to retrieve relevant information from the input data. It is obtained by linear transformation of the input matrix. K is the key matrix, which is used to match the query matrix to find relevant information. It is also obtained by linear transformation of the input matrix. V is the value matrix, which contains the actual information to be retrieved. It is also obtained by linear transformation of the input matrix. d k is the dimension of the key matrix. After calculating the dot product, divide by the square root of d k To scale, it is used to prevent the gradient from disappearing due to excessive values.

[0085] The core function of this mechanism is to find the input data fragment that is most relevant to the current emotional state through this correlation calculation.

[0086] 1.2.2 Multi-head attention mechanism: In order to improve the model's ability to capture information from different modalities, the system introduces multiple attention heads (multi-head attention). Each attention head independently calculates different attention distributions, then concatenates these distributions, and finally linearly transforms the output:

[0087] MultiHead(Q,K,V)=Concat(head1,head2...,head h )W O

[0088] Among them, head1, head2…, head h is the output of multiple attention heads. In the multi-head attention mechanism, the Q, K, and V matrices are divided into multiple parts (i.e., "heads"), and each head performs the attention mechanism calculation independently. The output of each head (e.g., head1, head2..., head h ) is an attention-weighted value matrix, which represents the information in the input sequence that the head pays attention to. Concat(head1,head2…,head h ) This operation represents the concatenation of the outputs of multiple attention heads. Since each head processes a part of the input data independently, their outputs are also independent. Through the concatenation operation, we can integrate these independent outputs to form a more comprehensive representation. Ois a learnable weight matrix used to perform a linear transformation on the concatenated output. Through this linear transformation, the concatenated output can be mapped to the desired dimension to obtain the final output. O The weights of the matrix are learned during model training, and can adjust the importance of each head output according to task requirements.

[0089] The formula for each attention head is:

[0090]

[0091] Among them, head i represents the output of the i-th head, are the query, key, and value transformation matrices of the i-th head, respectively. Attention(Q,K,V) is the single attention score calculated after function transformation.

[0092] This multi-head mechanism allows the model to focus on information in multiple dimensions when processing sentiment data, thereby comprehensively analyzing the user's emotional state.

[0093] 1.2 Large Model Sentiment Analysis

[0094] In order to improve the accuracy of emotion recognition and the scalability of the system, the present invention proposes the following specific solutions:

[0095] 1.2.1 Preprocessing of multimodal data: The input multimodal data (voice, expression, posture, etc.) is extracted through independent feature extraction modules to generate different high-dimensional vectors. These data are further processed using a feed-forward neural network (FFN) to generate a unified emotional representation.

[0096] FFN(x)=max(0,xW1+b1)W2+b2

[0097] Among them, x is the data input to the feedforward neural network, which can be a vector representation obtained after feature extraction of multimodal data such as speech, image, text, etc. W1 is the weight matrix of the first fully connected layer, which is used to map the input data x from the original feature space to the hidden layer feature space. b1 is the bias term of the first fully connected layer, which is used to adjust the offset of the output signal. max(0,xW1+b1) is the ReLU (Rectified Linear Unit, ReLU) activation function, which is used to increase the nonlinearity of the network and introduce sparsity to help the network learn more complex feature representations. It ensures that all negative values ​​are set to 0 and only positive values ​​are retained. W2 is the weight matrix of the second fully connected layer, which is used to map the hidden layer features to the output layer. b2 is the bias term of the second fully connected layer, which is used to further adjust the output signal.

[0098] Through feedforward neural networks, the system can learn more complex emotional features at each layer.

[0099] 1.2.2 Adaptive learning of emotion model: The large model automatically optimizes its parameters through a self-learning mechanism, so that it can adapt to the emotional expressions of different users and generate personalized psychological feedback suggestions. In the specific implementation, the model continuously adjusts the weight distribution of the multi-head attention mechanism based on the user emotional feedback in the training data to improve the dynamic adaptability of emotion recognition.

[0100] 2. Multimodal fusion technology

[0101] Emotion recognition systems need to comprehensively consider multiple user inputs (voice, expression, gesture, etc.). Information from different modalities provides rich emotional clues. Multimodal fusion technology generates accurate emotion analysis results by extracting features, calculating similarities, and fusing these different input modalities.

[0102] 2.1 Modal Fusion

[0103] Different modal inputs play a complementary role in emotion recognition. For example, speech can express emotional clues such as intonation and speaking speed, while facial expressions and gestures can provide non-verbal emotional information. Therefore, these modalities are combined through feature extraction and fusion algorithms.

[0104] 2.1.1 Feature extraction: Assuming that the input speech data is x1, the expression data is x2, and the posture data is x3, each modality is feature extracted through an independent encoder to generate vectors h1, h2, and h3. These vectors represent the important emotional information in each modality.

[0105] 2.1.2 Similarity calculation: In order to understand the correlation between these modes, the system calculates the similarity of different modes. The formula is as follows:

[0106]

[0107] Among them, h i ·h j represents the vector inner product of mode i and mode j, ||h i || and ||h j || represents the modulus of the modal vector. By calculating the similarity between modalities, the system can understand the relevance of speech and facial expressions in expressing emotions.

[0108] 2.1.3 Modal fusion: The system uses weighted averaging to fuse data from different modalities. The formula is:

[0109]

[0110] Among them, α i represents the weight of each modality, and these weights are dynamically adjusted according to the similarity and importance of the modalities. i is the sentiment representation of the ith modality. Each modality (such as text, audio, video, etc.) has a corresponding sentiment representation, which can be numerical, vector or other forms, used to quantify the sentiment information contained in the modality. fused To form a fusion of emotions.

[0111] 2.2 Multimodal Emotion Fusion System

[0112] In order to enhance the effect of multimodal fusion, the present invention proposes the following specific solutions:

[0113] Dynamic adjustment of modality weights: During the fusion process, the system dynamically adjusts the weights of modalities based on contextual needs. For example, when users express emotions through voice during consultation, the weight of voice modality increases, while when users convey emotions through facial expressions and body language, the weight of facial expressions and gestures increases.

[0114] Pyramid pooling technology: In order to reduce data dimensions and retain key information, the system uses pyramid pooling technology. This technology extracts multi-scale features through pooling operations at different scales, effectively improving the effect of multimodal fusion. The pooling operation formula is:

[0115] y=W·concat(p1,p1,...p n )+b

[0116] Where y is the fused feature vector obtained after pyramid pooling technology processing. W is the weight matrix used to map the spliced ​​multi-scale feature vector to the target feature space. p1,p1,...p n They represent the feature representations extracted at different pooling scales, which are concatenated and output through linear projection. concat(p1,p1,...p n ) is a concatenation operation, which combines the feature vectors p1, p1, ... p extracted at different pooling scales n concatenate into a longer vector. b is a bias term used to adjust the offset of the fused feature vector.

[0117] 3. Application of virtual reality technology

[0118] In order to enhance the user's sense of immersion during psychological counseling, the present invention introduces virtual reality technology, which provides personalized and dynamic emotional support through real-time linkage between virtual scenes and the user's emotional state.

[0119] 3.1 Virtual Reality and Emotional Connection

[0120] Virtual reality technology creates a highly immersive interactive environment for users. The system adjusts the elements in the virtual scene by sensing the user's emotional state. The system dynamically controls the light, sound and other factors of the virtual scene according to the user's emotional state to create a psychological counseling environment that meets the user's needs.

[0121] The linkage mechanism between virtual scenes and emotional states can be expressed by the following formula:

[0122] s=W e ·e+b s

[0123] Among them, e is the user's emotion vector, W e is the mapping matrix, b s is a bias term, and the output s is used to control the changes of elements in the virtual scene, such as light, sound effects, etc.

[0124] 3.2 Immersive Emotional Feedback System

[0125] In order to enhance the user's interactive experience, the present invention proposes the following virtual reality solution:

[0126] Emotion-driven virtual scenes: The virtual scenes respond in real time based on the user's current emotional state. For example, when the user shows tension or anxiety, the lighting in the scene becomes softer and the background music becomes more soothing, helping the user relax.

[0127] Multimodal emotion mapping: Changes in virtual scenes not only rely on the user's voice emotional expression, but also combine multimodal data such as facial expressions and gestures. By mapping the user's emotional state in real time, the system provides dynamic emotional feedback, enhancing the user's sense of immersion and experience.

[0128] 4. Cloud-edge collaborative computing

[0129] To ensure the real-time and high efficiency of the system under large-scale data processing, the present invention introduces cloud-edge collaborative computing. By distributing the emotion recognition tasks on the cloud and edge devices, the system can achieve efficient real-time emotion feedback.

[0130] 4.1 Cloud-Edge Collaborative Computing

[0131] The core of cloud-edge collaboration is that edge devices complete data collection and preliminary processing, while complex sentiment analysis is completed by the cloud. This collaborative approach not only improves computing efficiency but also reduces latency. The formula for edge device processing is as follows:

[0132] D edge =f edge (x1,x2,x3);

[0133] Among them, Dedge Data or results processed by edge devices. It is a preprocessed data set or a data structure containing preliminary analysis results. edge It will be uploaded to the cloud for further processing, analysis or storage. x1, x2, x3 are the inputs of multimodal data. They represent different types of data, such as x1 is video data, x2 is audio data, x3 is text data or other types of sensor data. edge It is a preliminary processing function executed on the edge device. Its role may be to pre-process the original multimodal data, such as data cleaning, feature extraction, compression, encryption, etc., so that it can be uploaded to the cloud for more complex processing or analysis.

[0134] 4.2 Real-time Emotion Computing and Privacy Protection

[0135] To protect user privacy and improve system performance, the present invention proposes the following cloud-edge collaboration solution:

[0136] Real-time processing: The edge devices complete the initial processing of multimodal data, and the cloud performs sentiment analysis and feedback generation, thus ensuring the efficiency and real-time performance of the system.

[0137] Privacy protection mechanism: The initial processing completed on the edge device includes the stripping of sensitive information, and the data uploaded to the cloud is only feature data, thereby reducing the risk of privacy leakage.

[0138] 5. Conclusion

[0139] It has significant advantages in emotion recognition, interactive experience and privacy protection:

[0140] Improved emotion recognition accuracy: The combination of large models and multimodal fusion technology improves the accuracy and reliability of the system in complex emotion recognition tasks.

[0141] Immersive interactive experience: The introduction of virtual reality technology provides a personalized and dynamic psychological counseling environment, enhancing the user's sense of immersion in the interaction process.

[0142] Real-time feedback and privacy protection: The cloud-edge collaborative computing mechanism not only ensures the real-time response capability of the system, but also protects the privacy and security of users through edge computing.

[0143] Embodiment 2

[0144] Reference Figure 2 , based on the above-mentioned embodiment 1, a psychological consultation robot with multimodal emotion recognition capability is disclosed, including a multimodal perception module, a data processing and analysis module, a real-time feedback generation module, an adaptive emotion regulation module, a human-computer interaction module, a storage and learning module, a power and control module and a power supply module;

[0145] The multimodal perception module is used to collect the user's emotional data such as voice, facial expression and text input. Its main function is to obtain the user's emotional expression information in multiple modes. It is installed on the head or front of the robot. It can capture the subtle changes in the user's facial expression through a high-definition camera, collect the user's voice intonation information through a high-sensitivity microphone, and obtain the user's written expression or touch screen input text information through the text input interface. It can capture the user's emotional expression in real time and generate a multimodal data stream.

[0146] The physiological signal module can also be used to obtain the user's physiological signals to assist in emotion recognition, which can be a heart rate monitor, skin conductance sensor, etc. The natural language processing module is used to realize natural language interaction between the robot and the user, including text input analysis, speech synthesis and recognition, etc.

[0147] The data processing and analysis module identifies the user's emotional state by comprehensively processing and analyzing the voice, image and text collected by the multimodal perception module. Specifically, the core processing unit inside the robot is installed; so that the module can first convert the voice into text through the voice recognition algorithm, and then combine the analysis of facial expressions, and use the emotion recognition algorithm (such as convolutional neural network and natural language processing technology) to comprehensively analyze the user's emotions. Through multimodal data fusion processing, accurate emotional state output is generated, such as "happy", "frustrated", "anxious", etc. The technical effect is that the accuracy of emotion recognition is significantly improved, especially in the face of complex emotional expression scenes. The data processing and analysis module is connected to the multimodal perception module by wired or wireless means, and is responsible for data collection and transmission. The multimodal perception module sends the raw data to the data processing module for processing.

[0148] Specifically, lightweight models can be deployed on edge computing devices for preliminary data processing and emotion recognition to achieve rapid response; the powerful computing power of the cloud can be used on the cloud computing platform to conduct deeper data analysis and model training to improve the accuracy and efficiency of emotion recognition; deep learning algorithms and neural network models can be used to integrate multimodal data, identify emotions, and analyze them;

[0149] The real-time feedback generation module generates instant psychological counseling feedback through data processing results, and transmits it to the user through language or display screen. The specific installation location is adjacent to the data processing module, so that it can generate personalized psychological counseling suggestions based on the results of emotion recognition, using a pre-set psychological counseling library or intelligent generation model, and feedback to the user in the form of voice or text. The technical effect of this module is that it can provide timely and targeted psychological counseling and improve the effect of counseling. The data processing and analysis module is connected to the real-time feedback generation module through an internal data bus, and the emotion analysis results are directly used as the input basis for feedback generation;

[0150] The real-time feedback generation module includes a feedback generation unit and a language synthesizer;

[0151] The feedback generation unit is used to generate psychological counseling suggestions based on the sentiment analysis results;

[0152] The speech synthesizer is used to convert the information of the feedback generation unit into speech feedback, or to display it visually through the display screen of the human-computer interaction module;

[0153] The adaptive emotion regulation module is responsible for adjusting the feedback method according to the identified emotional state, such as adjusting the language tone, content, etc., to achieve emotion regulation. The specific installation location is integrated in the data processing and analysis module. Specifically, the emotion generation model can enable the robot to communicate with the user in a more natural and empathetic way to enhance the interactive experience; and then it can rely on the adaptive algorithm to adjust the robot's tone, wording and interaction method according to the user's emotional changes. For example, when the user is in an anxious state, the robot will use a more soothing tone to provide comforting advice. The technical effect of this module is to help users effectively regulate their emotions and enhance the effect of mental health management. The adaptive emotion regulation module and the real-time feedback generation module are interdependent and dynamically adjust the feedback method through logical association. This module selects appropriate psychological counseling content and expression methods based on the user's current emotional state;

[0154] The human-computer interaction module is used to interact with the user and convey the emotion analysis results and psychological counseling feedback. It is installed on the front panel of the robot and includes a touch screen display, speakers and microphones. The human-computer interaction module is connected to the real-time feedback generation module through an interface and is responsible for conveying the feedback results to the user to achieve human-computer interaction.

[0155] The storage and learning module is used to store historical user data and feedback content, and continuously optimize the emotion recognition and feedback effects through learning algorithms. The specific installation location is integrated inside the robot. The storage and learning module has a direct data storage and retrieval relationship with the data processing and analysis module. It is responsible for storing historical data and using this data to continuously optimize the accuracy of emotion recognition and feedback quality.

[0156] It also includes a privacy protection module to ensure the security of user communication data; differential privacy and federated learning can be used to protect user privacy during model training;

[0157] The privacy protection module includes a data encryption module and an access control module;

[0158] The data encryption module is responsible for encrypting user data before transmission to ensure the security of data during transmission;

[0159] The access control module is used to ensure that data is used only within the scope of user authorization, preventing data from being illegally accessed or abused;

[0160] The power and control module is responsible for driving the robot's movement and performing various actions, enhancing the dynamics of human-computer interaction. The specific installation location is at the bottom of the robot. The power and control module is connected to the multimodal perception module, data processing and analysis module, real-time feedback generation module, adaptive emotion regulation module, human-computer interaction module, storage and learning module, and power module, cooperating with the robot's physical movement and emotional interaction to enhance the user experience.

[0161] The power module is used to provide the power required by the device to ensure continuous operation. The specific installation location is the back or bottom of the robot.

[0162] This embodiment also includes a network connection module for realizing the connection between the robot and the cloud computing platform, other devices or networks.

[0163] In this embodiment, during installation, the multimodal perception module is located at the head or front of the robot to ensure effective capture of the user's emotional signals; the data processing and analysis module and the storage and learning module are located at the internal core position; the human-computer interaction module is located at the front panel to facilitate communication between the user and the robot; the power module and the power and control module are located at the bottom of the robot to ensure the balance and operation of the equipment, and at the same time, the power and control module, the multimodal perception module, the data processing and analysis module, the real-time feedback generation module, the adaptive emotion regulation module, the human-computer interaction module, the storage and learning module and the power module are all connected through the internal data bus, and the data processing and analysis module is the core control unit of each functional module, and the functions of each module are coordinated through logical processing.

[0164] Achieve results

[0165] 1. Significantly improve the accuracy and robustness of emotion recognition:

[0166] Technical principle: This invention integrates multimodal emotion recognition technology, not only relying on traditional text input analysis, but also combining subtle changes in facial expressions captured by high-definition cameras, voice intonation information collected by high-sensitivity microphones, and possible physiological signals (such as heart rate, skin conductance, etc.). This multi-dimensional, deep-level emotional state analysis method uses deep learning algorithms and neural network models to effectively integrate and process these multi-source heterogeneous data, thereby being able to more comprehensively capture the user's emotional information.

[0167] Compared with existing technologies: Most existing technologies only rely on a single modality (such as text or voice) for emotion recognition, which is easily affected by factors such as environmental noise and user expression habits, resulting in low recognition accuracy.

[0168] Technical effect: Through the fusion processing of multimodal data, the present invention can still maintain a high accuracy of emotion recognition in complex and changeable environments, is more robust, and can more accurately understand the user's emotional state.

[0169] 2. Realize real-time and efficient emotion recognition and processing:

[0170] Technical principle: This invention adopts an architecture that combines edge computing and cloud computing. A lightweight model is deployed on the local device for preliminary processing to quickly respond to user needs. At the same time, the powerful computing power of the cloud is used for deeper analysis and model training. In addition, algorithm optimization and hardware acceleration technology are used to further shorten processing delays.

[0171] Compared with existing technologies: Existing technologies often have slow recognition speeds due to large amounts of data to be processed and high computational complexity, making it difficult to meet real-time requirements.

[0172] Technical effect: The real-time and efficient processing mechanism of the present invention can significantly reduce the delay of emotion recognition, improve user experience, and ensure the immediacy and effectiveness in application scenarios such as psychological counseling.

[0173] 3. Strengthen privacy protection and ensure user data security:

[0174] Technical principle: This invention designs a strict privacy protection mechanism, including encryption of user data before transmission, using data only within the scope of user authorization, and using advanced technologies such as differential privacy and federated learning to protect user privacy.

[0175] Compared with existing technologies: When existing technologies use large models for emotion recognition, there may be a risk of user data leakage and it is difficult to fully protect user privacy.

[0176] Technical effect: The privacy protection mechanism of the present invention can effectively prevent user data from being illegally obtained or abused, maintain user data sovereignty and security, and enhance users' trust in psychological counseling robots.

[0177] 4. Provide personalized interactive experience to improve the effectiveness of psychological counseling:

[0178] Technical principle: Based on the precise understanding of the user's emotions and psychological state, the invention can intelligently adjust the communication strategy and content and provide personalized psychological counseling advice. Through natural language processing technology and emotion generation model, the robot can communicate with users in a more natural and empathetic way.

[0179] Compared with existing technologies: Existing technologies often lack personalization and targeting, and are difficult to flexibly adjust according to the user's emotional state.

[0180] Technical effect: The personalized interactive experience of the present invention can enhance the user's sense of trust and immersion, promote the maximization of psychological counseling effects, and improve the user's psychological relief and support effects.

[0181] 5. Adaptive emotion regulation, effectively regulating user emotions:

[0182] Technical principle: The present invention dynamically adjusts the interaction mode with users through intelligent algorithms, and responds with more appropriate language, tone or expression, thereby effectively regulating users' emotions.

[0183] Compared with existing technologies: Most existing technologies lack emotion regulation mechanisms and are difficult to dynamically adjust according to the user's emotional state.

[0184] Technical effect: The adaptive emotion regulation mechanism of the present invention can help users restore a stable emotional state when they are under great psychological pressure, and improve the pertinence and effectiveness of psychological counseling.

[0185] Embodiment 3

[0186] The disclosure is made on the basis of the above-mentioned Embodiment 1 and Embodiment 2.

[0187] 1. Alternatives:

[0188] Based on simpler multimodal fusion: You can consider using simpler multimodal fusion methods, such as combining only visual and auditory information, or only visual and tactile information, to reduce technical difficulty and cost. This solution may not achieve the immersive experience you described, but it can serve as a simpler alternative.

[0189] Only use cloud processing: You can consider only computing and processing in the cloud and feeding the results back to the user. This solution can take advantage of the powerful computing resources of the cloud, but it may cause delays and network dependencies, affecting the real-time nature of the interaction.

[0190] Simplified virtual reality technology: Simplified virtual reality technology can be used, such as reducing the requirements for hardware devices or using simpler virtual scenes, thereby reducing costs and development difficulties. However, this solution may affect the immersion and realism of the interaction.

[0191] 2. Better solution:

[0192] More advanced cloud-edge collaborative computing models: More advanced cloud-edge collaborative computing models can be studied, such as combining the advantages of edge computing and cloud computing to improve processing speed and real-time performance and reduce dependence on the network.

[0193] More complex deep learning algorithms: More complex deep learning algorithms can be studied, such as applying generative adversarial networks (GANs) to multimodal fusion, or using more advanced natural language processing (NLP) techniques to achieve more natural and intelligent interactions.

[0194] Higher-precision motion capture technology: Higher-precision motion capture technology can be used, for example in combination with biofeedback signals, to increase the realism and immersion of the interaction.

[0195] More realistic virtual environments: More realistic virtual environments can be used, such as more advanced graphics rendering technology and more realistic physical simulations, to enhance the immersion and experience of the interaction.

[0196] 3. Second-best solution:

[0197] Reduce feedback frequency: The frequency of feedback to users can be reduced, for example, by providing feedback only at key moments or when specific events occur, thereby reducing the requirements for computing resources.

[0198] Reduce feedback dimensions: The dimensions of feedback can be reduced, for example, only visual feedback or auditory feedback can be provided instead of providing feedback in multiple modalities at the same time, thereby reducing technical difficulty and development costs.

[0199] Related patents: "An intelligent human-computer interaction system based on multimodal features", "A multimodal interaction system, method, device and medium based on a large model"

[0200] Related protocols:

[0201] User interface interaction protocol: used to define the way users interact with the system, such as clicking, sliding, voice input, etc.

[0202] Data collection protocol: used to collect user feedback data, such as user behavior data, physiological data, emotional data, etc.

[0203] Standards: Some organizations and institutions have developed standards for user feedback, such as:

[0204] ISO 9241-11: Standard for human-computer interaction, including guidance on user feedback.

[0205] Motion capture protocol: used to capture user movements, such as Leap Motion, ViveTracker, etc.

[0206] Standards: Some organizations and institutions have developed standards for virtual reality technology, such as:

[0207] OpenXR: Committed to developing cross-platform virtual reality development standards, such as standards for scenes, interactions, input, etc.

[0208] Data format protocol: used to define different modal data formats, such as image format, audio format, tactile data format, etc.

[0209] Data synchronization protocol: used to ensure the time synchronization of data of different modalities to ensure the accuracy of fusion.

[0210] Standards: Some organizations and institutions have developed standards for multimodal fusion.

[0211] IEEE Multimodal Standards Working Group: dedicated to developing standards and specifications for multimodal fusion, including standards in data formats, fusion methods, performance evaluation, etc.

[0212] The preferred embodiments of the present invention disclosed above are only used to help explain the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the present invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can understand and use the present invention well. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. A psychological counseling robot with multimodal emotion recognition capability, comprising a robot body, characterized in that: A multimodal perception module is installed on the head of the robot body, a data processing and analysis module is installed inside the robot body, a real-time feedback generation module is installed inside the robot body and on one side of the robot body, an adaptive emotion regulation module is arranged inside the data processing and analysis module, a human-computer interaction module is installed on the front of the robot body, a storage and learning module is installed inside the robot body, a power and control module is installed on the bottom of the robot body, and a power supply module is installed on the back of the robot body.

2. The psychological consultation robot with multimodal emotion recognition capability according to claim 1, characterized in that: The multimodal perception module is used to collect the user's voice, facial expression and text input emotion data, and obtain the user's emotion expression information in multiple modes; The data processing and analysis module identifies the user's emotional state by comprehensively processing and analyzing the voice, image and text collected by the multimodal perception module; The real-time feedback generation module generates instant psychological consultation feedback through data processing results, and transmits it to the user through language or display screen; The adaptive emotion regulation module is responsible for adjusting the feedback mode according to the identified emotion state; The human-computer interaction module is used to interact with the user and convey the sentiment analysis results and psychological counseling feedback; The storage and learning module is used to store historical user data and feedback content, and continuously optimize emotion recognition and feedback effects through learning algorithms; The power and control module is responsible for driving the robot to move and perform various actions; The power module is used to provide power required by the device.

3. The psychological consultation robot with multimodal emotion recognition capability according to claim 2, characterized in that: The multimodal perception module also includes a physiological signal module, which is used to obtain the user's physiological signals through the physiological signal module.

4. The psychological consultation robot with multimodal emotion recognition capability according to claim 2, characterized in that: The real-time feedback generation module includes a feedback generation unit and a language synthesizer; The feedback generation unit is used to generate psychological consultation suggestions according to the sentiment analysis results; The speech synthesizer is used to convert the information of the feedback generation unit into voice feedback, or to display it visually through the display screen of the human-computer interaction module.

5. The psychological consultation robot with multimodal emotion recognition capability according to claim 2, characterized in that: The storage and learning module also includes a privacy protection module for ensuring the security of data during user communication; The privacy protection module includes a data encryption module and an access control module; The data encryption module is responsible for encrypting user data before transmission; The access control module is used to implement data usage only within the scope of user authorization.

6. A consultation method for a psychological consultation robot with multimodal emotion recognition capability, used for the psychological consultation robot with multimodal emotion recognition capability according to any one of claims 1 to 5, characterized in that: Here are the steps: The collected multimodal data of users is processed through the emotion recognition model to identify different emotional information; Generate accurate sentiment analysis results by extracting features, calculating similarities, and fusing different input modalities; Through virtual reality technology, the virtual scene is linked to the user's emotional state in real time; Through cloud-edge collaborative computing, emotion recognition tasks are distributed on the cloud and edge devices.

7. The consulting method of the psychological consulting robot with multimodal emotion recognition capability according to claim 6, characterized in that: The steps of processing the collected user multimodal data through the emotion recognition big model to identify different emotion information are as follows: Using the Transformer architecture, the multi-head self-attention mechanism is used to improve the ability of multimodal emotion recognition; The feature extraction module extracts features from the information collected by the multimodal perception module and generates different high-dimensional vectors; The data is further processed using a feedforward neural network to generate a unified sentiment representation, which is expressed as follows: FFN(x)=max(0,xW1+b1)W2+b2 Among them, x is the data input to the feedforward neural network; W1 is the weight matrix of the first fully connected layer, which is used to map the input data x from the original feature space to the hidden layer feature space; b1 is the bias term of the first fully connected layer, which is used to adjust the offset of the output signal; max(0,xW1+b1) is the ReLU activation function, which is used to increase the nonlinearity of the network and introduce sparsity to help the network learn more complex feature representations; ensure that all negative values ​​are set to 0 and only retain positive values; W2 is the weight matrix of the second fully connected layer, which is used to map the hidden layer features to the output layer; b2 is the bias term of the second fully connected layer, which is used to further adjust the output signal; Based on the user emotional feedback in the training data, the weight distribution of the multi-head attention mechanism is continuously adjusted to form a large emotion recognition model.

8. The consulting method of the psychological consulting robot with multimodal emotion recognition capability according to claim 7, characterized in that: The steps of extracting features, calculating similarities and fusing different input modalities to generate accurate sentiment analysis results are as follows: Through the pyramid pooling technology, multi-scale features are extracted according to pooling operations at different scales. The expression is as follows: y=W·concat(p1,p1,...p n )+b Among them, y is the fused feature vector obtained after pyramid pooling technology processing; W is the weight matrix used to map the spliced ​​multi-scale feature vector to the target feature space; p1,p1,...p n They represent the feature representations extracted at different pooling scales, which are concatenated and output through linear projection; concat(p1,p1,...p n ) is a concatenation operation, which combines the feature vectors p1, p1, ... p extracted at different pooling scales n spliced ​​into a longer vector; b is the bias term, which is used to adjust the offset of the fused feature vector; The data of different modes are fused by weighted averaging, and the expression is as follows: Among them, α i represents the weight of each modality, and these weights are dynamically adjusted according to the similarity and importance of the modalities; h i is the sentiment representation of the ith modality. Each modality has a corresponding sentiment representation, which can be numerical, vector or other forms, used to quantify the sentiment information contained in the modality; h fused To form a fused emotion; The weights are adjusted according to the different proportions of user voice, facial expressions, and text input collected by the multimodal perception module.

9. The consulting method of the psychological consulting robot with multimodal emotion recognition capability according to claim 6, characterized in that: The virtual reality technology is used to link the virtual scene with the user's emotional state in real time, and the steps are as follows: The linkage mechanism between virtual scenes and emotional states is expressed as follows: s=W e ·e+b s Among them, e is the user's emotion vector, W e is the mapping matrix, b s is a bias term, and the output s is used to control the changes of elements in the virtual scene; Make real-time responses based on the user's current emotional state and change the output data of s.

10. The consulting method of the psychological consulting robot with multimodal emotion recognition capability according to claim 6, characterized in that: The emotion recognition task is distributed on the cloud and edge devices through cloud-edge collaborative computing. The steps are as follows: The formula processed by the edge device is as follows: D edge =f edge (x1,x2,x3); Among them, D edge Data or results processed by edge devices; a preprocessed data set or a data structure containing preliminary analysis results; D edge It will be uploaded to the cloud for further processing, analysis or storage; x1, x2, x3 are multimodal data inputs; they represent different types of data, for example, x1 is video data, x2 is audio data, x3 is text data or other types of sensor data; f edge It is the preliminary processing function executed on the edge device.

Images