Psychotherapy and healing robot based on user emotional state perception

Abstract

The invention relates to the field of intelligent treatment and healing, and particularly discloses a psychological treatment and healing robot based on user emotional state perception, which is characterized in that a consulting voice input signal of a target user object is acquired through a sound sensor, and a face image of the target user object is acquired through a camera; carrying out face feature extraction and emotion recognition analysis on the face image by adopting a signal processing and image analysis mode based on artificial intelligence, and carrying out embedded coding on the consultation voice input signal; in this way, psychotherapy recovery reply voice is intelligently generated according to cross-domain query modulation representation between the emotion type feature of the target object and the embedded feature of each consulting voice block. Therefore, the system can reflect the emotional state of the user more accurately. And moreover, more personalized psychotherapy recovery voice is generated, so that the requirements of different users can be better met.

Description

Technical Field

[0001] The present application relates to the field of intelligent healing, and more specifically, to a psychological healing robot based on the perception of the user's emotional state. Background Art

[0002] With the accelerated pace of life and increased pressure in modern society, more and more people are facing mental health problems. Factors such as fierce job competition, high cost of living, and complex interpersonal relationships have caused many people to feel anxious, depressed, and overly stressed. Long-term psychological stress not only affects an individual's work and quality of life, but also causes various physical and mental illnesses, such as insomnia and decreased immune system function. Therefore, timely and effective psychological support and intervention have become particularly important to help people better cope with stress and maintain mental health.

[0003] Traditional psychotherapy systems mainly rely on voice signals for emotion recognition, however, voice signals can only capture part of the emotional information. In fact, micro-expressions and facial expressions also play a vital role in emotional expression. If relying solely on voice signals, the system often finds it difficult to fully capture these subtle emotional changes, which affects the accuracy and comprehensiveness of emotion recognition. In addition, traditional psychotherapy emotion recognition methods are usually generic and lack optimization for individual differences. Everyone's emotional expression and mental state are unique, and general models have difficulty adapting to this diversity. For example, some users may express emotions mainly through changes in tone of voice, while others rely more on facial expressions. Because general models cannot capture these individual differences, the accuracy of emotion recognition and the level of personalized support are insufficient.

[0004] Therefore, a psychological healing robot based on the perception of the user's emotional state is desired. Summary of the invention

[0005] In order to solve the above technical problems, this application is proposed.

[0006] The embodiment of the present application provides a psychological healing robot based on user emotional state perception, which includes: a consultation voice signal acquisition module, which is used to acquire the consultation voice input signal of the target user object through a sound sensor; a face image acquisition module, which is used to acquire the face image of the target user object through a camera; a face emotion recognition module, which is used to extract face features and recognize emotions on the face image of the target user object to obtain an anchor representation of the target object's emotion type; a consultation voice block embedding module, which is used to embed and encode the consultation voice input signal to obtain a sequence of consultation voice block embedding representations; a cross-domain query encoding optimization module, which is used to optimize the target object emotion type anchor representation and the consultation voice block embedding representation. The sequence is cross-domain query coding optimized to obtain a sequence of embedded representations of target object emotion modulation consulting speech blocks, wherein the cross-domain query coding optimization module includes: a consulting speech essence capture unit, used to capture the essential features of the sequence of embedded representations of consulting speech blocks to obtain essential feature representations of consulting speech blocks; an attention allocation unit, used to perform hierarchical attention allocation on the sequence of target object emotion type anchor representation and the embedded representation of consulting speech blocks based on the essential feature representation of consulting speech blocks to obtain a sequence of embedded representations of target object emotion modulation consulting speech blocks; a psychological healing reply speech generation module, used to obtain psychological healing reply speech based on the sequence of embedded representations of target object emotion modulation consulting speech blocks.

[0007] In the above-mentioned psychological healing robot based on user emotional state perception, the facial emotion recognition module includes: a target object facial feature extraction unit, which is used to input the facial image of the target user object into a facial feature extractor based on a deep separable convolutional neural network model to obtain a target object facial semantic coding feature map; a target object emotion recognition unit, which is used to input the target object facial semantic coding feature map into a classifier-based emotion recognizer to obtain a target object emotion type anchor coding vector as the target object emotion type anchor representation.

[0008] In the above-mentioned psychological healing robot based on user emotional state perception, the consultation voice block embedding module is used to: perform tokenization and embedding encoding on the consultation voice input signal to obtain a sequence of consultation voice block embedding coding vectors as a sequence of target object emotion modulation consultation voice block embedding representations.

[0009] In the above-mentioned psychological healing robot based on user emotional state perception, the consultation speech essence capture unit is used to: input the sequence of the consultation speech block embedded coding vector into the essential feature capture network to obtain the consultation speech block essential feature representation vector as the essential feature representation of the consultation speech block.

[0010] In the above-mentioned psychological healing robot based on user emotional state perception, the consulting voice essence capturing unit includes: a consulting voice feature difference calculating subunit, which is used to calculate the feature difference coefficient of each consulting voice block embedding coding vector in the sequence of consulting voice block embedding coding vectors relative to other consulting voice block embedding coding vectors to obtain a sequence of consulting voice block feature difference coefficients; a consulting voice feature difference coefficient normalizing subunit, which is used to normalize the sequence of consulting voice block feature difference coefficients to obtain a sequence of normalized consulting voice block feature difference coefficients; a consulting voice essential feature calculating subunit, which is used to use the sequence of normalized consulting voice block feature difference coefficients as a sequence of weights to calculate the position-weighted sum of the sequence of consulting voice block embedding coding vectors to obtain the consulting voice block essential feature representation vector.

[0011] In the above-mentioned psychological healing robot based on user emotional state perception, the consultation speech feature difference calculation subunit is used to: extract a predetermined consultation speech block embedding coding vector from the sequence of consultation speech block embedding coding vectors; calculate the positional difference between the predetermined consultation speech block embedding coding vector and each other consultation speech block embedding coding vector in the sequence of consultation speech block embedding coding vectors, and calculate the norm of each consultation speech block feature difference vector in the obtained sequence of consultation speech block feature difference vectors to obtain a sequence of consultation speech block feature norm values; calculate the global mean of the sequence of consultation speech block feature norm values ​​to obtain the consultation speech block feature difference coefficient.

[0012] In the above-mentioned psychological healing robot based on user emotional state perception, the attention allocation unit includes: a consulting speech block semantic contribution coefficient calculation subunit, which is used to calculate the semantic contribution coefficient between each consulting speech block embedding coding vector in the sequence of the consulting speech block embedding coding vector and the essential feature representation vector of the consulting speech block to obtain a sequence of semantic contribution coefficients of the consulting speech block; a consulting speech feature hierarchical modulation subunit, which is used to hierarchically modulate the sequence of the consulting speech block embedding coding vector based on the sequence of the consulting speech block semantic contribution coefficients to widen the semantic essential expression difference between each consulting speech block embedding coding vector in the sequence of the consulting speech block embedding coding vector to obtain a sequence of modulated consulting speech block embedding coding vectors; a target object-consulting speech block query coding subunit, which is used to embed the target object emotion type anchor coding vector and the modulated consulting speech block embedding coding vector in the sequence The embedded coding vectors of each modulated consulting speech block are input into the cross-domain query coding attention network to obtain a sequence of target object-consulting speech block cross-domain query attention coefficients; the target object-consulting speech block attention weight conversion subunit is used to input the sequence of target object-consulting speech block cross-domain query attention coefficients into the weight conversion network based on the Sigmoid activation function to obtain a sequence of target object-consulting speech block cross-domain query attention weights; the target object-consulting speech block feature weighting subunit is used to use each target object-consulting speech block cross-domain query attention weight in the sequence of target object-consulting speech block cross-domain query attention weights as a weight, and weight each consulting speech block embedded coding vector in the sequence of consulting speech block embedded coding vectors by position to obtain a sequence of target object emotion modulation consulting speech block embedded coding vectors as a sequence of target object emotion modulation consulting speech block embedding representations.

[0013] In the above-mentioned psychological healing robot based on user emotional state perception, the consultation voice feature hierarchical modulation subunit is used to: compare the semantic contribution coefficients of each consultation voice block in the sequence of the semantic contribution coefficients of the consultation voice block with a preset threshold to obtain a sequence of the consultation voice block embedding coding vector after modulation; wherein, in response to the semantic contribution coefficient of the consultation voice block being greater than the second preset threshold, the vector obtained by positionally multiplying the consultation voice block embedding coding vector by constant two is used as the consultation voice block embedding coding vector after modulation; in response to the semantic contribution coefficient of the consultation voice block being greater than the first preset threshold and less than or equal to the second preset threshold, the original consultation voice block embedding coding vector is used as the consultation voice block embedding coding vector after modulation; in response to the semantic contribution coefficient of the consultation voice block being less than or equal to the first preset threshold, the vector obtained by positionally dividing the consultation voice block embedding coding vector by constant two is used as the consultation voice block embedding coding vector after modulation, wherein the second preset threshold is twice the first preset threshold.

[0014] In the above-mentioned psychological healing robot based on user emotional state perception, the target object-consulting speech block query coding subunit is used to: multiply the target object emotion type anchor coding vector with the transposed vector of each modulated consultation speech block embedded coding vector in the sequence of the modulated consultation speech block embedded coding vector to obtain a sequence of target object-consulting speech block association values; calculate the bi-norm of the target object emotion type anchor coding vector to obtain the target object emotion type anchor norm value; calculate the bi-norm of each modulated consultation speech block embedded coding vector in the sequence of the modulated consultation speech block embedded coding vector to obtain a sequence of modulated consultation speech block norm values; calculate the positional multiplication between the target object emotion type anchor norm value and the sequence of the modulated consultation speech block norm value to obtain a sequence of target object-consulting speech block joint values; divide the sequence of the target object-consulting speech block association values ​​and the sequence of the target object-consulting speech block joint values ​​by each group of corresponding target object-consulting speech block association values ​​and the target object-consulting speech block joint values ​​to obtain a sequence of target object-consulting speech block cross-domain query attention coefficients.

[0015] In the above-mentioned psychological healing robot based on user emotional state perception, the psychological healing reply speech generation module is used to: embed the target object emotion modulation consultation speech block into the sequence of coding vectors and input it into the end-to-end speech big model to obtain the psychological healing reply speech.

[0016] Compared with the prior art, the present application provides a psychological healing robot based on user emotional state perception, which collects the consulting voice input signal of the target user object through a sound sensor, and collects the face image of the target user object through a camera, and uses artificial intelligence-based signal processing and image analysis methods to extract facial features and perform emotion recognition analysis on the face image, and embeds the consulting voice input signal to encode, so as to intelligently generate psychological healing reply voice according to the cross-domain query modulation representation between the target object's emotional type characteristics and the embedded features of each consulting voice block. In this way, by simultaneously collecting voice signals and face images, the system can capture more emotional expression information to more accurately reflect the user's emotional state. In addition, it can better capture individual differences and generate more personalized psychological healing reply voices to better meet the needs of different users. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0018] Figure 1 This is a block diagram of a psychological healing robot based on user emotional state perception according to an embodiment of the present application.

[0019] Figure 2 This is a data flow diagram of a psychological healing robot based on user emotional state perception according to an embodiment of the present application.

[0020] Figure 3 This is a block diagram of a facial emotion recognition module in a psychological healing robot based on user emotional state perception according to an embodiment of the present application.

[0021] Figure 4 This is a block diagram of a cross-domain query coding optimization module in a psychological healing robot based on user emotional state perception according to an embodiment of the present application.

[0022] Figure 5 This is a block diagram of an attention allocation unit in a psychological healing robot based on user emotional state perception according to an embodiment of the present application. DETAILED DESCRIPTION

[0023] The embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, but rather these embodiments are provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0024] It should be noted that in this application, all actions to obtain signals, information or data are carried out in compliance with the relevant data protection laws and policies of the country where the application is located and with the authorization of the owner of the corresponding device.

[0025] With the accelerated pace of life and increased pressure in modern society, more and more people are facing mental health problems. Factors such as fierce job competition, high cost of living, and complex interpersonal relationships have caused many people to feel anxious, depressed, and overly stressed. Long-term psychological stress not only affects an individual's work and quality of life, but also causes various physical and mental illnesses, such as insomnia and decreased immune system function. Therefore, timely and effective psychological support and intervention have become particularly important to help people better cope with stress and maintain mental health.

[0026] Traditional psychotherapy systems mainly rely on voice signals for emotion recognition, however, voice signals can only capture part of the emotional information. In fact, micro-expressions and facial expressions also play a vital role in emotional expression. If relying solely on voice signals, the system often finds it difficult to fully capture these subtle emotional changes, which affects the accuracy and comprehensiveness of emotion recognition. In addition, traditional psychotherapy emotion recognition methods are usually generic and lack optimization for individual differences. Everyone's emotional expression and mental state are unique, and general models have difficulty adapting to this diversity. For example, some users may express emotions mainly through changes in tone of voice, while others rely more on facial expressions. Because general models cannot capture these individual differences, the accuracy of emotion recognition and the level of personalized support are insufficient.

[0027] Therefore, in response to the above technical problems, the technical concept of the present application is to collect the consultation voice input signal of the target user object through a sound sensor, and collect the face image of the target user object through a camera, and use artificial intelligence-based signal processing and image analysis methods to extract facial features and perform emotion recognition analysis on the face image, and embed the consultation voice input signal to encode it, so as to intelligently generate psychological healing reply voice according to the cross-domain query modulation representation between the target object's emotional type characteristics and the embedded features of each consultation voice block. In this way, by collecting voice signals and face images at the same time, the system can capture more emotional expression information to more accurately reflect the user's emotional state. In addition, it can better capture individual differences and generate more personalized psychological healing reply voices to better meet the needs of different users.

[0028] Figure 1 This is a block diagram of a psychological healing robot based on user emotional state perception according to an embodiment of the present application. Figure 2 Schematic diagram of data flow of a psychological healing robot based on user emotional state perception according to an embodiment of the present application. Figure 1 and Figure 2 As shown, according to the embodiment of the present application, the psychological healing robot 100 based on user emotional state perception includes: a consultation voice signal collection and acquisition module 110, which is used to collect the consultation voice input signal of the target user object through a sound sensor; a face image acquisition module 120, which is used to collect the face image of the target user object through a camera; a face emotion recognition module 130, which is used to perform face feature extraction and emotion recognition on the face image of the target user object to obtain the target object emotion type anchor representation; a consultation voice block embedding module 140, which is used to embed the consultation voice input signal to obtain a sequence of consultation voice block embedding representations; a cross-domain query coding optimization module 150, which is used to perform cross-domain query coding optimization on the target object emotion type anchor representation and the sequence of the consultation voice block embedding representation to obtain a sequence of target object emotion modulated consultation voice block embedding representations; a psychological healing reply voice generation module 160, which is used to obtain a psychological healing reply voice based on the sequence of target object emotion modulated consultation voice block embedding representations.

[0029] In the embodiment of the present application, the consulting voice signal acquisition module 110 is used to acquire the consulting voice input signal of the target user object through a sound sensor. It should be understood that the consulting voice input signal contains the specific problems or troubles encountered by the user, such as work pressure, interpersonal relationship conflicts, family problems, etc. Based on this, in order to help the robot understand the context of the conversation and provide targeted advice and support, in the technical solution of the present application, the consulting voice input signal of the target user object is acquired through a sound sensor.

[0030] In an embodiment of the present application, the facial image acquisition module 120 is used to collect the facial image of the target user object through a camera. Accordingly, it is considered that although voice can convey a lot of emotional information, some emotional states may not be fully expressed through voice. The facial image can display the user's facial expressions, such as smiling, frowning, surprise, etc. These expressions can directly reflect the user's emotional state, thereby providing rich facial expression information. Based on this, in the technical solution of the present application, the facial image of the target user object is collected by a camera, and this information is combined with the voice signal, which can make the robot's response more natural and humane, thereby providing more personalized services and enhancing user satisfaction.

[0031] In the embodiment of the present application, the facial emotion recognition module 130 is used to perform facial feature extraction and emotion recognition on the facial image of the target user object to obtain an anchor representation of the target object's emotion type. Figure 3 FIG. 1 is a block diagram of a facial emotion recognition module in a psychological healing robot based on user emotion state perception according to an embodiment of the present application. Specifically, Figure 3As shown, the facial emotion recognition module 130 includes: a target object facial feature extraction unit 131, which is used to input the facial image of the target user object into a facial feature extractor based on a deep separable convolutional neural network model to obtain a target object facial semantic coding feature map; a target object emotion recognition unit 132, which is used to input the target object facial semantic coding feature map into an emotion recognizer based on a classifier to obtain a target object emotion type anchor coding vector as the target object emotion type anchor representation. It should be understood that considering that the facial image of the target user object contains small and implicit facial semantic feature information, such as facial expressions, micro-expressions, eye contact, etc. And considering that the deep separable convolutional neural network model decomposes the traditional convolution operation into deep convolution and point-by-point convolution, it can effectively capture different levels of features in the facial image, including low-level texture and high-level expression emotion semantic information. Based on this, in the technical solution of the present application, the face image of the target user object is input into the face feature extractor based on the deep separable convolutional neural network model to capture and mine high-level semantic features through multi-layer convolution operations to obtain the target object face semantic coding feature map. Accordingly, in order to convert complex facial features into specific emotion labels or emotion type representations to provide more refined emotion labels, so that the robot can more accurately evaluate the user's emotional state, thereby providing more personalized support and suggestions, in the technical solution of the present application, the target object face semantic coding feature map is input into the classifier-based emotion recognizer to obtain the target object emotion type anchor coding vector. That is, the classifier-based emotion recognizer calculates the probability distribution of the target object face semantic coding feature map on each emotion category through forward propagation, and composes the probability distribution of each emotion category into the target object emotion type anchor coding vector to better characterize the emotional information expressed in the face of the target user object, providing richer data support for the subsequent generation of psychological healing reply speech.

[0032] In an embodiment of the present application, the consulting voice block embedding module 140 is used to embed the consulting voice input signal to obtain a sequence of consulting voice block embedding representations. Specifically, in an embodiment of the present application, the consulting voice block embedding module is used to: tokenize and embed the consulting voice input signal to obtain a sequence of consulting voice block embedding coding vectors as a sequence of target object emotion modulation consulting voice block embedding representations. It should be understood that, considering that the consulting voice input signal contains the semantic information expressed by the user, each semantic information is composed of discrete units (tokens), and each unit contains semantic information between contexts. Therefore, in order to capture and mine the semantic and contextual information of each token in a more fine-grained and detailed manner, in the technical solution of the present application, the consulting voice input signal is tokenized and embedded to obtain a sequence of consulting voice block embedding coding vectors, which can more effectively capture the semantic and emotional information in the voice. In particular, in a specific embodiment of the present application, after the consultation voice input signal is tokenized to obtain multiple consultation voice block signals, it is input into an encoder including an embedding layer to obtain a sequence of consultation voice block embedded coding vectors, which can more accurately capture and represent the semantic and emotional information in the voice and improve the accuracy of emotion recognition.

[0033] In an embodiment of the present application, the cross-domain query coding optimization module 150 is used to perform cross-domain query coding optimization on the sequence of the target object emotion type anchor representation and the consulting speech block embedded representation to obtain a sequence of the target object emotion modulation consulting speech block embedded representation. Figure 4 : is a block diagram of a cross-domain query encoding optimization module in a psychological healing robot based on user emotional state perception according to an embodiment of the present application. Specifically, Figure 4As shown, the cross-domain query coding optimization module 150 includes: a consulting speech essence capturing unit 151, which is used to capture the essential features of the sequence of the consulting speech block embedding representation to obtain the essential feature representation of the consulting speech block; an attention allocation unit 152, which is used to perform hierarchical attention allocation on the sequence of the target object emotion type anchor representation and the consulting speech block embedding representation based on the essential feature representation of the consulting speech block to obtain the sequence of the target object emotion modulation consulting speech block embedding representation. Furthermore, considering that the target object emotion type anchor coding vector represents the emotion category information of the target object. The sequence of the consulting speech block embedding coding vector expresses the semantic content information in each time speech block. Therefore, in order to integrate the emotional information into the embedded coding vector of each consulting speech block. This makes each speech block contain not only the information of the speech content, but also the emotional color, so as to better reflect the emotional state of the target object. In the technical solution of the present application, the target object emotion type anchor representation and the sequence of the consulting speech block embedding representation are cross-domain query coding optimized to obtain the target object emotion modulation consulting speech block embedding representation sequence.

[0034] In detail, first, the essential features of the sequence of embedded representations of the consulting speech blocks are captured to obtain the essential feature representation of the consulting speech blocks. Specifically, the sequence of embedded coding vectors of the consulting speech blocks is input into the essential feature capture network to obtain the essential feature representation vector of the consulting speech blocks as the essential feature representation of the consulting speech blocks. That is, the essential feature capture network can extract the most core and representative features from complex speech signals, reducing the redundancy and complexity of the data. The obtained essential feature representation vector can better capture the key information of the speech content, providing a solid foundation for subsequent emotional modulation and cross-domain query.

[0035] Next, based on the essential feature representation of the consulting speech block, hierarchical attention allocation is performed on the sequence of the target object emotion type anchor representation and the consulting speech block embedding representation to obtain the sequence of the target object emotion modulated consulting speech block embedding representation, so that the features of different importance levels can be finely adjusted so that the final embedding representation more accurately reflects the emotional state. Specifically, first, the semantic contribution between each consulting speech block embedding coding vector and the essential feature representation vector of the sequence of consulting speech block embedding coding vectors is calculated to identify which consulting speech blocks are most important to the expression of the overall consulting content. These key speech blocks may contain important information or emotional expressions, and a sequence of semantic contribution coefficients of consulting speech blocks is obtained. Then, based on the sequence of semantic contribution coefficients of consulting speech blocks, each consulting speech block embedding coding vector is hierarchically modulated to finely adjust the embedding coding vector according to the importance and influence of each speech block, so that the model can focus on these key information and obtain a sequence of modulated consulting speech block embedding coding vectors. Afterwards, the modulated vector and the target object emotion type anchor encoding vector are input into the cross-domain query encoding attention network to further enhance and refine the association between the target object emotion type and the different modalities of the consulting speech block, so as to generate a more accurate and expressive target object-consulting speech block cross-domain query attention coefficient sequence. Afterwards, in order to convert the attention coefficient into a smoother and more standardized weight value so that its output is limited to the interval (0, 1) to better reflect the importance of each consulting speech block in expressing the target object emotion, a weight conversion network based on the Sigmoid activation function is used to convert the sequence of attention coefficients to obtain the target object-consulting speech block cross-domain query attention weight sequence. Finally, each consulting speech block embedding encoding vector is weighted by each attention weight to obtain a sequence of target object emotion modulation consulting speech block embedding representations containing richer and more accurate information.

[0036] Specifically, the consulting speech essence capturing unit 151 is used to: input the sequence of the consulting speech block embedded coding vector into the essential feature capturing network to obtain the consulting speech block essential feature representation vector as the essential feature representation of the consulting speech block.

[0037] Specifically, in an embodiment of the present application, the consulting speech essence capturing unit includes: a consulting speech feature difference calculating subunit, which is used to calculate the feature difference coefficients of each consulting speech block embedding coding vector in the sequence of the consulting speech block embedding coding vector relative to other consulting speech block embedding coding vectors to obtain a sequence of consulting speech block feature difference coefficients; a consulting speech feature difference coefficient normalizing subunit, which is used to normalize the sequence of the consulting speech block feature difference coefficients to obtain a sequence of normalized consulting speech block feature difference coefficients; a consulting speech essential feature calculating subunit, which is used to use the sequence of the normalized consulting speech block feature difference coefficients as a sequence of weights to calculate the position-weighted sum of the sequence of the consulting speech block embedding coding vectors to obtain the essential feature representation vector of the consulting speech block. More specifically, in an embodiment of the present application, the consulting speech feature difference calculation subunit is used to: extract a predetermined consulting speech block embedded coding vector from the sequence of consulting speech block embedded coding vectors; calculate the positional difference between the predetermined consulting speech block embedded coding vector and each other consulting speech block embedded coding vector in the sequence of consulting speech block embedded coding vectors, and calculate the norm of each consulting speech block feature difference vector in the obtained sequence of consulting speech block feature difference vectors to obtain a sequence of consulting speech block feature norm values; calculate the global mean of the sequence of consulting speech block feature norm values ​​to obtain the consulting speech block feature difference coefficient.

[0038] More specifically, in an embodiment of the present application, the consultation speech feature difference coefficient normalization subunit is used to: calculate an exponential function value with the natural constant e as the base and the negative of each consultation speech block feature difference coefficient in the sequence of consultation speech block feature difference coefficients as the exponential power to obtain a sequence of consultation speech block feature difference indices; calculate the position-by-position sum of the sequence of consultation speech block feature difference indices to obtain a total sum of consultation speech block feature difference indices; and divide the sequence of consultation speech block feature difference indices by the total sum of consultation speech block feature difference indices by position to obtain a sequence of normalized consultation speech block feature difference coefficients.

[0039] Figure 5 : is a block diagram of an attention allocation unit in a psychological healing robot based on user emotional state perception according to an embodiment of the present application. Specifically, Figure 5As shown, the attention allocation unit 152 includes: a consulting speech block semantic contribution coefficient calculation subunit 1521, which is used to calculate the semantic contribution coefficient between each consulting speech block embedded coding vector in the sequence of the consulting speech block embedded coding vector and the essential feature representation vector of the consulting speech block to obtain a sequence of semantic contribution coefficients of the consulting speech block; a consulting speech feature hierarchical modulation subunit 1522, which is used to hierarchically modulate the sequence of the consulting speech block embedded coding vector based on the sequence of the consulting speech block semantic contribution coefficients to widen the semantic essential expression difference between each consulting speech block embedded coding vector in the sequence of the consulting speech block embedded coding vector to obtain a sequence of modulated consulting speech block embedded coding vectors; a target object-consulting speech block query coding subunit 1523, which is used to embed the target object emotion type anchor coding vector and each modulated consulting speech block embedded coding vector in the sequence of the modulated consulting speech block The consulting speech block embedding coding vector is input into the cross-domain query coding attention network to obtain a sequence of target object-consulting speech block cross-domain query attention coefficients; the target object-consulting speech block attention weight conversion subunit 1524 is used to input the sequence of target object-consulting speech block cross-domain query attention coefficients into the weight conversion network based on the Sigmoid activation function to obtain a sequence of target object-consulting speech block cross-domain query attention weights; the target object-consulting speech block feature weighting subunit 1525 is used to use each target object-consulting speech block cross-domain query attention weight in the sequence of target object-consulting speech block cross-domain query attention weights as a weight, and weight each consulting speech block embedding coding vector in the sequence of consulting speech block embedding coding vectors by position to obtain a sequence of target object emotion modulation consulting speech block embedding coding vectors as a sequence of target object emotion modulation consulting speech block embedding representations.

[0040] Specifically, in an embodiment of the present application, the consulting speech block semantic contribution coefficient calculation subunit is used to: calculate the Mahalanobis distance between each consulting speech block embedding coding vector in the sequence of the consulting speech block embedding coding vector and the essential feature representation vector of the consulting speech block as the semantic contribution coefficient to obtain a sequence of the semantic contribution coefficients of the consulting speech block.

[0041] Specifically, in an embodiment of the present application, the consulting speech feature hierarchical modulation subunit is used to: compare the semantic contribution coefficients of each consulting speech block in the sequence of semantic contribution coefficients of the consulting speech block with a preset threshold to obtain a sequence of embedded coding vectors of the modulated consulting speech block; wherein, in response to the semantic contribution coefficient of the consulting speech block being greater than a second preset threshold, the vector obtained by positionally multiplying the embedded coding vector of the consulting speech block with a constant of two is used as the embedded coding vector of the modulated consulting speech block; in response to the semantic contribution coefficient of the consulting speech block being greater than a first preset threshold and less than or equal to the second preset threshold, the original embedded coding vector of the consulting speech block is used as the embedded coding vector of the modulated consulting speech block; in response to the semantic contribution coefficient of the consulting speech block being less than or equal to the first preset threshold, the vector obtained by positionally dividing the embedded coding vector of the consulting speech block with a constant of two is used as the embedded coding vector of the modulated consulting speech block, wherein the second preset threshold is twice the first preset threshold.

[0042] Specifically, in an embodiment of the present application, the target object-consulting speech block query coding subunit is used to: multiply the target object emotion type anchor coding vector with the transposed vector of each modulated consulting speech block embedded coding vector in the sequence of the modulated consulting speech block embedded coding vector to obtain a sequence of target object-consulting speech block association values; calculate the binary norm of the target object emotion type anchor coding vector to obtain the target object emotion type anchor norm value; calculate the binary norm of each modulated consulting speech block embedded coding vector in the sequence of the modulated consulting speech block embedded coding vector to obtain a sequence of modulated consulting speech block norm values; calculate the positional multiplication between the target object emotion type anchor norm value and the sequence of the modulated consulting speech block norm value to obtain a sequence of target object-consulting speech block joint values; divide the sequence of the target object-consulting speech block association values ​​and the sequence of the target object-consulting speech block joint values ​​by each group of corresponding target object-consulting speech block association values ​​and the target object-consulting speech block joint values ​​to obtain a sequence of the target object-consulting speech block cross-domain query attention coefficients.

[0043] In the embodiment of the present application, specifically, the cross-domain query coding optimization is performed on the sequence of the target object emotion type anchor coding vector and the consultation voice block embedded coding vector, which can be expressed by the formula: .

[0044] in, is the sequence of embedding coding vectors of the consulting speech blocks, and are respectively the first, second, and third in the sequence of the consultation speech block embedded coding vector , and The consulting speech block embedding coding vector, represents the norm of the eigenvector, is the number of vectors in the sequence of the consulting speech block embedded coding vector minus one, yes The corresponding consultation speech block feature difference coefficient, Expressed as a natural constant The exponential function value with base , is the number of vectors in the sequence of embedding coding vectors of the consulting speech block, is the essential feature representation vector of the consulting speech block, is the transpose of a vector, yes and The inverse matrix of the covariance matrix between yes The corresponding semantic contribution coefficient of the consultation speech block, is the preset threshold, For Perform mask modulation operation, for The modulated consultation speech block is embedded with the coding vector after modulation, is vector multiplication, is the target object emotion type anchor coding vector, is the bi-norm of the vector for The corresponding target object - the cross-domain query attention coefficient of the consulting voice block, yes function, for The corresponding target object - consultation voice block cross-domain query attention weight, is a sequence of target object emotion modulation consultation speech block embedded coding vectors.

[0045] In an embodiment of the present application, the psychological healing reply speech generation module 160 is used to obtain the psychological healing reply speech based on the sequence of the target object emotion modulation consultation speech block embedding representation. Specifically, in an embodiment of the present application, the psychological healing reply speech generation module is used to: input the sequence of the target object emotion modulation consultation speech block embedding coding vector into the end-to-end speech large model to obtain the psychological healing reply speech. That is, the sequence of the target object emotion modulation consultation speech block embedding representation obtained by cross-domain query encoding using the target object emotion type anchor coding vector and the sequence of the consultation speech block embedding coding vector is used to intelligently generate the psychological healing reply speech. In this way, by simultaneously collecting voice signals and facial images, the system can capture more emotional expression information to more accurately reflect the user's emotional state. Moreover, it can better capture individual differences and generate more personalized psychological healing reply speech to better meet the needs of different users.

[0046] In particular, in a specific embodiment of the present application, the sequence of the target object emotion modulation consultation voice block embedded coding vector is input into the end-to-end voice big model to obtain the psychological healing response voice, which can be achieved in the following ways: 1. Convert the sequence of the target object emotion modulation consultation voice block embedded coding vector into a format suitable for the end-to-end voice big model. 2. Load the pre-trained end-to-end voice big model. This model is usually a deep neural network, such as Transformer, RNN or LSTM, which can process sequence data. 3. Input the prepared input vector into the model, perform forward propagation, and generate the embedded representation of the psychological healing response voice. 4. Decode the embedded representation generated by the model into an actual speech waveform. This step is usually completed using a vocoder (such as WaveNet, Tacotron 2, etc.). 5. Convert the decoded speech waveform into a playable audio file or stream through digital signal processing technology to obtain the final psychological healing response voice.

[0047] It should be understandable that, considering that in the generative regression process, when the regression weight matrix acts on the sequence of target object emotion modulation consultation speech block embedded coding vectors, the distribution diversity of the sequence of target object emotion modulation consultation speech block embedded coding vectors will lead to the uncertainty of the weight stimulus parameters of the regression weight matrix, making the posterior probability density inference of the sequence of target object emotion modulation consultation speech block embedded coding vectors through the weight matrix missing, affecting the probability reasoning regression effect of the generative regression.

[0048] Preferably, when the target object emotion type anchor coding vector represents the emotion recognition coding feature based on the image semantic feature of the facial image of the target user object, and the sequence of the consulting speech block embedded coding vector represents the embedded coding sequence feature of the consulting speech input signal, when performing cross-domain query coding optimization based on essential feature hierarchical modulation, the sequence of the target object emotion modulation consulting speech block embedded coding vector will also be based on the multi-dimensional expression semantic diversity of the target user's multimodal emotion-related source data expression, thereby, when it is input into the end-to-end speech large model for generative regression, it is expected to improve its generative probabilistic reasoning regression effect.

[0049] Based on this, the present application optimizes the sequence of target object emotion modulation consultation speech block embedded coding vectors when the sequence of target object emotion modulation consultation speech block embedded coding vectors is input into the end-to-end speech big model to obtain the psychological healing reply speech, which specifically includes the steps of: cascading the sequence of target object emotion modulation consultation speech block embedded coding vectors into a target object emotion modulation consultation speech block embedded cascade feature vector; determining the feature mean and feature variance of the target object emotion modulation consultation speech block embedded cascade feature vector, and dividing the feature mean by the feature variance to obtain the target object emotion modulation consultation speech block embedded cascade probability feature value; performing dot multiplication of the target object emotion modulation consultation speech block embedded cascade feature vector and the inverse of the maximum eigenvalue in the target object emotion modulation consultation speech block embedded cascade feature vector to obtain the target object emotion Emotion modulation consulting speech block embedding cascade probability constraint vector: after adding the target object emotion modulation consulting speech block embedding cascade probability constraint vector and the target object emotion modulation consulting speech block embedding cascade probability eigenvalue, calculate the logarithm value with base 2 to obtain the target object emotion modulation consulting speech block embedding cascade information interaction vector: after subtracting the target object emotion modulation consulting speech block embedding cascade probability constraint vector with the target object emotion modulation consulting speech block embedding cascade probability eigenvalue, calculate the inverse of each eigenvalue to obtain the target object emotion modulation consulting speech block embedding cascade sequence constraint vector: after adding the target object emotion modulation consulting speech block embedding cascade information interaction vector and the target object emotion modulation consulting speech block embedding cascade sequence constraint vector to obtain the optimized target object emotion modulation consulting speech block embedding cascade feature vector.

[0050] Here, the target object emotion modulation consultation speech block is embedded in the cascade feature vector The optimization expression is: .

[0051] in, The feature mean of the concatenated feature vector embedded in the target object emotion modulation consultation speech block, The feature variance of the concatenated feature vector embedded in the target object emotion modulation consultation speech block, embedding the cascade probability feature value of the target object emotion modulation consultation speech block, embedding the concatenated feature vectors for the target object emotion modulation consultation speech block, It means point multiplication by position. represents the inverse of the maximum eigenvalue of the target object emotion modulation consultation speech block embedded in the cascade feature vector, Embedding a cascade probability constraint vector for the target object emotion modulation consultation speech block, Indicates adding by position point. represents the logarithmic function with base 2, Embedding the cascade information interaction vector for the target object emotion modulation consultation speech block, It means to subtract by position point. To calculate the inverse of each feature in the eigenvector, Embedding a concatenated sequence constraint vector for the target object emotion modulation consultation speech block, is the optimized target object emotion modulation consultation speech block embedded cascade feature vector.

[0052] Therefore, in this case, the probability statistical characteristics of the target object emotion modulation consultation speech block embedded cascade feature vector obtained by serial cascading of the target object emotion modulation consultation speech block embedded coding vector are used to simulate the mesoscale interaction structure under probability constraints between the eigenvalue scale and the eigenvector scale of the target object emotion modulation consultation speech block embedded cascade feature vector, so as to construct a bidirectional latent variable motif based on the mesoscale short sequence relative to the probability statistical value to perform mesoscale bidirectional migration, and to perform posterior recovery based on the sequence constraints on the interactive information, thereby improving the convergence effect in the probability density domain and improving the speech quality of the psychological healing reply speech.

[0053] In summary, the psychological healing robot 100 based on user emotional state perception according to the embodiment of the present application is explained, which collects the consultation voice input signal of the target user object through a sound sensor, and collects the face image of the target user object through a camera, and uses artificial intelligence-based signal processing and image analysis methods to extract facial features and perform emotion recognition analysis on the face image, and embeds the consultation voice input signal to encode, so as to intelligently generate psychological healing reply voice according to the cross-domain query modulation representation between the target object's emotional type characteristics and the embedded features of each consultation voice block. In this way, by simultaneously collecting voice signals and face images, the system can capture more emotional expression information to more accurately reflect the user's emotional state. In addition, it can better capture individual differences and generate more personalized psychological healing reply voices to better meet the needs of different users.

[0054] As described above, the psychological healing robot 100 based on user emotional state perception according to the embodiment of the present application can be implemented in various wireless terminals, such as a server with a psychological healing algorithm based on user emotional state perception. In one possible implementation, the psychological healing robot 100 based on user emotional state perception according to the embodiment of the present application can be integrated into the wireless terminal as a software module and / or a hardware module. For example, the psychological healing robot 100 based on user emotional state perception can be a software module in the operating system of the wireless terminal, or can be an application developed for the wireless terminal; of course, the psychological healing robot 100 based on user emotional state perception can also be one of the many hardware modules of the wireless terminal.

[0055] Alternatively, in another example, the psychological healing robot 100 based on user emotional state perception and the wireless terminal may also be separate devices, and the psychological healing robot 100 based on user emotional state perception may be connected to the wireless terminal via a wired and / or wireless network and transmit interactive information in accordance with an agreed data format.

[0056] The above descriptions of various implementations of the present disclosure are exemplary and not exhaustive. They are not limited to the disclosed implementations, and many modifications and variations are obvious to those skilled in the art without departing from the scope and spirit of the described implementations.

Claims

1. A psychological healing robot based on user emotional state perception, characterized in that: include: A consultation voice signal acquisition module is used to collect the consultation voice input signal of the target user object through a sound sensor; A face image acquisition module, used to collect the face image of the target user object through a camera; A facial emotion recognition module is used to perform facial feature extraction and emotion recognition on the facial image of the target user object to obtain an anchor representation of the target object emotion type; a consulting speech block embedding module is used to embed the consulting speech input signal to obtain a sequence of consulting speech block embedding representations; a cross-domain query coding optimization module is used to perform cross-domain query coding optimization on the sequence of the target object emotion type anchor representation and the consulting speech block embedding representation to obtain a sequence of target object emotion modulated consulting speech block embedding representations, wherein the cross-domain query coding optimization module includes: a consulting speech essence capture unit, used to capture the essence characteristics of the sequence of consulting speech block embedding representations to obtain an essential feature representation of the consulting speech block; an attention allocation unit, used to perform hierarchical attention allocation on the sequence of the target object emotion type anchor representation and the consulting speech block embedding representation based on the essential feature representation of the consulting speech block to obtain a sequence of target object emotion modulated consulting speech block embedding representations; a psychological healing reply speech generation module is used to obtain a psychological healing reply speech based on the sequence of target object emotion modulated consulting speech block embedding representations.

2. The psychological healing robot based on user emotional state perception according to claim 1, characterized in that: The facial emotion recognition module includes: a target object facial feature extraction unit, which is used to input the facial image of the target user object into a facial feature extractor based on a deep separable convolutional neural network model to obtain a target object facial semantic coding feature map; a target object emotion recognition unit, which is used to input the target object facial semantic coding feature map into a classifier-based emotion recognizer to obtain a target object emotion type anchor coding vector as the target object emotion type anchor representation.

3. The psychological healing robot based on user emotional state perception according to claim 2, characterized in that: The consulting speech block embedding module is used to: perform tokenization embedding coding on the consulting speech input signal to obtain a sequence of consulting speech block embedding coding vectors as a sequence of embedding representations of the target object emotion modulation consulting speech blocks.

4. The psychological healing robot based on user emotional state perception according to claim 3 is characterized in that: The consulting speech essence capturing unit is used to: input the sequence of the consulting speech block embedded coding vector into the essential feature capturing network to obtain the consulting speech block essential feature representation vector as the essential feature representation of the consulting speech block.

5. The psychological healing robot based on user emotional state perception according to claim 4, characterized in that: The consulting speech essence capturing unit comprises: a consulting speech feature difference calculating subunit, used for calculating the feature difference coefficient of each consulting speech block embedding coding vector in the sequence of the consulting speech block embedding coding vector relative to other consulting speech block embedding coding vectors to obtain a sequence of consulting speech block feature difference coefficients; a consulting speech feature difference coefficient normalizing subunit, used for normalizing the sequence of the consulting speech block feature difference coefficients to obtain a sequence of normalized consulting speech block feature difference coefficients; a consulting speech essential feature calculating subunit, used for calculating the position-weighted sum of the sequence of the consulting speech block embedding coding vector using the sequence of the normalized consulting speech block feature difference coefficients as a sequence of weights to obtain the essential feature representation vector of the consulting speech block.

6. The psychological healing robot based on user emotional state perception according to claim 5, characterized in that: The consulting speech feature difference calculation subunit is used to: extract a predetermined consulting speech block embedding coding vector from the sequence of consulting speech block embedding coding vectors; Calculate the positional differences between the predetermined consulting speech block embedding coding vector and each other consulting speech block embedding coding vector in the sequence of the consulting speech block embedding coding vector, and calculate the norm of each consulting speech block feature difference vector in the obtained sequence of consulting speech block feature difference vectors to obtain a sequence of consulting speech block feature norm values; calculate the global mean of the sequence of consulting speech block feature norm values ​​to obtain the consulting speech block feature difference coefficient.

7. The psychological healing robot based on user emotional state perception according to claim 6, characterized in that: The attention allocation unit includes: a consulting speech block semantic contribution coefficient calculation subunit, which is used to calculate the semantic contribution coefficient between each consulting speech block embedded coding vector in the sequence of the consulting speech block embedded coding vector and the essential feature representation vector of the consulting speech block to obtain a sequence of semantic contribution coefficients of the consulting speech block; a consulting speech feature hierarchical modulation subunit, which is used to hierarchically modulate the sequence of the consulting speech block embedded coding vector based on the sequence of semantic contribution coefficients of the consulting speech block to increase the semantic essential expression difference between each consulting speech block embedded coding vector in the sequence of the consulting speech block embedded coding vector to obtain a sequence of modulated consulting speech block embedded coding vectors; a target object-consulting speech block query coding subunit, which is used to anchor the target object emotion type coding vector and each modulated consulting speech block in the sequence of the modulated consulting speech block embedded coding vector The embedded coding vector is input into the cross-domain query coding attention network to obtain a sequence of target object-consulting speech block cross-domain query attention coefficients; the target object-consulting speech block attention weight conversion subunit is used to input the sequence of target object-consulting speech block cross-domain query attention coefficients into the weight conversion network based on the Sigmoid activation function to obtain a sequence of target object-consulting speech block cross-domain query attention weights; the target object-consulting speech block feature weighting subunit is used to use each target object-consulting speech block cross-domain query attention weight in the sequence of target object-consulting speech block cross-domain query attention weights as a weight, and weight each consulting speech block embedded coding vector in the sequence of consulting speech block embedded coding vectors by position to obtain a sequence of target object emotion modulation consulting speech block embedded coding vectors as a sequence of target object emotion modulation consulting speech block embedded representations.

8. The psychological healing robot based on user emotional state perception according to claim 7, characterized in that: The consulting speech feature hierarchical modulation subunit is used to: compare the semantic contribution coefficients of each consulting speech block in the sequence of semantic contribution coefficients of the consulting speech block with a preset threshold to obtain a sequence of embedded coding vectors of the modulated consulting speech block; wherein, in response to the semantic contribution coefficient of the consulting speech block being greater than a second preset threshold, the vector obtained by positionally multiplying the embedded coding vector of the consulting speech block with a constant of two is used as the embedded coding vector of the modulated consulting speech block; in response to the semantic contribution coefficient of the consulting speech block being greater than a first preset threshold and less than or equal to the second preset threshold, the original embedded coding vector of the consulting speech block is used as the embedded coding vector of the modulated consulting speech block; in response to the semantic contribution coefficient of the consulting speech block being less than or equal to the first preset threshold, the vector obtained by positionally dividing the embedded coding vector of the consulting speech block with a constant of two is used as the embedded coding vector of the modulated consulting speech block, wherein the second preset threshold is twice the first preset threshold.

9. The psychological healing robot based on user emotional state perception according to claim 8, characterized in that: The target object-consulting speech block query coding subunit is used to: multiply the target object emotion type anchor coding vector by the transposed vector of each modulated consulting speech block embedded coding vector in the sequence of modulated consulting speech block embedded coding vectors to obtain a sequence of target object-consulting speech block association values; Calculating the binary norm of the target object emotion type anchor coding vector to obtain the target object emotion type anchor norm value; Calculating the binary norm of each modulated consultation speech block embedding coding vector in the sequence of modulated consultation speech block embedding coding vectors to obtain a sequence of modulated consultation speech block norm values; Calculate the positional multiplication between the target object emotion type anchor norm value and the sequence of modulated consultation speech block norm values ​​to obtain a sequence of target object-consultation speech block joint values; divide the sequence of target object-consultation speech block association values ​​and the target object-consultation speech block joint values ​​by each group of corresponding target object-consultation speech block association values ​​to obtain a sequence of target object-consultation speech block cross-domain query attention coefficients.

10. The psychological healing robot based on user emotional state perception according to claim 9, characterized in that: The psychological healing reply speech generation module is used to: embed the target object emotion modulation consultation speech block into a sequence of coding vectors and input it into an end-to-end speech large model to obtain the psychological healing reply speech.

Images