Millimeter wave radar-based psychological counseling multi-mode emotion recognition method and device

Abstract

The invention discloses a psychological counseling multi-modal emotion recognition method and device based on a millimeter wave radar, and the method comprises the steps: obtaining a low-frequency and high-frequency ratio feature sequence and a voice emotion feature sequence of a to-be-recognized target user, and carrying out the time sequence alignment and feature fusion, and obtaining a multi-modal feature vector set; and inputting the multi-modal feature vector set into an emotion recognition model to obtain an emotion state classification result corresponding to each multi-modal feature vector in the multi-modal feature vector set so as to form an emotion state classification result output sequence. According to the embodiment of the invention, after the current millimeter-wave radar data set and the current user voice data set of the to-be-recognized target user are collected, feature extraction, time sequence alignment and feature fusion are respectively carried out to obtain the multi-modal feature vector set, the multi-modal feature vector set is input to the emotion recognition model to obtain the recognition result, and the accuracy of emotion recognition is improved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent decision-making technology of artificial intelligence, and in particular to a multimodal emotion recognition method and device for psychological consultation based on millimeter-wave radar. Background Art

[0002] In the fields of psychological counseling and emotion recognition, there are many technologies and solutions for emotion recognition, mainly emotion recognition technology based on visual recognition, which usually relies on cameras to collect facial images or videos of users, and uses computer vision algorithms (such as facial key point detection and facial expression recognition) to analyze facial expression changes to determine emotional states. Camera vision technology can capture micro-expression changes, but its application in psychological counseling has significant limitations. The use of cameras will bring privacy concerns to visitors, affect their natural expression of emotions, and thus affect the accuracy of emotion recognition results.

[0003] At present, speech-based emotion recognition technology is also used, that is, judging the emotional state based on the extracted user voice features combined with machine learning or deep learning models. Speech emotion recognition has been applied in many fields, but it is challenging to distinguish the source of the sound in psychological counseling scenarios. During the consultation process, the voices of the counselor and the visitor need to be accurately distinguished, otherwise it may affect the accuracy of emotion analysis. Summary of the invention

[0004] The embodiments of the present invention provide a multimodal emotion recognition method and device for psychological counseling based on millimeter-wave radar, aiming to solve the problem that when the emotion recognition technology based on visual recognition is used in the field of psychological counseling in the prior art, it often relies on cameras to collect facial images or videos of users and uses computer vision algorithms to judge the emotional state of users. However, the use of cameras will bring privacy concerns to visitors, affect their natural expression of emotions, and thus affect the accuracy of emotion recognition results.

[0005] In a first aspect, an embodiment of the present invention provides a multimodal emotion recognition method for psychological counseling based on millimeter wave radar, which includes:

[0006] In response to a multimodal emotion recognition instruction, obtaining to-be-recognized user information corresponding to the multimodal emotion recognition instruction and a to-be-recognized target user corresponding to the to-be-recognized user information;

[0007] Acquire a current millimeter-wave radar data set corresponding to the target user to be identified; wherein the current millimeter-wave radar data set is composed of millimeter-wave radar data collected by the millimeter-wave radar module within the target collection time interval corresponding to the multimodal emotion recognition instruction after the millimeter-wave radar module is aimed at the target user to be identified;

[0008] Acquire a current user voice data set corresponding to the target user to be identified; wherein the current user voice data set is composed of user voice data collected by a sound pickup and processing module from the target user to be identified within a target collection time interval;

[0009] Based on a preset millimeter-wave radar feature extraction strategy, a heart rate variability data sequence corresponding to the current millimeter-wave radar data set and a low-frequency and high-frequency ratio feature sequence corresponding to the heart rate variability data sequence are acquired in time sequence;

[0010] Based on a preset speech feature extraction model, a speech emotion feature sequence corresponding to the current user speech data set is acquired in time sequence;

[0011] Based on a preset data fusion strategy, each low-frequency and high-frequency ratio feature in the low-frequency and high-frequency ratio feature sequence is time-aligned and feature-fused with each voice emotion feature in the voice emotion feature sequence to obtain a multimodal feature vector set corresponding to the target user to be identified; wherein the multimodal feature vector set includes a plurality of multimodal feature vectors, and each multimodal feature vector is composed of a low-frequency and high-frequency ratio feature that has completed time alignment and a corresponding voice emotion feature;

[0012] The multimodal feature vector set is input into a pre-trained emotion recognition model to obtain an emotion state classification result corresponding to each multimodal feature vector in the multimodal feature vector set to form an emotion state classification result output sequence.

[0013] In a second aspect, an embodiment of the present invention further provides a multimodal emotion recognition device for psychological counseling based on millimeter wave radar, which includes:

[0014] a target user information acquisition unit, configured to, in response to a multimodal emotion recognition instruction, acquire user information to be identified corresponding to the multimodal emotion recognition instruction and a target user to be identified corresponding to the user information to be identified;

[0015] A millimeter-wave radar data acquisition unit, used to acquire a current millimeter-wave radar data set corresponding to the target user to be identified; wherein the current millimeter-wave radar data set is composed of millimeter-wave radar data collected by the millimeter-wave radar module within the target collection time interval corresponding to the multimodal emotion recognition instruction after the millimeter-wave radar module is aimed at the target user to be identified;

[0016] A sound data acquisition unit, used to acquire a current user voice data set corresponding to the target user to be identified; wherein the current user voice data set is composed of user voice data collected by the sound pickup and processing module from the target user to be identified within a target collection time interval;

[0017] A low-frequency and high-frequency ratio feature acquisition unit, configured to acquire, in time sequence, a heart rate variability data sequence corresponding to the current millimeter-wave radar data set and a low-frequency and high-frequency ratio feature sequence corresponding to the heart rate variability data sequence based on a preset millimeter-wave radar feature extraction strategy;

[0018] A speech emotion feature acquisition unit, used to acquire a speech emotion feature sequence corresponding to the current user speech data set in time sequence based on a preset speech feature extraction model;

[0019] A multimodal feature fusion unit, for performing time alignment and feature fusion on each low-frequency and high-frequency ratio feature in the low-frequency and high-frequency ratio feature sequence and each voice emotion feature in the voice emotion feature sequence based on a preset data fusion strategy, so as to obtain a multimodal feature vector set corresponding to the target user to be identified; wherein the multimodal feature vector set includes a plurality of multimodal feature vectors, and each multimodal feature vector is composed of a low-frequency and high-frequency ratio feature with time alignment completed and a corresponding voice emotion feature;

[0020] The classification result output unit is used to input the multimodal feature vector set into a pre-trained emotion recognition model to obtain the emotion state classification results corresponding to each multimodal feature vector in the multimodal feature vector set to form an emotion state classification result output sequence.

[0021] In a third aspect, an embodiment of the present invention further provides a computer device, which includes a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0022] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the method described in the first aspect above can be implemented.

[0023] The embodiment of the present invention provides a method and device for multimodal emotion recognition in psychological counseling based on millimeter-wave radar, the method comprising: in response to a multimodal emotion recognition instruction, obtaining user information to be recognized corresponding to the multimodal emotion recognition instruction, and a target user to be recognized corresponding to the user information to be recognized; obtaining a current millimeter-wave radar data set corresponding to the target user to be recognized; wherein the current millimeter-wave radar data set is composed of millimeter-wave radar data collected by the millimeter-wave radar module within a target collection time interval corresponding to the multimodal emotion recognition instruction after aiming at the target user to be recognized; obtaining a current user voice data set corresponding to the target user to be recognized; wherein the current user voice data set is composed of user voice data collected by the sound pickup and processing module for the target user to be recognized within the target collection time interval; based on a preset millimeter-wave radar feature extraction strategy, obtaining the corresponding to the current millimeter-wave radar data set in time sequence A heart rate variability data sequence, and a low-frequency and high-frequency ratio feature sequence corresponding to the heart rate variability data sequence; based on a preset voice feature extraction model, a voice emotion feature sequence corresponding to the current user voice data set is obtained in time sequence; based on a preset data fusion strategy, each low-frequency and high-frequency ratio feature in the low-frequency and high-frequency ratio feature sequence is time-sequentially aligned and feature-fused with each voice emotion feature in the voice emotion feature sequence, so as to obtain a multimodal feature vector set corresponding to the target user to be identified; wherein the multimodal feature vector set includes a plurality of multimodal feature vectors, and each multimodal feature vector is composed of a low-frequency and high-frequency ratio feature that has completed time sequence alignment and a corresponding voice emotion feature; the multimodal feature vector set is input into a pre-trained emotion recognition model to obtain an emotion state classification result corresponding to each multimodal feature vector in the multimodal feature vector set, so as to form an emotion state classification result output sequence. The embodiment of the present invention can collect the current millimeter wave radar data set and the current user voice data set of the target user to be identified, respectively extract features, time sequence align and feature fuse to obtain a multimodal feature vector set, and then input it into the emotion recognition model to obtain a recognition result, thereby improving the accuracy of emotion recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0025] Figure 1 A schematic diagram of an application scenario of a multimodal emotion recognition method for psychological counseling based on millimeter-wave radar provided in an embodiment of the present invention;

[0026] Figure 2A schematic diagram of a flow chart of a multimodal emotion recognition method for psychological counseling based on millimeter wave radar provided in an embodiment of the present invention;

[0027] Figure 3 A schematic diagram of a sub-process of a multi-modal emotion recognition method for psychological counseling based on millimeter-wave radar provided in an embodiment of the present invention;

[0028] Figure 4 A schematic diagram of a sub-process of a multi-modal emotion recognition method for psychological counseling based on millimeter-wave radar provided in an embodiment of the present invention;

[0029] Figure 5 A schematic diagram of a sub-process of a multi-modal emotion recognition method for psychological counseling based on millimeter-wave radar provided in an embodiment of the present invention;

[0030] Figure 6 A schematic block diagram of a multimodal emotion recognition device for psychological counseling based on millimeter-wave radar provided in an embodiment of the present invention;

[0031] Figure 7 A schematic block diagram of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0032] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0033] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0034] It should also be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0035] It should be further understood that the term "and / or" used in the present description and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0036] Please also refer to Figure 1 and Figure 2 ,in Figure 1 Schematic diagram of a scenario of a multimodal emotion recognition method for psychological counseling based on millimeter wave radar according to an embodiment of the present invention. Figure 2 FIG. 1 is a flow chart of a multimodal emotion recognition method for psychological counseling based on millimeter wave radar provided by an embodiment of the present invention. Figure 1 As shown, the millimeter wave radar-based multimodal emotion recognition method for psychological counseling provided by the embodiment of the present invention is applied to a millimeter wave radar-based multimodal emotion recognition device for psychological counseling, wherein the millimeter wave radar-based multimodal emotion recognition device for psychological counseling includes a server 10, a millimeter wave radar module 20, and a sound pickup and processing module 30, and the millimeter wave radar module 20 and the sound pickup and processing module 30 are both communicatively connected to the server 10. Moreover, during specific implementation, the millimeter wave radar module 20 and the sound pickup and processing module 30 can be arranged in the same indoor place such as a psychological counseling room, and the server 10 can be arranged in a computer room outside the psychological counseling room (of course, in order to ensure the data transmission effect, the server 10 can also be arranged in the psychological counseling room).

[0037] like Figure 2 As shown, the method includes the following steps S110-S170.

[0038] S110 . In response to a multimodal emotion recognition instruction, obtain user information to be recognized corresponding to the multimodal emotion recognition instruction and a target user to be recognized corresponding to the user information to be recognized.

[0039] In this embodiment, when a user enters the psychological counseling room to communicate with the user in the role of counselor, the user in the role of counselor can first use his user terminal (such as a desktop computer, tablet computer, smart phone, etc.) to log in to the user interaction interface corresponding to the server, and after obtaining the user information to be identified by manual entry or based on voice recognition (such as the user information to be identified at least includes the user's unique identification code, user age, user gender, user contact number, etc.), click the start virtual button on the user interaction interface to trigger the generation of a multimodal emotion recognition instruction, and send the multimodal emotion recognition instruction to the server, and at the same time send the target user to be identified corresponding to the user information to be identified to the server to notify the target user to be identified as the current object to be identified. In the time interval between detecting the multimodal emotion recognition instruction and detecting the stop recognition instruction, the server will control the millimeter-wave radar module to collect the millimeter-wave radar data of the target user to be identified, and control the sound pickup and processing module to collect the user voice data of the target user to be identified.

[0040] S120: Acquire a current millimeter-wave radar data set corresponding to the target user to be identified.

[0041] The current millimeter-wave radar data set is composed of millimeter-wave radar data collected by the millimeter-wave radar module within the target collection time interval corresponding to the multimodal emotion recognition instruction after the millimeter-wave radar module is aimed at the target user to be identified.

[0042] In this embodiment, when determining the target collection time interval corresponding to the multimodal emotion recognition instruction, in addition to obtaining the generation time corresponding to the multimodal emotion recognition instruction, it is also necessary to obtain the generation time of the stop recognition instruction generated by the user of the consultant role clicking the end virtual button on the user interaction interface using his user terminal. Afterwards, the generation time corresponding to the multimodal emotion recognition instruction is used as the starting time point of the target collection time interval, and the generation time of the stop recognition instruction is used as the end time point of the target collection time interval.

[0043] The millimeter-wave radar module installed in the psychological counseling room is specifically aimed at the chest area of ​​the target user to be identified (in specific implementation, the millimeter-wave radar module can also be set as a clip-on card-type millimeter-wave radar module and worn on the chest of the user playing the role of the counselor. When the server is specifically a desktop computer, the millimeter-wave radar module can be connected to the server via a USB interface, and powered and transmitted data via USB), and detects tiny movements of the chest of the target user to be identified (chest fluctuations caused by heartbeat) to form a current millimeter-wave radar data set corresponding to the target user to be identified.

[0044] S130: Acquire a current user voice data set corresponding to the target user to be identified.

[0045] The current user voice data set is composed of user voice data collected by the sound pickup and processing module from the target user to be identified within a target collection time interval.

[0046] In this embodiment, the target collection time interval corresponding to the current user voice data set is exactly the same as the target collection time interval corresponding to the current millimeter-wave radar data set. The sound pickup and processing module arranged in the psychological counseling room can collect the current user voice data corresponding to the target user to be identified in the target collection time interval and form the current user voice data set.

[0047] In one embodiment, before step S130, the method further includes:

[0048] Acquire a current communication process speech data set corresponding to the target user to be identified;

[0049] Based on the pre-collected voiceprint features of the user to be filtered, the voice signal in the voice data set of the current communication process and the voiceprint features of the user to be filtered are filtered to obtain the current user voice data set.

[0050] In this embodiment, in order to make the current user voice data set only include the voice data of the target user to be identified, the sound pickup and processing module (in the specific implementation, the sound pickup and processing module can also use a microphone module) can first collect the user voice data of the consultant role as the initial sample sound data, and then perform audio noise reduction, normalization processing, time domain feature extraction (such as extracting energy features, short-time average amplitude), and frequency domain feature extraction (such as extracting linear prediction cepstral coefficients, Mel-frequency cepstral coefficients, spectrum centroid, etc.) on the initial sample sound data, and then form the voiceprint features of the user to be filtered based on the time domain features and frequency domain features corresponding to the initial sample sound data (at this time, the user to be filtered refers to the user of the consultant role).

[0051] After obtaining the current communication process speech data set including the voice data of the target user to be identified and the voice data of the user to be filtered, the speech signal in the current communication process speech data set with the voiceprint features of the user to be filtered can be matched and filtered based on the dynamic time warping (DTW) algorithm or the Gaussian mixture model (GMM), so as to obtain the current user speech data set that only includes the voice data of the target user to be identified, thereby retaining only the voice of the target user to be identified for subsequent emotion analysis.

[0052] In one embodiment, after step S130, the method further includes:

[0053] Speech recognition is performed on the current user speech data set based on a pre-trained speech recognition model to obtain current user text data including text time sequence information.

[0054] In this embodiment, the server also includes an automatic speech recognition module (also referred to as an ASR module). After the sound pickup and processing module obtains the current user speech data set corresponding to the target user to be identified, the sound pickup and processing module can send the current user speech data set to the server, and the automatic speech recognition module in the server performs speech recognition. More specifically, the speech recognition model deployed in the automatic speech recognition module (such as a hidden Markov model, a Gaussian mixture model, a deep neural network, a convolutional neural network, etc.) performs speech recognition on the current user speech data set to obtain the current user text data including text timing information. It should be noted that after each speech data in the current user speech data set is identified and converted into text, the feature of the collection time corresponding to the speech data is also inherited by the text, which is also to facilitate the subsequent timing alignment of the emotion recognition results based on the sequence arranged in time sequence.

[0055] In one embodiment, after step S130 and before step S140, the method further includes:

[0056] Get the preset sliding window period;

[0057] Dividing the current millimeter-wave radar data set based on the sliding window period to obtain a plurality of current millimeter-wave radar data subsets; wherein the plurality of current millimeter-wave radar data subsets constitute the current millimeter-wave radar data set;

[0058] The current user voice data set is divided based on the sliding window period to obtain multiple current user voice data subsets; wherein the multiple current user voice data subsets constitute the current user voice data.

[0059] In this embodiment, since the target acquisition time interval corresponding to the current user voice data set is exactly the same as the target acquisition time interval corresponding to the current millimeter-wave radar data set, the same sliding window period is used (such as setting it to 5s, 10s, 15s, 20s, etc., of course, the specific real-time performance is not limited to the time lengths listed in the above examples, and can also be flexibly set according to the actual needs of the user) to divide the current millimeter-wave radar data set and the current user voice data set respectively, and the first total number of subsets corresponding to the multiple current millimeter-wave radar data subsets is equal to the second total number of subsets corresponding to the multiple current user voice data subsets.

[0060] Afterwards, when performing millimeter-wave radar feature extraction on the current millimeter-wave radar data set, it can be decomposed into performing millimeter-wave radar feature extraction on each current millimeter-wave radar data subset, thereby forming a heart rate variability data sequence and a low-frequency and high-frequency ratio feature sequence. Similarly, when performing voice emotion recognition on the current user voice data set, it can be decomposed into performing voice emotion feature extraction on each current user voice data subset, thereby forming a voice emotion feature sequence. It can be seen that through the above method, the current millimeter-wave radar data set and the current user voice data set can be divided into equal sliding window lengths, which is convenient for subsequent time series alignment. Among them, the current millimeter-wave radar data set, the current user voice data set, and various processed data obtained subsequently can all be encrypted and stored in the server to improve data security.

[0061] S140. Based on a preset millimeter-wave radar feature extraction strategy, a heart rate variability data sequence corresponding to the current millimeter-wave radar data set and a low-frequency to high-frequency ratio feature sequence corresponding to the heart rate variability data sequence are acquired in time sequence.

[0062] In this embodiment, after the server obtains the current millimeter wave radar data set, a waveform diagram can be fitted according to the order of the acquisition time corresponding to each current millimeter wave radar data in the current millimeter wave radar data set, and the heart rate data corresponding to the target user to be identified can be extracted from the waveform diagram, and finally the heart rate variability data sequence is determined in combination with the interval between consecutive heartbeats (such as NN interval, RR interval, etc.). The heart rate variability data sequence includes multiple heart rate variability data (i.e., HRV data, the full name of HRV is Heart Rate Variability), and each heart rate variability data can be further decomposed into a low-frequency component (i.e., LF component, and the low-frequency frequency band corresponding to the low-frequency component is 0.04-0.15Hz) and a high-frequency component (i.e., HF component, and the high-frequency frequency band corresponding to the high-frequency component is 0.15-0.4Hz). By calculating the low-frequency component / high-frequency cost ratio of each heart rate variability data, the corresponding low-frequency high-frequency ratio feature can be obtained. After the heart rate variability data sequence corresponding to the current millimeter wave radar data set is obtained in time sequence, the low-frequency high-frequency ratio feature sequence of the same time sequence can also be obtained.

[0063] In one embodiment, if Figure 3 As shown, step S140 includes:

[0064] S141, acquiring a current heart rate fitting data set corresponding to the current millimeter-wave radar data according to the acquisition time sequence of each current millimeter-wave radar data subset in the current millimeter-wave radar data set;

[0065] S142, performing data cleaning on the current heart rate fitting data set based on a preset data cleaning strategy to obtain a cleaned current heart rate fitting data set;

[0066] S143, obtaining a current RR interval sequence corresponding to the cleaned current heart rate fitting data set, and obtaining the heart rate variability data sequence based on the current RR interval sequence and a preset power spectrum density estimation model;

[0067] S144. For each heart rate variability data in the heart rate variability data sequence, a ratio of low frequency component / high frequency cost is obtained as a low frequency and high frequency ratio feature, and the low frequency and high frequency ratio feature of each heart rate variability data in the heart rate variability data sequence is arranged in time series to form the low frequency and high frequency ratio feature sequence.

[0068] In this embodiment, after the current millimeter-wave radar data set is obtained, a waveform diagram can be fitted according to the acquisition time sequence corresponding to each current millimeter-wave radar data subset in the current millimeter-wave radar data set, and the current heart rate fitting data set corresponding to the target user to be identified can be extracted from the waveform diagram. Then, the current heart rate fitting data set is removed by a data cleaning strategy such as an outlier cleaning strategy to obtain the cleaned current heart rate fitting data set. After that, the time interval between two adjacent heartbeat signals is determined in sequence according to the time sequence in the cleaned current heart rate fitting data set, thereby obtaining the current RR interval sequence (wherein the RR interval refers to the time interval between two adjacent heartbeat signals), and the current RR interval sequence can be further combined with the power spectrum density estimation model to perform a fast Fourier transform on the current RR interval sequence to perform power spectrum density estimation, thereby decomposing the variability of the current RR interval sequence into the power of different frequency components.

[0069] Finally, because each heart rate variability data has been decomposed into low-frequency components (i.e., LF components, and the low-frequency frequency band corresponding to the low-frequency components is 0.04-0.15Hz) and high-frequency components (i.e., HF components, and the high-frequency frequency band corresponding to the high-frequency components is 0.15-0.4Hz) based on the power spectrum density estimation model, the corresponding low-frequency and high-frequency ratio features can be obtained by calculating the low-frequency component / high-frequency cost ratio of each heart rate variability data. After obtaining the heart rate variability data sequence corresponding to the current millimeter-wave radar data set in time sequence, the low-frequency and high-frequency ratio feature sequence of the same time sequence can also be obtained. Among them, the total number of low-frequency and high-frequency ratio features included in the low-frequency and high-frequency ratio feature sequence is the same as the number of the first total subset. It can be seen that the above method, combined with the data cleaning strategy and the power spectrum density estimation model, can quickly extract the low-frequency and high-frequency ratio feature sequence corresponding to the current millimeter-wave radar data set.

[0070] S150. Based on a preset speech feature extraction model, a speech emotion feature sequence corresponding to the current user speech data set is acquired in time sequence.

[0071] In this embodiment, since the current user voice data set includes multiple current user voice data subsets arranged in chronological order, voice emotion recognition can be performed on multiple current user voice data subsets corresponding to the current user voice data set, and after obtaining the voice emotion features corresponding to each current user voice data subset, the voice emotion feature sequence is formed. The total number of voice emotion features included in the voice emotion feature sequence is the same as the number of the first total subsets.

[0072] In one embodiment, if Figure 4 As shown, step S150 includes:

[0073] S151, for each current user voice data subset in the current user voice data set, input the current user voice data subset into the voice feature extraction model to extract voice emotion features, and obtain current voice emotion features corresponding to the current user voice data subset; wherein the current voice emotion features at least include user pitch feature data, user sound intensity feature data, and user speech speed feature data;

[0074] S152, arranging the current voice emotion features corresponding to each current user voice data subset in the current user voice data set in time sequence to form the voice emotion feature sequence.

[0075] In this embodiment, the speech feature extraction model preset in the server at least includes a speech noise reduction processing submodel, a speech framing submodel, a pitch feature extraction submodel, a sound intensity feature extraction submodel and a speech rate feature extraction submodel. When the current user speech data subset is input into the speech feature extraction model, the current user speech data subset is obtained after the noise reduction processing (such as adaptive filtering noise reduction processing) of the extraction submodel and the framing processing (such as first determining the frame length and frame shift, and then performing a windowing operation to achieve framing processing of the speech signal) of the speech framing submodel. Then, the pre-processed current user speech data subset is respectively extracted based on the pitch feature extraction submodel (such as a harmonic peak detection model), the sound intensity feature extraction submodel and the speech rate feature extraction submodel (such as a phoneme interval statistical model), and the current speech emotion feature including the user pitch feature data, the user sound intensity feature data and the user speech rate feature data can be obtained. It can be seen that based on the above method, the speech feature extraction model can quickly extract the speech emotion feature sequence corresponding to the current user speech data set.

[0076] S160. Based on a preset data fusion strategy, time-series alignment and feature fusion are performed on each low-frequency and high-frequency ratio feature in the low-frequency and high-frequency ratio feature sequence and each voice emotion feature in the voice emotion feature sequence to obtain a multimodal feature vector set corresponding to the target user to be identified.

[0077] The multimodal feature vector set includes a plurality of multimodal feature vectors, and each multimodal feature vector is composed of a low-frequency and high-frequency ratio feature with time alignment and a corresponding speech emotion feature.

[0078] In this embodiment, after the low-frequency and high-frequency ratio feature sequence and the voice emotion feature sequence are obtained in the server, in order to make the obtained multimodal feature vector set also have time series characteristics, the low-frequency and high-frequency ratio features in the low-frequency and high-frequency ratio feature sequence and the voice emotion features in the voice emotion feature sequence can be time-aligned based on a preset data fusion strategy, and then feature fusion is performed to obtain a multimodal feature vector set corresponding to the target user to be identified.

[0079] In one embodiment, if Figure 5 As shown, step S160 includes:

[0080] S161, obtaining a first timestamp for each low-frequency and high-frequency ratio feature in the low-frequency and high-frequency ratio feature sequence, and obtaining a second timestamp for each voice emotion feature in the voice emotion feature sequence;

[0081] S162, aligning the low-frequency and high-frequency ratio features with the same first timestamp and second timestamp with the speech emotion feature and performing feature fusion to form a multimodal feature vector;

[0082] S163. Arrange the multiple multimodal feature vectors in chronological order to form the multimodal feature vector set.

[0083] In this embodiment, when each low-frequency and high-frequency ratio feature in the low-frequency and high-frequency ratio feature sequence is time-aligned with each voice emotion feature in the voice emotion feature sequence, it is necessary to obtain the first timestamp corresponding to each low-frequency and high-frequency ratio feature in the low-frequency and high-frequency ratio feature sequence, and obtain the second timestamp corresponding to each voice emotion feature in the voice emotion feature sequence. For example, the first low-frequency and high-frequency ratio feature in the low-frequency and high-frequency ratio feature sequence corresponds to the first timestamp T10, and the first voice emotion feature in the voice emotion feature sequence corresponds to the second timestamp T20. When the first timestamp T10 is equal to the second timestamp T20, it can be regarded that the two features are time-aligned, and the first low-frequency and high-frequency ratio feature in the low-frequency and high-frequency ratio feature sequence and the first voice emotion feature in the voice emotion feature sequence can also be feature-fused to form a multimodal feature vector corresponding to the first timestamp T10 or the second timestamp T20. Afterwards, when other low-frequency and high-frequency ratio features in the low-frequency and high-frequency ratio feature sequence are time-aligned with the corresponding voice emotion features in the voice emotion feature sequence, the above process is also referred to. Since the low-frequency and high-frequency ratio feature includes the low-frequency and high-frequency ratio, and the voice emotion feature includes the user pitch feature data, the user sound intensity feature data and the user speech rate feature data, the obtained multimodal feature vector includes the low-frequency and high-frequency ratio, the user pitch feature data, the user sound intensity feature data and the user speech rate feature data. Moreover, the multiple multimodal feature vectors included in the multimodal feature vector set corresponding to the target user to be identified are also arranged in sequence according to the time sequence of the timestamp. It can be seen that through the above method, the low-frequency and high-frequency ratio feature sequence can be quickly aligned and fused with the voice emotion feature sequence to obtain multiple multimodal feature vectors arranged in time sequence.

[0084] S170, inputting the multimodal feature vector set into a pre-trained emotion recognition model to obtain an emotion state classification result corresponding to each multimodal feature vector in the multimodal feature vector set, so as to form an emotion state classification result output sequence.

[0085] In this embodiment, the emotion recognition model adopted is a long short-term memory network integrated with a self-attention mechanism or a multimodal convolutional neural network integrated with a self-attention mechanism. The multiple multimodal feature vectors included in the multimodal feature vector set also have the temporal sorting characteristics of the time series. Each of the multimodal feature vectors in the multimodal feature vector set is input into the pre-trained emotion recognition model in sequence according to the temporal sorting order, and the emotion state classification results corresponding to each multimodal feature vector can be obtained. The above-mentioned multiple emotion state classification results are combined in chronological order to obtain an emotion state classification result output sequence. Specifically, each of the emotion state classification results corresponding to the multimodal feature vectors corresponds to a 1*2 vector, and the two vector values ​​included therein are probability values ​​ranging from 0 to 1. The first probability value can determine the emotion category according to the specific value interval to which it belongs, and the second probability value can determine the emotion intensity according to the specific value interval to which it belongs; for example, the emotion state classification result is [0.3, 0.8], and the emotion category is determined to be anxiety based on the value interval to which the first probability value 0.3 in the emotion state classification result belongs, and the emotion intensity is determined to be high intensity based on the value interval to which the second probability value 0.8 in the emotion state classification result belongs. Because the emotion state classification results in the emotion state classification result output sequence are also arranged in chronological order, an emotion state diagram with the time axis as the horizontal axis and the probability values ​​corresponding to the emotion categories in the emotion state classification results as the vertical axis can also be constructed to characterize the emotional fluctuation state of the target user to be identified. Of course, after performing speech recognition on the current user speech data set and obtaining the current user text data including text timing information, the current user text data can also be integrated and displayed based on the timeline of the emotional state diagram, that is, each text record is accompanied by a timestamp and displayed together with the corresponding emotional state, so that users in the role of consultants can view the communication text content and emotional state at different times. When the emotional category corresponding to a time point in the emotional state diagram changes greatly, for example, from relaxation to anger, abnormal fluctuation prompt information can be automatically generated at this time, and the abnormal fluctuation prompt information can be displayed on the display device corresponding to the server.

[0086] Among them, when the emotion recognition model adopts the long short-term memory network integrated with the self-attention mechanism, it can focus more on capturing the dynamic changes of the low-frequency and high-frequency ratio and the voice emotion features; and when the emotion recognition model adopts the multimodal convolutional neural network integrated with the self-attention mechanism, it can perform more refined fusion at the feature level. Moreover, the self-attention mechanism adopted is to dynamically adjust the weight of each feature in the multimodal feature vector when each multimodal feature vector is input into the emotion recognition model to cope with the difference in the importance of features under different emotional states. For example, when the low-frequency and high-frequency ratio in the multimodal feature vector increases significantly, the weight of the low-frequency and high-frequency ratio feature will be automatically increased; when the change of voice features is more significant, the weight of voice emotion features will be increased.

[0087] Moreover, when the emotion state classification result corresponding to each multimodal feature vector is obtained based on the emotion recognition model in chronological order, specifically, an emotion state classification sub-result may be output for the low-frequency and high-frequency ratio feature in the multimodal feature vector, or an emotion state classification sub-result may be output for the voice emotion feature in the multimodal feature vector, and the above two emotion state classification sub-results are weighted and summed to obtain a comprehensive emotion state classification result. However, it should be noted that when there is a significant difference between outputting an emotion state classification sub-result for the low-frequency and high-frequency ratio feature in the multimodal feature vector and outputting an emotion state classification sub-result for the voice emotion feature in the multimodal feature vector, the weighted summation can balance the emotion judgment result.

[0088] It can be seen that the embodiment of the method can collect the current millimeter-wave radar data set and the current user voice data set of the target user to be identified, and then perform feature extraction, time alignment and feature fusion to obtain a multimodal feature vector set, and then input it into the emotion recognition model to obtain the recognition result, thereby improving the accuracy of emotion recognition.

[0089] Figure 6 is a schematic block diagram of a multi-modal emotion recognition device for psychological counseling based on millimeter wave radar provided by an embodiment of the present invention. Figure 6 As shown, corresponding to the above-mentioned psychological counseling multimodal emotion recognition method based on millimeter wave radar, the present invention also provides a psychological counseling multimodal emotion recognition device 100 based on millimeter wave radar. The psychological counseling multimodal emotion recognition device 100 based on millimeter wave radar includes a unit for executing the above-mentioned psychological counseling multimodal emotion recognition method based on millimeter wave radar. Figure 6The millimeter-wave radar-based psychological counseling multimodal emotion recognition device 100 includes: a target user information acquisition unit 110, a millimeter-wave radar data acquisition unit 120, a sound data acquisition unit 130, a low-frequency and high-frequency ratio feature acquisition unit 140, a voice emotion feature acquisition unit 150, a multimodal feature fusion unit 160 and a classification result output unit 170.

[0090] The target user information acquisition unit 110 is used to respond to the multimodal emotion recognition instruction and acquire the user information to be identified corresponding to the multimodal emotion recognition instruction and the target user to be identified corresponding to the user information to be identified.

[0091] In this embodiment, when a user enters the psychological counseling room to communicate with the user in the role of counselor, the user in the role of counselor can first use his user terminal (such as a desktop computer, tablet computer, smart phone, etc.) to log in to the user interaction interface corresponding to the server, and after obtaining the user information to be identified by manual entry or based on voice recognition (such as the user information to be identified at least includes the user's unique identification code, user age, user gender, user contact number, etc.), click the start virtual button on the user interaction interface to trigger the generation of a multimodal emotion recognition instruction, and send the multimodal emotion recognition instruction to the server, and at the same time send the target user to be identified corresponding to the user information to be identified to the server to notify the target user to be identified as the current object to be identified. In the time interval between detecting the multimodal emotion recognition instruction and detecting the stop recognition instruction, the server will control the millimeter-wave radar module to collect the millimeter-wave radar data of the target user to be identified, and control the sound pickup and processing module to collect the user voice data of the target user to be identified.

[0092] The millimeter wave radar data acquisition unit 120 is used to acquire a current millimeter wave radar data set corresponding to the target user to be identified.

[0093] The current millimeter-wave radar data set is composed of millimeter-wave radar data collected by the millimeter-wave radar module within the target collection time interval corresponding to the multimodal emotion recognition instruction after the millimeter-wave radar module is aimed at the target user to be identified.

[0094] In this embodiment, when determining the target collection time interval corresponding to the multimodal emotion recognition instruction, in addition to obtaining the generation time corresponding to the multimodal emotion recognition instruction, it is also necessary to obtain the generation time of the stop recognition instruction generated by the user of the consultant role clicking the end virtual button on the user interaction interface using his user terminal. Afterwards, the generation time corresponding to the multimodal emotion recognition instruction is used as the starting time point of the target collection time interval, and the generation time of the stop recognition instruction is used as the end time point of the target collection time interval.

[0095] The millimeter-wave radar module installed in the psychological counseling room is specifically aimed at the chest area of ​​the target user to be identified (in specific implementation, the millimeter-wave radar module can also be set as a clip-on card-type millimeter-wave radar module and worn on the chest of the user playing the role of the counselor. When the server is specifically a desktop computer, the millimeter-wave radar module can be connected to the server via a USB interface, and powered and transmitted data via USB), and detects tiny movements of the chest of the target user to be identified (chest fluctuations caused by heartbeat) to form a current millimeter-wave radar data set corresponding to the target user to be identified.

[0096] The voice data acquisition unit 130 is used to acquire a current user voice data set corresponding to the target user to be identified.

[0097] The current user voice data set is composed of user voice data collected by the sound pickup and processing module from the target user to be identified within a target collection time interval.

[0098] In this embodiment, the target collection time interval corresponding to the current user voice data set is exactly the same as the target collection time interval corresponding to the current millimeter-wave radar data set. The sound pickup and processing module arranged in the psychological counseling room can collect the current user voice data corresponding to the target user to be identified in the target collection time interval and form the current user voice data set.

[0099] In one embodiment, the millimeter wave radar-based psychological counseling multimodal emotion recognition device 100 further includes:

[0100] A current communication process speech data set acquisition unit, used to acquire a current communication process speech data set corresponding to the target user to be identified;

[0101] The voice signal filtering unit is used to filter the voice signal of the voice data set of the current communication process and the voice print feature of the user to be filtered based on the pre-collected voice print feature of the user to be filtered, so as to obtain the current user voice data set.

[0102] In this embodiment, in order to make the current user voice data set only include the voice data of the target user to be identified, the sound pickup and processing module (in the specific implementation, the sound pickup and processing module can also use a microphone module) can first collect the user voice data of the consultant role as the initial sample sound data, and then perform audio noise reduction, normalization processing, time domain feature extraction (such as extracting energy features, short-time average amplitude), and frequency domain feature extraction (such as extracting linear prediction cepstral coefficients, Mel-frequency cepstral coefficients, spectrum centroid, etc.) on the initial sample sound data, and then form the voiceprint features of the user to be filtered based on the time domain features and frequency domain features corresponding to the initial sample sound data (at this time, the user to be filtered refers to the user of the consultant role).

[0103] After obtaining the current communication process speech data set including the voice data of the target user to be identified and the voice data of the user to be filtered, the speech signal in the current communication process speech data set with the voiceprint features of the user to be filtered can be matched and filtered based on the dynamic time warping (DTW) algorithm or the Gaussian mixture model (GMM), so as to obtain the current user speech data set that only includes the voice data of the target user to be identified, thereby retaining only the voice of the target user to be identified for subsequent emotion analysis.

[0104] In one embodiment, the millimeter wave radar-based psychological counseling multimodal emotion recognition device 100 further includes:

[0105] The speech-to-text recognition unit is used to perform speech recognition on the current user speech data set based on a pre-trained speech recognition model to obtain current user text data including text time sequence information.

[0106] In this embodiment, the server also includes an automatic speech recognition module (also referred to as an ASR module). After the sound pickup and processing module obtains the current user speech data set corresponding to the target user to be identified, the sound pickup and processing module can send the current user speech data set to the server, and the automatic speech recognition module in the server performs speech recognition. More specifically, the speech recognition model deployed in the automatic speech recognition module (such as a hidden Markov model, a Gaussian mixture model, a deep neural network, a convolutional neural network, etc.) performs speech recognition on the current user speech data set to obtain the current user text data including text timing information. It should be noted that after each speech data in the current user speech data set is identified and converted into text, the feature of the collection time corresponding to the speech data is also inherited by the text, which is also to facilitate the subsequent timing alignment of the emotion recognition results based on the sequence arranged in time sequence.

[0107] In one embodiment, the millimeter wave radar-based psychological counseling multimodal emotion recognition device 100 further includes:

[0108] A sliding window period acquisition unit, used to acquire a preset sliding window period;

[0109] A first division unit is used to divide the current millimeter-wave radar data set based on the sliding window period to obtain a plurality of current millimeter-wave radar data subsets; wherein the plurality of current millimeter-wave radar data subsets constitute the current millimeter-wave radar data set;

[0110] The second division unit is used to divide the current user voice data set based on the sliding window period to obtain multiple current user voice data subsets; wherein the multiple current user voice data subsets constitute the current user voice data.

[0111] In this embodiment, since the target acquisition time interval corresponding to the current user voice data set is exactly the same as the target acquisition time interval corresponding to the current millimeter-wave radar data set, the same sliding window period is used (such as setting it to 5s, 10s, 15s, 20s, etc., of course, the specific real-time performance is not limited to the time lengths listed in the above examples, and can also be flexibly set according to the actual needs of the user) to divide the current millimeter-wave radar data set and the current user voice data set respectively, and the first total number of subsets corresponding to the multiple current millimeter-wave radar data subsets is equal to the second total number of subsets corresponding to the multiple current user voice data subsets.

[0112] Afterwards, when performing millimeter-wave radar feature extraction on the current millimeter-wave radar data set, it can be decomposed into performing millimeter-wave radar feature extraction on each current millimeter-wave radar data subset, thereby forming a heart rate variability data sequence and a low-frequency and high-frequency ratio feature sequence. Similarly, when performing voice emotion recognition on the current user voice data set, it can be decomposed into performing voice emotion feature extraction on each current user voice data subset, thereby forming a voice emotion feature sequence. It can be seen that through the above method, the current millimeter-wave radar data set and the current user voice data set can be divided into equal sliding window lengths, which is convenient for subsequent time series alignment. Among them, the current millimeter-wave radar data set, the current user voice data set, and various processed data obtained subsequently can all be encrypted and stored in the server to improve data security.

[0113] The low-frequency and high-frequency ratio feature acquisition unit 140 is used to acquire the heart rate variability data sequence corresponding to the current millimeter-wave radar data set and the low-frequency and high-frequency ratio feature sequence corresponding to the heart rate variability data sequence in time sequence based on a preset millimeter-wave radar feature extraction strategy.

[0114] In this embodiment, after the server obtains the current millimeter wave radar data set, a waveform diagram can be fitted according to the order of the acquisition time corresponding to each current millimeter wave radar data in the current millimeter wave radar data set, and the heart rate data corresponding to the target user to be identified can be extracted from the waveform diagram, and finally the heart rate variability data sequence is determined in combination with the interval between consecutive heartbeats (such as NN interval, RR interval, etc.). The heart rate variability data sequence includes multiple heart rate variability data (i.e., HRV data, the full name of HRV is Heart Rate Variability), and each heart rate variability data can be further decomposed into a low-frequency component (i.e., LF component, and the low-frequency frequency band corresponding to the low-frequency component is 0.04-0.15Hz) and a high-frequency component (i.e., HF component, and the high-frequency frequency band corresponding to the high-frequency component is 0.15-0.4Hz). By calculating the low-frequency component / high-frequency cost ratio of each heart rate variability data, the corresponding low-frequency high-frequency ratio feature can be obtained. After the heart rate variability data sequence corresponding to the current millimeter wave radar data set is obtained in time sequence, the low-frequency high-frequency ratio feature sequence of the same time sequence can also be obtained.

[0115] In one embodiment, the low-frequency and high-frequency ratio feature acquisition unit 140 is specifically used to:

[0116] Acquire a current heart rate fitting data set corresponding to the current millimeter-wave radar data according to the acquisition time sequence of each current millimeter-wave radar data subset in the current millimeter-wave radar data set;

[0117] Performing data cleaning on the current heart rate fitting data set based on a preset data cleaning strategy to obtain a cleaned current heart rate fitting data set;

[0118] Acquire a current RR interval sequence corresponding to the cleaned current heart rate fitting data set, and acquire the heart rate variability data sequence based on the current RR interval sequence and a preset power spectrum density estimation model;

[0119] For each heart rate variability data in the heart rate variability data sequence, the ratio of low frequency component / high frequency cost is obtained as a low frequency and high frequency ratio feature, and the low frequency and high frequency ratio feature of each heart rate variability data in the heart rate variability data sequence is arranged in time series to form the low frequency and high frequency ratio feature sequence.

[0120] In this embodiment, after the current millimeter-wave radar data set is obtained, a waveform diagram can be fitted according to the acquisition time sequence corresponding to each current millimeter-wave radar data subset in the current millimeter-wave radar data set, and the current heart rate fitting data set corresponding to the target user to be identified can be extracted from the waveform diagram. Then, the current heart rate fitting data set is removed by a data cleaning strategy such as an outlier cleaning strategy to obtain the cleaned current heart rate fitting data set. After that, the time interval between two adjacent heartbeat signals is determined in sequence according to the time sequence in the cleaned current heart rate fitting data set, thereby obtaining the current RR interval sequence (wherein the RR interval refers to the time interval between two adjacent heartbeat signals), and the current RR interval sequence can be further combined with the power spectrum density estimation model to perform a fast Fourier transform on the current RR interval sequence to perform power spectrum density estimation, thereby decomposing the variability of the current RR interval sequence into the power of different frequency components.

[0121] Finally, because each heart rate variability data has been decomposed into low-frequency components (i.e., LF components, and the low-frequency frequency band corresponding to the low-frequency components is 0.04-0.15Hz) and high-frequency components (i.e., HF components, and the high-frequency frequency band corresponding to the high-frequency components is 0.15-0.4Hz) based on the power spectrum density estimation model, the corresponding low-frequency and high-frequency ratio features can be obtained by calculating the low-frequency component / high-frequency cost ratio of each heart rate variability data. After obtaining the heart rate variability data sequence corresponding to the current millimeter-wave radar data set in time sequence, the low-frequency and high-frequency ratio feature sequence of the same time sequence can also be obtained. Among them, the total number of low-frequency and high-frequency ratio features included in the low-frequency and high-frequency ratio feature sequence is the same as the number of the first total subset. It can be seen that the above method, combined with the data cleaning strategy and the power spectrum density estimation model, can quickly extract the low-frequency and high-frequency ratio feature sequence corresponding to the current millimeter-wave radar data set.

[0122] The speech emotion feature acquisition unit 150 is used to acquire the speech emotion feature sequence corresponding to the current user speech data set in time sequence based on a preset speech feature extraction model.

[0123] In this embodiment, since the current user voice data set includes multiple current user voice data subsets arranged in chronological order, voice emotion recognition can be performed on multiple current user voice data subsets corresponding to the current user voice data set, and after obtaining the voice emotion features corresponding to each current user voice data subset, the voice emotion feature sequence is formed. The total number of voice emotion features included in the voice emotion feature sequence is the same as the number of the first total subsets.

[0124] In one embodiment, the speech emotion feature acquisition unit 150 is specifically used to:

[0125] For each current user voice data subset in the current user voice data set, the current user voice data subset is input into the voice feature extraction model to extract voice emotion features, so as to obtain a current voice emotion feature corresponding to the current user voice data subset; wherein the current voice emotion feature at least includes user pitch feature data, user sound intensity feature data and user speech speed feature data;

[0126] The current voice emotion features corresponding to each current user voice data subset in the current user voice data set are arranged in time sequence to form the voice emotion feature sequence.

[0127] In this embodiment, the speech feature extraction model preset in the server at least includes a speech noise reduction processing submodel, a speech framing submodel, a pitch feature extraction submodel, a sound intensity feature extraction submodel and a speech rate feature extraction submodel. When the current user speech data subset is input into the speech feature extraction model, the current user speech data subset is obtained after the noise reduction processing (such as adaptive filtering noise reduction processing) of the extraction submodel and the framing processing (such as first determining the frame length and frame shift, and then performing a windowing operation to achieve framing processing of the speech signal) of the speech framing submodel. Then, the pre-processed current user speech data subset is respectively extracted based on the pitch feature extraction submodel (such as a harmonic peak detection model), the sound intensity feature extraction submodel and the speech rate feature extraction submodel (such as a phoneme interval statistical model), and the current speech emotion feature including the user pitch feature data, the user sound intensity feature data and the user speech rate feature data can be obtained. It can be seen that based on the above method, the speech feature extraction model can quickly extract the speech emotion feature sequence corresponding to the current user speech data set.

[0128] The multimodal feature fusion unit 160 is used to perform time alignment and feature fusion on each low-frequency and high-frequency ratio feature in the low-frequency and high-frequency ratio feature sequence and each voice emotion feature in the voice emotion feature sequence based on a preset data fusion strategy to obtain a multimodal feature vector set corresponding to the target user to be identified.

[0129] The multimodal feature vector set includes a plurality of multimodal feature vectors, and each multimodal feature vector is composed of a low-frequency and high-frequency ratio feature with time alignment and a corresponding speech emotion feature.

[0130] In this embodiment, after the low-frequency and high-frequency ratio feature sequence and the voice emotion feature sequence are obtained in the server, in order to make the obtained multimodal feature vector set also have time series characteristics, the low-frequency and high-frequency ratio features in the low-frequency and high-frequency ratio feature sequence and the voice emotion features in the voice emotion feature sequence can be time-aligned based on a preset data fusion strategy, and then feature fusion is performed to obtain a multimodal feature vector set corresponding to the target user to be identified.

[0131] In one embodiment, the multimodal feature fusion unit 160 is specifically used for:

[0132] Acquire a first timestamp for each low-frequency and high-frequency ratio feature in the low-frequency and high-frequency ratio feature sequence, and acquire a second timestamp for each voice emotion feature in the voice emotion feature sequence;

[0133] Aligning the low-frequency and high-frequency ratio features with the same first timestamp and second timestamp with the speech emotion features and performing feature fusion to form a multimodal feature vector;

[0134] Arrange multiple multimodal feature vectors in chronological order to form the multimodal feature vector set.

[0135] In this embodiment, when each low-frequency and high-frequency ratio feature in the low-frequency and high-frequency ratio feature sequence is time-aligned with each voice emotion feature in the voice emotion feature sequence, it is necessary to obtain the first timestamp corresponding to each low-frequency and high-frequency ratio feature in the low-frequency and high-frequency ratio feature sequence, and obtain the second timestamp corresponding to each voice emotion feature in the voice emotion feature sequence. For example, the first low-frequency and high-frequency ratio feature in the low-frequency and high-frequency ratio feature sequence corresponds to the first timestamp T10, and the first voice emotion feature in the voice emotion feature sequence corresponds to the second timestamp T20. When the first timestamp T10 is equal to the second timestamp T20, it can be regarded that the two features are time-aligned, and the first low-frequency and high-frequency ratio feature in the low-frequency and high-frequency ratio feature sequence and the first voice emotion feature in the voice emotion feature sequence can also be feature-fused to form a multimodal feature vector corresponding to the first timestamp T10 or the second timestamp T20. Afterwards, when other low-frequency and high-frequency ratio features in the low-frequency and high-frequency ratio feature sequence are time-aligned with the corresponding voice emotion features in the voice emotion feature sequence, the above process is also referred to. Since the low-frequency and high-frequency ratio feature includes the low-frequency and high-frequency ratio, and the voice emotion feature includes the user pitch feature data, the user sound intensity feature data and the user speech rate feature data, the obtained multimodal feature vector includes the low-frequency and high-frequency ratio, the user pitch feature data, the user sound intensity feature data and the user speech rate feature data. Moreover, the multiple multimodal feature vectors included in the multimodal feature vector set corresponding to the target user to be identified are also arranged in sequence according to the time sequence of the timestamp. It can be seen that through the above method, the low-frequency and high-frequency ratio feature sequence can be quickly aligned and fused with the voice emotion feature sequence to obtain multiple multimodal feature vectors arranged in time sequence.

[0136] The classification result output unit 170 is used to input the multimodal feature vector set into a pre-trained emotion recognition model to obtain the emotion state classification results corresponding to each multimodal feature vector in the multimodal feature vector set to form an emotion state classification result output sequence.

[0137] In this embodiment, the emotion recognition model adopted is a long short-term memory network integrated with a self-attention mechanism or a multimodal convolutional neural network integrated with a self-attention mechanism. The multiple multimodal feature vectors included in the multimodal feature vector set also have the temporal sorting characteristics of the time series. Each of the multimodal feature vectors in the multimodal feature vector set is input into the pre-trained emotion recognition model in sequence according to the temporal sorting order, and the emotion state classification results corresponding to each multimodal feature vector can be obtained. The above-mentioned multiple emotion state classification results are combined in chronological order to obtain an emotion state classification result output sequence. Specifically, each of the emotion state classification results corresponding to the multimodal feature vectors corresponds to a 1*2 vector, and the two vector values ​​included therein are probability values ​​ranging from 0 to 1. The first probability value can determine the emotion category according to the specific value interval to which it belongs, and the second probability value can determine the emotion intensity according to the specific value interval to which it belongs; for example, the emotion state classification result is [0.3, 0.8], and the emotion category is determined to be anxiety based on the value interval to which the first probability value 0.3 in the emotion state classification result belongs, and the emotion intensity is determined to be high intensity based on the value interval to which the second probability value 0.8 in the emotion state classification result belongs. Because the emotion state classification results in the emotion state classification result output sequence are also arranged in chronological order, an emotion state diagram with the time axis as the horizontal axis and the probability values ​​corresponding to the emotion categories in the emotion state classification results as the vertical axis can also be constructed to characterize the emotional fluctuation state of the target user to be identified. Of course, after performing speech recognition on the current user speech data set and obtaining the current user text data including text timing information, the current user text data can also be integrated and displayed based on the timeline of the emotional state diagram, that is, each text record is accompanied by a timestamp and displayed together with the corresponding emotional state, so that users in the role of consultants can view the communication text content and emotional state at different times. When the emotional category corresponding to a time point in the emotional state diagram changes greatly, for example, from relaxation to anger, abnormal fluctuation prompt information can be automatically generated at this time, and the abnormal fluctuation prompt information can be displayed on the display device corresponding to the server.

[0138] Among them, when the emotion recognition model adopts the long short-term memory network integrated with the self-attention mechanism, it can focus more on capturing the dynamic changes of the low-frequency and high-frequency ratio and the voice emotion features; and when the emotion recognition model adopts the multimodal convolutional neural network integrated with the self-attention mechanism, it can perform more refined fusion at the feature level. Moreover, the self-attention mechanism adopted is to dynamically adjust the weight of each feature in the multimodal feature vector when each multimodal feature vector is input into the emotion recognition model to cope with the difference in the importance of features under different emotional states. For example, when the low-frequency and high-frequency ratio in the multimodal feature vector increases significantly, the weight of the low-frequency and high-frequency ratio feature will be automatically increased; when the change of voice features is more significant, the weight of voice emotion features will be increased.

[0139] Moreover, when the emotion state classification result corresponding to each multimodal feature vector is obtained based on the emotion recognition model in chronological order, specifically, an emotion state classification sub-result may be output for the low-frequency and high-frequency ratio feature in the multimodal feature vector, or an emotion state classification sub-result may be output for the voice emotion feature in the multimodal feature vector, and the above two emotion state classification sub-results are weighted and summed to obtain a comprehensive emotion state classification result. However, it should be noted that when there is a significant difference between outputting an emotion state classification sub-result for the low-frequency and high-frequency ratio feature in the multimodal feature vector and outputting an emotion state classification sub-result for the voice emotion feature in the multimodal feature vector, the weighted summation can balance the emotion judgment result.

[0140] It can be seen that the implementation embodiment of the device can collect the current millimeter-wave radar data set and the current user voice data set of the target user to be identified, and then perform feature extraction, time alignment and feature fusion to obtain a multimodal feature vector set, and then input it into the emotion recognition model to obtain the recognition result, thereby improving the accuracy of emotion recognition.

[0141] The above-mentioned millimeter wave radar-based psychological counseling multimodal emotion recognition device can be implemented in the form of a computer program, which can be used in Figure 7 Runs on the computer device shown.

[0142] See also Figure 7 , Figure 7 400 is a schematic block diagram of a computer device provided by an embodiment of the present invention. The computer device 400 integrates any of the millimeter wave radar-based multimodal emotion recognition devices for psychological counseling provided by an embodiment of the present invention.

[0143] See also Figure 7 The computer device 400 includes a processor 402 , a memory and a network interface 405 connected via a system bus 401 , wherein the memory may include a storage medium 403 and an internal memory 404 .

[0144] The storage medium 403 can store an operating system 4031 and a computer program 4032. The computer program 4032 includes program instructions, and when the program instructions are executed, the processor 402 can execute a multi-modal emotion recognition method for psychological counseling based on millimeter wave radar.

[0145] The processor 402 is used to provide computing and control capabilities to support the operation of the entire computer device.

[0146] The internal memory 404 provides an environment for the operation of the computer program 4032 in the storage medium 403. When the computer program 4032 is executed by the processor 402, the processor 402 can execute the above-mentioned millimeter-wave radar-based multimodal emotion recognition method for psychological counseling.

[0147] The network interface 405 is used to communicate with other devices over the network. Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the scheme of the present invention, and does not constitute a limitation on the computer device to which the scheme of the present invention is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0148] The processor 402 is used to run the computer program 4032 stored in the memory to implement the multimodal emotion recognition method for psychological counseling based on millimeter-wave radar as described in any of the above embodiments.

[0149] It should be understood that in the embodiment of the present invention, the processor 402 may be a central processing unit (CPU), and the processor 402 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0150] It can be understood by those skilled in the art that all or part of the processes in the method for implementing the above embodiment can be completed by instructing the relevant hardware through a computer program. The computer program includes program instructions, and the computer program can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the embodiment of the above method.

[0151] Therefore, the present invention also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions. When the program instructions are executed by a processor, the processor executes the multimodal emotion recognition method for psychological counseling based on millimeter-wave radar as described in any of the above embodiments.

[0152] The storage medium may be any computer-readable storage medium that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a magnetic disk, or an optical disk.

[0153] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0154] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of each unit is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0155] The steps in the method of the embodiment of the present invention can be adjusted in order, combined and deleted according to actual needs. The units in the device of the embodiment of the present invention can be combined, divided and deleted according to actual needs. In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0156] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, terminal, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention.

[0157] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A multimodal emotion recognition method for psychological counseling based on millimeter wave radar, characterized in that: include: In response to a multimodal emotion recognition instruction, obtaining to-be-recognized user information corresponding to the multimodal emotion recognition instruction and a to-be-recognized target user corresponding to the to-be-recognized user information; Acquire a current millimeter-wave radar data set corresponding to the target user to be identified; wherein the current millimeter-wave radar data set is composed of millimeter-wave radar data collected by the millimeter-wave radar module within the target collection time interval corresponding to the multimodal emotion recognition instruction after the millimeter-wave radar module is aimed at the target user to be identified; Acquire a current user voice data set corresponding to the target user to be identified; wherein the current user voice data set is composed of user voice data collected by a sound pickup and processing module from the target user to be identified within a target collection time interval; Based on a preset millimeter-wave radar feature extraction strategy, a heart rate variability data sequence corresponding to the current millimeter-wave radar data set and a low-frequency and high-frequency ratio feature sequence corresponding to the heart rate variability data sequence are acquired in time sequence; Based on a preset speech feature extraction model, a speech emotion feature sequence corresponding to the current user speech data set is acquired in time sequence; Based on a preset data fusion strategy, each low-frequency and high-frequency ratio feature in the low-frequency and high-frequency ratio feature sequence is time-aligned and feature-fused with each voice emotion feature in the voice emotion feature sequence to obtain a multimodal feature vector set corresponding to the target user to be identified; wherein the multimodal feature vector set includes a plurality of multimodal feature vectors, and each multimodal feature vector is composed of a low-frequency and high-frequency ratio feature that has completed time alignment and a corresponding voice emotion feature; The multimodal feature vector set is input into a pre-trained emotion recognition model to obtain an emotion state classification result corresponding to each multimodal feature vector in the multimodal feature vector set to form an emotion state classification result output sequence.

2. The method according to claim 1, characterized in that: Before the step of acquiring the current user voice data set corresponding to the target user to be identified, the method further includes: Acquire a current communication process speech data set corresponding to the target user to be identified; Based on the pre-collected voiceprint features of the user to be filtered, the voice signal in the voice data set of the current communication process and the voiceprint features of the user to be filtered are filtered to obtain the current user voice data set.

3. The method according to claim 1, characterized in that After the step of acquiring the current user voice data set corresponding to the target user to be identified, the method further includes: Speech recognition is performed on the current user speech data set based on a pre-trained speech recognition model to obtain current user text data including text time sequence information.

4. The method according to claim 1, characterized in that: After the step of acquiring the current user voice data set corresponding to the target user to be identified and before the step of acquiring, in time sequence based on a preset millimeter-wave radar feature extraction strategy, a heart rate variability data sequence corresponding to the current millimeter-wave radar data set and a low-frequency and high-frequency ratio feature sequence corresponding to the heart rate variability data sequence, the method further includes: Get the preset sliding window period; Dividing the current millimeter-wave radar data set based on the sliding window period to obtain a plurality of current millimeter-wave radar data subsets; wherein the plurality of current millimeter-wave radar data subsets constitute the current millimeter-wave radar data set; The current user voice data set is divided based on the sliding window period to obtain multiple current user voice data subsets; wherein the multiple current user voice data subsets constitute the current user voice data.

5. The method according to claim 1, characterized in that The method of extracting the heart rate variability data sequence corresponding to the current millimeter wave radar data set and the low-frequency and high-frequency ratio feature sequence corresponding to the heart rate variability data sequence in time sequence based on the preset millimeter wave radar feature extraction strategy includes: Acquire a current heart rate fitting data set corresponding to the current millimeter-wave radar data according to the acquisition time sequence of each current millimeter-wave radar data subset in the current millimeter-wave radar data set; Performing data cleaning on the current heart rate fitting data set based on a preset data cleaning strategy to obtain a cleaned current heart rate fitting data set; Acquire a current RR interval sequence corresponding to the cleaned current heart rate fitting data set, and acquire the heart rate variability data sequence based on the current RR interval sequence and a preset power spectrum density estimation model; For each heart rate variability data in the heart rate variability data sequence, the ratio of low frequency component / high frequency cost is obtained as a low frequency and high frequency ratio feature, and the low frequency and high frequency ratio feature of each heart rate variability data in the heart rate variability data sequence is arranged in time series to form the low frequency and high frequency ratio feature sequence.

6. The method according to claim 1, characterized in that The method of obtaining a speech emotion feature sequence corresponding to the current user speech data set in time sequence based on a preset speech feature extraction model includes: For each current user voice data subset in the current user voice data set, the current user voice data subset is input into the voice feature extraction model to extract voice emotion features, so as to obtain a current voice emotion feature corresponding to the current user voice data subset; wherein the current voice emotion feature at least includes user pitch feature data, user sound intensity feature data and user speech speed feature data; The current voice emotion features corresponding to each current user voice data subset in the current user voice data set are arranged in time sequence to form the voice emotion feature sequence.

7. The method according to claim 1, characterized in that The method of performing time sequence alignment and feature fusion of each low-frequency and high-frequency ratio feature in the low-frequency and high-frequency ratio feature sequence and each voice emotion feature in the voice emotion feature sequence based on a preset data fusion strategy to obtain a multimodal feature vector set corresponding to the target user to be identified includes: Acquire a first timestamp for each low-frequency and high-frequency ratio feature in the low-frequency and high-frequency ratio feature sequence, and acquire a second timestamp for each voice emotion feature in the voice emotion feature sequence; Aligning the low-frequency and high-frequency ratio features with the same first timestamp and second timestamp with the speech emotion features and performing feature fusion to form a multimodal feature vector; Arrange multiple multimodal feature vectors in chronological order to form the multimodal feature vector set.

8. A multimodal emotion recognition device for psychological counseling based on millimeter wave radar, characterized in that: include: a target user information acquisition unit, configured to, in response to a multimodal emotion recognition instruction, acquire user information to be identified corresponding to the multimodal emotion recognition instruction and a target user to be identified corresponding to the user information to be identified; A millimeter-wave radar data acquisition unit, used to acquire a current millimeter-wave radar data set corresponding to the target user to be identified; wherein the current millimeter-wave radar data set is composed of millimeter-wave radar data collected by the millimeter-wave radar module within the target collection time interval corresponding to the multimodal emotion recognition instruction after the millimeter-wave radar module is aimed at the target user to be identified; A sound data acquisition unit, used to acquire a current user voice data set corresponding to the target user to be identified; wherein the current user voice data set is composed of user voice data collected by the sound pickup and processing module from the target user to be identified within a target collection time interval; A low-frequency and high-frequency ratio feature acquisition unit, configured to acquire, in time sequence, a heart rate variability data sequence corresponding to the current millimeter-wave radar data set and a low-frequency and high-frequency ratio feature sequence corresponding to the heart rate variability data sequence based on a preset millimeter-wave radar feature extraction strategy; A speech emotion feature acquisition unit, used to acquire a speech emotion feature sequence corresponding to the current user speech data set in time sequence based on a preset speech feature extraction model; A multimodal feature fusion unit, for performing time alignment and feature fusion on each low-frequency and high-frequency ratio feature in the low-frequency and high-frequency ratio feature sequence and each voice emotion feature in the voice emotion feature sequence based on a preset data fusion strategy, so as to obtain a multimodal feature vector set corresponding to the target user to be identified; wherein the multimodal feature vector set includes a plurality of multimodal feature vectors, and each multimodal feature vector is composed of a low-frequency and high-frequency ratio feature with time alignment completed and a corresponding voice emotion feature; The classification result output unit is used to input the multimodal feature vector set into a pre-trained emotion recognition model to obtain the emotion state classification results corresponding to each multimodal feature vector in the multimodal feature vector set to form an emotion state classification result output sequence.

9. A computer device, characterized in that: The computer device includes a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the multimodal emotion recognition method for psychological counseling based on millimeter-wave radar as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which includes program instructions. When the program instructions are executed by a processor, the multimodal emotion recognition method for psychological counseling based on millimeter-wave radar as described in any one of claims 1 to 7 can be implemented.

Images