Intelligent voice interaction method and device

Abstract

The application discloses an intelligent voice interaction method and device, which is applied to the technical field of data processing, and the method comprises the following steps: acquiring user image, voice and text multi-modal original data, processing the multi-modal original data through a specific model and a tool, detecting and cutting a face through RetinaFace, extracting image emotion features through a ResNet-50 network model after the face is cut, extracting voice MFCC features through a torchaudio library, extracting text semantic features through a Sentence-BERT Chinese model, and generating three standardized single-modal feature data; uniformly processing the dimensions of the three standardized single-modal feature data through linear projection, combining a cross-modal attention mechanism and a Transformer encoder to fuse the features, inputting an emotion classifier to obtain an emotion recognition result; then, according to a Plutchik emotion wheel theory, combining a business scene and a role constraint to match a text and a voice double-response strategy, generating multi-modal interaction response content, finally combining intelligent hardware and cyclic monitoring to realize continuous interaction, and outputting an intelligent interaction result meeting the emotion and scene demands of a user.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and particularly relates to an intelligent voice interaction method and device. BACKGROUND

[0002] Existing emotion recognition technologies mostly rely on single modal data, such as recognizing emotions only through text semantic analysis or single facial image detection, and have the problem of insufficient feature dimensions, and it is difficult to fully capture the real emotional state of the user, resulting in limited recognition accuracy. Part of the multi-modal emotion recognition scheme does not effectively unify and deeply interact different modal features, only simply splices the features, ignores the correlation information between the modes, and results in weak emotion representation ability of the fused features.

[0003] Existing interaction response technologies mostly generate feedback based on fixed rules, do not fully combine emotion intensity, business scenarios and role constraints for dynamic adjustment, have the problems of asynchronous text and voice response, low emotion adaptation degree, and the like, and it is difficult to meet the personalized interaction needs of the user. Most of the schemes lack a cyclic monitoring and dynamic updating mechanism, cannot capture the emotional change trajectory of the user in real time, and it is difficult to achieve emotion adaptation response in the whole process, resulting in poor continuity of the interaction and user experience. SUMMARY

[0004] To solve the above technical problems, the present application provides the following technical solutions:

[0005] An intelligent voice interaction method, comprising: acquiring user image, voice, text multi-modal original data; detecting and cropping the face through the RetinaFace model for the image data, first performing BGR to RGB format conversion, then pre-processing, the pre-processing process includes normalization and tensor conversion, at the same time, adjusting the image to 224x224 RGB format, then extracting 128-dimensional emotion features through the ResNet-50 network model; for the 16kHz sampling rate voice data, extracting 40-dimensional MFCC coefficients through the torchaudio library, configuring n_fft as 400, hop_length as 160, n_mels as 128 in the extraction process, then performing time dimension average pooling after the extraction is completed; extracting 768-dimensional semantic embedding vectors through the paraphrase-multilingual-MiniLM-L12-v2 model of the Sentence-BERT for the Chinese text data, generating standardized image, voice, text single-modal feature data; processing the standardized single-modal feature data, the voice feature is accurately converted from 40 dimensions to 256 dimensions through a dedicated linear projection layer, the tone and energy information related to the voice emotion are preserved in the conversion process, after the linear projection unifies the dimensions, the feature fusion is performed through the cross-modal attention mechanism and the Transformer encoder, the emotion label is predicted through the emotion classifier, and the user emotion recognition result is generated; processing the user emotion recognition result, taking the Plutchik emotion wheel theory as the basis, combining the business scene and the role constraint, matching the corresponding text strategy and voice strategy, and generating the emotion-adapted double-response strategy, wherein the voice strategy clearly responds to the reference pitch, speech speed coefficient, volume size and pause rules of the voice, and different emotions are configured differently; processing the emotion-adapted double-response strategy, generating multi-modal interaction response content; processing the multi-modal interaction response content, combining the intelligent hardware device, realizing continuous interaction through the loop listening mechanism, playing emotional voice through the loudspeaker, at the same time, continuously collecting user subsequent voice feedback through the microphone array, converting the text in real time and monitoring the emotion change, and generating the intelligent interaction response result meeting the user emotion and scene demand.

[0006] An intelligent voice interaction device, the device comprising: an acquisition module configured to acquire user image, voice, text multi-modal original data;

[0007] The processing module is used for detecting and cropping the human face through the RetinaFace model for the image data, performing BGR to RGB format conversion first, and then performing preprocessing, the preprocessing process including normalization and tensor conversion, while adjusting the image to 224x224 RGB format, and then extracting 128-dimensional emotion features through the ResNet-50 network model; for the 16kHz sampling rate voice data, 40-dimensional MFCC coefficients are extracted through the torchaudio library, and during the extraction process, the parameters of n_fft, hop_length and n_mels are configured as 400, 160 and 128 respectively; after the extraction is completed, time dimension average pooling is performed; the paraphrase-multilingual-MiniLM-L12-v2 model of the Sentence-BERT is used to extract 768-dimensional semantic embedding vectors for the Chinese text data, and standardized image, voice and text single-modal feature data are generated; the standardized single-modal feature data is processed, the voice feature is accurately converted from 40 dimensions to 256 dimensions through a dedicated linear projection layer, the tone and energy information related to the voice emotion are preserved during the conversion process, after the linear projection is performed to unify the dimensions, the feature fusion is performed through the cross-modal attention mechanism and the Transformer encoder, the emotion label is predicted by inputting the emotion classifier, and the user emotion recognition result is generated; the user emotion recognition result is processed, the Plutchik emotion wheel theory is used as a basis, the business scene and role constraints are combined, the corresponding text strategy and voice strategy are matched, and the dual-response strategy adapted to the emotion is generated, wherein the voice strategy clearly responds to the reference pitch, speech speed coefficient, volume size and pause rules of the voice, and different emotions are configured differently; the dual-response strategy adapted to the emotion is processed to generate multi-modal interactive response content; the multi-modal interactive response content is processed, and the intelligent hardware device is combined to realize continuous interaction through a loop listening mechanism.

[0008] The response module is used for playing emotional voice through a loudspeaker, continuously collecting subsequent voice feedback of a user through a microphone array, converting the voice feedback into text in real time, monitoring emotion changes, and generating an intelligent interactive response result meeting the needs of the user emotion and the scene.

[0009] The beneficial effects are that the application provides an intelligent voice interaction method, through acquiring user image, voice, text multi-modal original data, then using RetinaFace+ResNet-50 network model to extract 128-dimensional image emotion features, torchaudio library to extract 40-dimensional voice MFCC features, Sentence-BERT Chinese model to extract 768-dimensional text semantic features, to generate standardized single-modal data. After linear projection unifies the dimension to 256, the cross-modal attention mechanism and the Transformer encoder are used to fuse the features, and the 6-class emotion classifier is input to obtain the results of anger, sadness, etc. Based on the Plutchik emotion wheel theory, combined with the business scene and role constraints, the text and voice double response strategy is matched to generate multi-modal interaction content, finally combined with the intelligent hardware and the cycle listening to realize the continuous interaction, the problems of single-modal recognition inaccuracy and poor response adaptability are solved, and the human-computer interaction experience is improved. BRIEF DESCRIPTION OF DRAWINGS

[0010] Figure 1 A flow chart of an intelligent voice interaction method provided for the embodiment of the application is shown in the figure.

[0011] Figure 2 A module schematic diagram of an intelligent voice interaction device provided for the embodiment of the application is shown in the figure. DETAILED DESCRIPTION

[0012] The preferred embodiments of the application are described below in conjunction with the accompanying drawings, and it should be understood that the preferred embodiments described herein are only used to illustrate and explain the application, and are not used to limit the application. The intelligent voice interaction method according to the exemplary embodiments of the present application is described below in conjunction with the accompanying drawings. Figure 1

[0013] In the embodiment of the present application, an intelligent voice interaction method is shown in the figure. Figure 1

[0014] S101, acquiring user image, voice, text multi-modal original data.

[0015] In an embodiment, the video stream is collected by the camera of the intelligent hardware terminal (such as Xiaozhi AI robot), and the effective image frame containing the human face is extracted from the video stream as the image original data, which provides the basis data for subsequent face detection and emotion feature extraction. The intelligent hardware terminal (Xiaozhi Client) is equipped with a high-definition camera, and the camera needs to support real-time video stream collection, has certain resolution and light self-adaptation ability, and can clearly capture the user's face picture in different light environments (such as normal light, weak light and strong light in indoor environment).

[0016] ​​After the user starts the intelligent interaction function, the camera is automatically turned on and continuously collects video streams. The system intercepts image frames from the video streams at fixed time intervals (e.g., every 300 ms). At the same time, the intercepted image frames are preliminarily screened, and only image frames with a face area ratio of ≥30% and no serious obstruction (e.g., a mask completely covering the mouth and nose, or a hand covering a key area of the face) are retained, and invalid frames with blurring, overexposure, underexposure, or no face are excluded. The final obtained image raw data is a single static image, in BGR (opencv default format), which needs to be converted to RGB format for feature extraction. The image resolution is uniformly adjusted to 224x224 RGB (adapted to the input requirements of the ResNet-50 network model), and the color depth is 8 bits (0-255 value range).

[0017] Taking the intelligent customer service scenario as an example, when the user consults equipment failure problems through the intelligent hardware terminal, the camera of the intelligent hardware terminal is automatically turned on, and the user's face is real-time photographed, and a video stream with a resolution of 1920x1080 is collected. The system intercepts an image frame every 300 ms, and in one image frame, the user's face is clear and unobstructed, with a face area ratio of about 45%, meeting the effective frame screening standard, and is determined as image raw data. The raw image frame will be subsequently subjected to BGR to RGB format conversion, size scaling to 224x224 RGB, normalization, and other preprocessing operations, to prepare for RetinaFace model detection of faces and ResNet-50 network model extraction of emotional features.

[0018] The user's voice signal is collected by the microphone array of the intelligent hardware terminal to form standardized audio stream data as the original input for voice emotional feature extraction. The microphone array equipped with the intelligent hardware terminal needs to support noise reduction and echo cancellation functions, and can directionally collect user voice and filter environmental noise (such as indoor air conditioner sound and background conversation sound) to ensure the clarity of the voice signal. At the same time, the microphone array needs to support a 16 kHz sampling rate to match the parameter requirements of subsequent MFCC feature extraction.

[0019] When the user speaks, the microphone array captures the voice signal in real time and preliminarily filters the environmental noise through a hardware noise reduction algorithm. The voice duration is monitored in real time during the collection process to ensure that the duration of a single segment of voice raw data is ≥1 second (to avoid incomplete feature extraction due to too short data). If the duration of a single user utterance is less than 1 second, the system automatically pads zeros to 1 second. After the collection is completed, the audio signal is subjected to amplitude normalization to avoid affecting the accuracy of subsequent feature extraction due to excessively large or small volume. The voice raw data is a 16 kHz sampling rate, single-channel, 16-bit depth PCM (Pulse Code Modulation) audio stream, and 40-dimensional MFCC features will be extracted based on this data subsequently. The amplitude range of the audio signal is controlled between [-1, 1] to ensure data standardization.

[0020] Still taking the intelligent customer service scenario as an example, when the user says “your director station crashes after a while, about 15 minutes or so, and it crashes, please handle it quickly”, the intelligent hardware terminal microphone array directionally collects the voice of this segment, filters the air conditioner running sound in the background through the noise reduction algorithm, and forms a clear voice signal. The system detects that the voice duration is about 8 seconds, which meets the duration requirement, and directly uses the voice as the original data; the subsequent 16kHz sampling rate audio data will be used to extract 40-dimensional MFCC coefficients (with n_fft=400, hop_length=160, n_mels=128 parameters) using the torchaudio library, and 40-dimensional voice features will be obtained through time dimension average pooling.

[0021] The collected user voice is converted into Chinese text by voice-to-text (ASR) technology, or the user manually inputs (such as touch screen input, external keyboard input) Chinese text, forming text original data for subsequent text semantic feature extraction. There are two core sources, one is “voice-to-text”, which converts the above-mentioned voice original data into Chinese text based on an industrial-level ASR model, ensuring that the text accuracy is ≥95%, supporting Chinese Mandarin and common dialects (such as Cantonese, Sichuanese); the second is “manual input”, the user directly inputs Chinese text through the touch screen, external keyboard, etc. of the intelligent hardware terminal, and the system receives and stores it in real time. The validity of the obtained text original data is verified, and invalid texts such as meaningless characters (such as random codes, special symbols), too long (more than 500 characters per text) or too short (less than 1 character per text) are excluded; the text encoding format is unified as UTF-8, supporting Chinese punctuation symbols (such as comma, period, exclamation mark), retaining the tone features (such as exclamation mark in “handle it quickly!”) in user expression.

[0022] The voice-to-text example is as follows: the user's voice “your director station crashes after a while, about 15 minutes or so, and it crashes, please handle it quickly” is converted by the ASR model to get the text original data “your director station crashes after a while, about 15 minutes or so, and it crashes, please handle it quickly”, which is free of errors and has complete semantics, meeting the requirements of subsequent feature extraction.

[0023] The manual input example is as follows: if the user manually inputs “today the weather is really good, the mood is very happy” through the touch screen of the intelligent hardware terminal, the system directly receives the string as the text original data, and the subsequent 768-dimensional text semantic embedding vector will be extracted using the “paraphrase-multilingual-MiniLM-L12-v2” model of Sentence-BERT.

[0024] S102, the image data is detected and cropped by the RetinaFace model, first BGR to RGB format conversion is performed, then preprocessing is performed, the preprocessing process includes normalization and tensor conversion, at the same time, the image is adjusted to 224x224 RGB format, then 128-dimensional emotion features are extracted through the ResNet-50 network model; for the 16kHz sampling rate voice data, 40-dimensional MFCC coefficients are extracted through the torchaudio library, during the extraction process, the parameters of n_fft, hop_length and n_mels are configured as 400, 160 and 128 respectively, and after the extraction is completed, time dimension average pooling is performed; the paraphrase-multilingual-MiniLM-L12-v2 model of the Sentence-BERT is used to extract 768-dimensional semantic embedding vectors for the Chinese text data, and standardized image, voice and text single modal feature data are generated.

[0025] In an embodiment, the user image, voice and text multi-modal original data are processed, the image data is detected and cropped by the RetinaFace model, first BGR to RGB format conversion is performed, then preprocessing is performed, the preprocessing process includes normalization and tensor conversion, at the same time, the image is adjusted to 224x224 RGB format, then 128-dimensional emotion features are extracted through the ResNet-50 network model. The image original data is sequentially completed face detection and cropping, format conversion, preprocessing and emotion feature extraction, and finally a 128-dimensional standardized image emotion feature vector is generated, which provides core data in the image dimension for cross-modal fusion.

[0026] For face detection and cropping, the RetinaFace model is used to accurately detect the target face of the image original data, locate the coordinate position of the face in the image, crop the independent face region according to the coordinates, and eliminate the background and irrelevant interference elements, so as to ensure that the subsequent feature extraction focuses on the key area of the face emotion (eyebrows, eyes, mouth, nose, etc.). The cropped face image is converted from BGR format to RGB format, which adapts to the input format requirements of the ResNet-50 network model, and avoids feature extraction distortion caused by format difference.

[0027] The preprocessing operation is as follows: first, the RGB format face image size is uniformly adjusted to 224x224 to ensure that the image specifications of the input model are consistent; then, normalization processing is performed to map the image pixel value from the value range of 0-255 to 0-1, eliminating the influence of the pixel value magnitude difference on model training and feature extraction; finally, tensor conversion is performed to convert the image data into a tensor form recognizable by the model. The preprocessed tensor data is input into the ResNet-50 network model, and through the deep network structure of the model's convolutional layer, pooling layer, etc., the deep features related to emotions in the face image are mined, and finally a 128-dimensional emotion feature vector is output, which can accurately represent the core information of the face emotion.

[0028] Taking image processing when a user consults a device fault in an intelligent customer service scenario as an example, the original image data is a user face image in BGR format with a resolution of 1920x1080, the RetinaFace model detects the face coordinates (x1=350, y1=200, x2=680, y2=530), and accordingly, a region image containing only the face is cropped. The cropped face image is converted from BGR format to RGB format to restore the true color information; the RGB face image is preprocessed to adjust the size to 224x224, and after normalization, the pixel value is in the interval of 0-1, and then converted to tensor format; the tensor data is input into the ResNet-50 network model, and the model extracts a 128-dimensional emotion feature vector, in which the feature dimension number related to the "discontent" emotion is significantly higher than that of other dimensions, providing image dimension support for subsequent emotion recognition.

[0029] For 16kHz sampling rate speech data, 40-dimensional MFCC coefficients are extracted through the torchaudio library, and during the extraction process, the parameters n_fft, hop_length, and n_mels are configured as 400, 160, and 128, respectively. After extraction, time dimension average pooling is performed. Specifically, for 16kHz sampling rate speech original data, 40-dimensional standardized speech feature vectors are generated through parameter configuration, MFCC coefficient extraction, and time dimension average pooling, accurately capturing emotion-related features in speech (such as tone, speech rate, and energy change). For 16kHz sampling rate speech data, the key parameters for extracting MFCC coefficients through the torchaudio library are configured, where n_fft is set to 400 (to control the window size of Fourier transform), hop_length is set to 160 (to control the window moving step), and n_mels is set to 128 (to control the number of mel filter banks), ensuring that the extracted MFCC coefficients can fully reflect the spectral features and emotion-related information of the speech.

[0030] Based on the configured parameters, the MFCC transformation is performed on the voice raw data through the torchaudio library to convert the voice signal from the time domain to the frequency domain, and 40-dimensional MFCC coefficients are extracted, which can effectively represent the frequency distribution, energy change and other features closely related to emotional expression of the voice. Since the extracted MFCC coefficients contain time series information, to simplify the features and highlight the overall emotional features, the MFCC coefficients in the time dimension are averaged and pooled, and finally a 40-dimensional fixed-dimensional voice feature vector is obtained, realizing the standardization of voice features.

[0031] Taking the processing of the user's angry voice expressing "your director station crashes in a while, handle it quickly" as an example, the voice raw data is a PCM audio stream with a sampling rate of 16kHz and a length of 7 seconds, and the torchaudio library is configured according to n_fft=400, hop_length=160, n_mels=128; 40-dimensional MFCC coefficients of the voice are extracted through the torchaudio library, and the values corresponding to the middle and high frequency bands in the coefficients show obvious fluctuations, which are consistent with the characteristics of high-pitched and energy-concentrated voice under angry emotion; the MFCC coefficients in the time dimension are averaged and pooled to obtain a 40-dimensional standardized voice feature vector, which can accurately reflect the overall emotional features of the voice and provide support for cross-modal emotion recognition in the voice dimension.

[0032] The 768-dimensional semantic embedding vector is extracted from the Chinese text data through the paraphrase-multilingual-MiniLM-L12-v2 model of Sentence-BERT, and the standardized image, voice and text single-modal feature data are generated. The specified Sentence-BERT model is used to encode the semantic information of the Chinese text raw data, and a 768-dimensional standardized text semantic embedding vector is generated to accurately capture the semantic information and emotional tendency in the text. The "paraphrase-multilingual-MiniLM-L12-v2" model of Sentence-BERT is selected, which is designed for multi-language text semantic embedding and has good adaptability and semantic capture ability for Chinese text, and can efficiently extract deep semantic features of the text.

[0033] The Chinese text raw data that has passed the effectiveness check (no garbled code, length compliance) is input into the model, and the model encodes the semantic information of the text through a multi-layer Transformer network structure, converts the text into a high-dimensional vector form, and finally outputs a 768-dimensional semantic embedding vector, which can comprehensively represent the semantic connotation and emotional tendency (positive, negative, neutral) of the text. The extracted 768-dimensional semantic embedding vector has a unified dimension and numerical range, and does not need additional dimension adjustment or normalization processing, and can be directly used as standardized text feature data for subsequent cross-modal feature fusion.

[0034] The voice-to-text scenario is as follows: the user's angry voice is converted into text data by ASR, which is "your director station will crash in a while, about 15 minutes, please handle it quickly". The text is input into the "paraphrase-multilingual-MiniLM-L12-v2" model, and a 768-dimensional semantic embedding vector is extracted. The values of the semantic dimensions related to "discontent" and "urgency" in the vector are significantly higher than those of other dimensions.

[0035] The manual input scenario is as follows: the user manually inputs the text "Today the weather is really good, and I am very happy", and the 768-dimensional semantic embedding vector extracted after inputting the model has a higher proportion of values of semantic dimensions related to "joy" and "positivity", accurately capturing the positive emotional tendency in the text and providing text dimension support for cross-modal emotion recognition.

[0036] In S103, the standardized single-modal feature data is processed. The voice feature is accurately converted from 40 dimensions to 256 dimensions through a dedicated linear projection layer. The tone and energy information related to the voice emotion are preserved during the conversion process. After the dimensions are unified through linear projection, the feature fusion is performed through the cross-modal attention mechanism and the Transformer encoder. The emotion classifier is input to predict the emotion label, and the user emotion recognition result is generated.

[0037] In one implementation, the standardized single-modal feature data is processed. Linear projection layers are introduced to convert the dimensions of image, voice, and text single-modal features. The image feature is projected from 128 dimensions, the voice feature is projected from 40 dimensions, and the text feature is projected from 768 dimensions to 256 dimensions, generating single-modal feature data with unified dimensions. By introducing independent linear projection layers, the dimensions of image, voice, and text standardized single-modal features are converted, and different dimensional features are mapped to 256 dimensions, generating single-modal feature data with consistent dimensions, laying a foundation for subsequent cross-modal feature fusion.

[0038] To address the differences in the original dimensions of the three types of features, three independent linear projection layers are designed to adapt to the input dimensions of image 128 dimensions, voice 40 dimensions, and text 768 dimensions. Each projection layer can accurately map the corresponding modal feature to the target dimension of 256 dimensions. The linear projection layer performs linear transformation on the input feature through the learned parameter matrix, realizes dimension unification under the premise of preserving modal core emotional information, and avoids feature fusion distortion caused by dimension differences. The converted three types of features are 256-dimensional vectors, and the numerical range is normalized to ensure that different modal features have comparability and interactivity in the same dimensional space.

[0039] The energy information in the voice emotion refers to the intensity-related characteristics of the voice signal in the propagation process, reflecting the intensity, volume change and time distribution of the sound energy when the user speaks, directly related to the emotional state (such as anger with concentrated energy, sadness with smooth energy), and is an important voice dimension for distinguishing different emotions.

[0040] When extracting 40-dimensional MFCC (Mel Frequency Cepstral Coefficient) features through the torchaudio library, the energy information of the voice is captured synchronously - the MFCC coefficients themselves contain the spectral energy distribution characteristics, and the subsequent time dimension average pooling processing will retain the overall trend of the energy in the entire voice segment (such as continuous high intensity, strong first and weak later). In the process of converting voice features from 40 dimensions to 256 dimensions through a dedicated linear projection layer, the energy features that are strongly related to emotions are specifically retained, avoiding the loss of core emotional information caused by dimension conversion. For example, the voice energy corresponding to the angry emotion is concentrated and has a high amplitude, and this feature will still maintain its relevance to the emotional label after projection.

[0041] Together with features such as tone and speed, it constitutes a complete representation of voice emotion, providing emotional basis for cross-modal feature fusion (combined with image and text features) in the voice dimension. For example, high-intensity energy information often corresponds to strong emotions such as anger and surprise, while low-intensity and smooth energy information may correspond to sadness and neutral emotions. It provides data support for the differentiated configuration of voice strategies. Voice strategies need to adjust parameters such as volume and speed according to emotions, and energy information is the core basis for judging emotional intensity and matching corresponding volume (such as high-intensity energy corresponding to anger, requiring response voice to control volume to avoid exacerbating conflicts).

[0042] Taking the processing of three types of features corresponding to the angry emotion of users in the intelligent customer service scenario as an example, the image standardized feature is a 128-dimensional vector, which is converted to a 256-dimensional image feature vector through a dedicated linear projection layer, and the key feature information of the angry facial expression is retained in the vector. The voice standardized feature is a 40-dimensional MFCC vector, which is converted to a 256-dimensional voice feature vector through a voice linear projection layer, and the core features such as tone and energy of the angry voice are completely retained; the text standardized feature is a 768-dimensional semantic embedding vector, which is mapped to a 256-dimensional text feature vector through a text linear projection layer, effectively retaining the emotional tendency semantic information of "discontent" and "eager" in the text; finally, three groups of 256-dimensional single-modal feature data are obtained, with completely consistent dimensions, which can be directly used for subsequent feature sequence splicing and cross-modal interaction.

[0043] The dimension-unified single-modal feature data is processed, spliced into a feature sequence, and then the cross-modal attention mechanism composed of multi-head self-attention is used to realize feature interaction, and then the Transformer encoder with 2 layers of encoder is input to further refine the feature relationship and generate the fused multi-modal feature data. The three types of single-modal features with unified dimensions are spliced into a feature sequence, and the cross-modal attention mechanism composed of multi-head self-attention is used to realize the interaction between the modal features, and then the 2-layer Transformer encoder is used to refine the feature relationship, and generate the feature data fused with multi-modal information.

[0044] The 256-dimensional image, speech, and text features are spliced into a feature sequence with a dimension of (3, 256) in a fixed order (image→speech→text), forming the basic input form of cross-modal fusion. The multi-head self-attention mechanism is used to calculate the similarity between different modal features to generate dynamic attention weights, allowing each modal feature to focus on the key information of other modal features, realizing deep interaction and complementarity between modalities, such as speech features focusing on facial feature dimensions related to emotions in image features. The feature sequence after attention interaction is input into the Transformer structure with 2 layers of encoder, and through the -feedforward network and layer normalization operation of the encoder, the association between modalities is further refined, the emotion-related features are strengthened, the irrelevant noise is weakened, and more representative fused feature data is generated.

[0045] The three types of 256-dimensional features of the above anger emotion are spliced in order to form a feature sequence with a dimension of (3, 256), which contains the multi-dimensional original information of the user's emotion. The cross-modal attention mechanism calculates a dynamic weight matrix, in which the attention weight of image features to speech features is 0.3, the attention weight of text features is 0.2, the attention weight of speech features to image features is 0.3, the attention weight of text features is 0.2, the attention weight of text features to image features is 0.1, and the attention weight of speech features is 0.3. The interaction and fusion of the three types of features are realized through the weight weighting. The feature sequence after interaction is input into the 2-layer Transformer encoder, and after feature refinement, a fused multi-modal feature data with a dimension of (3, 256) is generated, which integrates the angry emotion information in facial expressions, speech intonation, and text semantics, and has more emotion representation than single-modal features.

[0046] The fused multimodal feature data is processed by mean pooling to obtain a 256-dimensional fused feature vector. This vector is then input into an emotion classifier with six output nodes. After Softmax normalization, the probability of each emotion is calculated, and the label corresponding to the maximum probability is selected to generate user emotion recognition results containing six categories: anger, sadness, happiness, fear, surprise, and neutrality. The fused multimodal feature data is further processed by mean pooling to obtain a unified 256-dimensional fused feature vector. This vector is then input into an emotion classifier with six output nodes. After Softmax normalization, the probability of each emotion is calculated, and the label corresponding to the maximum probability is selected to generate the final user emotion recognition results. The (3,256) fused feature sequence is subjected to mean pooling along the time dimension, compressing the fused features of the three modalities into a single 256-dimensional vector. This vector integrates the core emotional information of all modalities, simplifying the feature dimensions while retaining key representations.

[0047] The emotion classifier is designed as follows: it is a fully connected neural network with six nodes in the output layer, corresponding to six emotions: anger, sadness, happiness, fear, surprise, and neutral. Emotion classification is achieved by learning the mapping relationship between multimodal fusion features and emotion labels. The raw scores output by the classifier are converted into a probability distribution between 0 and 1 using Softmax normalization, with the sum of the probabilities of each emotion being 1. The emotion label corresponding to the highest probability is selected as the final recognition result to ensure the clarity of the recognition result.

[0048] Taking user anger processing as an example, mean pooling is performed on the fused multimodal features (3,256), and the average value of the three types of features in each dimension is calculated to obtain a 256-dimensional fused feature vector. This vector comprehensively reflects the multimodal features of user anger. This vector is input into an emotion classifier, which outputs the following raw scores: anger 3.2, sadness -1.2, happiness 0.8, fear 1.5, surprise -0.3, and neutral 0.4. After Softmax normalization, the probabilities of each emotion are: anger 0.65, sadness 0.02, happiness 0.12, fear 0.15, surprise 0.03, and neutral 0.03. The emotion label corresponding to the maximum probability of 0.65 is selected as "anger," generating the user emotion recognition result and accurately matching the user's actual emotional state.

[0049] S104 processes the user's emotion recognition results. Based on the Plutchik emotion wheel theory and combined with business scenarios and role constraints, it matches the corresponding text strategy and voice strategy to generate a dual-response strategy that adapts to emotions. The voice strategy specifies the reference pitch, speech rate coefficient, volume, and pause rules of the response speech, with different configurations for different emotions.

[0050] In an implementation, the user emotion recognition result is processed to construct a structured emotion vector containing a dominant emotion, a secondary emotion, an intensity, a polarity, and a triggering scene based on the Plutchik emotion wheel theory, and a emotion-scene-role three-way matching result is generated in combination with a business scene priority matrix and a role constraint condition. The structured emotion vector is constructed as follows: based on the eight basic emotions of the Plutchik emotion wheel and the three-dimensional dynamic relationship (intensity, similarity, and opposition), a five-dimensional emotion vector containing a dominant emotion, a secondary emotion, an intensity, a polarity, and a triggering scene is constructed. The dominant emotion is the core emotion in the recognition result, the secondary emotion is the adjacent associated emotion in the emotion wheel, the intensity quantifies the emotion saturation (0-1 interval), the polarity distinguishes the positive and negative emotions (-1 to 1), and the triggering scene labels the business scene tag where the emotion is generated. According to the preset scene priority matrix, the emotion response priority in the current scene is determined, for example, in a crisis handling scene, the priority of anger and fear emotions is higher than that of trust and joy. In combination with the behavior specification of the interactive role, the strategy direction that does not meet the role requirement is filtered, such as the customer service role is prohibited from using high-intensity aversion emotion related expressions.

[0051] Taking the user anger emotion handling in the intelligent customer service scene as an example, the user emotion recognition result is "anger", and the structured emotion vector is constructed: dominant emotion = anger, secondary emotion = [aversion] (adjacent emotion in the emotion wheel), intensity = 0.85 (high intensity), polarity = -0.9 (strong negative), and triggering scene = service_failure (device failure complaint). The business scene is "complaint handling", according to the priority matrix, the priority of the anger emotion is the highest, which matches the core requirement "quick response + problem solving" of the scene; the role is customer service, and the constraint condition is "prohibition of high-intensity conflict expression", which excludes the strategy direction that may aggravate the contradiction, and finally generates the three-way matching result: the calming of the anger emotion and the solution type strategy are preferentially matched, and aggressive expression is avoided.

[0052] The three-party matching result is processed, and a multi-modal expression strategy library is called to match the text strategy and the voice strategy corresponding to the emotional combination. The text strategy specifies the DeepSeek prompt word requirements and generates parameters, and the voice strategy determines the voice feature parameters including the speech speed, tone, and volume. The strategy library calling logic is as follows: the multi-modal expression strategy library includes a text strategy engine and a voice strategy engine, which respectively store the response rules corresponding to different emotional combinations, scenes, and roles. The rules are accurately retrieved and matched through the dominant emotion, intensity, and scene label in the emotional vector. The text strategy matching is as follows: the core requirements of the DeepSeek prompt word (such as empathy preposition and solution priority) are specified, and personalized generation parameters (temperature, repetition_penalty, etc.) are configured to ensure that the generated text fits the emotion and scene. The voice strategy matching is as follows: the core feature parameters of the voice are determined, including the reference pitch, speech speed coefficient, and pause strategy, and the adjustment rules of the volume and tone are specified to make the voice expression consistent with the emotional state.

[0053] Based on the above three-party matching result of the angry emotion in the customer service scene, the multi-modal expression strategy library is called to match the text strategy corresponding to "angry + high intensity + complaint handling": the text requirements are "recognize feelings, state facts, and inform that a special officer will contact for processing", and the text parameters temperature=0.4 (low randomness) and repetition_penalty=1.5 (low repetition rate) are configured; the voice parameters are volume=45, rate=0.7 (slow), and pitch=0.8 (low tone), and a 0.2s pause strategy after the key word is configured to meet the calming needs under the angry emotion. The final preliminary matching result is obtained, the text strategy specifies the prompt word template and generation parameters, and the voice strategy determines the three-dimensional voice features and pause rules.

[0054] The matched text strategy and voice strategy are processed, a conflict resolution mechanism is started, an intensity attenuation and compensation strategy is executed for text voice conflict, an emotion neutralization algorithm is used for emotion opposite conflict, and an emotion-adapted dual-response strategy is generated. The dual-response strategy includes emotion-adapted text generation rules, voice parameter configuration and conflict handling scheme. The emotion neutralization algorithm is final_valence=(joy_valence+sadness_valence) / 2, where final_valence is the final emotion valence, which is the core quantitative result after emotion neutralization, used to match the neutral bias-adapted dual-response strategy (text+voice), and the value range is -1 to 1 (negative to positive). joy_valence is the joy emotion valence, which represents the positive and negative degree and intensity of the user's joy emotion, and the value range is -1 to 1 (the closer to 1, the stronger the joy emotion). sadness_valence is the sadness emotion valence, which represents the positive and negative degree and intensity of the user's sadness emotion, and the value range is -1 to 1 (the closer to -1, the stronger the sadness emotion). The preliminary matched text and voice strategy are conflict detected, various conflicts are solved through intensity attenuation, compensation strategy and emotion neutralization algorithm, and finally a complete dual-response strategy including text generation rules, voice parameter configuration and conflict handling scheme is generated. Specifically, the conflict detection dimensions include two core conflict scenarios: text and voice strategy conflict (such as the contradiction between text soothing and voice urgency), and emotion opposite conflict (such as detecting joy and sadness at the same time).

[0055] For text voice conflict, an intensity attenuation function (adjusted_intensity=max(0.5,original_intensity*0.6)) is started to reduce the conflict strategy intensity, where adjusted_intensity is the adjusted strategy intensity, which is the final intensity value used for execution after conflict resolution, ensuring that the strategy intensity is moderate, and the value range is ≥0.5; original_intensity is the original strategy intensity, which refers to the intensity parameter (such as voice speed, text emotion expression intensity) when the text or voice strategy is initially matched, and the value range is 0-1, and a buffer phrase compensation is added; for emotion opposite conflict, an emotion neutralization algorithm (final_valence=(joy_valence+sadness_valence) / 2) is used to calculate the neutralized emotion polarity, and the adaptive strategy is matched. The text strategy and voice strategy after resolution are integrated to determine the text prompt word template, generation parameter, voice feature parameter and conflict handling plan, forming a standardized strategy that can be directly executed.

[0056] Assuming a conflict arises in the initial matching between the text strategy "slow and calming" and the speech strategy "fast pace": Intensity attenuation is activated, the speech rate coefficient is adjusted from 0.8 to 0.7, and a buffer phrase "Please calm down, we will handle it for you immediately" is added to resolve the conflict. If there is a conflict in the user's emotion recognition results (e.g., both joy and sadness are recognized), the neutralization algorithm calculates final_valence=0.1 (weakly positive), matching a neutral-to-soothing text strategy with a smooth speech strategy (pitch 180Hz, speech rate 1.0x). The final generated dual-response strategy is as follows: the text strategy is "The user's emotional state is [anger]. Please provide a professional response. Requirements: acknowledge the feelings, state the facts, and inform that a specialist will contact you for handling," with parameters temperature=0.4 and repetition_penalty=1.5; the speech strategy is volume=45, rate=0.7, pitch=0.8, with a 0.2s pause after the keyword; the conflict resolution scheme is "If the speech and text rhythms conflict, intensity attenuation + buffer phrase compensation is activated."

[0057] S105 processes the dual-response strategy for emotion adaptation to generate multimodal interactive response content.

[0058] In one implementation, the dual-response strategy for emotion adaptation, along with user input text and historical dialogue records, are extracted and processed. A preset prompt word template is used to complete the text generation rules and parameters, and a DeepSeek API call request is constructed. The text generation rules and parameters from the emotion-adaptive dual-response strategy are extracted, combined with user input text and historical dialogue records, and key information is completed using a preset prompt word template to construct a standardized DeepSeek API call request, providing a basis for response text generation.

[0059] Extract textual requirements (such as empathy-first approach and solution-first approach) and generation parameters (temperature, repetition_penalty, etc.) from the dual-response strategy; simultaneously extract the core demands of the user's current input text, as well as information already clarified in historical dialogues (such as details of device malfunctions mentioned by the user), ensuring that the call request contains complete context. Use a preset "emotional label + textual requirements + context" template to populate the extracted information, clarifying the direction of model generation, for example, "User's emotional state is [emotional label], please provide a professional answer, requirements: [textual requirements], user's current demand: [core demand], historical dialogue information: [key historical content]". In accordance with the requirements of DeepSeekAPI, integrate prompts, generation parameters, dialogue history, and other information to construct a complete call request that includes model selection, message list, and generation control parameters, ensuring that the request format is compliant and the information is complete.

[0060] Take the double-response strategy for user anger in the intelligent customer service scenario as an example, extract the text requirements in the double-response strategy "acknowledge feelings, state facts, inform that a specialist will contact for processing", generate parameters temperature=0.4, repetition_penalty=1.5, extract the core appeal of the user input text "the director's platform crashes after 15 minutes of use and needs urgent processing", and there is no relevant pre-information in the historical dialogue record. Complete the preset prompt word template to get "the user's emotional state is angry, please give a professional answer, requirements: acknowledge feelings, state facts, inform that a specialist will contact for processing, user's current appeal: the director's platform crashes after 15 minutes of use and needs urgent processing, historical dialogue information: none". The model selects deepseek-chat, the message list contains system prompt words (completed template content) and user input text, and the generation parameter configuration is temperature=0.4, repetition_penalty=1.5, max_tokens=100.

[0061] Process the DeepSeek API call request, call the target large language model to generate a response text that meets the emotional adaptation requirements. Send the constructed API call request to the DeepSeek API, call the target large language model, generate a response text that meets the emotional adaptation requirements and fits the context, and ensure that the text content is empathetic, accurate and targeted. Send the constructed API call request to the DeepSeek server in the specified format, ensure stable network transmission and no missing request parameters. Receive the response text returned by the model and verify whether the text meets the text requirements in the double-response strategy (such as whether it embodies empathy and whether it clearly states the solution), and if there is a deviation, re-adjust the parameters and initiate the call (such as reducing temperature to improve certainty). The text generated by the model is screened again to remove redundant information and repetitive expressions, ensuring that the text is concise, smooth and suitable for the interaction scenario (such as formal and friendly in the customer service scenario).

[0062] Based on the above API call request for angry emotion, send the call request to the DeepSeek API, and the model generates a response text based on the prompt word template and generation parameters. Receive the text returned by the model "I'm very sorry for the inconvenience caused by your device! We have received the problem of the director's platform crashing after 15 minutes of use, and we will immediately arrange a professional specialist to contact you to troubleshoot and solve the problem, please wait patiently." Verify that the text meets the requirements of "acknowledge feelings (apologize), state facts (confirm the problem), and inform the specialist contact (solution)", and there is no redundant information, so it is determined as the final response text.

[0063] The response text and voice strategy parameters are processed, a preset emotion TTS interface is called, corresponding speech speed, tone, volume parameters and 48000 Hz sampling rate, wav format output settings are configured, and response voice with emotional characteristics is synthesized. The voice feature parameters in the double response strategy are extracted, combined with the generated response text, the preset emotion TTS interface is called, the sampling rate, output format and other parameters are configured, and the emotional voice file is synthesized. The voice feature parameters are extracted from the double response strategy, including volume, rate, pitch and pause strategy (such as pause after keywords), to ensure that the parameters match the emotional state. The Ali Cloud SAM-BERT emotion TTS interface is called, the output parameters are configured, the sampling rate is fixed at 48000 Hz, the output format is wav, and the extracted voice feature parameters are transmitted, to ensure that the voice synthesis effect meets the requirements. The generated response text is transmitted to the TTS interface, and the interface synthesizes voice with emotional characteristics based on the text content and voice parameters, such as slow speed, low tone, and keyword accent voice corresponding to angry emotion.

[0064] Still taking the double response strategy of the user's angry emotion as an example, the voice parameters in the double response strategy are extracted: volume = 45, rate = 0.7, pitch = 0.8, and the pause strategy is to pause 0.2s after keywords such as "specialist" and "contact". The Ali Cloud SAM-BERT emotion TTS interface is called, the sampling rate is configured as 48000 Hz, the output format is wav, the voice parameters and the response text "I am very sorry for the inconvenience caused by the use of your device! We have received the problem of the director's station crashing after 15 minutes of use, and we will immediately arrange a professional specialist to contact you to troubleshoot and solve the problem, please wait patiently." are transmitted. The interface synthesizes emotional voice, specifically, the speech speed is slow (0.7 times the normal speech speed), the tone is low (0.8 times the reference tone), there is a pause of 0.2s after "specialist" and "contact", the volume is 45, and it meets the needs of appeasement under angry emotion.

[0065] The response text and the response voice are integrated to generate multi-modal interactive response content containing emotion-adapted text content and emotional voice files. The generated emotion-adapted response text and emotional voice files are integrated to ensure that the content of the two is consistent and the emotions are coordinated, and standardized multi-modal interactive response content containing text content and voice files is generated. The core information (such as solutions, time commitments) of the response text and the voice file are checked for consistency to avoid contradictions such as the text mentioning "contact within 1 hour" and the voice stating "contact within 2 hours". The response text is saved as a UTF-8 encoded text file, and the voice file is archived in wav format with a sampling rate of 48000 Hz, with a uniform naming convention (such as "response_emotion label_timestamp. Text / voice"), which facilitates subsequent calling and management. The standardized text file and voice file are stored in association to generate index information containing file paths, emotion labels, and generation times, forming complete multi-modal interactive response content to ensure that intelligent hardware can be quickly called.

[0066] The response text and voice of the above-mentioned angry emotion are received, the response text and voice file are verified, and it is confirmed that the core information is consistent and contains three key contents of "apology, confirmation of problem, and specialist contact" without information contradiction. The response text is saved as "response_anger_202405201030.txt" with a UTF-8 encoding format, and the voice file is saved as "response_anger_202405201030.wav" with a wav format and a sampling rate of 48000 Hz. Index information is generated, specifically emotion label = anger, text path = xxx / response_anger_202405201030.txt, voice path = xxx / response_anger_202405201030.wav, and generation time = 2024-05-20-10:30, and the multi-modal interactive response content integration is completed.

[0067] S106, processing the multi-modal interactive response content, combining the intelligent hardware device, realizing continuous interaction through a loop listening mechanism, playing emotional voice through a loudspeaker, and continuously collecting user's subsequent voice feedback through a microphone array, converting the voice feedback into text in real time and monitoring the emotional changes, and generating intelligent interactive response results that meet the user's emotional and scene needs.

[0068] In an embodiment, the multi-modal interactive response content is adapted and integrated with the smart hardware device, a real-time data acquisition channel is constructed through the terminal camera and microphone array to generate a hardware-content linkage interactive basis. The compatibility of the multi-modal interactive response content (text file, wav format voice) with the smart hardware terminal is ensured, the text display resolution, voice playback interface and audio decoding format of the hardware are adapted to ensure that the content can be normally output. The camera and microphone array of the smart hardware terminal are activated, the camera is responsible for continuously capturing the user's face image, the microphone array is responsible for directional collection of the user's voice, and the two work together to construct an uninterrupted data acquisition channel to support real-time transmission and conversion (voice to text) of image and voice data. A linkage mechanism of hardware state and content output is established, the hardware real-time feedback running state (such as whether the camera is normally started and whether the microphone is picking up sound) is ensured to ensure that the content output and data acquisition are synchronized to avoid interruption of interaction.

[0069] Taking the hardware-content adaptation in the smart customer service scene as an example, the multi-modal interactive response content is a text file (UTF-8 encoding) and a wav format voice (48000 Hz sampling rate) corresponding to an "angry" emotion, the smart hardware terminal adapts the text display resolution to 1080P and supports wav audio decoding, and the content adaptation is successfully completed. The camera (supporting 1080P real-time shooting) and microphone array of the smart hardware terminal are started, the camera continuously captures the user's face picture, the microphone array directional collects the user's subsequent response voice, and the voice is converted into text in real time, and a "image-voice-text" three-in-one data acquisition channel is constructed. The hardware real-time feedback state information "camera normal, microphone pickup normal" confirms that the content output and data acquisition can be executed synchronously to generate a hardware-content linkage interactive basis.

[0070] Start the cycle listening mechanism, continuously collect user's subsequent image, voice, text multi-modal feedback data, combine historical interaction records and emotional change trajectory, dynamically update user's emotional state and scene demand, generate real-time interaction update data. Continuously collect user's multi-modal feedback data, combine historical interaction records and emotional change trajectory, dynamically update user's emotional state and scene demand, form real-time interaction update data. The cycle listening mechanism runs as follows: set a fixed listening period (e.g. every 500ms), continuously detect whether the user has image changes, voice input or text input, to ensure that no user interaction feedback is missed, while avoiding excessive occupation of hardware resources. The camera collects the user's facial expression change image, the microphone array collects the user's response voice and converts it into text, and integrates it into image, voice and text multi-modal feedback data to ensure data integrity. Combine historical interaction records (e.g. previous user emotion is angry, demand is device failure) with current feedback data (e.g. user facial expression is relaxed, voice tone is smooth), dynamically adjust user's emotional state (angry→neutral), update scene demand (from "emergency fault handling" to "fault progress consultation"), and generate interaction update data containing real-time emotion and demand changes.

[0071] Continuously interact with the above customer service scene, start the cycle listening mechanism, collect user feedback every 500ms, the user responds "when can the specialist contact me", the microphone array collects the voice and converts it into text, the camera captures the user's facial expression which is more relaxed than before, without obvious angry expression. Combine historical interaction records (user's previous emotion is angry, demand is to handle the dead director station), analyze current feedback data: user's voice tone is smooth, face has no angry features, text demand focuses on "specialist contact time", dynamically update user's emotional state to "neutral", and scene demand to "consult fault handling progress"; generate real-time interaction update data: current emotion = neutral, current demand = consult specialist contact time, historical emotion trajectory = angry→neutral, scene type = complaint handling progress consultation.

[0072] Input the real-time interaction update data into the emotion analysis module and the strategy matching module, iteratively optimize the text strategy and the voice strategy, adjust the response text content and the voice feature parameters, and generate a dynamic response scheme that adapts to the real-time emotion and the scene. Input the real-time interaction update data into the emotion analysis module and the strategy matching module, iteratively optimize the text strategy and the voice strategy, adjust the response text content and the voice feature parameters, and form a dynamic response scheme that adapts to the real-time emotion and the scene. Input the real-time interaction update data (image, voice, text feedback) into the emotion analysis module, re-execute the emotion recognition process (feature extraction→cross-modal fusion→emotion classification), and confirm the real-time emotion label.

[0073] The policy matching module adjusts the text strategy (e.g., from "pacification + solution" to "progress notification + patient answer") and the voice strategy (e.g., from 0.7 times to 1.0 times in speech speed, from 45 to 50 in volume) based on the real-time emotion and the updated scene demand, ensuring that the strategy fits the current interaction state. According to the optimized strategy, the response text content (focusing on progress answer) and voice feature parameters (speech speed, tone, volume) are adjusted to generate a dynamic response scheme, ensuring that the response matches the user's real-time state.

[0074] With the above user emotion turning to "neutral" interaction update data, the user's real-time facial image, response voice, and converted text are input into the emotion analysis module, and the emotion is re-identified as "neutral" and the scene demand is "consultation officer contact time". The text strategy is adjusted to "inform progress + clear time + friendly pacification", and the text parameter temperature = 1.0 (normal randomness); the voice strategy is adjusted to volume = 50, rate = 1.0 (normal speed), pitch = 1.0 (baseline tone). The response text content is adjusted to "Hello! The officer responsible for handling your device failure will contact you within 30 minutes, and if you have any other questions, you can inform us at any time ~", and the voice parameters are configured according to the optimized strategy to generate a dynamic response scheme.

[0075] The dynamic response scheme is executed through intelligent hardware devices, playing emotional voice and synchronously feeding back response text, while continuously listening to user interaction instructions until detecting an end instruction, generating an intelligent interaction response result that meets the user's emotion and scene demand. The intelligent hardware terminal outputs the response text on the display screen and plays the emotional voice (according to the optimized voice parameters) through the loudspeaker according to the dynamic response scheme, ensuring that the text and voice are output synchronously and improving the interaction experience. During the loop listening process, it continuously detects whether the user has issued an end instruction, which includes explicit voice instructions (such as "end consultation"), text instructions (such as input "exit"), and specific action instructions (such as the camera detecting that the user has left the screen for more than 30 seconds). If an end instruction is detected, the loop listening and data collection are terminated, and the whole process of interaction records (emotion change, appeal response, processing progress) are integrated to generate an intelligent interaction response result containing the whole process of interaction information and the final processing result; if no end instruction is detected, it returns to the "loop listening" step and continues the continuous interaction.

[0076] After the dynamic response scheme is executed, the intelligent hardware terminal displays the text "Hello! The staff responsible for handling your device failure will contact you within 30 minutes, and if you have any other questions, please let us know at any time" on the screen, and plays the corresponding voice according to the parameters of volume=50, rate=1.0, and pitch=1.0. The user's subsequent instructions are looped and listened to. The user responds "OK, I know" and does not issue an end instruction, so data collection and listening are continued. About 5 minutes later, the user says "end consultation", and the microphone array captures the voice and identifies it as an end instruction, terminating the looped listening. The whole process of interaction records (initial user emotion anger -> neutral, appeal from device failure handling -> consultation contact time, response result is to inform the staff to contact within 30 minutes) are integrated to generate the final intelligent interaction response result, and the interaction closed loop is clear.

[0077] As shown in Figure 2 An intelligent voice interaction device includes:

[0078] An acquisition module 201 is configured to acquire user image, voice, and text multi-modal original data.

[0079] The processing module 202 is configured to detect and crop the human face in the image data by the RetinaFace model, perform BGR to RGB format conversion, and then perform preprocessing, which includes normalization and tensor conversion, while adjusting the image to 224x224 RGB format, and then extracting 128-dimensional emotion features through the ResNet-50 network model; for the 16 kHz sampling rate voice data, 40-dimensional MFCC coefficients are extracted through the torchaudio library, and during the extraction process, the parameters of n_fft, hop_length and n_mels are configured as 400, 160 and 128 respectively, and then time dimension average pooling is performed after the extraction is completed; the Chinese text data is extracted through the paraphrase-multilingual-MiniLM-L12-v2 model of the Sentence-BERT to extract 768-dimensional semantic embedding vectors, and standardized image, voice and text single-modal feature data is generated; the standardized single-modal feature data is processed, the voice feature is accurately converted from 40 dimensions to 256 dimensions through a dedicated linear projection layer, the tone and energy information related to the voice emotion are preserved during the conversion process, and after the linear projection is unified in dimension, the feature fusion is performed through the cross-modal attention mechanism and the Transformer encoder, the emotion label is predicted by the emotion classifier, and the user emotion recognition result is generated; the user emotion recognition result is processed, the Plutchik emotion wheel theory is used as a basis, the business scene and role constraints are combined, the corresponding text strategy and voice strategy are matched, and the dual-response strategy adapted to the emotion is generated, wherein the voice strategy clearly responds to the reference pitch, speech speed coefficient, volume size and pause rules of the voice, and different emotions are configured differently; the dual-response strategy adapted to the emotion is processed to generate multi-modal interactive response content; the multi-modal interactive response content is processed, and the intelligent hardware device is combined to realize continuous interaction through a loop listening mechanism.

[0080] The response module 203 is configured to play emotional voice through a loudspeaker, continuously collect subsequent voice feedback of a user through a microphone array, convert the voice feedback into text in real time, monitor emotion changes, and generate an intelligent interactive response result meeting the needs of the user emotion and the scene.

[0081] A computing device includes a memory for storing computer program instructions and a processor for executing the computer program instructions, wherein when the computer program instructions are executed by the processor, the device is triggered to perform any one of the intelligent voice interaction methods.

[0082] The methods and / or embodiments in the present application can be implemented as a computer software program. For example, embodiments of the present disclosure include a computer program product comprising a computer program that is carried on a computer readable medium, the computer program comprising program code for executing the methods illustrated in the flowcharts. When the computer program is executed by a processing unit, the above-mentioned functions defined in the methods of the present application are performed.

[0083] It should be noted that the computer readable medium in the present application can be a computer readable signal medium or a computer readable storage medium or any combination of the two. More specific examples of the computer readable storage medium can include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any appropriate combination of the foregoing. In the present application, the computer readable medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus or device.

[0084] Computer program code for carrying out operations of the present application can be written in one or more programming languages or combinations of languages including object oriented programming languages such as Java, Smalltalk, C++ Python, conventional procedural programming languages such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0085] It is apparent that a person skilled in the art would not limit the present application to the details of the above exemplary embodiments, but would understand that the present application can be implemented in other concrete forms without departing from the spirit or essential characteristics of the present application. Therefore, the embodiments should be considered in all respects as illustrative and not restrictive, the scope of the present application being defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and range of equivalents of the claims be embraced therein.

Claims

1. An intelligent voice interaction method, characterized in that, include: Acquire raw multimodal data of user images, voice, and text; The RetinaFace model is used to detect and crop faces in the image data. First, the BGR to RGB format conversion is performed, and then preprocessing is performed. The preprocessing process includes normalization and tensor transformation. At the same time, the image is adjusted to 224×224 RGB format. Then, 128-dimensional emotion features are extracted by the ResNet-50 network model. For speech data with a sampling rate of 16kHz, 40-dimensional MFCC coefficients are extracted using the torchaudio library. During the extraction process, parameters n_fft is configured as 400, hop_length as 160, and n_mels as 128. After extraction, time-dimensional average pooling is performed. For Chinese text data, 768-dimensional semantic embedding vectors are extracted using the Sentence-BERT paraphrase-multilingual-MiniLM-L12-v2 model to generate standardized image, speech, and text single-modal feature data. Standardized single-modal feature data is processed. Speech features are accurately transformed from 40 dimensions to 256 dimensions through a dedicated linear projection layer. During the transformation process, tone and energy information related to speech emotion are preserved. After unifying the dimensions through linear projection, the features are fused with the Transformer encoder through a cross-modal attention mechanism. The feature is then input into the emotion classifier to predict the emotion label and generate the user emotion recognition result. The user emotion recognition results are processed based on the Plutchik emotion wheel theory, combined with business scenarios and role constraints, to match corresponding text and voice strategies and generate emotion-adaptive dual-response strategies. This includes processing the user emotion recognition results and constructing a structured emotion vector containing dominant emotion, secondary emotion, intensity, polarity, and triggering scenarios, based on the Plutchik emotion wheel theory. Combined with the business scenario priority matrix and role constraints, a three-way matching result of emotion-scenario-role is generated. The three-way matching result is then processed by calling a multimodal expression strategy library to match the corresponding text and voice strategies for emotion combinations. The text strategy specifies the DeepSeek prompt word requirements and generation parameters, while the voice strategy determines voice feature parameters including speech rate, tone, and volume. The matched text and speech strategies are processed, and a conflict resolution mechanism is activated. Intensity attenuation and compensation strategies are implemented for text-speech conflicts, and an emotion neutralization algorithm is used for emotion-opposing conflicts. A dual-response strategy with emotion adaptation is generated. The speech strategy specifies the reference pitch, speech rate coefficient, volume, and pause rules of the responding speech, and different emotions correspond to different configurations. The dual-response strategy for emotion adaptation is processed to generate multimodal interactive response content; The system processes multimodal interactive responses and, in conjunction with smart hardware devices, achieves continuous interaction through a loop monitoring mechanism. It plays emotional voice messages through a speaker while a microphone array continuously collects subsequent voice feedback from the user, converts it into text in real time, monitors emotional changes, and generates intelligent interactive response results that match the user's emotions and the needs of the scenario.

2. The intelligent voice interaction method according to claim 1, characterized in that, Standardized single-modal feature data is processed. Speech features are precisely transformed from 40 dimensions to 256 dimensions through a dedicated linear projection layer. During the transformation, tone and energy information related to speech emotion are preserved. After unifying the dimensions through linear projection, features are fused with a cross-modal attention mechanism and a Transformer encoder. The fused features are then input into an emotion classifier to predict emotion labels, generating user emotion recognition results, including: Standardized single-modal feature data is processed by introducing a linear projection layer to transform the dimensions of image, speech, and text single-modal features respectively. The image features are projected from 128 dimensions, the speech features from 40 dimensions, and the text features from 768 dimensions to 256 dimensions, generating single-modal feature data with unified dimensions. The unimodal feature data with uniform dimensions is processed by first concatenating them into a feature sequence, then using a cross-modal attention mechanism composed of multi-head self-attention to achieve feature interaction, and then inputting it into a Transformer encoder with two layers of encoders to further refine the feature relationships and generate fused multimodal feature data. The fused multimodal feature data is processed, and a 256-dimensional fused feature vector is obtained through mean pooling. This vector is then input into an emotion classifier with 6 output nodes. After Softmax normalization, the probability of each emotion is calculated, and the label corresponding to the maximum probability is taken to generate user emotion recognition results containing six categories: anger, sadness, happiness, fear, surprise, and neutrality.

3. The intelligent voice interaction method according to claim 1, characterized in that, The dual-response strategy for emotion adaptation is processed to generate multimodal interactive response content, including: Extract and process the dual-response strategy for emotion adaptation, user input text, and historical dialogue records. Use preset prompt word templates to complete the text generation rules and parameters, and construct the DeepSeekAPI call request. Process DeepSeekAPI call requests and call the target large language model to generate response text that meets the emotion adaptation requirements; The response text and speech strategy parameters are processed, the preset emotional TTS interface is called, and the corresponding speech rate, tone, volume parameters and output settings of 48000Hz sampling rate and wav format are configured to synthesize response speech with emotional features. The response text and response speech are integrated to generate multimodal interactive response content that includes emotion-adapted text content and emotional speech files.

4. The intelligent voice interaction method according to claim 1, characterized in that, The system processes multimodal interactive responses, integrates with smart hardware devices, and achieves continuous interaction through a loop monitoring mechanism. It plays emotionally charged voice messages through a speaker while a microphone array continuously collects subsequent user voice feedback, converts it into text in real time, monitors emotional changes, and generates intelligent interactive response results that match the user's emotions and the context's needs. These results include: The multimodal interactive response content is adapted and integrated with smart hardware devices, and a real-time data acquisition channel is built through the terminal camera and microphone array to generate the basis for hardware-content linkage interaction. A loop monitoring mechanism is initiated to continuously collect subsequent multimodal feedback data from users, including images, voice, and text. Combined with historical interaction records and emotional change trajectories, the system dynamically updates users' emotional states and scenario needs, generating real-time interaction update data. Real-time interactive update data is input into the sentiment analysis module and the strategy matching module to iteratively optimize the text strategy and voice strategy, adjust the response text content and voice feature parameters, and generate a dynamic response scheme that adapts to real-time emotions and scenarios. The system executes a dynamic response scheme through smart hardware devices, plays emotional voice messages and provides simultaneous feedback text, while continuously listening for user interaction commands until an end command is detected, generating intelligent interactive response results that match the user's emotions and the needs of the scenario.

5. An intelligent voice interaction device, characterized in that, The device includes: The acquisition module is used to acquire raw multimodal data of user images, voice, and text. The processing module detects and crops faces in image data using the RetinaFace model. It first performs BGR to RGB conversion, then preprocesses the data, including normalization and tensor transformation, adjusting the image to 224×224 RGB format. Subsequently, it extracts 128-dimensional emotion features using a ResNet-50 network model. For 16kHz sampling rate speech data, it extracts 40-dimensional MFCC coefficients using the torchaudio library, configuring parameters n_fft to 400, hop_length to 160, and n_mels to 128. After extraction, it performs time-dimensional average pooling. For Chinese text... This data uses the Sentence-BERT paraphrase-multilingual-MiniLM-L12-v2 model to extract 768-dimensional semantic embedding vectors, generating standardized image, speech, and text unimodal feature data. The standardized unimodal feature data is then processed: speech features are precisely transformed from 40 dimensions to 256 dimensions using a dedicated linear projection layer, preserving emotion-related intonation and energy information during the transformation. After unifying the dimensions through linear projection, features are fused with a cross-modal attention mechanism and a Transformer encoder, then input into an emotion classifier to predict emotion labels, generating user emotion recognition results. The user emotion recognition results are processed based on the Plutchik emotion wheel theory, combined with business scenarios and role constraints, to match corresponding text and voice strategies, generating a dual-response strategy for emotion adaptation. This includes processing the user emotion recognition results, constructing a structured emotion vector containing dominant emotion, secondary emotion, intensity, polarity, and triggering scenarios based on the Plutchik emotion wheel theory, and combining it with a business scenario priority matrix and role constraints to generate a three-way matching result of emotion-scenario-role. The three-way matching result is then processed by calling a multimodal expression strategy library to match the corresponding text and voice strategies for emotion combinations. The text strategy explicitly defines the DeepSe... The system defines the requirements and generation parameters for eK prompt words, and determines speech feature parameters including speech rate, tone, and volume. It processes the matched text and speech strategies, initiates a conflict resolution mechanism, implements intensity attenuation and compensation strategies for text-speech conflicts, and employs an emotion neutralization algorithm for emotion-contradictory conflicts, generating an emotion-adapted dual-response strategy. The speech strategy specifies the baseline pitch, speech rate coefficient, volume, and pause rules for the response speech, with differentiated configurations for different emotions. The emotion-adapted dual-response strategy is then processed to generate multimodal interactive response content. This multimodal interactive response content is then processed and, in conjunction with smart hardware devices, achieves continuous interaction through a loop monitoring mechanism. The response module plays emotional voice through a speaker, while the microphone array continuously collects the user's subsequent voice feedback, converts it into text in real time, monitors emotional changes, and generates intelligent interactive response results that match the user's emotions and the needs of the scenario.

6. An electronic device, characterized in that, include: First processor; and memory for storing executable instructions of the first processor; The first processor is configured to execute the intelligent voice interaction method according to any one of claims 1 to 4 by executing the executable instructions.

7. A computing device, the device comprising a memory for storing computer program instructions and a second processor for executing the computer program instructions, wherein, When the computer program instructions are executed by the second processor, the device is triggered to execute the intelligent voice interaction method according to any one of claims 1 to 4.

Images