Multi-modal feature fused AI interactive voice intention recognition method and system

Abstract

The invention relates to the technical field of voice processing, and discloses an AI interactive voice intention recognition method fusing multi-modal features, and the method comprises the steps: collecting a voice signal, a face video and a depth image of a user, and carrying out the preprocessing; determining facial expression features, a head attitude angle and a gesture category according to the facial video and the depth image; converting the voice signal, the facial expression feature, the head attitude angle and the gesture category into a voice semantic vector, a text semantic vector and a visual semantic vector, and inputting the voice semantic vector, the text semantic vector and the visual semantic vector into a cross-modal alignment module for semantic space alignment to obtain a high-dimensional intention representation; inputting the high-dimensional intention representation into a lightweight intention classification head, outputting intention category probability distribution, dynamically evaluating intention recognition confidence, and if the intention recognition confidence is lower than a preset threshold, triggering a clarification interaction strategy; feedback signals are collected, and a final intent is determined based on the feedback signals. According to the method, the intention recognition accuracy is improved through multi-mode collaborative perception fusion.

Description

A method and system for AI interactive voice intent recognition that integrates multimodal features Technical Field

[0001] This invention relates to the field of speech processing technology, and in particular to an AI interactive speech intent recognition method and system that integrates multimodal features. Background Technology

[0002] With the rapid development of artificial intelligence and human-computer interaction technologies, application scenarios such as intelligent voice assistants, service robots, and in-vehicle interaction systems are placing higher demands on the accuracy, robustness, and naturalness of user intent recognition. Traditional voice intent recognition methods mainly rely on a single voice modality, converting speech into text through automatic speech recognition (ASR) and then using natural language processing models for intent classification. However, such methods often fail to accurately capture the user's true intent in noisy environments, with different accents, semantic ambiguity, or incomplete user expression, leading to interaction failures or erroneous responses.

[0003] In recent years, multimodal fusion technology has gradually become an important direction for improving the performance of intent recognition. By introducing visual information (such as facial expressions, head posture, and gestures), it is possible to effectively supplement the missing or ambiguous semantic content of speech signals. However, existing technologies generally have the following shortcomings: multimodal data acquisition devices are scattered and difficult to synchronize, resulting in low spatiotemporal alignment accuracy; the feature extraction methods of each modality are isolated, lacking a deep alignment mechanism under a unified semantic space; the fusion strategies are simple (such as concatenation or weighted averaging), failing to fully model the dynamic correlation between modalities; and there is a lack of quantitative evaluation of the confidence of recognition results, making it impossible to proactively initiate clarification interactions in low-confidence scenarios, affecting user experience and system reliability.

[0004] Therefore, there is an urgent need for an AI-powered interactive voice intent recognition method that can efficiently integrate multimodal information from speech, text, and vision, and possess adaptive confidence assessment and interactive feedback capabilities. Summary of the Invention

[0005] The purpose of this invention is to provide an AI interactive voice intent recognition method and system that integrates multimodal features, in order to solve the above-mentioned problems.

[0006] This invention provides an AI interactive voice intent recognition method that integrates multimodal features, comprising: acquiring user voice signals, facial videos, and depth images through an integrated microphone array and RGB-D camera, and preprocessing them; detecting and extracting facial expression features and head posture angles from the facial videos, and determining the user's gesture category based on the depth images; inputting the voice signals into a pre-trained voice encoder to output a voice semantic vector, inputting the text transcribed from the voice signals into a large voice model to output a text semantic vector, and encoding the facial expression features, head posture angles, and gesture categories to obtain a visual semantic vector; inputting the voice semantic vector, text semantic vector, and visual semantic vector into a cross-modal alignment module for semantic space alignment, introducing a multi-head cross-attention mechanism to determine the correlation weights between each modality, and obtaining a high-dimensional intent representation based on a hierarchical intent fusion network; inputting the high-dimensional intent representation into a lightweight intent classification head to output an intent category probability distribution, introducing an uncertainty perception module to dynamically evaluate the intent recognition confidence, and triggering a clarification interaction strategy if the intent recognition confidence is lower than a preset threshold; collecting feedback signals from the clarification interaction strategy, and determining the final intent based on the feedback signals.

[0007] Preferably, preprocessing is performed, including: noise reduction, endpoint detection, and sampling rate unification of the speech signal; and temporal alignment of the speech signal, facial video, and depth image.

[0008] Preferably, the head pose angle is extracted by detecting the facial video, including: converting the video frames of the facial video into RGB channel format and normalizing the pixel values ​​to the [0,1] interval; using a deep learning-based face detection model to detect the video frames and locate all face bounding boxes; cropping the face bounding boxes to obtain a face cropping region, inputting the face cropping region into a face key point detection model to obtain facial key point coordinates; loading a standard 3D face model, matching the facial key point coordinates with the model points of the standard 3D face model to determine the rotation vector and translation vector; and calculating Euler angles based on the rotation vector and translation vector to obtain the head pose angle.

[0009] Preferably, determining the user's gesture category based on the depth image includes: filling the depth image with invalid values ​​and segmenting it using a depth threshold to filter out the foreground region; locating and cropping the upper body region of the human body within the foreground region based on the face bounding box; inputting the cropped depth image into a pre-trained lightweight 3D human pose estimation model to output the 3D coordinates of the upper body key points; constructing the user's upper body skeleton based on the 3D coordinates of the upper body key points and defining bone vectors to calculate key joint angles and relative limb positions; cropping a local area of ​​the hand based on the key joint angles and relative limb positions, calculating gesture features based on 3D coordinates, and determining the gesture category based on the gesture features, wherein the gesture features include finger bending angle, palm opening, and fingertip spacing.

[0010] Preferably, the speech semantic vector, text semantic vector, and visual semantic vector are input into a cross-modal alignment module for semantic space alignment, and a multi-head cross-attention mechanism is introduced to determine the relevance weights between each modality. This includes: unifying the embedding dimensions of the speech semantic vector, text semantic vector, and visual semantic vector through the cross-modal alignment module, stacking them into an embedding sequence, and assigning a learnable positional code to each modality to obtain a modal input with positional information; setting the hyperparameters of the multi-head cross-attention mechanism, performing linear projection on the modal input to generate a query, key, and value matrix, and splitting the query, key, and value matrix into several heads; calculating scaled dot product attention for each head, and determining the relevance weights between each modality based on the scaled dot product attention.

[0011] Preferably, obtaining a high-dimensional intent representation based on a hierarchical intent fusion network includes: generating a modality fusion feature vector based on the scaled dot product attention; inputting the modality fusion feature vector into a residual feedforward network to obtain an enhanced fusion feature vector; and inputting the enhanced fusion feature vector into a lightweight explicit intent classifier to obtain a predefined explicit intent.

[0012] Preferably, the high-dimensional intent representation obtained based on the hierarchical intent fusion network includes: concatenating the dialogue history state vector output by the memory-enhanced LSTM with the current enhanced fusion feature vector, and dynamically adjusting the fusion ratio of historical information and current information through a gating mechanism; using the memory-enhanced LSTM to perform contextual semantic modeling on the concatenated vector, capturing long-distance dependencies, and outputting implicit intent features; hierarchically fusing predefined explicit intent and implicit intent features, highlighting key intent cues through an attention mechanism, and finally generating a high-dimensional intent representation containing explicit and implicit intent information.

[0013] Preferably, inputting the high-dimensional intent representation into a lightweight intent classification head to output the intent category probability distribution includes: inputting the high-dimensional intent representation into a lightweight intent classification head composed of two fully connected layers, wherein the first layer uses the ReLU activation function for nonlinear transformation, and the second layer outputs the intent category probability distribution through the Softmax function.

[0014] Preferably, an uncertainty perception module is introduced to dynamically evaluate the confidence level of intent recognition, including: in the uncertainty perception module, analyzing the probability distribution of intent categories output by the lightweight intent classification head and calculating the entropy value of the intent category probability distribution; simultaneously, introducing a Monte Carlo algorithm to perform multiple random dropout operations on the fully connected layer of the lightweight intent classification head during the inference phase to obtain multiple different intent category probability distribution samples, and quantifying the cognitive uncertainty of the model by calculating the prediction variance between the samples; and weightedly fusing the entropy value and the prediction variance to obtain a comprehensive confidence level evaluation value for intent recognition.

[0015] This invention also discloses an AI interactive voice intent recognition method and system that integrates multimodal features, used to apply the aforementioned AI interactive voice intent recognition method integrating multimodal features. The system includes: a data acquisition and processing module, configured to acquire the user's voice signal, facial video, and depth image through an integrated microphone array and an RGB-D camera, and perform preprocessing; detect and extract facial expression features and head pose angles from the facial video, and determine the user's gesture category based on the depth image; and a semantic conversion module, configured to input the voice signal into a pre-trained voice encoder to output a voice semantic vector, input the text transcribed from the voice signal into a large voice model to output a text semantic vector, and convert the facial expression... Features, head pose angle, and gesture category are encoded to obtain a visual semantic vector; the semantic fusion module is configured to input the speech semantic vector, text semantic vector, and visual semantic vector into a cross-modal alignment module for semantic space alignment, introduce a multi-head cross-attention mechanism to determine the relevance weights between each modality, and obtain a high-dimensional intent representation based on a hierarchical intent fusion network; the intent determination module is configured to input the high-dimensional intent representation into a lightweight intent classification head to output the intent category probability distribution, introduce an uncertainty perception module to dynamically evaluate the intent recognition confidence, and if the intent recognition confidence is lower than a preset threshold, a clarification interaction strategy is triggered; feedback signals of the clarification interaction strategy are collected, and the final intent is determined based on the feedback signals.

[0016] Compared with the prior art, the beneficial effects of the present invention are that by integrating a microphone array and an RGB-D camera, the present invention simultaneously acquires voice signals, facial videos and depth images, and combines time dimension alignment, deep learning-based facial key point detection and 3D pose estimation, and depth image-based gesture category recognition, it achieves accurate quantification of non-verbal cues such as user expressions, head postures and gestures, providing rich and reliable visual semantic input for intent recognition.

[0017] By utilizing a cross-modal alignment module to map speech semantic vectors, text semantic vectors, and visual semantic vectors to a unified embedding space, and introducing a multi-head cross-attention mechanism to dynamically calculate the relevance weights between each modality, the system can adaptively focus on the most discriminative modal information based on the current interaction context (such as relying more on visual cues in noisy environments), significantly improving the accuracy and robustness of intent recognition.

[0018] By using a hierarchical intent fusion network, on the one hand, the residual feedforward network is used to enhance the multimodal fusion features and output predefined explicit intents; on the other hand, memory-enhanced LSTM is introduced to model the dialogue history state and dynamically fuse it with the current fusion features through a gating mechanism to capture long-distance contextual dependencies and generate high-dimensional representations containing implicit intent information such as emotions and habits, so that the system can not only understand "what was said" but also infer "what was intended to be done".

[0019] Based on the lightweight intent classification head, an uncertainty perception module is introduced. By weighted fusion of probability distribution entropy and Monte Carlo Dropout prediction variance, the cognitive uncertainty and data uncertainty of the model are quantified, enabling dynamic evaluation of intent recognition confidence. When the confidence is lower than the threshold, a clarification interaction strategy (such as asking questions or confirming options) is automatically triggered, and the final intent is corrected based on user feedback signals. This significantly improves the system's fault tolerance and user experience in scenarios with ambiguous, conflicting, or low-quality input. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0021] Figure 1 is a flowchart illustrating an AI interactive voice intent recognition method that integrates multimodal features according to the present invention.

[0022] Figure 2 is a functional block diagram of an AI interactive voice intent recognition system that integrates multimodal features according to the present invention. Detailed Implementation

[0023] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0024] As shown in Figure 1, this invention provides an AI interactive voice intent recognition method that integrates multimodal features, comprising: acquiring user voice signals, facial videos, and depth images through an integrated microphone array and RGB-D camera, and performing preprocessing; detecting and extracting facial expression features and head posture angles from the facial videos, and determining the user's gesture category based on the depth images; inputting the voice signals into a pre-trained voice encoder to output a voice semantic vector, inputting the text transcribed from the voice signals into a large voice model to output a text semantic vector, and encoding the facial expression features, head posture angles, and gesture categories to obtain a visual semantic vector; inputting the voice semantic vector, text semantic vector, and visual semantic vector into a cross-modal alignment module for semantic space alignment, introducing a multi-head cross-attention mechanism to determine the correlation weights between each modality, and obtaining a high-dimensional intent representation based on a hierarchical intent fusion network; inputting the high-dimensional intent representation into a lightweight intent classification head to output an intent category probability distribution, introducing an uncertainty perception module to dynamically evaluate the intent recognition confidence, and triggering a clarification interaction strategy if the intent recognition confidence is lower than a preset threshold; collecting feedback signals from the clarification interaction strategy, and determining the final intent based on the feedback signals.

[0025] This invention effectively overcomes the limitations of single-modal speech recognition, which is susceptible to environmental noise, accent differences, and semantic ambiguity, through deep fusion of multimodal features. Specifically, it integrates a microphone array and an RGB-D camera to collaboratively acquire speech, facial video, and depth images, achieving complementary multi-dimensional information: the speech signal is processed by a pre-trained encoder and a large language model to construct speech semantic vectors and text semantic vectors from the acoustic and text semantic levels, respectively; the encoding of facial expression features, head posture angles, and gesture categories forms visual semantic vectors, intuitively reflecting the user's emotional state and body language intentions. The cross-modal alignment module accurately captures the dynamic correlation between different modalities using a multi-head cross-attention mechanism, dynamically allocating weights to enable deep interaction of different modal information in a unified semantic space. The hierarchical intent fusion network further integrates multimodal semantic information, generating a more comprehensive and robust high-dimensional intent representation. The lightweight intent classification head, while ensuring recognition efficiency, combines an uncertainty perception module to evaluate recognition confidence in real time. When confidence is insufficient, a clarification interaction strategy is triggered. By actively acquiring user feedback signals, the accuracy and reliability of intent recognition in complex scenarios are significantly improved.

[0026] In some embodiments of this application, preprocessing is performed, including: noise reduction, endpoint detection, and sampling rate unification of the speech signal; and temporal alignment of the speech signal, facial video, and depth image.

[0027] Understandably, by ensuring that multimodal input data is strictly synchronized in time, uniform in format, and minimizes noise interference, a high-quality, aligned input foundation is provided for subsequent feature extraction and cross-modal fusion.

[0028] Specifically, the speech signal is subjected to noise reduction, endpoint detection, and unified sampling rate processing, including: using noise reduction algorithms based on spectral subtraction or deep neural networks to suppress the interference of environmental noise on the speech content; identifying effective speech segments in the speech signal through endpoint detection (Voice Activity Detection, VAD) technology, removing preceding and following silences or non-speech segments, and reducing irrelevant information; and resampling the speech signal to a sampling rate uniformly set by the system (e.g., 16 kHz) to ensure consistency with the input requirements of subsequent speech encoders and text transcription modules.

[0029] Specifically, the audio signal, facial video, and depth image are aligned in time dimension, including: mapping the audio frames captured by the microphone array, the facial video frames captured by the RGB camera, and the depth image frames captured by the depth camera to the same time reference based on hardware synchronization signals or software timestamps; using the audio signal as a reference time axis, interpolating or dropping frames in the facial video and depth image to match their frame rate with the temporal granularity of the audio frames; the three types of data are aligned one-to-one in time, ensuring that the audio, facial expressions, head posture, and gesture information at the same moment come from the same user interaction behavior, avoiding semantic misalignment caused by asynchrony.

[0030] In some embodiments of this application, detecting and extracting head pose angles from the facial video includes: converting the video frames of the facial video into RGB channel format and normalizing the pixel values ​​to the [0,1] interval; using a deep learning-based face detection model to detect the video frames and locate all face bounding boxes; cropping based on the face bounding boxes to obtain a face cropping region, inputting the face cropping region into a face key point detection model to obtain facial key point coordinates; loading a standard 3D face model, matching the facial key point coordinates with the model points of the standard 3D face model to determine rotation and translation vectors; and calculating Euler angles based on the rotation and translation vectors to obtain head pose angles.

[0031] It is understandable that accurately and robustly extracting the orientation information of the user's head in three-dimensional space (i.e., head pose angle) from facial videos can provide key visual cues that reflect the user's attention direction, interaction intention, and non-verbal behavior, thus providing structured and quantifiable pose features for multimodal intent recognition.

[0032] Specifically, each frame of the facial video is converted to a standard RGB three-channel format to ensure that the input data is consistent with the expected input of the subsequent deep learning model. Simultaneously, pixel values ​​are normalized to the [0,1] range to eliminate numerical differences caused by varying lighting, equipment, or acquisition conditions, thus improving the stability and generalization ability of the model. Next, a deep learning-based face detection model (such as MTCNN, RetinaFace, or YOLOv5-face) is used to process the normalized video frames, detecting all faces present in the image and outputting the bounding box coordinates for each face to locate its position in the image. Then, based on the detected face bounding boxes, the corresponding face regions are cropped from the original video frames to form independent face image blocks. This cropping operation eliminates background interference, focuses on local facial details, and improves the accuracy of keypoint detection. The cropped face images are then input into a pre-trained face keypoint detection model (such as HRNet, 3DDFA, etc.) to obtain a set of semantically meaningful facial keypoint coordinates, typically including 68 or more two-dimensional image coordinates for keypoints such as the eyes, nose tip, and corners of the mouth. Subsequently, a standard 3D face model (such as a general 3D Morphable Model or a predefined 68-point 3D reference template) is loaded. This model contains 3D spatial coordinates that correspond one-to-one with the aforementioned 2D keypoints. By solving the perspective n-point problem, the 2D keypoints are matched with the 3D model points, and an optimization algorithm is used to calculate the rotation and translation vectors describing the face's pose in the camera coordinate system. Finally, the rotation vectors are converted into rotation matrices and further decomposed into Euler angles about the X-axis (pitch), Y-axis (yaw), and Z-axis (roll) to obtain the user's head pose angles. These three angles collectively characterize the specific orientation of the head in three-dimensional space and can be used to determine whether the user is facing the device, nodding / shaking their head, or engaging in other interactive behaviors, providing important contextual information for intent recognition.

[0033] In some embodiments of this application, determining the user's gesture category based on the depth image includes: padding the depth image with invalid values, segmenting it using a depth threshold, filtering out the foreground region, locating and cropping the upper body region of the human body in the foreground region based on the face bounding box; inputting the cropped depth image into a pre-trained lightweight 3D human pose estimation model, outputting the 3D coordinates of the upper body key points; constructing the user's upper body skeleton based on the 3D coordinates of the upper body key points, defining bone vectors, and calculating key joint angles and relative limb positions; cropping a local area of ​​the hand based on the key joint angles and relative limb positions, calculating gesture features based on 3D coordinates, and determining the gesture category based on the gesture features, wherein the gesture features include finger bending angle, palm opening, and fingertip spacing.

[0034] Understandably, identifying a user's gesture category from depth images involves extracting gesture features through a series of processing and analysis steps, and then accurately classifying the gestures based on these features.

[0035] Specifically, a depth image is a two-dimensional image that records the distance information from each pixel in a scene to the camera (or sensor). Unlike ordinary color images (RGB images) that only represent color, each pixel value in a depth image represents the depth (i.e., distance) in physical space, usually measured in millimeters (mm) or meters (m). Determining the user's gesture category involves: padding the depth image with invalid values, i.e., filling in missing data areas caused by sensor occlusion or reflections, ensuring data integrity for subsequent processing. Then, an appropriate depth threshold is set to segment the foreground (human body) from the background. This threshold is used to filter out the foreground region, which contains the user's body information. Based on the previously determined face bounding box, the upper body region is located and cropped within the foreground region. This process helps reduce computational load, focusing on key areas that may contain gestures. The cropped upper body depth image is input into a pre-trained lightweight 3D human pose estimation model (e.g., OpenPose 3D), which outputs the coordinates of upper body key points in 3D space, including the positions of joints such as shoulders, elbows, and wrists. Based on the obtained 3D coordinates, a user's upper body skeleton model is constructed, and the vector relationships between each bone segment are defined. Next, the angles of key joints (such as the elbow flexion angle) and the relative positions of each limb are calculated to describe the movement state of the hand and its surrounding area. Subsequently, based on the key joint angles and relative limb positions, the local area of ​​the hand is further precisely located, and detailed gesture features are extracted again based on 3D coordinates, including but not limited to the degree of finger flexion, whether the palm is open and the degree of opening, and the distance between fingertips. Finally, based on the extracted gesture features, a pre-trained classifier (such as Support Vector Machine (SVM), Random Forest (RF), or neural network) is used to determine the gesture category. These gesture categories can cover a variety of common gestures, such as "clenching a fist," "extending the index finger," and "OK gesture," providing input signals for subsequent application logic. In this way, the system can understand the user's intentions and execute corresponding commands through gestures.

[0036] In some embodiments of this application, the speech semantic vector, text semantic vector, and visual semantic vector are input into a cross-modal alignment module for semantic space alignment. A multi-head cross-attention mechanism is introduced to determine the relevance weights between each modality. This includes: unifying the embedding dimensions of the speech semantic vector, text semantic vector, and visual semantic vector through the cross-modal alignment module, stacking them into an embedding sequence, and assigning a learnable positional code to each modality to obtain a modal input with positional information; setting the hyperparameters of the multi-head cross-attention mechanism, performing linear projection on the modal input to generate a query, key, and value matrix, and splitting the query, key, and value matrix into several heads; calculating scaled dot product attention for each head, and determining the relevance weights between each modality based on the scaled dot product attention.

[0037] It is understandable that heterogeneous semantic vectors from different perceptual modalities (speech, text, vision) are mapped into a unified semantic space, and the semantic correlation between modalities is dynamically modeled through a multi-head cross-attention mechanism, thereby achieving effective alignment and complementarity of cross-modal information and providing semantically consistent and mutually reinforcing feature representations for subsequent intent fusion.

[0038] Specifically, since the speech semantic vectors, text semantic vectors, and visual semantic vectors are generated by different encoders, their original dimensions are usually inconsistent. Therefore, a linear projection layer in the cross-modal alignment module is needed to transform the three vectors to the same embedding dimension to achieve dimensional alignment. Then, these three aligned vectors are stacked in a fixed order (e.g., speech-text-vision) into an embedding sequence of length 3, which serves as the joint input for the multimodal model. To enable the model to distinguish different modal sources and retain their identity information, a learnable positional encoding is assigned to each modality position in the stacked sequence. This positional encoding is bound to the modality type (rather than the time step) and is incorporated into the corresponding vector through addition, forming an input representation with modality identification. Next, key hyperparameters of the multi-head cross-attention mechanism are set, including the number of heads (e.g., 8 heads) and the dimension of each head. Three independent linear transformations are performed on the aforementioned modal inputs with positional information to generate corresponding query, key, and value matrices. Then, these three matrices are evenly divided into several "heads" along the feature dimensions, and each head independently computes attention within its subspace. Within each head, the query vector of a particular modality is multiplied by the key vectors of all other modalities, and then divided by a scaling factor (usually the square root of the head dimension) to obtain a scaled dot product attention score. After normalization using the Softmax function, the attention weight distribution of that query modality to other modalities is obtained. This weight reflects the degree of dependence of the current modality on other modalities when understanding semantics. The value matrix is ​​then weighted and summed using this weight to obtain the output of that head. Finally, the outputs of all heads are concatenated and fused through a linear layer to generate aligned features enhanced by cross-modal interaction. In this process, the correlation weights between modalities are automatically learned by the attention mechanism; for example, when speech is interfered with by noise, the system may automatically increase the attention to text or visual modalities. This dynamic, data-driven weight allocation mechanism effectively solves the problem of multimodal semantic heterogeneity and significantly improves the discriminativeness and robustness of the fused features.

[0039] In some embodiments of this application, obtaining a high-dimensional intent representation based on a hierarchical intent fusion network includes: generating a modality fusion feature vector based on the scaled dot product attention; inputting the modality fusion feature vector into a residual feedforward network to obtain an enhanced fusion feature vector; and inputting the enhanced fusion feature vector into a lightweight explicit intent classifier to obtain a predefined explicit intent.

[0040] Understandably, after completing cross-modal alignment, the multimodal fusion information is refined and enhanced layer by layer to generate explicit intent representations with clear structure and semantics, providing highly discriminative intermediate features for subsequent intent classification and interaction decision-making.

[0041] Specifically, based on the scaled dot product attention weights output by the aforementioned multi-head cross-attention mechanism, the value vectors corresponding to each modality are weighted and summed to obtain a modality fusion feature vector that comprehensively reflects the interactive information of speech, text, and visual modalities. This vector has initially integrated the complementary semantics between different modalities. This modality fusion feature vector is then input into a residual feedforward network, which typically consists of two fully connected layers with a non-linear activation function such as ReLU in between, and introduces residual connections (i.e., skip connections) to directly add the input features to the network output. This structure helps retain key information in the original fusion features while enhancing feature expressiveness through non-linear transformations, outputting a semantically richer and more robust enhanced fusion feature vector. The enhanced fusion feature vector is then input into a lightweight explicit intent classifier, which typically consists of one or two fully connected layers with fewer parameters, specifically designed to map high-dimensional fusion features to a predefined intent category space (such as "play music," "turn on lights," "check schedule," etc.). Its output is the predefined explicit intent corresponding to the current user interaction behavior, which serves as a structured semantic label that can be directly used for system response and also provides basic input for subsequent implicit intent modeling.

[0042] In some embodiments of this application, a high-dimensional intent representation is obtained based on a hierarchical intent fusion network, including: concatenating the dialogue history state vector output by the memory-enhanced LSTM with the current enhanced fusion feature vector, and dynamically adjusting the fusion ratio of historical information and current information through a gating mechanism; using the memory-enhanced LSTM to perform contextual semantic modeling on the concatenated vector, capturing long-distance dependencies, and outputting implicit intent features; hierarchically fusing predefined explicit intents and implicit intent features, highlighting key intent cues through an attention mechanism, and finally generating a high-dimensional intent representation containing explicit and implicit intent information.

[0043] Understandably, based on explicit intent recognition, it is necessary to further integrate multi-turn dialogue history context information, model the deep intents (such as preferences, emotions, task continuity, etc.) that users do not explicitly express but are implicit in the interaction process, and organically combine explicit instructions with implicit semantics through a hierarchical fusion mechanism to generate a high-dimensional intent representation that has both immediacy and context awareness capabilities, so as to improve the system's understanding accuracy and response intelligence in continuous and complex human-computer interaction scenarios.

[0044] Specifically, a Long Short-Term Memory (LSTM) augmented with memory is used to model the intent, behavior, and state extracted from multiple rounds of historical interactions, outputting a dialogue history state vector that represents the user's long-term interaction context. This vector encodes key semantics and temporal dependencies from past rounds. This dialogue history state vector is concatenated with the augmented fusion feature vector obtained after processing the current round's residual feedforward network, forming a joint input containing both historical and current information. Based on this, a gating mechanism (e.g., a gating unit consisting of a fully connected layer and a sigmoid activation function) is introduced to adaptively calculate a fusion weight based on the current input content, dynamically adjusting the contribution ratio of historical and current information in subsequent processing to avoid overloading historical information or diluting current key signals. The gated concatenated vector is then input back into the LSTM augmented with memory for deeper contextual semantic modeling. This LSTM, through its extended memory unit structure, effectively captures long-distance dependencies across rounds, outputting an implicit intent feature vector reflecting the user's potential needs, interaction habits, or unspoken goals. The aforementioned predefined explicit intents (usually in the form of their embedded representations or intermediate features of the classifier) ​​are hierarchically fused with implicit intent features: An attention mechanism (such as additive attention or scaled dot product attention) is used to calculate the relevance weights between explicit and implicit features, highlighting the most critical intent cues in the current interaction; then, the weighted features are concatenated or summed to generate the final high-dimensional intent representation. This representation simultaneously includes the user's explicitly expressed instructions (explicit intent) and deep semantics inferred from the context (implicit intent), providing a comprehensive and fine-grained semantic foundation for subsequent intent classification, confidence assessment, and interaction clarification.

[0045] In some embodiments of this application, inputting the high-dimensional intent representation into a lightweight intent classification head to output the intent category probability distribution includes: inputting the high-dimensional intent representation into a lightweight intent classification head composed of two fully connected layers, wherein the first layer uses the ReLU activation function for nonlinear transformation, and the second layer outputs the intent category probability distribution through the Softmax function.

[0046] Understandably, the high-dimensional intent representation obtained through multimodal fusion and contextual modeling is efficiently mapped to a predefined intent category space, outputting the probability distribution of each intent category, providing the system with quantifiable and interpretable classification results, while also taking into account the lightweight nature of the model to meet the limitations of computational efficiency and resource consumption in actual deployment.

[0047] Specifically, the high-dimensional intent representation generated above is used as input and fed into a lightweight intent classification head consisting of two fully connected layers. The first fully connected layer performs dimensionality reduction or adaptation transformation on the high-dimensional features and introduces non-linear expressive power through the ReLU activation function, enabling the model to learn the complex decision boundaries between intent categories. The output dimension of the second fully connected layer is equal to the total number of intent categories predefined by the system. Its output vector is normalized by the Softmax function and transformed into a probability distribution, where each element represents the predicted probability of the corresponding intent category. This probability distribution is not only used to select the category with the highest probability as the final recognition result, but also serves as the input basis for the subsequent uncertainty perception module. The entire classification head structure is simple and has few parameters, significantly reducing inference latency and memory consumption while ensuring classification performance. It is suitable for resource-constrained scenarios such as embedded devices, mobile terminals, or real-time voice interaction systems.

[0048] In some embodiments of this application, an uncertainty perception module is introduced to dynamically evaluate the confidence level of intent recognition, including: in the uncertainty perception module, analyzing the probability distribution of intent categories output by the lightweight intent classification head and calculating the entropy value of the intent category probability distribution; simultaneously, introducing a Monte Carlo algorithm to perform multiple random dropout operations on the fully connected layer of the lightweight intent classification head during the inference phase to obtain multiple different intent category probability distribution samples, and quantifying the cognitive uncertainty of the model by calculating the prediction variance between the samples; and weightedly fusing the entropy value and the prediction variance to obtain a comprehensive intent recognition confidence level evaluation value.

[0049] Understandably, dynamic and quantitative evaluation of the reliability of intent recognition results during the inference stage is necessary to distinguish between data uncertainty caused by input ambiguity and cognitive uncertainty caused by insufficient model knowledge. This provides a scientific basis for whether the system needs to initiate clarification interaction, thereby improving the robustness of human-computer interaction and user experience.

[0050] Specifically, the probability distribution of intent categories output by the lightweight intent classification head is analyzed, and its information entropy value is calculated. Entropy reflects the concentration of the probability distribution: when the probability of a certain intent category is close to 1 and the rest are close to 0, the entropy value is low, indicating that the model's judgment is clear; conversely, if the probabilities of each category are close to a uniform distribution, the entropy value is high, indicating that the current input has semantic ambiguity or conflict, belonging to data uncertainty (i.e., the input itself is unclear). Monte Carlo Dropout is enabled during the inference phase: the fully connected layers in the lightweight intent classification head retain the Dropout operation (usually enabled during training and disabled during inference, but enabled here during inference), and forward propagation is repeated multiple times (e.g., 10–20 times). Each time, due to the random masking of some neurons by Dropout, slightly different intent category probability distribution samples are obtained. By calculating the prediction mean and variance of these samples for each intent category, the cognitive uncertainty caused by parameter uncertainty (i.e., the model lacks sufficient confident knowledge about the current input) can be quantified. The entropy value (representing data uncertainty) and prediction variance (representing cognitive uncertainty) are weighted and fused together. The weights can be set based on application scenario experience or determined through validation set optimization to obtain a comprehensive intent recognition confidence assessment value. The lower the assessment value, the less reliable the system's judgment of the current intent is, thus triggering a clarification interaction strategy (such as asking "Do you want to play music?") to guide the user to supplement information to correct the intent understanding.

[0051] This invention also discloses an AI interactive voice intent recognition method and system that integrates multimodal features, used to apply the aforementioned AI interactive voice intent recognition method integrating multimodal features. The system includes: a data acquisition and processing module, configured to acquire the user's voice signal, facial video, and depth image through an integrated microphone array and an RGB-D camera, and perform preprocessing; detect and extract facial expression features and head pose angles from the facial video, and determine the user's gesture category based on the depth image; and a semantic conversion module, configured to input the voice signal into a pre-trained voice encoder to output a voice semantic vector, input the text transcribed from the voice signal into a large voice model to output a text semantic vector, and convert the facial expression... Features, head pose angle, and gesture category are encoded to obtain a visual semantic vector; the semantic fusion module is configured to input the speech semantic vector, text semantic vector, and visual semantic vector into a cross-modal alignment module for semantic space alignment, introduce a multi-head cross-attention mechanism to determine the relevance weights between each modality, and obtain a high-dimensional intent representation based on a hierarchical intent fusion network; the intent determination module is configured to input the high-dimensional intent representation into a lightweight intent classification head to output the intent category probability distribution, introduce an uncertainty perception module to dynamically evaluate the intent recognition confidence, and if the intent recognition confidence is lower than a preset threshold, a clarification interaction strategy is triggered; feedback signals of the clarification interaction strategy are collected, and the final intent is determined based on the feedback signals.

[0052] This invention, through an integrated microphone array and RGB-D camera acquisition and processing module, can simultaneously acquire multimodal information such as user voice signals, facial expressions, head posture, and gestures, achieving comprehensive capture of user expression intentions and avoiding the problems of single voice modalities being easily interfered with by environmental noise or semantic ambiguity. The semantic conversion module, through a pre-trained voice encoder, a large language model, and visual feature encoding, transforms different modal information into unified semantic vector representations, laying the foundation for cross-modal fusion. The voice semantic vector retains the prosody and emotional features of the voice signal, the text semantic vector enhances the accuracy of language logic, and the visual semantic vector supplements non-verbal emotional attitudes and action intentions. The semantic fusion module innovatively employs a cross-modal alignment module to achieve semantic space unification and dynamically learns the relevance weights between modalities through a multi-head cross-attention mechanism, enabling the system to adapt to the importance of modal information in different scenarios. The hierarchical intent fusion network gradually aggregates fine-grained semantic features to construct a more comprehensive high-dimensional intent representation. The intent determination module introduces a lightweight classification head and an uncertainty perception module. While ensuring recognition efficiency, it effectively solves the recognition problem in ambiguous intent scenarios by dynamically evaluating confidence and triggering clarification interaction. Combined with feedback signals, it optimizes the final intent judgment, significantly improving the accuracy and robustness of intent recognition.

[0053] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program goods. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program goods embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0054] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program goods according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more blocks of the flowchart illustrations and / or one or more blocks of the block diagrams.

[0055] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more block diagrams.

[0056] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more flowcharts and / or one or more block diagrams.

[0057] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for AI interactive voice intent recognition that integrates multimodal features, characterized in that, include: The system acquires the user's voice signals, facial video, and depth images through an integrated microphone array and RGB-D camera, and then preprocesses them. The facial video is processed to extract facial expression features and head pose angles. The user's gesture category is determined based on the depth image. The speech signal is input into a pre-trained speech encoder to output a speech semantic vector. The text transcribed from the speech signal is input into a large speech model to output a text semantic vector. The facial expression features, head pose angles, and gesture categories are encoded to obtain a visual semantic vector. The speech semantic vector, text semantic vector, and visual semantic vector are input into a cross-modal alignment module for semantic space alignment. A multi-head cross-attention mechanism is introduced to determine the relevance weights between modalities, and a high-dimensional intent representation is obtained based on a hierarchical intent fusion network. The high-dimensional intent representation is input into a lightweight intent classification head to output the intent category probability distribution. An uncertainty perception module is introduced to dynamically evaluate the intent recognition confidence. If the intent recognition confidence is lower than a preset threshold, a clarification interaction strategy is triggered. Feedback signals from the clarification interaction strategy are collected, and the final intent is determined based on the feedback signals.

2. The AI ​​interactive voice intent recognition method integrating multimodal features according to claim 1, characterized in that, Preprocessing includes: noise reduction, endpoint detection, and sampling rate unification of the speech signal; and temporal alignment of the speech signal, facial video, and depth image.

3. The AI ​​interactive voice intent recognition method integrating multimodal features according to claim 1, characterized in that, The process of detecting and extracting head pose angles from the facial video includes: converting the video frames of the facial video into RGB channel format and normalizing the pixel values ​​to the [0,1] range; using a deep learning-based face detection model to detect the video frames and locate all face bounding boxes; cropping the face bounding boxes to obtain a face cropping region, inputting the face cropping region into a face key point detection model to obtain facial key point coordinates; loading a standard 3D face model, matching the facial key point coordinates with the model points of the standard 3D face model to determine rotation and translation vectors; and calculating Euler angles based on the rotation and translation vectors to obtain head pose angles.

4. The AI ​​interactive voice intent recognition method integrating multimodal features according to claim 3, characterized in that, Determining the user's gesture category based on the depth image includes: padding the depth image with invalid values, segmenting it using a depth threshold, filtering out the foreground region, locating and cropping the upper body region within the foreground region based on the face bounding box; inputting the cropped depth image into a pre-trained lightweight 3D human pose estimation model, outputting the 3D coordinates of the upper body key points; constructing the user's upper body skeleton based on the 3D coordinates of the upper body key points, defining bone vectors, and calculating key joint angles and relative limb positions; cropping a local area of ​​the hand based on the key joint angles and relative limb positions, calculating gesture features based on 3D coordinates, and determining the gesture category based on the gesture features, which include finger bending angle, palm opening, and fingertip spacing.

5. The AI ​​interactive voice intent recognition method integrating multimodal features according to claim 1, characterized in that, The speech semantic vector, text semantic vector, and visual semantic vector are input into a cross-modal alignment module for semantic space alignment. A multi-head cross-attention mechanism is introduced to determine the relevance weights between modalities. This includes: unifying the embedding dimensions of the speech semantic vector, text semantic vector, and visual semantic vector through the cross-modal alignment module, stacking them into an embedding sequence, and assigning a learnable positional code to each modality to obtain a modal input with positional information; setting the hyperparameters of the multi-head cross-attention mechanism, performing linear projection on the modal input to generate a query, key, and value matrix, and splitting the query, key, and value matrix into several heads; calculating scaled dot product attention for each head, and determining the relevance weights between modalities based on the scaled dot product attention.

6. The AI ​​interactive voice intent recognition method integrating multimodal features according to claim 5, characterized in that, The high-dimensional intent representation obtained based on the hierarchical intent fusion network includes: generating a modality fusion feature vector based on the scaled dot product attention; inputting the modality fusion feature vector into a residual feedforward network to obtain an enhanced fusion feature vector; and inputting the enhanced fusion feature vector into a lightweight explicit intent classifier to obtain a predefined explicit intent.

7. The AI ​​interactive voice intent recognition method integrating multimodal features according to claim 6, characterized in that, A high-dimensional intent representation is obtained based on a hierarchical intent fusion network, including: concatenating the dialogue history state vector output by memory-enhanced LSTM with the current enhanced fusion feature vector, and dynamically adjusting the fusion ratio of historical and current information through a gating mechanism; using memory-enhanced LSTM to perform contextual semantic modeling on the concatenated vector, capturing long-distance dependencies, and outputting implicit intent features; hierarchically fusing predefined explicit intent and implicit intent features, highlighting key intent cues through an attention mechanism, and finally generating a high-dimensional intent representation containing explicit and implicit intent information.

8. The AI ​​interactive voice intent recognition method integrating multimodal features according to claim 1, characterized in that, The high-dimensional intent representation is input into a lightweight intent classification head to output the intent category probability distribution, including: inputting the high-dimensional intent representation into a lightweight intent classification head composed of two fully connected layers, wherein the first layer uses the ReLU activation function for nonlinear transformation, and the second layer outputs the intent category probability distribution through the Softmax function.

9. The AI ​​interactive voice intent recognition method integrating multimodal features according to claim 1, characterized in that, An uncertainty perception module is introduced to dynamically evaluate the confidence level of intent recognition. This includes: analyzing the probability distribution of intent categories output by the lightweight intent classification head and calculating the entropy value of the probability distribution; simultaneously, a Monte Carlo algorithm is introduced to perform multiple random dropout operations on the fully connected layer of the lightweight intent classification head during the inference phase to obtain multiple samples of different intent category probability distributions, and quantifying the cognitive uncertainty of the model by calculating the prediction variance between the samples; finally, the entropy value and the prediction variance are weighted and fused to obtain a comprehensive confidence level assessment value for intent recognition.

10. A method and system for AI interactive speech intent recognition that integrates multimodal features, used to apply the AI ​​interactive speech intent recognition method that integrates multimodal features as described in any one of claims 1-9, characterized in that, The system includes: a data acquisition and processing module configured to acquire user voice signals, facial videos, and depth images through an integrated microphone array and an RGB-D camera, and perform preprocessing; detect and extract facial expression features and head posture angles from the facial videos, and determine the user's gesture category based on the depth images; a semantic conversion module configured to input the voice signals into a pre-trained voice encoder to output a voice semantic vector, input the text transcribed from the voice signals into a large voice model to output a text semantic vector, and encode the facial expression features, head posture angles, and gesture categories to obtain a visual semantic vector; a semantic fusion module configured to input the voice semantic vector, text semantic vector, and visual semantic vector into a cross-modal alignment module for semantic space alignment, introduce a multi-head cross-attention mechanism to determine the correlation weights between each modality, and obtain a high-dimensional intent representation based on a hierarchical intent fusion network; and an intent determination module configured to input the high-dimensional intent representation into a lightweight intent classification head to output an intent category probability distribution, introduce an uncertainty perception module to dynamically evaluate the intent recognition confidence, and if the intent recognition confidence is lower than a preset threshold, trigger a clarification interaction strategy; collect feedback signals from the clarification interaction strategy, and determine the final intent based on the feedback signals.

Images