An intelligent speech recognition and interaction system and method integrating AI visual information

Abstract

This invention discloses an intelligent speech recognition and interaction system and method that integrates AI visual information, belonging to the field of speech recognition technology. The invention simultaneously acquires audio and RGB-D video streams, and utilizes a dual-stream convolutional neural network to extract acoustic features, visual lip-reading features, and three-dimensional spatial interaction features in parallel. By constructing a cross-modal dynamic gating fusion network, and combining real-time signal-to-noise ratio and visual confidence, it dynamically allocates audio-visual weights, achieving highly robust lip-reading-based recognition even in high-noise environments. Addressing referential ambiguity in natural language, it utilizes the projection and collision detection of gaze and gesture rays in a three-dimensional semantic map to achieve "what you see is what you control" intent parsing. This invention effectively solves the problems of low recognition rate, unclear referentiality, and cumbersome wake-up in complex sound field environments.

Description

Technical Field

[0001] This invention belongs to the field of speech recognition technology, and more specifically, relates to an intelligent speech recognition and interaction system and method that integrates AI visual information. Background Technology

[0002] With the rapid development of artificial intelligence and the Internet of Things (IoT) technologies, intelligent human-computer interaction has become a core technology in fields such as smart homes, smart cockpits, and healthcare. The global intelligent voice market continues to grow rapidly every year, and users' demand for natural interaction is becoming increasingly urgent. From early button control to today's voice control, human-computer interaction methods are evolving towards a more intuitive and efficient direction.

[0003] Current mainstream voice interaction solutions heavily rely on audio monomodality. Their core process typically involves: capturing user speech through beamforming and sound source localization using a microphone array; activating the device with a specific wake word to enter listening mode; and then performing automatic speech recognition (ASR) on the speech to convert the acoustic signal into text commands. Although some cutting-edge research attempts to introduce visual sensors, their application is relatively shallow, either using cameras for face detection and tracking to assist in sound source localization and improve the ability to identify who is speaking, or simply performing post-processing concatenation or weighted averaging of extracted lip-reading visual features and audio features. These methods fail to achieve deep interaction and collaborative representation learning of audiovisual features at the encoder level, and lack a mechanism for adaptive weight adjustment based on dynamic conditions such as real-time environmental noise and visual occlusion, resulting in limited improvement in robustness in complex scenarios.

[0004] This reveals the following problems with existing technologies: 1. Poor noise resistance: In high-noise environments, the recognition rate relying solely on audio drops sharply, and it is easily interfered with by background voices or television sounds; 2. Unclear semantic reference: Users are accustomed to using natural language with pronouns such as "turn this on" or "turn that up a bit," but existing technologies cannot understand the semantic information brought about by spatial location, leading to interaction failure; 3. Unnatural interactive experience: Forced wake words interrupt the user's natural expression logic and are prone to false wake-ups or wake-up failures. Summary of the Invention

[0005] (a) Technical problems to be solved

[0006] To address the problems in related technologies, this invention provides an intelligent speech recognition and interaction system and method that integrates AI visual information, thereby overcoming the aforementioned technical problems existing in the prior art.

[0007] (II) Technical Solution

[0008] To solve the above-mentioned technical problems, the present invention is achieved through the following technical solution:

[0009] S1. Acquire sound field signals and light field signals in the environment, and align them with timestamps to obtain audio streams and RGB-D video streams;

[0010] S2. Extract the acoustic feature vector of the audio stream in S1, as well as the visual speech vector and three-dimensional spatial interaction vector of the RGB-D video stream;

[0011] S3. Through a cross-modal dynamic gating fusion network, the acoustic feature vectors and visual speech vectors extracted in S2 are mapped to the latent semantic space. Based on the signal-to-noise ratio index and the lip occlusion rate index, the fusion weights of the auditory channel and the visual channel are dynamically allocated to generate a multimodal speech semantic representation and decode the preliminary speech text sequence.

[0012] S4. For the preliminary speech text sequence generated by S3, perform pronoun trigger detection. When a pronoun is detected, use the three-dimensional spatial interaction vector extracted by S2 to emit a gaze ray and a gesture ray in the constructed local three-dimensional semantic map. Calculate the intersection probability and spatial distance score between the ray and the bounding box of the object in the scene, and parse out the physical object ID corresponding to the pronoun.

[0013] S5. Real-time monitoring of the dwell time and lip movement state of the three-dimensional spatial interaction vector in S2 relative to the preset interaction area; when it is determined to be an intention trigger, the preliminary voice text sequence generated in S3 is combined with the physical object ID parsed in S4 to generate the final device control command.

[0014] Preferably, step S1 includes the following steps:

[0015] S11: Configure a six-microphone circular array, set the audio sampling rate and bit depth; simultaneously configure an RGB-D depth camera, set the video resolution, frame rate and depth map measurement range to obtain a multimodal perception hardware environment;

[0016] S12. Perform hardware-level time synchronization, use FPGA to generate a unified synchronization pulse signal, and simultaneously trigger the analog-to-digital conversion of the microphone array and the exposure acquisition of the depth camera to obtain the original multi-channel audio stream and RGB-D video stream.

[0017] S13. Beamforming is performed on the multi-channel audio stream. The generalized cross-correlation algorithm is used to calculate the direction of arrival (DOA) of the sound source, enhance the speech signal in the target direction and suppress background noise, and output the audio stream.

[0018] Based on the RGB-D video stream, a face detection algorithm is used to locate the user's facial region, and background pixels are removed based on depth information, retaining only the foreground visual information of the user and the interaction range to obtain the denoised RGB-D video stream.

[0019] Preferably, step S2 includes the following steps:

[0020] S21. Using a pre-trained Conformer encoder structure, the audio stream output by S1 is divided into frames and windowed to extract Mel filter bank features. The first-order difference and the second-order difference are then concatenated. Long-term dependence is captured through a multi-layer self-attention mechanism, and the acoustic feature vector is output.

[0021] S22. Extract the region of interest (ROI) of the user's lips from the RGB-D video stream and normalize it into a grayscale image sequence; use the ResNet-3D network to extract the continuous motion pattern of the lips in the time dimension, and output the acoustic feature vector and the visual speech vector.

[0022] S23. Utilize a skeletal detection model to detect key skeletal points of the user's hand and the center point of the eyeball in the RGB-D video stream; combine the depth map of the skeletal key points and the center point of the eyeball. Z The axis information is used to back-project the two-dimensional pixel coordinates to the camera coordinate system, generating a three-dimensional hand key point coordinate set and a three-dimensional eye gaze direction vector, thus obtaining a three-dimensional spatial interaction vector.

[0023] Preferably, step S3 includes the following steps:

[0024] S31. Calculate the ratio of the signal energy of the current audio frame to the background noise energy of the non-speech segment to obtain the environmental signal-to-noise ratio;

[0025] S32. Calculate the initial visual confidence score; detect whether there are obstructions in the lip area, whether the light intensity is lower than the preset threshold, and whether the head posture deflection angle exceeds the head yaw angle threshold; if so, reduce the initial visual confidence score to obtain the final visual confidence score; otherwise, use the initial visual confidence score as the final visual confidence score.

[0026] S33. Construct a cross-modal dynamic gating unit; define the audio gating coefficient and visual gating coefficient of the cross-modal dynamic gating unit; the gating coefficients are calculated by a fully connected layer and a Sigmoid activation function;

[0027] The acoustic feature vector, visual speech vector, normalized environmental signal-to-noise ratio, and final visual confidence are input into the cross-modal dynamic gating unit to obtain the current audio gating coefficient and the current visual gating coefficient.

[0028] S34. Calculate the fusion features using the fusion formula based on the current audio gating coefficient and the current visual gating coefficient;

[0029] S35. Input the fused features into the Transformer decoder, combine them with the beam search algorithm, and output the preliminary speech-text sequence with the highest probability.

[0030] Preferably, step S4 includes the following steps:

[0031] S41. Construct a dictionary of pronouns containing words with spatial orientation; traverse the speech-text sequence generated in S3, and if a word in the dictionary of pronouns is matched, mark it as a trigger point, and extract the three-dimensional spatial interaction vector within a preset time window before and after the trigger point to obtain the average gaze vector and the average gesture vector.

[0032] S42; Using point cloud data acquired by a depth camera, perform instance segmentation and bounding box fitting on objects in the scene to generate a list of objects containing object category label ID, center coordinates and 3D dimensions, thus obtaining a local 3D semantic map.

[0033] S43. In the local three-dimensional semantic map, using the average gaze vector and average gesture vector in S41 as directions, construct the corresponding gaze ray and gesture ray starting from the user's brow and index fingertip respectively.

[0034] S44. For each object in the scene, calculate the intersection depth and Euclidean distance between the view ray and gesture ray and the object's bounding box; define the fusion pointing score formula, and calculate the final score by combining adjustable weights.

[0035] S45. Select the object with the highest fusion pointing score as the referent object, and bind the object ID of the referent object to the speech intent.

[0036] Preferably, S44 defines a fusion-oriented scoring formula, and the calculation of the final score using adjustable weights includes the following steps:

[0037] S441, Define the line-of-sight ray as R eye The gesture ray is R ha The bounding box of the object is O i ;

[0038] S442. Calculate the Euclidean distance between the line-of-sight ray, the gesture ray, and the center of the object's bounding box, and the Boolean value indicating whether they intersect.

[0039] S443. Define the confidence weight of eye contact and the confidence weight of gesture; detect the height of the user's wrist relative to the abdomen in real time; if the height of the user's wrist relative to the abdomen is greater than the preset height threshold, it is judged as explicit pointing; otherwise, it is judged as implicit gazing.

[0040] Define a single-ray score calculation formula; based on the Euclidean distances between the line-of-sight ray, the gesture ray, and the center of the object's bounding box, and the Boolean value indicating whether they intersect, calculate the line-of-sight ray score and the gesture ray score using the single-ray score calculation formula;

[0041] The final fusion pointing score is calculated based on the gaze ray score and gesture ray score.

[0042] Preferably, step S5 includes the following steps:

[0043] S51. A cone-shaped virtual space is set in front of the user's field of vision; the cone-shaped virtual space covers the physical area containing the smart device and its sensors, thus obtaining the interaction area ROI;

[0044] S52. Calculate the angle between the user's gaze vector and the interaction area ROI in real time; if the angle is less than the angle threshold and the duration exceeds the duration threshold, it is determined to be a gaze lock state.

[0045] S53. Calculate the opening amplitude and rate of change of the key lip points in S2; if continuous lip movement signals exceeding the silence threshold are detected in the gaze lock state, it is determined to be intention triggering.

[0046] S54. Upon receiving the intent trigger, the system transitions from standby monitoring state to active listening state, activates modules S3 and S4 for processing, combines the preliminary voice text sequence generated by S3 with the physical object ID parsed by S4 to generate the final device control command.

[0047] S55. After processing, if the confidence level of the generated device control command is higher than the first confidence level threshold, the control command is issued directly.

[0048] If the confidence level is between the second confidence threshold and the first confidence threshold, a voice inquiry for confirmation will be initiated; if the gaze moves out of the ROI area for more than the gaze-out time threshold, it will automatically revert to standby monitoring mode.

[0049] Preferably, the present invention also includes an adaptive online learning mechanism, comprising the following steps:

[0050] S61. Define positive and negative feedback signals;

[0051] S62. Construct a contrastive loss function; for negative samples, calculate the feature distance between the predicted object and the true intention object as a penalty term; for positive samples, reduce the prediction error under the current environmental parameters.

[0052] S63. Collect user feedback behavior on the interaction results as the current positive and negative feedback signals; based on the current positive and negative feedback signals and the loss function, iteratively update the weight parameters of the cross-modal dynamic gating fusion network and the ray weighting coefficients of the 3D spatial semantic alignment engine.

[0053] Preferably, the iterative update of the weight parameters of the cross-modal dynamic gating fusion network and the ray weighting coefficients of the 3D spatial semantic alignment engine in S63 includes the following steps:

[0054] S631. Construct a chromosome population; use the chromosomes in the population as weight parameters of a cross-modal dynamic gated fusion network and the ray weighting coefficients of a 3D spatial semantic alignment engine;

[0055] S632. By comparing the loss function, select, crossover, and mutate chromosomes in the chromosome population; repeat the iteration until the set maximum number of iterations is reached or less than the preset loss value threshold, to obtain the optimal chromosome; apply the weight parameters of the cross-modal dynamic gating fusion network corresponding to the optimal chromosome and the ray weighting coefficient of the three-dimensional spatial semantic alignment engine to the cross-modal dynamic gating and the three-dimensional spatial semantic alignment engine, respectively.

[0056] An intelligent speech recognition and interaction system integrating AI visual information, used to implement the aforementioned intelligent speech recognition and interaction method integrating AI visual information, includes:

[0057] Multimodal perception front-end module: integrates microphone array and depth camera, responsible for synchronous data acquisition and preprocessing, to obtain multi-channel audio stream and RGB-D video stream;

[0058] Dual-stream feature extraction module: Includes parallel acoustic feature extraction unit and visual feature extraction unit, used to process multi-channel audio stream and RGB-D video stream, and generate acoustic feature vector, visual speech vector and three-dimensional spatial interaction vector;

[0059] Fusion Inference Engine Module: Runs a cross-modal dynamic gating network and a 3D spatial semantic alignment algorithm to process acoustic feature vectors, visual speech vectors, and 3D spatial interaction vectors, perform intent parsing and object locking, and obtain preliminary speech-text sequences and physical object IDs;

[0060] Interactive Control Center Module: Includes state machine logic and instruction distribution interface, based on preliminary voice and text sequences and physical object IDs, to realize the system's state transitions and device control.

[0061] (III) Beneficial Effects

[0062] The present invention has the following beneficial effects:

[0063] This invention provides an intelligent speech recognition and interaction method that integrates AI visual information, exhibiting significant noise robustness. Traditional speech recognition technologies suffer a significant drop in accuracy in noisy environments with a signal-to-noise ratio below 10dB (such as family gatherings or in-vehicle driving). This invention, by constructing a cross-modal dynamic gating fusion network, can perceive environmental noise levels and visual confidence in real time, dynamically adjusting the fusion weights of auditory and visual features. When noise interference is severe, the system automatically reduces the weights of contaminated audio features, significantly increasing the reliance on clear lip-reading visual features, thereby maintaining an extremely high command recognition rate even in high-noise environments, effectively solving the technical problem of unclear hearing.

[0064] This invention achieves accurate spatial semantic understanding and object referencing. Existing voice assistants cannot understand common natural language commands containing demonstrative pronouns, such as "turn this on" or "turn that up," resulting in limited interaction. This invention introduces a three-dimensional spatial semantic alignment engine, using a depth camera to construct a local three-dimensional semantic map and projecting the user's eye gaze vector and gesture pointing vector as spatial rays. By calculating the intersection probability and fusion score between the rays and the bounding boxes of environmental objects, the system can, like a human, interpret whether "this" refers to a floor lamp or an air conditioner simply by "seeing," achieving what is seen is what is controlled, and what is pointed to is what is controlled, thus improving the naturalness and accuracy of the interaction.

[0065] This invention provides a more natural, efficient, and evolutionary interactive experience. Traditional voice interaction relies on specific wake words, which is not only cumbersome but also easily interrupts the user's train of thought. This invention constructs a wake-free interactive state machine based on gaze persistence and lip movement detection. Users only need to look at the device and speak directly to trigger commands, simulating the natural eye contact mechanism between people. Combined with an adaptive online learning mechanism based on genetic algorithms, the system can automatically fine-tune the fusion weights according to the user's positive and negative feedback (such as correction commands and facial expressions), thereby adapting to different user preferences (such as some people are used to looking with their eyes, while others are used to using their fingers), realizing the continuous evolution and personalized adaptation of the system.

[0066] Of course, any product implementing this invention does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0067] To more clearly illustrate the technical solutions of the embodiments of the invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the invention. For those skilled in the art, the drawings can be obtained from these drawings without creative effort.

[0068] Figure 1 This is a flowchart illustrating an intelligent speech recognition and interaction method that integrates AI visual information according to the present invention.

[0069] Figure 2 This is a schematic diagram of a module of an intelligent speech recognition and interaction system that integrates AI visual information according to the present invention. Detailed Implementation

[0070] The technical solutions of the embodiments of the invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the invention, and not all embodiments. Based on the embodiments of the invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the invention.

[0071] To address the technical problems raised in the background section, please refer to [link / reference]. Figure 1 This invention provides an intelligent speech recognition and interaction system and method that integrates AI visual information, including:

[0072] S1. Acquire sound field signals and light field signals in the environment, and align them with timestamps to obtain audio streams and RGB-D video streams;

[0073] S2. Extract the acoustic feature vector of the audio stream in S1, as well as the visual speech vector and three-dimensional spatial interaction vector of the RGB-D video stream;

[0074] S3. Through a cross-modal dynamic gating fusion network, the acoustic feature vectors and visual speech vectors extracted in S2 are mapped to the latent semantic space. Based on the signal-to-noise ratio index and the lip occlusion rate index, the fusion weights of the auditory channel and the visual channel are dynamically allocated to generate a multimodal speech semantic representation and decode the preliminary speech text sequence.

[0075] S4. For the preliminary speech text sequence generated by S3, perform pronoun trigger detection. When a pronoun is detected, use the three-dimensional spatial interaction vector extracted by S2 to emit a gaze ray and a gesture ray in the constructed local three-dimensional semantic map. Calculate the intersection probability and spatial distance score between the ray and the bounding box of the object in the scene, and parse out the physical object ID corresponding to the pronoun.

[0076] S5. Real-time monitoring of the dwell time and lip movement state of the three-dimensional spatial interaction vector in S2 relative to the preset interaction area; when it is determined to be an intention trigger, the preliminary voice text sequence generated in S3 is combined with the physical object ID parsed in S4 to generate the final device control command.

[0077] The above embodiments synchronously acquire audio and video signals through hardware and extract acoustic, visual speech, and 3D interactive features; utilize cross-modal dynamic gating networks to dynamically fuse audiovisual features based on environmental signal-to-noise ratio and visual quality, significantly improving the robustness of speech recognition in noisy environments; accurately parse spatial references in natural language by constructing local 3D semantic maps and projecting user gaze and gesture rays; achieve wake-free interaction based on gaze dwell and lip movement detection, and personalize adaptation to different user habits through an online learning mechanism; solve the three major pain points of high-noise recognition, unclear references, and unnatural interaction, and achieve more efficient, accurate, and human-centered natural human-computer interaction.

[0078] The above embodiment S1 includes the following steps:

[0079] S11 is equipped with a six-microphone circular array, with an audio sampling rate of 16kHz and a bit depth of 16bit; it is also equipped with an RGB-D depth camera, with a video resolution of 1080P, a frame rate of 30fps, and a depth map measurement range of 0.5 meters to 5 meters, to obtain a multimodal perception hardware environment.

[0080] In specific implementation, the above embodiment S11 is as follows: hardware is deployed on a smart terminal (such as the top of a smart TV or the main body of a smart speaker); the microphone array adopts a uniform circular array composed of 6 MEMS microphones with a radius of 40mm to ensure uniform sampling of the 360-degree sound field; the RGB-D camera uses a depth camera with active infrared projection function (such as Intel RealSense D435i), the RGB module is responsible for capturing color texture for face recognition, and the depth module is responsible for acquiring environmental geometric information; the video stream resolution is set to 1920x1080, and the frame rate is fixed at 30fps to ensure that fast lip movements can be captured; the depth measurement range is set to 0.5m-5m, covering typical interaction distances such as family living rooms and vehicle cabins;

[0081] S12. Perform hardware-level time synchronization, use FPGA to generate a unified synchronization pulse signal, and simultaneously trigger the analog-to-digital conversion of the microphone array and the exposure acquisition of the depth camera, control the timestamp deviation of the audio and video frames within 5 milliseconds, and obtain the original multi-channel audio stream and RGB-D video stream.

[0082] In specific implementation, the above embodiment S12 is as follows: In order to ensure lip-sync, an FPGA is introduced as a clock source to generate a synchronization trigger pulse with a frequency of 30Hz; the rising edge of the pulse is connected to the external trigger pin of the camera and the GPIO port of the microphone array acquisition card; whenever the pulse arrives, the camera starts a frame exposure, and the microphone array timestamps the current audio buffer at the same time; through this hardware synchronization method, the time deviation of the audio and video stream is controlled to <5ms, which is better than the 30-50ms deviation of software synchronization, providing a strict timing alignment basis for feature fusion in S3; S13, beamforming is performed on the multi-channel audio stream, the generalized cross-correlation algorithm is used to calculate the direction of arrival (DOA) of the sound source, the speech signal in the target direction is enhanced and the background noise is suppressed, and the enhanced single-channel audio stream is output;

[0083] Based on the RGB-D video stream, a face detection algorithm is used to locate the user's facial region, and background pixels are removed based on depth information, retaining only the foreground visual information within the user's and interaction range to obtain a denoised RGB-D video stream. Specifically, in embodiment S13, the time delay difference between each microphone pair is calculated using a generalized cross-correlation algorithm to estimate the azimuth and pitch angles of the sound source (user) relative to the array. In this embodiment, the user is detected at an azimuth angle of 30 degrees. By dynamically adjusting the delay of each microphone signal, the signals from the 30-degree direction are in-phase and enhanced, while noise from other directions (such as sound from a television, located at 0 degrees) is incoherently canceled, resulting in a single-channel enhanced audio stream with a signal-to-noise ratio improvement of approximately 10-15d. For each frame of the RGB image, the RetinaFace algorithm is used to detect the face bounding box. Combined with the aligned depth map, the average depth value of the face region is calculated. Z face Set depth threshold range [ Z face -0.3m, Z face +0.3m] sets the pixel values ​​outside the range to 0 (black), thereby removing interference from moving people or television images in the background on lip reading recognition; the 0.3m in the depth threshold range in this embodiment is based on the average diameter of an adult head of about 20cm, and a 10cm margin for slight forward and backward movement is reserved to prevent pixel loss due to slight head movements; this threshold supports dynamic adjustment, and the adjustment formula is Thr= Z face ±(0.15+0.05 * Z faceThis indicates that as the user's distance increases, considering the increase in depth measurement error, the depth retention interval is automatically expanded to ensure the integrity of facial information at long distances. The above embodiment builds the hardware foundation by deploying a six-microphone array and an RGB-D depth camera, and introduces an FPGA hardware-level synchronization mechanism to strictly control the timestamp deviation of the audio and video streams within 5 milliseconds, providing millisecond-level timing alignment guarantees for subsequent cross-modal feature fusion. Combined with generalized cross-correlation beamforming and depth-based foreground segmentation technology, the audio and video streams are enhanced and denoised respectively. This ensures the synchronization, purity, and high signal-to-noise ratio of the input data, laying the data foundation for the entire system to achieve accurate audiovisual fusion, lip reading recognition, and 3D spatial perception.

[0084] The above embodiment S2 includes the following steps (this embodiment aims to solve the problem of high dimensionality and low semantics of the original data, and transform it into a high-dimensional vector that can be understood by a computer):

[0085] S21. Using a pre-trained Conformer encoder structure, the single-channel enhanced audio stream output from S1 is framed and windowed to extract 80-Vimel filter bank features. First-order and second-order differences are then concatenated, and a multi-layer self-attention mechanism is used to capture long-term dependencies. The output time step is [missing information]. T The sequence of acoustic feature vectors;

[0086] In specific implementation, the above embodiment S21 is as follows: After the audio stream undergoes pre-emphasis, framing (frame length 25ms, frame shift 10ms), and Hamming windowing, an FFT transformation is performed to extract 80-dimensional Fbank features; the 80-dimensional Fbank features are input to the Conformer encoder, which consists of 12 stacked layers, each layer containing a feedforward module, a multi-head self-attention module (MHSA), a convolutional module, and layer normalization; the convolutional module uses a depthwise separable convolution with a kernel size of 31 to capture local acoustic textures; MHSA is used to capture global semantic dependencies; the final output sequence is... H au ={ h a1 , h a2 ,..., h aTThe feature dimension D=512; the multi-head self-attention module contains 8 attention heads; heads 1-4 are used to focus on the phoneme features of short-term speech, and heads 5-8 are used to focus on the prosodic and contextual features of long-term speech; the Conformer encoder is pre-trained on a public speech dataset before use, and self-supervised learning is performed using the CTC loss function to enable the model to initially have the ability to map acoustic features to phoneme sequences, and then fine-tuned on the multimodal dataset of this system; the pre-training dataset uses a large-scale monolingual speech corpus (such as LibriSpeech or Common). Voice); during fine-tuning, the system uses a multimodal dataset with precise audio and video synchronization alignment; the first half of the Conformer encoder's low-level parameters (6 layers) are frozen to retain its learned general acoustic feature extraction capabilities; all weight parameters of the second half of the encoder (6 layers), the dynamic gating network, and the Transformer decoder are unfrozen and jointly fine-tuned; a combined loss function (CTC loss to ensure phoneme-level temporal alignment combined with cross-entropy loss to optimize text generation) is adopted, and the model is adjusted by combining gradient pruning and dynamic learning rate adjustment strategies to make the acoustic feature vector sequence extracted by the model consistent with the label feature data in the multimodal dataset;

[0087] S22. Extract the region of interest (ROI) of the user's lips from the RGB-D video stream and normalize it into a grayscale image sequence; use the ResNet-3D network to extract the continuous motion pattern of the lips in the time dimension, and output a visual speech vector sequence aligned with the acoustic feature vector sequence and the time step.

[0088] In specific implementation, the above embodiment S21 is as follows: The RetinaFace face detection algorithm is used to locate the face in the RGB image, and the Face Alignment Network is used to detect 68 facial key points; based on 20 key points in the lip region, an 88*88 pixel area at the center of the lips is cropped; the data is input into a ResNet-3D network, where the first layer of the ResNet-3D network has a (5,7,7) 3D convolutional kernel, corresponding to the time, height, and width dimensions respectively, used to capture dynamic micro-expressions such as lip opening and closing, lip rounding, and lip spreading; after processing through 4 residual blocks, a global average pooling layer is used to output a visual feature sequence. H vi Because the video frame rate (30fps) and audio frame rate (100fps) are inconsistent, linear interpolation is used to... H vi Upsampling, making its length the same as H au Strict consistency;

[0089] S23. Utilize a skeletal detection model to detect 21 skeletal key points of the user's hand and the center point of the eyeball in the RGB-D video stream; combine the depth maps of the skeletal key points and the center point of the eyeball. Z The axis information is used to back-project the two-dimensional pixel coordinates to the camera coordinate system, generating a three-dimensional hand key point coordinate set and a three-dimensional eye gaze direction vector, thus obtaining a three-dimensional spatial interaction vector.

[0090] In specific implementation, the above embodiment S23 specifically involves: using the OpenPose or MediaPipe Hands model to extract the 2D coordinates of 21 key points of the hand (such as the wrist joint and the tip of the index finger). u , v Read the depth value of the corresponding depth map location. Z ;Calculate 3D coordinates using the camera intrinsic parameter matrix K: x =( u - cx )* Z / f x y=( v - cy )*Z / f y ;in, u , v Represents the two-dimensional pixel coordinates (unit: pixels) of key points of the hand or eyes on the image, such as the position of the fingertip of the index finger in the image in this embodiment; Z The depth value is read from the depth map at the (u, v) coordinates and represents the vertical distance from the key point to the camera's imaging plane. f x , f y This indicates the camera's focal length parameter; cx、cy Indicates the coordinates of the camera's principal point; x , y This represents the calculated X and Y coordinates of the keypoint in the 3D camera coordinate system; at this point, the 3D coordinates of the point are (x, y, ...). Z The origin is the optical center of the camera; thus, the gesture vector is constructed. V ha (Wrist joint pointing towards fingertips); Simultaneously, using an existing gaze estimation model as input, the eye image is used to output the eye's pitch and yaw angles, which are then converted into unit gaze vectors. V eye ;

[0091] S24. Perform temporal alignment interpolation on the various feature vectors output by S21, S22, and S23 to ensure that acoustic features, lip features, and spatial interaction features have a one-to-one index relationship under the same time slice.

[0092] The above embodiments extract acoustic features containing contextual information through a pre-trained Conformer encoder, capture temporal patterns of lip movements using a ResNet-3D network, and obtain three-dimensional spatial vectors of gestures and gaze using skeleton detection and depth back projection techniques. Temporal alignment ensures that all features are synchronized on the time axis. Multimodal, low-level sensor data is uniformly transformed into high-dimensional, semantic, and temporally aligned feature vectors, providing an accurate and consistent data foundation for subsequent cross-modal dynamic fusion and spatial intent understanding.

[0093] The above embodiment S3 includes the following steps (this embodiment aims to solve the problem of low recognition rate when relying solely on speech recognition in high-noise environments, and allows visual information to take over the dominant role when there is a lot of noise through a soft fusion mechanism):

[0094] S31. Calculate the ratio of the signal energy of the current audio frame to the background noise energy of the non-speech segment to obtain the environmental signal-to-noise ratio (SNR).

[0095] S32. Calculate the initial visual confidence VC1; detect whether there are obstructions in the lip area, whether the light intensity is lower than the preset threshold, and whether the head posture deflection angle exceeds 45 degrees; if so, reduce the initial visual confidence VC1 to obtain the final visual confidence VC2; otherwise, use the initial visual confidence as the final visual confidence.

[0096] In specific implementation, the above embodiment S32 is as follows: the range of visual confidence VC value is [0, 1]; if a hand or cup is detected covering the lips (occlusion rate > 30%), VC2 = VC1 * 0.1; if the ambient light intensity Lux < 50, VC2 = VC1 * 0.5; if the head yaw angle Y aw >Head yaw angle threshold 45 degrees (at this time, the face is turned to the side), VC2=VC1*cos( Y aw The initial visual confidence score VC1 is obtained by multiplying the confidence score of the RetinaFace face detection algorithm in S22 with the Laplacian variance of the lip ROI region; an occlusion rate >30% is the validity boundary based on lip reading recognition, and key phoneme features will be invisible if this ratio is exceeded; the coefficient 0.1 is to strongly suppress visual weights to prevent misleading when occlusion occurs; Lux <50 is the illumination threshold based on the obvious noise generated by a normal RGB camera, at which point image details are severely lost, so the coefficient in the VC2 calculation formula is set to 0.5; the head yaw angle threshold is set to 45 degrees, which is the geometric critical point at which the lip shape causes self-occlusion;

[0097] S33. Construct a cross-modal dynamic gating unit; define the audio gating coefficients of the cross-modal dynamic gating unit. Ga and visual gating coefficient G v The gating coefficients are calculated using a fully connected layer and a Sigmoid activation function.

[0098] The acoustic feature vector, visual-speech vector, normalized environmental signal-to-noise ratio (SNR), and final visual confidence score (VC2) are input into the cross-modal dynamic gating unit to obtain the current audio gating coefficients. G a1 and current visual gating coefficient G v1 ;

[0099] In specific implementation, the above embodiment S33 is as follows: the gated network structure is MLP([ H au , H vi , SNR, VC])→[ G a , G v ], that is, the gating coefficient [ G a , G v It is calculated by a two-layer fully connected neural network (MLP), and the calculation formula is as follows:

[0100] ;in G a , G v ∈[0, 1]; The quantization rule is set to introduce a regularization term during the training of the gated network, forcing a condition where SNR is low and VC is high. G v Tend to 1, G a The value tends towards 0; conversely, it also tends towards 0. In this embodiment, when the SNR is 5dB (high noise) and the network is facing the camera, the network output... G a =0.2, G v=0.8; The structure of the cross-modal dynamic gating unit includes an input layer (dimensionality of acoustic features 512 + visual features 512 + environmental parameters 2 = 1026), a hidden layer (64-dimensional, ReLU activation), and an output layer (2-dimensional, Sigmoid activation); The first layer of the fully connected network is a hidden layer with an input dimension of 1026, an output dimension of 64, a weight matrix size of 1026×64 (65,664 parameters in total), and a 64-dimensional bias vector. The output is activated by the ReLU function; The second layer is an output layer with an input dimension of 64, an output dimension of 2, a weight matrix size of 64×2 (128 parameters in total), and a 2-dimensional bias vector. The output is activated by the Sigmoid function, and the final output is [ G a , G v Two gating coefficients; before use, the network is pre-trained on a synthetic dataset containing clear video and audio with Gaussian white noise superimposed at different signal-to-noise ratios (-5dB to 20dB). The training objective is to maximize decoding accuracy, so that the network can automatically learn to adjust weights according to the noise level.

[0101] S34, Based on the current audio gating coefficient G a1 and current visual gating coefficient G v1 The fusion formula is used to calculate the fusion features; under conditions of low signal-to-noise ratio and high visual confidence, the network automatically learns to achieve... G v Significantly greater than G a This allows them to dominate the decoding process;

[0102] In specific implementation, the above embodiment S34 specifically refers to: fusion features F fu = G a1 * 1 H au + G v1 * 1 H vi ;in,* 1The term "element-wise multiplication" indicates that the weighting here is not a simple linear summation, but rather a gating selection at the feature channel level. This allows the network to suppress noisy audio frequency band features while preserving clear lip movement visual features. This dynamic gating network is a two-layer fully connected structure. Its input layer receives the concatenated 1026-dimensional features, the hidden layer has a dimension of 64 and uses the ReLU activation function, and the output layer has a dimension of 2 and uses the Sigmoid activation function, ultimately generating the audio and visual gating coefficients. G a and G v ;

[0103] S35. Input the fused features into the Transformer decoder, combine them with the beam search algorithm, and output the preliminary speech-text sequence with the highest probability.

[0104] In specific implementation, the above embodiment S35 specifically involves: the decoder outputting a text sequence. Y ={ y 1, y 2,..., y L For example, the output text is "Open that"; the Transformer decoder uses a multi-head cross-attention mechanism to fuse features. F fu Using the word vectors predicted at the previous time step as the key and value, the key time steps in the feature sequence are considered; the bundle search algorithm sets the bundle width to 5, retains the 5 candidate paths with the highest probability at each decoding step, and selects the one with the highest cumulative probability at the end of the sentence as the final output preliminary speech-text sequence.

[0105] The above embodiments introduce environmental signal-to-noise ratio and visual confidence as control signals, and calculate the optimal fusion weights of audio and visual features in real time and dynamically. This enables the system to automatically enhance lip-reading ability when there is high noise and clear vision, and to rely on pure audio when there is quiet or poor visual quality, thus achieving the effect of feature splicing and intelligent selection. The fused features are decoded by Transformer to generate accurate text, solving the problem of single speech recognition failure in high-noise environments and providing reliable input for subsequent referential resolution.

[0106] The above embodiment S4 includes the following steps (this embodiment aims to solve the problem of unclear references to "this" and "that" in natural language):

[0107] S41. Construct a dictionary of pronouns, containing words with spatial orientation; traverse the speech-text sequence generated in S3, and if a word in the dictionary of pronouns is matched, mark it as a trigger point, and extract the three-dimensional spatial interaction vector within a preset time window before and after the trigger point to obtain the average gaze vector. V1 eye and average gesture vector V 1 ha ;

[0108] In specific implementation, the above embodiment S41 specifically involves: the dictionary including {this, that, left, it, here}; if the text is "turn that on", and "that" is detected, the average gaze vector within 0.5 seconds before and after the pronunciation of that word is extracted. V 1 eye and average gesture vector V 1 ha ;

[0109] S42; Using point cloud data acquired by a depth camera, perform instance segmentation and bounding box fitting on objects in the scene to generate a list of objects containing object category label ID, center coordinates and 3D dimensions, thus obtaining a local 3D semantic map.

[0110] In specific implementation, the above embodiment S42 specifically involves: processing RGB-D data using YOLOv8 combined with PointNet++; detecting the 2D bounding box of the object on the RGB image using YOLOv8; back-projecting the depth pixels within the 2D bounding box to generate a 3D point cloud view frustum based on camera intrinsic parameters; performing semantic segmentation on the point cloud within the view frustum using the PointNet++ network to remove background points; calculating the minimum bounding box (OBB) of the remaining foreground points using the PCA (Principal Component Analysis) algorithm to generate the 3D coordinates and dimensions of the object; and identifying objects in the scene. O A (Floor lamp, coordinates [1.5, -0.5, 3.0], enclosure dimensions [0.3, 1.5, 0.3]) O B (Air conditioner, coordinates [-1.0, 1.0, 3.5]);

[0111] S43. In the local three-dimensional semantic map, using the average gaze vector and average gesture vector in S41 as directions, construct the corresponding gaze ray and gesture ray starting from the user's brow and index fingertip respectively.

[0112] S44. For each object in the scene, calculate the intersection depth and Euclidean distance between the view ray and gesture ray and the object's bounding box; define the fusion pointing score formula, and calculate the final score by combining adjustable weights.

[0113] The above embodiment S44 defines a fusion-oriented scoring formula, and the calculation of the final score in combination with adjustable weights includes the following steps:

[0114] S441, Define the line-of-sight ray as R eye The gesture ray is R ha The bounding box of the object is O i ;

[0115] S442, Calculate the Euclidean distance Dist between the line-of-sight ray, the gesture ray, and the center of the object's bounding box. R , O i ) and the Boolean value Hit indicating whether they intersect. R , O i );

[0116] S443, Define the confidence weight of gaze level α and gesture confidence weights β Real-time detection of the user's wrist height relative to their abdomen. H wr If the user's wrist is at a height relative to their abdomen H wr If the height exceeds a preset threshold, it is determined to be an explicit pointing gesture; otherwise, it is determined to be an implicit gaze.

[0117] The formula for calculating the single-ray score is defined as Score( R , O i )=Hit( R , O i )*exp(- l *Dist( R , O i )),in l The distance attenuation coefficient ranges from 0.5 to 1.0. Based on the Euclidean distance between the line-of-sight ray, the gesture ray, and the center of the object's bounding box, and the Boolean value indicating whether they intersect, the line-of-sight ray score and the gesture ray score are calculated using the single-ray score calculation formula. l The value is dynamically adjusted based on the spatial scale and density of the interaction scene; in open, large spaces (such as living rooms), where objects are scattered, a smaller value is used. l This reduces the distance penalty, enabling the system to effectively respond to the pointing of distant targets; in compact, high-density environments (such as vehicle cabins), a larger [size / size] is used. l This enhances distance attenuation, making the system more sensitive and accurate in selecting nearby objects; l The relationship between the average distance (davg) of interactive objects in the scene and the actual distance (davg) is as follows: l= 0.67 / davg + 0.33; Ensure that the λ value remains within the effective range of 0.5 to 1.0 within the common distance range (0.5 meters to 5 meters). For example, in the living room above, the davg is 4 meters, so λ = 0.5; in the vehicle cabin, the davg is 1 meter, so... l =1.0; through l With scene adaptability, the system can maintain optimal pointing selection accuracy and response speed under different spatial layouts;

[0118] The final fused pointing score is calculated based on the gaze ray score and gesture ray score. S to = α *Score( R eye , O i )+ β * Score( R ha , O i );

[0119] In specific implementation, the above embodiment S44 is as follows: In this embodiment, the user looks at the floor lamp and points at it with their finger, and the line of sight... R eye The origin is the user's forehead, and the direction is... V 1 eye Gesture Ray R ha The origin is the user's index fingertip, and the direction is... V 1 ha ; Calculate the intersection of the ray with the bounding boxes of all objects in the scene; for a floor lamp O A The line of sight intersects the bounding box; the intersection depth is... D eye =2.8m, center offset Off eye =0.05m (very close to the center); the gesture ray intersects the bounding box, and the intersection depth is... D ha =2.5m, center offset Off ha =0.1m;

[0120] The Slab algorithm is used to calculate the intersection of the ray and the axis-aligned bounding box (AABB); if there is no intersection, Hit( R , O i =0; If there is an intersection, calculate the distance Dist from the origin of the ray to the intersection; S( R ,O i ) is the matching degree function between the ray and the object: ;in This indicates that the hit indicator function has been executed, and can be either 1 or 0. off Indicates center offset;

[0121] α and β Weighting rules: When explicitly pointing, set β =0.7, α =0.3; determined as latent gaze, set α =0.8, β =0.2; the α , β The value is determined by drawing on the "Pareto Principle" and the "70 / 30" rule, ensuring that the dominant modality (overt gestures or latent gaze) has an overwhelming decision weight (>70%), while retaining the corrective ability of the secondary modality (approximately 20%-30%). This ratio, determined through repeated adjustments in real-world testing, ensures that even with reasonable errors in gesture recognition, the overall pointing accuracy of the system remains above 95%. It detects whether the user's hand is raised (wrist coordinates). y (A value higher than the abdomen indicates the hand is raised); if the hand is raised and the finger points towards the floor lamp, H wr When the distance is greater than 0.2m (relative to the abdomen), the system trusts gestures more. β Take the higher value of 0.7; if the user is simply sitting on the sofa looking at the floor lamp and says "turn it on" without raising their hand, the system trusts the gaze more. α Take the highest value, 0.9; in this example, the user raises their hand, and the floor lamp score is calculated. la =0.3*e−0.05+0.7*e−0.1≈0.3*0.95+0.7*0.90=0.285+0.63=0.915; Calculate the air conditioner score: Neither the line of sight nor the gesture hit the air conditioner enclosure. ac =0; the system determines that "that" refers to "floor lamp";

[0122] S45. Select the object with the highest fusion pointing score as the referent object, and bind the object ID of the referent object to the speech intent.

[0123] The above embodiments construct a local three-dimensional semantic map and generate spatial rays based on the user's gaze and gestures to perform precise object pointing calculations in a geometric manner; they introduce a dynamic weight allocation mechanism based on wrist height, which trusts gestures more when pointing explicitly and gaze more when looking implicitly, making decisions conform to human natural intentions; they map abstract language references to specific physical object IDs, laying a key foundation for achieving natural interaction where what you see is what you control and what you point to is what you control.

[0124] The above embodiment S5 includes the following steps (this embodiment aims to remove cumbersome wake words such as "Xiao Ai" and "Tmall Genie"):

[0125] S51. A cone-shaped virtual space is set in front of the user's field of vision; the cone-shaped virtual space covers the physical area containing the smart device and its sensors, thus obtaining the interaction area ROI;

[0126] In specific implementation, the above embodiment S51 specifically involves: constructing a wake-up-free interactive state machine; the states include: idle, ready, listening, and processing;

[0127] S52. Calculate the angle between the user's gaze vector and the interaction area ROI in real time; if the angle is less than the angle threshold of 15 degrees and the duration exceeds the duration threshold of 500 milliseconds, it is determined to be a gaze lock state.

[0128] In specific implementation, the above embodiment S52 is as follows: when the angle between the user's gaze vector and the interaction area ROI is less than the angle threshold of 15 degrees, and the duration T>500ms, the state transitions from idle to ready; at this time, the LED light on the device is dimly lit; the angle threshold of 15 degrees is set based on the effective visual field range of the human eye's foveal vision, representing that the user is looking directly at the device; the duration threshold of 500ms is to distinguish between unconscious saccades (less than 300ms) and conscious fixation;

[0129] S53. Calculate the opening amplitude and rate of change of the key lip points in S2; if continuous lip movement signals exceeding the silence threshold are detected in the gaze lock state, it is determined to be intention triggering.

[0130] In specific implementation, the above embodiment S53 is as follows: In the READY state, key points of the lips are detected; if the lip opening amplitude is detected to be >5mm and the lip movement frequency is >2Hz (speaking begins), the state is immediately switched to listening; at the same time, the cached audio data of the past 2 seconds is reviewed to ensure that the beginning of the sentence is not lost; the silence threshold is set according to the average value of the background lip micro-movement amplitude in the environment in the idle state plus 3 times the standard deviation, which is used to filter mouth micro-movements when not speaking;

[0131] S54. Upon receiving the intent trigger, the system transitions from standby monitoring state to active listening state, activates modules S3 and S4 for processing, combines the preliminary voice text sequence generated by S3 with the physical object ID parsed by S4 to generate the final device control command.

[0132] In specific implementation, the above embodiment S54 is as follows: after the user says "turn that on", he closes his mouth and keeps his gaze fixed; the system determines that the speech has ended, and combines the text "turn on" recognized in S3 and the object "floor lamp" parsed in S4 to generate the instruction "turn on the floor lamp";

[0133] S55. After processing, if the confidence level of the generated device control command is higher than the first confidence level threshold of 0.9, then the control command is issued directly.

[0134] If the confidence level is between the second confidence threshold of 0.6 and the first confidence threshold of 0.9, a voice inquiry for confirmation will be initiated; if the gaze moves out of the ROI area for more than 2 seconds, the system will automatically return to the standby monitoring state.

[0135] In specific implementation, the above embodiment S55 is as follows: the instruction confidence is the geometric mean of the probability score of the decoder output text sequence in S3 and the final fusion pointing score in S4; the first confidence threshold of 0.9 is set to ensure that the directly executed operation must have extremely high accuracy and avoid misoperation. The second confidence threshold of 0.6 is the minimum threshold for the system to perform effective interaction; if it is lower than this value, it is considered to be noise or an invalid instruction; after the instruction is executed, the device provides voice feedback "Okay, the floor lamp has been turned on" and returns to the idle state; if the user's gaze is removed for more than 2 seconds during the speaking process (such as turning their head to talk to a family member), the system determines that it is a non-interactive intention, automatically discards the currently recognized content, and returns to the idle state, effectively preventing false triggering;

[0136] The above embodiments determine the interaction intent by detecting the user's continuous gaze dwell in the interaction area (>500ms) and subsequent lip movements, simulating the natural process of initiating dialogue between people through eye contact; eliminating the reliance on traditional wake words and improving the smoothness of interaction; by introducing confidence-based hierarchical decision-making and automatic backtracking logic for gaze interruption, it effectively prevents false triggering while ensuring high accuracy of intent recognition, making the entire interaction process both efficient and reliable.

[0137] The intelligent speech recognition and interaction method integrating AI visual information in this invention also possesses an adaptive online learning mechanism, including the following steps:

[0138] S61. Define positive and negative feedback signals;

[0139] S62. Construct a contrastive loss function; for negative samples, calculate the feature distance between the predicted object and the true intention object as a penalty term; for positive samples, reduce the prediction error under the current environmental parameters.

[0140] S63. Collect user feedback behavior on the interaction results as the current positive and negative feedback signals; based on the current positive and negative feedback signals and the loss function, iteratively update the weight parameters of the cross-modal dynamic gating fusion network and the ray weighting coefficients of the three-dimensional spatial semantic alignment engine to adapt to the personalized interaction habits and environmental changes of different users.

[0141] In specific implementation, the above embodiment S6 is as follows: if the user does not issue a new correction instruction (such as "No", "That's it") within 5 seconds after the instruction is executed, or shows a nodding or smiling facial expression, it is marked as a positive sample; if the user issues a correction instruction or shows a frowning or shaking facial expression, it is marked as a negative sample; the formula for the contrast loss function is as follows: ,in N Indicates the batch sample quantity; u i =1 represents a positive sample. u i =0 represents a negative sample; d i To predict the Euclidean distance between the feature vector pointing to the object and the feature vector of the real target object. March The set boundary threshold is used to define a desired separation boundary in negative samples. In this embodiment, the empirical distance distribution between positive and negative sample feature vector pairs in the system feature space is analyzed, and the model's separation requirement for erroneous samples is balanced with the stability of gradient updates to obtain the desired boundary. March It is 0.5; March At a loss threshold of 0.5, a clear boundary is effectively formed, causing positive sample pairs to cluster tightly while pushing negative sample pairs to a safe distance. The loss threshold is set to 0.01; at 0.01, when the change in the average fitness of the population is less than this value, it indicates that the algorithm has converged to a local optimum and the parameters have stabilized. For positive samples... u i =1, the loss term is d 2 i The optimization objective is to minimize the distance between the prediction and the target; for negative samples, when the distance of the incorrect prediction is... d Less than the boundary March At that time, the loss function will be calculated March - d i ) 2 As a penalty, the feature distance between the incorrect prediction and the target is forcibly increased through gradient updates until it exceeds the set safety boundary, pushing the incorrect prediction further away. March In addition;

[0142] In this embodiment, if user A1 is accustomed to using eye contact for control, the system initially misidentifies the gesture due to the high default weighting (negative sample). After the user corrects the gesture, the system automatically increases the confidence weight of the gaze pattern configured by user A through gradient descent. α Lower the confidence weight of gestures β After approximately 20 interactive iterations, the system's accuracy in predicting user A's intent increased from 85% to 98%, achieving the goal of understanding users better with repeated use.

[0143] The iterative update of the weight parameters of the cross-modal dynamic gating fusion network and the ray weighting coefficients of the 3D spatial semantic alignment engine in the above embodiment S63 includes the following steps:

[0144] S631. Construct a chromosome population; use the chromosomes in the population as weight parameters of a cross-modal dynamic gated fusion network and the ray weighting coefficients of a 3D spatial semantic alignment engine, wherein the ray weighting coefficients are the line-of-sight confidence weights. α Lower the confidence weight of gestures β ;

[0145] S632. By comparing the loss function, select, crossover and mutate chromosomes in the chromosome population; repeat the iteration, and when the set maximum number of iterations is reached or less than the preset loss value threshold, the optimal chromosome is obtained; apply the weight parameters of the cross-modal dynamic gating fusion network corresponding to the optimal chromosome and the ray weighting coefficient of the three-dimensional spatial semantic alignment engine to the cross-modal dynamic gating and the three-dimensional spatial semantic alignment engine respectively.

[0146] In specific implementation; S631 and S632 above are specifically as follows: S6311, constructing a chromosome population, setting the size of the chromosome population as... c This embodiment sets c If the value is 50, then the chromosome population is represented as follows: , p i Represents the first chromosome in the population. A population of chromosomes; each chromosome in the chromosome population is used as a combination of weight parameters of a random cross-modal dynamic gated fusion network in the search space and ray weighting coefficients of a 3D spatial semantic alignment engine; a maximum number of optimization iterations is set; chromosome encoding uses real number encoding; each chromosome is encoded as a sequence of lengths... L A floating-point vector, where the first mf Each gene corresponds to a trainable weight in a cross-modal dynamic gated fusion network, and then... mh Each gene corresponds to a ray-weighted coefficient (i.e., line-of-sight confidence weight) in the 3D spatial semantic alignment engine. α Weighted by gesture confidence β For example, a chromosome can be represented as [w 1, w 2, ..., w mf , α , β ];in w mf Indicates the first mf Trainable weights for a cross-modal dynamic gated fusion network;

[0147] S6312. Begin iteration. In each iteration, the fitness value of each chromosome in the chromosome population is calculated based on the contrastive loss function, resulting in a set of chromosome fitness values. The smaller the loss, the higher the fitness. Based on the chromosome fitness values ​​in the set, chromosomes in the chromosome population are selected from high to low to obtain a selected chromosome population. Crossover and mutation operations are performed on the chromosomes in the selected chromosome population to obtain the operated chromosome population. In each iteration, the current best combination is selected based on the chromosome performance values ​​in the set. The crossover is calculated by averaging the parent chromosome values, and the mutation is Gaussian mutation. The crossover operation uses arithmetic crossover, where the weighted average (e.g., taking the mean) of each gene value corresponding to the two selected parent chromosomes is used to generate offspring. In this embodiment, the crossover rate is set to 0.8. The mutation operation uses Gaussian mutation, where a Gaussian random perturbation with a mean of 0 and a standard deviation of 0.05 is applied to the selected genes, i.e., the mutation rate is set to 0.05. The selection strategy is roulette wheel selection based on fitness values.

[0148] Repeat the iteration until the maximum number of optimization iterations is reached or the preset loss value threshold is reached, then stop the iteration and take the current best combination as the optimal solution;

[0149] The above embodiments collect users' explicit corrections or implicit facial expressions as feedback, construct a contrastive loss function, and use a genetic algorithm to globally optimize network weights and ray fusion coefficients. This enables the system to learn from interaction errors and dynamically adjust its internal trust weight allocation for different users (such as those who prefer eye contact or gesture control). After several iterations, the system can significantly improve the accuracy of predicting the intent of specific users, achieving user adaptation that becomes more intuitive with use, and enabling the entire interaction system to continuously update.

[0150] Taking a smart home central control screen scenario as an example, this invention is as follows:

[0151] The user is sitting in a noisy living room (with loud TV volume) and wants to turn on the humidifier in the corner;

[0152] S1: The microphone array on the central control screen collects audio mixed with TV sound and human voice, and the depth camera captures the user's image;

[0153] S2: The system extracts audio Log-Mel features, and also extracts user lip features, gaze, and gesture vectors;

[0154] S3: When the ambient SNR is low (5dB), the system automatically increases the weight of the visual lip reading feature (Gv=0.8), ignores the background noise of the TV in the audio, and accurately identifies the text "Open this" corresponding to the user's lip movements.

[0155] S4: The pronoun "this" was detected, triggering spatial alignment. The system detected that the user's gaze was directed towards the humidifier. O C And the hand is slightly raised and pointing in that direction; calculate the ray score, the humidifier has the highest score;

[0156] S5: Since the user is looking at the central control screen and speaking, the system does not need a wake word and directly determines it as a valid command;

[0157] S6: The system issues a command to turn on the humidifier; the user sees the humidifier spraying and smiles (positive feedback). The system records the interaction parameters of this successful interaction and reinforces the current weight configuration.

[0158] For further details, please refer to Figure 2 A smart speech recognition and interaction system integrating AI visual information, used to implement the aforementioned smart speech recognition and interaction method integrating AI visual information, includes:

[0159] Multimodal perception front-end module: integrates microphone array and depth camera, responsible for synchronous data acquisition and preprocessing, to obtain multi-channel audio stream and RGB-D video stream;

[0160] Dual-stream feature extraction module: Includes parallel acoustic feature extraction unit and visual feature extraction unit, used to process multi-channel audio stream and RGB-D video stream, and generate acoustic feature vector, visual speech vector and three-dimensional spatial interaction vector;

[0161] Fusion Inference Engine Module: Runs a cross-modal dynamic gating network and a 3D spatial semantic alignment algorithm to process acoustic feature vectors, visual speech vectors, and 3D spatial interaction vectors, perform intent parsing and object locking, and obtain preliminary speech-text sequences and physical object IDs;

[0162] Interactive Control Center Module: Includes state machine logic and instruction distribution interface, based on preliminary voice and text sequences and physical object IDs, to realize the system's state transitions and device control.

[0163] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0164] The preferred embodiments of the invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention.

Claims

1. An intelligent speech recognition and interaction method integrating AI visual information, characterized in that, Includes the following steps: S1. Acquire sound field signals and light field signals in the environment, and align them with timestamps to obtain audio streams and RGB-D video streams; S2. Extract the acoustic feature vector of the audio stream in S1, as well as the visual speech vector and three-dimensional spatial interaction vector of the RGB-D video stream; S3. Through a cross-modal dynamic gating fusion network, the acoustic feature vectors and visual speech vectors extracted in S2 are mapped to the latent semantic space. Based on the signal-to-noise ratio index and the lip occlusion rate index, the fusion weights of the auditory channel and the visual channel are dynamically allocated to generate a multimodal speech semantic representation and decode the preliminary speech text sequence. S4. For the preliminary speech text sequence generated by S3, perform indicator pronoun trigger detection. When an indicator pronoun is detected, use the three-dimensional spatial interaction vector extracted by S2 to emit gaze rays and gesture rays in the constructed local three-dimensional semantic map, calculate the intersection probability and spatial distance score between the rays and the bounding boxes of objects in the scene, and parse out the physical object ID corresponding to the indicator pronoun. S5. Real-time monitoring of the dwell time and lip movement state of the three-dimensional spatial interaction vector in S2 relative to the preset interaction area; when it is determined to be an intention trigger, the preliminary voice text sequence generated in S3 is combined with the physical object ID parsed in S4 to generate the final device control command. S4 includes the following steps: S41. Construct a dictionary of indicator pronouns containing words with spatial orientation; traverse the speech-text sequence generated in S3, and if a word in the dictionary of indicator pronouns is matched, mark it as a trigger point, and extract the three-dimensional spatial interaction vector within a preset time window before and after the trigger point to obtain the average gaze vector and the average gesture vector. S42; Using point cloud data acquired by a depth camera, perform instance segmentation and bounding box fitting on objects in the scene to generate a list of objects containing object category label ID, center coordinates and 3D dimensions, thus obtaining a local 3D semantic map. S43. In the local three-dimensional semantic map, using the average gaze vector and average gesture vector in S41 as directions, construct the corresponding gaze ray and gesture ray starting from the user's brow and index fingertip respectively. S44. For each object in the scene, calculate the intersection depth and Euclidean distance between the view ray and gesture ray and the object's bounding box; define the fusion pointing score formula, and calculate the final fusion pointing score by combining adjustable weights. S45. Select the object with the highest final fusion pointing score as the referent object, and bind the object ID of the referent object to the speech intent. The S44 definition of the fusion-oriented scoring formula, combined with adjustable weights, includes the following steps in calculating the final score: S441, Define the line-of-sight ray as R eye The gesture ray is R ha The bounding box of the object is O i ; S442, Calculate the Euclidean distance Dist between the line-of-sight ray, the gesture ray, and the center of the object's bounding box. R , O i ) and the Boolean value Hit indicating whether they intersect. R , O i ); S443, Define the confidence weight of gaze level α and gesture confidence weights β Real-time detection of the user's wrist height relative to their abdomen. H wr If the user's wrist is at a height relative to their abdomen H wr If the height exceeds a preset threshold, it is determined to be an explicit pointing gesture; otherwise, it is determined to be an implicit gaze. Based on the Euclidean distances between the gaze ray, gesture ray, and the center of the object's bounding box, and the Boolean value indicating whether they intersect, the gaze ray score and gesture ray score are calculated using the single-ray score calculation formula. The single-ray score calculation formula is defined as Score( R , O i )=Hit( R , O i )*exp(- λ *Dist( R , O i )),in λ This is the distance attenuation coefficient; The final fusion pointing score is calculated based on the gaze ray score and the gesture ray score.

2. The intelligent speech recognition and interaction method integrating AI visual information as described in claim 1, characterized in that, S1 includes the following steps: S11: Configure a six-microphone circular array, set the audio sampling rate and bit depth; simultaneously configure an RGB-D depth camera, set the video resolution, frame rate and depth map measurement range to obtain a multimodal perception hardware environment; S12. Perform hardware-level time synchronization, use FPGA to generate a unified synchronization pulse signal, and simultaneously trigger the analog-to-digital conversion of the microphone array and the exposure acquisition of the depth camera to obtain the original multi-channel audio stream and RGB-D video stream. S13. Beamforming is performed on the multi-channel audio stream. The generalized cross-correlation algorithm is used to calculate the direction of arrival (DOA) of the sound source, enhance the speech signal in the target direction and suppress background noise, and output the audio stream. Based on the RGB-D video stream, a face detection algorithm is used to locate the user's facial region, and background pixels are removed based on depth information, retaining only the foreground visual information within the user's and interaction range to obtain a denoised RGB-D video stream.

3. The intelligent speech recognition and interaction method integrating AI visual information as described in claim 1, characterized in that, S2 includes the following steps: S21. Using a pre-trained Conformer encoder structure, the audio stream output by S1 is divided into frames and windowed to extract Mel filter bank features. The first-order difference and the second-order difference are then concatenated. Long-term dependence is captured through a multi-layer self-attention mechanism, and the acoustic feature vector is output. S22. Extract the region of interest (ROI) of the user's lips from the RGB-D video stream and normalize it into a grayscale image sequence. Use the ResNet-3D network to extract the continuous motion pattern of the lips in the time dimension of the grayscale image sequence and output the acoustic feature vector and the visual speech vector. S23. Detect key points of the user's hand skeleton and the center point of the eyeball in the RGB-D video stream using a skeleton detection model; combine the depth map of the key points of the skeleton and the center point of the eyeball. Z The axis information is used to back-project the two-dimensional pixel coordinates to the camera coordinate system, generating a three-dimensional hand key point coordinate set and a three-dimensional eye gaze direction vector, thus obtaining a three-dimensional spatial interaction vector.

4. The intelligent speech recognition and interaction method integrating AI visual information as described in claim 1, characterized in that, S3 includes the following steps: S31. Calculate the ratio of the signal energy of the current audio frame to the background noise energy of the non-speech segment to obtain the environmental signal-to-noise ratio; S32. Calculate the initial visual confidence score; detect whether there are obstructions in the lip area, whether the light intensity is lower than the preset threshold, or whether the head posture deflection angle exceeds the head yaw angle threshold; if there are obstructions, the light intensity is lower than the preset threshold, or the head posture deflection angle exceeds the head yaw angle threshold, the initial visual confidence score is reduced to obtain the final visual confidence score; otherwise, the initial visual confidence score is used as the final visual confidence score. S33. Construct a cross-modal dynamic gating unit; define the audio gating coefficient and visual gating coefficient of the cross-modal dynamic gating unit; the gating coefficients are calculated by a fully connected layer and a Sigmoid activation function; The acoustic feature vector, visual speech vector, normalized environmental signal-to-noise ratio, and final visual confidence are input into the cross-modal dynamic gating unit to obtain the current audio gating coefficient and the current visual gating coefficient. S34. Calculate the fusion features using the fusion formula based on the current audio gating coefficient and the current visual gating coefficient; S35. Input the fused features into the Transformer decoder, and combine them with the beam search algorithm to output the preliminary speech-text sequence with the highest probability.

5. The intelligent speech recognition and interaction method integrating AI visual information as described in claim 1, characterized in that, S5 includes the following steps: S51. A cone-shaped virtual space is set in front of the user's field of vision; the cone-shaped virtual space covers the physical area containing the smart device and its sensors, thus obtaining the interaction area ROI; S52. Calculate the angle between the user's gaze vector and the interaction area ROI in real time; if the angle is less than the angle threshold and the duration exceeds the duration threshold, it is determined to be a gaze lock state. S53. Calculate the opening amplitude and rate of change of the key lip points in S2; if continuous lip movement signals exceeding the silence threshold are detected in the gaze lock state, it is determined to be intention triggering. S54. Upon receiving the intent trigger, the device transitions from standby monitoring state to active listening state, combines the preliminary voice text sequence generated in S3 with the physical object ID parsed in S4, and generates the final device control command. S55. After processing, if the confidence level of the generated device control command is higher than the first confidence level threshold, the control command is issued directly. If the confidence level is between the second confidence threshold and the first confidence threshold, a voice inquiry for confirmation will be initiated; if the gaze moves out of the ROI area for more than the gaze-out time threshold, it will automatically revert to standby monitoring mode.

6. The intelligent speech recognition and interaction method integrating AI visual information as described in claim 1, characterized in that, It also features an adaptive online learning mechanism, including the following steps: S61. Define positive and negative feedback signals; S62. Construct a contrastive loss function; for negative samples, calculate the feature distance between the predicted object and the true intention object as a penalty term; for positive samples, reduce the prediction error under the current environmental parameters. S63. Collect user feedback behavior on the interaction results as the current positive and negative feedback signals; based on the current positive and negative feedback signals and the loss function, iteratively update the weight parameters of the cross-modal dynamic gating fusion network and the ray weighting coefficients of the 3D spatial semantic alignment engine.

7. The intelligent speech recognition and interaction method integrating AI visual information as described in claim 6, characterized in that, The iterative update of the weight parameters of the cross-modal dynamic gating fusion network and the ray weighting coefficients of the 3D spatial semantic alignment engine in S63 includes the following steps: S631. Construct a chromosome population; use the chromosomes in the population as weight parameters of a cross-modal dynamic gated fusion network and the ray weighting coefficients of a 3D spatial semantic alignment engine; S632. By comparing the loss function, select, crossover, and mutate chromosomes in the chromosome population; repeat the iteration until the set maximum number of iterations is reached or less than the preset loss value threshold, to obtain the optimal chromosome; apply the weight parameters of the cross-modal dynamic gating fusion network corresponding to the optimal chromosome and the ray weighting coefficient of the three-dimensional spatial semantic alignment engine to the cross-modal dynamic gating and the three-dimensional spatial semantic alignment engine, respectively.

8. An intelligent voice recognition and interaction system integrating AI visual information, characterized in that, The system implementing the intelligent speech recognition and interaction method integrating AI visual information as described in any one of claims 1-7 includes: Multimodal perception front-end module: integrates microphone array and depth camera, responsible for synchronous data acquisition and preprocessing, to obtain multi-channel audio stream and RGB-D video stream; Dual-stream feature extraction module: Includes parallel acoustic feature extraction unit and visual feature extraction unit, used to process multi-channel audio stream and RGB-D video stream, and generate acoustic feature vector, visual speech vector and three-dimensional spatial interaction vector; Fusion Inference Engine Module: Runs a cross-modal dynamic gating network and a 3D spatial semantic alignment algorithm to process acoustic feature vectors, visual speech vectors, and 3D spatial interaction vectors, perform intent parsing and object locking, and obtain preliminary speech-text sequences and physical object IDs; Interactive Control Center Module: Includes state machine logic and instruction distribution interface, based on preliminary voice and text sequences and physical object IDs, to realize the system's state transitions and device control.

Images