Lip-reading-based voice interaction methods, devices, equipment, and storage media
By collecting and fusing lip reading and speech features in real time, the problem of low speech recognition accuracy in noisy environments was solved, enabling more efficient human-computer interaction and improving the efficiency and satisfaction of visitor machines.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHENZHEN WANRUI INTELLIGENT TECH CO LTD
- Filing Date
- 2025-06-27
- Publication Date
- 2026-07-03
AI Technical Summary
Existing speech recognition technologies struggle to effectively integrate lip reading and speech recognition in noisy environments, leading to decreased speech recognition accuracy and impacting the interactive experience between visitors and the visitor machine.
By synchronously acquiring facial video streams and audio signals in real time, deep learning face detection and lip keypoint detection algorithms are used to extract lip region features. Combined with cross-modal fusion coding technology, lip reading features and audio features are input into a pre-trained large language model to generate semantic responses and play or display them.
It significantly improves the robustness and accuracy of speech recognition, achieves a more natural and intelligent human-computer interaction experience, reduces environmental noise interference, and enhances the adaptability and smoothness of the visitor machine in noisy environments.
Smart Images
Figure CN120600019B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of artificial intelligence, and specifically relates to a voice interaction method, device, equipment and storage medium based on lip reading enhancement. Background Technology
[0002] With the development of artificial intelligence technology, self-service visitor kiosks are widely used in enterprises, industrial parks, communities, and other locations. These devices use cameras and microphones to collect images and sound signals from visitors, enabling functions such as identity verification, visitor registration, and information retrieval. However, in practical applications, environmental noise often leads to a decrease in the accuracy of speech recognition results, affecting the normal interactive experience between visitors and the kiosk.
[0003] Traditional speech recognition technology relies primarily on the analysis of sound signals. However, in noisy environments, background noise can easily drown out a visitor's valid speech, making it difficult for speech recognition systems to accurately extract and identify the visitor's intent. While some existing noise reduction techniques can partially address these issues, their effectiveness remains limited.
[0004] In recent years, researchers have begun exploring methods to assist speech recognition using visual information, with lip reading being an important approach. Human lip movements are closely related to pronunciation; even when the sound is unclear, some or all of the spoken content can be inferred by observing the speaker's lip movements. Combining lip reading with speech recognition can significantly improve speech recognition accuracy in noisy environments.
[0005] However, existing technologies that apply lip reading to assisted speech recognition often perform speech and lip reading independently, followed by simple result fusion. This approach fails to fully leverage the synergistic relationship between speech and lip reading. In particular, effectively integrating lip reading information into advanced large language models to improve the focus and accuracy of interactions remains a challenge. Summary of the Invention
[0006] In view of this, the present invention provides a voice interaction method, apparatus, device and storage medium based on lip reading enhancement, which solves the technical problem of the ineffective integration of traditional lip reading recognition and voice recognition.
[0007] To address the aforementioned problems, according to a first aspect of the present invention, embodiments of the present invention provide a voice interaction method based on lip-reading enhancement, comprising: real-time synchronous acquisition of a facial video stream and a speech signal of an interactive object; extraction of a facial region from the facial video stream using a deep learning-based face detection model, and location of the lip region of the facial region using a lip keypoint detection algorithm, and real-time tracking of the lip region using a deep learning-based target tracking algorithm to obtain an image sequence of the lip region; extraction of lip-reading features based on the image sequence of the lip region, preprocessing the acquired speech signal and extracting features to obtain audio features; cross-modal fusion encoding of the extracted lip-reading features and audio features to generate a hybrid feature containing audiovisual information; inputting the hybrid encoded feature into a pre-trained large language model to understand the intention of the interactive object and generate a corresponding semantic response; synthesizing the semantic response into speech for playback, and / or converting the semantic response into text for visualization display.
[0008] In some embodiments, the step of extracting face regions from a facial video stream using a deep learning-based face detection model, locating the lip region of the face region using a lip keypoint detection algorithm, and tracking the lip region in real time using a deep learning-based target tracking algorithm to obtain an image sequence of the lip region includes: performing frame-by-frame detection on the facial video stream using the YOLOv8-face algorithm, cropping the face region according to the detection box coordinates, and scaling it to a uniform size; extracting the facial feature vector of the face region as the corresponding identity feature using the ArcFace model, and tracking the same person in consecutive frames of the facial video stream based on the identity feature; predicting the motion trajectory through Kalman filtering during the tracking process, and using LSTM temporal modeling to enhance tracking robustness; locating the lip region in the face region using the lip keypoint detection algorithm, cropping the lip region and scaling it to a standard size to obtain an image sequence of the lip region of the same person.
[0009] In some embodiments, the extraction of lip-reading features from image sequences based on the lip region includes: extracting short-term spatiotemporal features of the image sequence of the lip region using 3D-ResNet, capturing local patterns of inter-frame micro-movements through multi-level convolutional kernels; converting the short-term spatiotemporal features into lip-reading features using a visual Transformer encoder; wherein a compression-excitation module is introduced at the front end of the visual Transformer encoder.
[0010] In some embodiments, the step of cross-modal fusion encoding of the extracted lip-reading features and audio features to generate hybrid features containing audiovisual information includes: linearly interpolating the audio features to synchronize them with the lip-reading features; standardizing the audio features and lip-reading features respectively to eliminate dimensional differences; projecting the audio features and lip-reading features onto the same latent space using independent learnable weight matrices, and transforming the audio features and lip-reading features to align their dimensions; using dot product attention to measure the correlation between the audio features and lip-reading features to obtain a relevance score, normalizing the relevance score using the Softmax function to generate probabilistic attention weights; and weighting and concatenating the audio features and lip-reading features using the attention weights to obtain hybrid features containing audiovisual information.
[0011] In some embodiments, the step of performing cross-modal fusion encoding on the extracted lip-reading features and audio features to generate hybrid features containing audiovisual information further includes: automatically adjusting the weights according to the distribution of audio features and lip-reading features, while establishing cross-modal and intramodal relationships to dynamically capture complementary information.
[0012] In some embodiments, the step of performing cross-modal fusion encoding on the extracted lip-reading features and audio features to generate hybrid features containing audiovisual information further includes: forming a sequence block with the hybrid features and text tokens based on a multimodal time synchronization strategy, extracting the local context of the sequence block through a causal convolutional layer, and adding a learnable positional encoding mark to indicate the temporal order.
[0013] In some embodiments, the step of inputting the hybrid encoded hybrid features into a pre-trained large language model to understand the intent of the interactive object and generate a corresponding semantic response includes: inputting the sequence block into the pre-trained large language model in the form of a token stream, and constraining the cross-modal visibility range in the self-attention layer of the large language model through a masking mechanism.
[0014] According to a second aspect of the present invention, embodiments of the present invention provide a voice interaction device based on lip-reading enhancement, comprising: an acquisition unit for real-time synchronous acquisition of a facial video stream and a speech signal of an interactive object; an image processing unit for extracting a facial region from the facial video stream using a deep learning-based face detection model, locating the lip region of the facial region using a lip keypoint detection algorithm, and tracking the lip region in real time using a deep learning-based target tracking algorithm to obtain an image sequence of the lip region; a feature extraction unit for extracting lip-reading features based on the image sequence of the lip region, preprocessing the acquired speech signal, and extracting features to obtain audio features; a fusion unit for cross-modal fusion encoding of the extracted lip-reading features and audio features to generate a hybrid feature containing audiovisual information; a generation unit for inputting the hybrid encoded feature into a pre-trained large language model to understand the intention of the interactive object and generate a corresponding semantic response; and a playback and display unit for synthesizing the semantic response into speech for playback, and / or converting the semantic response into text for visual display.
[0015] According to a third aspect of the present invention, embodiments of the present invention provide a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the lip-reading-enhanced speech interaction method as described in the first aspect.
[0016] According to a fourth aspect of the present invention, embodiments of the present invention provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the lip-reading-enhanced speech interaction method as described in the first aspect.
[0017] This invention introduces lip features to provide additional visual cues for speech recognition, significantly improving its robustness and accuracy, especially in noisy environments. It effectively fuses and encodes lip-reading and vocal features, enabling the model to utilize both auditory and visual information for understanding and reasoning, avoiding the semantic information fragmentation caused by simple independent recognition. The hybrid-encoded audiovisual features are then input into a pre-trained large language model, fully leveraging the capabilities of the large model to achieve a more natural and intelligent interactive experience.
[0018] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, the preferred embodiments of the present invention are described in detail below with reference to the accompanying drawings. Attached Figure Description
[0019] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0020] Figure 1 This is a flowchart illustrating the voice interaction method based on lip reading enhancement provided in an embodiment of the present invention;
[0021] Figure 2 This is a schematic block diagram of a voice interaction device based on lip reading enhancement provided in an embodiment of the present invention. Detailed Implementation
[0022] To further illustrate the technical means and effects adopted by the present invention to achieve the intended purpose, the specific embodiments, structures, features, and effects according to the present invention will be described in detail below with reference to the accompanying drawings and preferred embodiments. In the following description, different "an embodiment" or "an embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.
[0023] like Figure 1 As shown, this embodiment provides a voice interaction method based on lip-reading enhancement, which includes:
[0024] S101. Real-time synchronous acquisition of facial video streams and voice signals of interactive objects;
[0025] S102. A face region is extracted from the facial video stream using a deep learning-based face detection model, and the lip region of the face region is located using a lip key point detection algorithm. A deep learning-based target tracking algorithm is used to track the lip region in real time to obtain an image sequence of the lip region.
[0026] S103. Extract lip-reading features from image sequences of the lip region, preprocess the acquired speech signal and extract features to obtain audio features;
[0027] S104. Perform cross-modal fusion encoding on the extracted lip-reading features and audio features to generate hybrid features containing audiovisual information;
[0028] S105. Input the hybrid encoded features into a pre-trained large language model to understand the intention of the interactive object and generate the corresponding semantic response;
[0029] S106. The semantic response is synthesized into speech and played, and / or the semantic response is converted into text and displayed visually.
[0030] This invention introduces lip features to provide additional visual cues for speech recognition, significantly improving its robustness and accuracy, especially in noisy environments. It effectively fuses and encodes lip-reading and vocal features, enabling the model to utilize both auditory and visual information for understanding and reasoning, avoiding the semantic information fragmentation caused by simple independent recognition. The hybrid-encoded audiovisual features are then input into a pre-trained large language model, fully leveraging the capabilities of the large model to achieve a more natural and intelligent interactive experience.
[0031] Specifically, the method of this invention can be applied to a self-service visitor machine, and a specific example of its hardware configuration is as follows:
[0032] Perception layer: The self-service visitor machine is equipped with a high-definition camera (200W pixels, 1080P resolution, 30fps frame rate, supports automatic white balance) to collect visitor video, and a MEMS high-sensitivity microphone (6 channels, noise reduction level ≥40dB) to collect visitor sound.
[0033] Computation layer: High-performance computing unit (integrated NVIDIA Jetson AGX Orin computing platform, 200 TOPS computing power), supporting real-time feature extraction and model inference on the edge;
[0034] Interaction layer: Equipped with a 21.5-inch capacitive touchscreen (1920×1080 resolution) and speakers (signal-to-noise ratio ≥85dB).
[0035] Software implementation: Develop the software system corresponding to the modules for each step.
[0036] Model Training and Optimization: Collect a multimodal dataset containing facial video streams and speech signals for training and fine-tuning the fusion model (corresponding to step S104) and the large language model (corresponding to step S105). Furthermore, optimize the model for specific scenarios of self-service visitor machines, such as fine-tuning for specific industry terminology or interaction flows.
[0037] Specifically, in step S101, the high-definition camera and high-sensitivity microphone equipped with the self-service visitor machine are used to collect visitor facial video streams (resolution ≥1080p, frame rate ≥30fps) and voice signals (sampling rate ≥48kHz) in real time, providing raw data for subsequent multimodal processing.
[0038] In step S102, a deep learning-based face detection model (such as MTCNN, a multi-task cascaded convolutional neural network) is used to locate the visitor's facial region. Combined with an Active Shape Model (ASM) and keypoint detection technology (68-point facial landmark localization), lip keypoints (including 28 inner and outer lip keypoints such as the cupid's bow, valley, and corners of the lips) are extracted. Furthermore, a deep learning-based target tracking algorithm can be used to track the lip region in continuous video frames in real time, achieving sub-pixel-level localization accuracy and ensuring the continuity of motion feature extraction.
[0039] In some embodiments, step S102 includes:
[0040] The YOLOv8-face algorithm is used to detect the facial video stream frame by frame, and the face region is cropped according to the coordinates of the detection box and scaled to a uniform size;
[0041] The ArcFace model is used to extract the facial feature vector of the face region as the corresponding identity feature, and the same person in consecutive frames of the facial video stream is tracked based on the identity feature; during the tracking process, the motion trajectory is predicted by Kalman filtering and LSTM temporal modeling is used to enhance the tracking robustness.
[0042] The lip region in the face region is located using a lip keypoint detection algorithm, and the lip region is cropped and scaled to a standard size to obtain an image sequence of the lip region of the same person.
[0043] In this embodiment, in order to ensure lightweight design and achieve sufficient accuracy, face detection technology based on the YOLO series models is adopted, combined with lip key point localization and feature vector tracking technology, to realize a complete process of face capture, standardized extraction of lip features, and continuous tracking of personnel identity.
[0044] Specifically, the YOLOv8-face algorithm is used to perform frame-by-frame fast detection of the facial video stream. The face region is cropped according to the coordinates of the detection box and scaled to a fixed size (224x224) to achieve size standardization. To avoid deformation, a scaling + padding strategy that maintains the aspect ratio is used to fill the shorter side to the target size.
[0045] The lip landmark detection algorithm uses a 68-point face landmark model (MTCNN extension) to locate the lip region. The lip image normalization method is to uniformly crop the lip region and scale it to a standard size (96x96).
[0046] Face feature encoding uses the ArcFace model to extract face feature vectors (128 dimensions), which serve as important identity features in multi-frame tracking. When a face is detected in the first frame, its feature vector is stored as an identity identifier. For faces in subsequent frames, the Euclidean distance between the feature vectors and the registered vectors is calculated. If the distance is less than a threshold (e.g., 0.6), the face is identified as belonging to the same person.
[0047] During tracking, Kalman filtering is used to predict motion trajectories, and LSTM temporal modeling is employed to enhance tracking robustness. In cases of occlusion or loss, blank images are inserted to avoid mismatches in the lip sequence data. LSTM (Long Short-Term Memory) is a special type of recurrent neural network (RNN) generally used for modeling temporal data. Its steps typically include: data preparation and preprocessing, building the LSTM model, model training, and model evaluation and prediction. This embodiment uses LSTM temporal modeling, which allows for more stable gradient flow during backpropagation, avoiding gradient vanishing or exploding phenomena, thus enabling stable training over longer time spans.
[0048] In step S103, lip reading features and audio features are extracted respectively.
[0049] The image sequence of the tracked lip region is analyzed to extract lip-reading features that characterize lip movement and shape changes. Lip-reading features can be obtained using image-based static methods or motion-based dynamic methods.
[0050] Image-based static methods involve extracting visual features of the lip region, such as pixel values, Local Binary Patterns (LBP), and Histogram of Oriented Gradients (HOG). Simultaneously, convolutional neural networks (CNNs) are used to directly learn high-level visual features from the lip images.
[0051] Motion-based dynamic methods analyze the motion information of the lip region between consecutive frames, such as optical flow and keypoint displacement. Temporal motion features can be modeled using recurrent neural networks (RNNs) or Transformer models, outputting a 128-dimensional lip-reading feature vector.
[0052] In some embodiments, the extraction of lip-reading features based on the image sequence of the lip region includes:
[0053] 3D-ResNet is used to extract short-term spatiotemporal features of the image sequence of the lip region, and multi-level convolutional kernels are used to capture local patterns of micro-movements between frames;
[0054] The short-term spatiotemporal features are converted into lip-reading features using a visual Transformer encoder; wherein, a compression-excitation module is introduced at the front end of the visual Transformer encoder.
[0055] The lip-reading feature embedding model used in this embodiment is a hybrid model based on 3D CNN and visual Transformer, that is, it combines 3D-ResNet and visual Transformer encoder.
[0056] 3D Convolutional Layer: 3D-ResNet is used to extract short-term spatiotemporal features (such as lip movement trajectory and opening and closing frequency) from lip videos (image sequences of the lip region), and multi-level convolutional kernels are used to capture local patterns of micro-movements between frames.
[0057] Visual Transformer Encoder: Converts short-term spatiotemporal features output from 3D convolutional layers into temporal vectors to obtain lip reading features. It utilizes a multi-head self-attention mechanism to model long-distance dependencies, solving the gradient vanishing problem of traditional RNNs and enhancing global perception of cross-frame lip dynamics.
[0058] This invention also employs an SE module to enhance the visual Transformer encoder: a squeeze-and-excitation module is introduced at the Transformer front end. This module achieves adaptive recalibration of channel feature responses by explicitly modeling inter-channel dependencies. It includes compression and excitation operations. The compression operation aggregates the spatial dimension information of the transformed features to generate channel descriptors; the excitation operation uses a self-gating mechanism, taking the compression result as input to generate a set of modulation weights for each channel. In this embodiment, combined with the SE module, the feature responses of keyframes (such as high-frequency information during lip opening and closing) are enhanced by dynamically adjusting the channel weights, while suppressing redundant background noise.
[0059] The hybrid model used in this embodiment leverages the joint modeling capability of spatiotemporal features, resulting in improved recognition rates compared to pure CNN models. Furthermore, it supports dynamic temporal resolution adjustment, adapting to lip-sync videos with varying speech rates through hierarchical feature fusion.
[0060] Audio features can be extracted using the following methods: Preprocess the acquired speech signal, such as through noise reduction and voice activity detection (VAD), to extract valid speech segments. Then, perform feature extraction on the speech signal, such as using Mel-frequency cepstral coefficients (MFCC) and filter bank features (FBANK), to generate a 256-dimensional sound feature vector.
[0061] In step S104, the main focus is on fusing multimodal features. The extracted lip-reading and audio features are fused and encoded across modally to generate hybrid features containing audiovisual information. Specifically, feature-level fusion or model-level fusion can be used. Feature-level fusion refers to calibrating the temporal deviations of the audiovisual signals using a spatiotemporal alignment algorithm (Dynamic Time Warping (DTW),) and generating joint features through channel splicing (lip-reading features + audio features, dimension 384) or adaptive weighting (weights dynamically generated by a cross-modal attention mechanism). Model-level fusion involves constructing a multimodal Transformer network, capturing the semantic relationships between audiovisual features through a multi-head self-attention mechanism (8 heads), and outputting a 256-dimensional fused feature vector to achieve end-to-end cross-modal feature generation.
[0062] In some embodiments, step S104 includes:
[0063] Linear interpolation is performed on the audio features to synchronize them with the lip-reading features; the audio features and lip-reading features are then standardized to eliminate dimensional differences.
[0064] The audio features and lip-reading features are projected into the same latent space through independent learnable weight matrices, and the audio features and lip-reading features are transformed to align their dimensions.
[0065] Dot product attention is used to measure the correlation between audio features and lip reading features to obtain a correlation score. The correlation score is then normalized using the Softmax function to generate probabilistic attention weights.
[0066] The audio features and lip-reading features are weighted and concatenated using the attention weights to obtain a hybrid feature that includes audiovisual information.
[0067] Due to the time-varying nature of the features, weight configuration needs to be combined with feature quality. Firstly, cross-modal alignment and interpolation fusion are required. Specifically, linear interpolation is performed on the audio features to synchronize their frame rate with the lip-reading features. For example, the sliding window FastDTW algorithm is used to control the temporal deviation between the 300ms audio window (30 frames) and the 8 frames of video (320ms) within ±5ms, and cubic spline interpolation is used to compensate for minute time differences. Z-score standardization is then applied to both audio and lip-reading features to eliminate dimensional differences. Finally, a weighted fusion is performed by calculating the contribution weights of the audio and lip-reading features using a fully connected layer and Softmax.
[0068] Audio and lip-reading features are projected onto the same latent space through independent learnable weight matrices (fully connected layers). The audio and video features are then transformed to align their dimensions. These steps map features from different modalities to a comparable space, facilitating subsequent correlation calculations.
[0069] The cross-modal attention mechanism calculates attention weights as follows: Dot-product attention is used to measure the correlation between features to obtain a relevance score. The relevance score is then normalized using a softmax function to generate probabilistic attention weights, ensuring the total weight sum is 1. This step transforms the raw scores into interpretable contributions, reflecting the importance of each modal feature. Essentially, the weight generation process described above involves learning the correlation between modalities through a neural network.
[0070] The attention weights mentioned above are not fixed but dynamically generated based on the input data. This dynamism is reflected in the fact that the learnable weight matrix is updated via backpropagation. During training, the neural network automatically adjusts the weight parameters by minimizing a loss function (such as cross-entropy). The optimization objective is to maximize the importance of task-relevant features.
[0071] In some embodiments, step S104 further includes:
[0072] The weights are automatically adjusted based on the distribution of audio features and lip-reading features, while cross-modal and intra-modal relationships are established to dynamically capture complementary information.
[0073] In this embodiment, the weights can be automatically adjusted based on the distribution of input features, while simultaneously modeling cross-modal and intra-modal relationships to dynamically capture complementary information. For example, in noisy environments (such as audio being interfered with by background noise), the model will reduce the weights of noisy modes to improve robustness.
[0074] In some embodiments, step S104 further includes:
[0075] Based on a multimodal time synchronization strategy, the hybrid features and text tokens are combined to form a sequence block. The local context of the sequence block is extracted through a causal convolutional layer, and a learnable positional encoding is added to mark the temporal order.
[0076] In this embodiment, in addition to the aforementioned audio features and lip-reading features, a text token can also be added, meaning a text token is also incorporated. Specifically, the weights of each modality can be dynamically calculated (e.g., the lip-reading feature weight drops to 0.2 when video is occluded, the audio feature weight drops to 0.3 when there is audio noise, and the text token weight increases to 0.5 when the text confidence score is >0.9). This embodiment can concatenate the weighted audio features, lip-reading features, and text features into a 1360-dimensional vector, which is then compressed into a 1024-dimensional unified representation space through a fully connected layer. The text token refers to the text content obtained after recognizing the speech signal using automatic speech recognition technology.
[0077] Finally, each sequence block serves as the input unit of the large language model, extracting local context through causal convolutional layers and adding learnable positional encodings to mark the temporal order.
[0078] The position encoding of this invention can employ frame-level encoding, window-level encoding, and modality-type encoding. Specifically, frame-level encoding marks the relative position of a video frame within an 8-frame window (0-7), window-level encoding marks the global position of a 300ms window in a long sequence (0-N), and modality-type encoding distinguishes the sources of audio features, lip-reading features, and text features (e.g., 0x0001 for audio, 0x0002 for lip-reading, and 0x0003 for text).
[0079] In step S105, the visitor's intent is understood more accurately by combining the visual context provided by the lip reading information, and a corresponding response is generated.
[0080] In some embodiments, step S105 includes:
[0081] The sequence blocks are input into a pre-trained large language model in the form of a token stream, and the visibility range across modalities is constrained in the self-attention layer of the large language model through a masking mechanism.
[0082] In this embodiment, the sequence block is input to the LLM (Large Language Model) in the form of a token stream, and the visibility range of cross-modal tokens is constrained in the self-attention layer through a masking mechanism.
[0083] Finally, in step S106, the semantic response generated by the large language model is converted into speech synthesis and played. Alternatively, a visual interface can be used to display the text, or both methods can be used simultaneously to achieve a closed loop of human-computer interaction.
[0084] In one specific embodiment, visitors register using a self-service visitor kiosk in a noisy environment. The kiosk's camera captures a video stream of the visitor's lips, and its microphone picks up the visitor's speech. The system first detects and tracks the visitor's lip region, extracts the shape and motion features of the lips, and uses a convolutional neural network to extract spatiotemporal lip-reading features. Simultaneously, the system performs noise reduction on the visitor's speech signal and extracts MFCC features. Then, the extracted lip-reading features and speech features are fused using a multimodal Transformer encoder to generate a joint representation containing audiovisual information. Finally, this joint representation is input into a pre-trained large-scale Chinese language model, which has been fine-tuned using audiovisual data containing lip-reading information. Even in noisy environments, thanks to the assistance of lip-reading information, the large-scale model can accurately recognize the visitor's speech commands, such as "I want to visit [company name]," and process them accordingly.
[0085] This invention, in special scenarios such as noisy environments, can significantly guide the model to focus more on the visitor's speech content by introducing lip-reading information. This provides additional visual cues for speech recognition, reduces interference from irrelevant information such as environmental noise, makes the interaction more accurate, reduces recognition errors, and improves the smoothness of the interaction, thus significantly enhancing the visitor experience and improving the efficiency and satisfaction of the visitor management system. This method improves the adaptability of the visitor management system in complex environments, enabling it to provide reliable interactive services under various noise conditions. By effectively fusing and encoding lip-reading and vocal features, the model can simultaneously utilize auditory and visual information for understanding and reasoning, avoiding the semantic information fragmentation caused by simple independent recognition.
[0086] like Figure 2 As shown, this embodiment of the invention also provides a voice interaction device 200 based on lip-reading enhancement, which includes:
[0087] Acquisition unit 201 is used to acquire facial video streams and voice signals of interactive objects in real time and synchronously.
[0088] The image processing unit 202 is used to extract the face region from the facial video stream using a deep learning-based face detection model, locate the lip region of the face region using a lip key point detection algorithm, and track the lip region in real time using a deep learning-based target tracking algorithm to obtain an image sequence of the lip region.
[0089] The feature extraction unit 203 is used to extract lip-reading features based on the image sequence of the lip region, preprocess the acquired speech signal and extract features to obtain audio features;
[0090] The fusion unit 204 is used to perform cross-modal fusion encoding on the extracted lip-reading features and audio features to generate hybrid features containing audiovisual information;
[0091] The generation unit 205 is used to input the hybrid encoded features into a pre-trained large language model to understand the intention of the interactive object and generate the corresponding semantic response;
[0092] The playback display unit 206 is used to synthesize the semantic response into speech for playback, and / or to convert the semantic response into text for visual display.
[0093] This invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of a lip-reading-enhanced speech interaction method.
[0094] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of a lip-reading-enhanced speech interaction method.
[0095] In summary, it is readily understood by those skilled in the art that, without conflict, the aforementioned advantageous technical features can be freely combined and superimposed.
[0096] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any way. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention shall still fall within the scope of the technical solution of the present invention.
Claims
1. A method for speech interaction based on lip enhancement, characterized in that, include: Real-time synchronous acquisition of facial video streams and voice signals of interactive objects; A deep learning-based face detection model is used to extract face regions from facial video streams, and a lip key point detection algorithm is used to locate the lip region of the face region. A deep learning-based target tracking algorithm is used to track the lip region in real time to obtain an image sequence of the lip region. Lip reading features are extracted from image sequences of the lip region, and audio features are obtained by preprocessing the acquired speech signals and extracting features. The extracted lip-reading features and audio features are fused and encoded across modally to generate hybrid features containing audiovisual information; The hybrid encoded features are input into a pre-trained large language model to understand the intent of the interacting object and generate corresponding semantic responses; The semantic response is synthesized into speech for playback, and / or the semantic response is converted into text for visual display; The step of performing cross-modal fusion encoding on the extracted lip-reading features and audio features to generate hybrid features containing audiovisual information includes: Linear interpolation is performed on the audio features to synchronize them with the lip-reading features; the audio features and lip-reading features are then standardized to eliminate dimensional differences. The audio features and lip-reading features are projected into the same latent space through independent learnable weight matrices, and the audio features and lip-reading features are transformed to align their dimensions. Dot product attention is used to measure the correlation between audio features and lip reading features to obtain a correlation score. The correlation score is then normalized using the Softmax function to generate probabilistic attention weights. The audio features and lip-reading features are weighted and concatenated using the attention weights to obtain a hybrid feature containing audiovisual information; The weights are automatically adjusted based on the distribution of audio features and lip-reading features, while cross-modal and intra-modal relationships are established to dynamically capture complementary information. Based on a multimodal time synchronization strategy, the hybrid features and text tokens are combined to form a sequence block. The local context of the sequence block is extracted through a causal convolutional layer, and a learnable positional encoding is added to mark the temporal order. 2.The lip-reading enhanced speech interaction method of claim 1, wherein, The process involves extracting face regions from a facial video stream using a deep learning-based face detection model, locating the lip region of the face using a lip keypoint detection algorithm, and then using a deep learning-based target tracking algorithm to track the lip region in real time, resulting in an image sequence of the lip region. This includes: The YOLOv8-face algorithm is used to detect the facial video stream frame by frame, and the face region is cropped according to the coordinates of the detection box and scaled to a uniform size; The ArcFace model is used to extract the facial feature vector of the face region as the corresponding identity feature, and the same person in consecutive frames of the facial video stream is tracked based on the identity feature; during the tracking process, the motion trajectory is predicted by Kalman filtering and LSTM temporal modeling is used to enhance the tracking robustness. The lip region in the face region is located using a lip keypoint detection algorithm, and the lip region is cropped and scaled to a standard size to obtain an image sequence of the lip region of the same person. 3.The lip-reading enhanced speech interaction method of claim 1, wherein, The extraction of lip-reading features based on image sequences of the lip region includes: 3D-ResNet is used to extract short-term spatiotemporal features of the image sequence of the lip region, and multi-level convolutional kernels are used to capture local patterns of micro-movements between frames; The short-term spatiotemporal features are converted into lip-reading features using a visual Transformer encoder; wherein, a compression-excitation module is introduced at the front end of the visual Transformer encoder. 4.The lip-reading enhanced speech interaction method of claim 1, wherein, The process of inputting the hybrid encoded features into a pre-trained large language model to understand the intent of the interacting object and generate a corresponding semantic response includes: The sequence blocks are input into a pre-trained large language model in the form of a token stream, and the visibility range across modalities is constrained in the self-attention layer of the large language model through a masking mechanism.
5. A speech interaction device based on lip-reading enhanced, characterized by, include: The acquisition unit is used to synchronously acquire facial video streams and voice signals of interactive objects in real time. The image processing unit is used to extract the face region from the facial video stream using a deep learning-based face detection model, locate the lip region of the face region using a lip key point detection algorithm, and track the lip region in real time using a deep learning-based target tracking algorithm to obtain an image sequence of the lip region. The feature extraction unit is used to extract lip-reading features based on the image sequence of the lip region, preprocess the acquired speech signal and extract features to obtain audio features; The fusion unit is used to perform cross-modal fusion encoding of the extracted lip-reading features and audio features to generate hybrid features containing audiovisual information; The generation unit is used to input the hybrid encoded features into a pre-trained large language model to understand the intent of the interacting object and generate corresponding semantic responses; A playback display unit is used to synthesize the semantic response into speech for playback, and / or to convert the semantic response into text for visual display; The step of performing cross-modal fusion encoding on the extracted lip-reading features and audio features to generate hybrid features containing audiovisual information includes: Linear interpolation is performed on the audio features to synchronize them with the lip-reading features; the audio features and lip-reading features are then standardized to eliminate dimensional differences. The audio features and lip-reading features are projected into the same latent space through independent learnable weight matrices, and the audio features and lip-reading features are transformed to align their dimensions. Dot product attention is used to measure the correlation between audio features and lip reading features to obtain a correlation score. The correlation score is then normalized using the Softmax function to generate probabilistic attention weights. The audio features and lip-reading features are weighted and concatenated using the attention weights to obtain a hybrid feature containing audiovisual information; The weights are automatically adjusted based on the distribution of audio features and lip-reading features, while cross-modal and intra-modal relationships are established to dynamically capture complementary information. Based on a multimodal time synchronization strategy, the hybrid features and text tokens are combined to form a sequence block. The local context of the sequence block is extracted through a causal convolutional layer, and a learnable positional encoding is added to mark the temporal order.
6. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the lip-reading-enhanced voice interaction method as described in any one of claims 1 to 4.
7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the lip-reading-enhanced voice interaction method as described in any one of claims 1 to 4.