Method and apparatus for visual speech recognition

Abstract

Embodiments of the present specification provide a method and device for visual speech recognition, which can reconstruct the visual feature vector in the video to be recognized during the visual speech recognition process using a large language model, to obtain a reconstructed visual feature vector. The reconstructed visual feature vector can be determined based on the embedding vector of the vocabulary token as the basis vector. Specifically, the visual feature vector in the video to be recognized can be converted into a linear combination form of the basis vector as the reconstructed visual feature vector. The reconstructed visual feature vector is provided as a prompt information to the large language model, and the large language model outputs the speech information in the video. Since the visual feature vector is represented by the embedding vector of the vocabulary token when constructing the prompt information of the large language model, the visual speech feature and the text feature can be effectively aligned, and the accuracy of the recognition result is greatly improved.

Description

Technical Field

[0001] This specification relates to the field of artificial intelligence technology, and more particularly to methods and apparatus for visual speech recognition. Background Technology

[0002] Visual Speech Recognition (VSR) is a technology that identifies speech content based solely on visual information from the speaker's lips and face (rather than audio). It is also known as "lip reading" or "lip reading." This technology has significant applications in noisy environments, assisting the hearing impaired, and security monitoring.

[0003] Visual speech recognition faces the following core technical challenges: visual signals are inherently ambiguous, exhibiting numerous homophenes, meaning different speech content may correspond to extremely similar lip movements. This makes accurate recognition based solely on visual information extremely difficult. In recent years, Large Language Models (LLMs) have demonstrated powerful language understanding and generation capabilities, providing a new technical approach for visual speech recognition—by mapping visual feature vectors to the semantic space of an LLM, and leveraging the contextual reasoning capabilities of LLMs to disambiguate and reconstruct sentences.

[0004] However, a significant distribution shift exists between the visual feature vector space and the LLM text embedding space, potentially leading to problems such as unstable training, loss of semantic information, or low computational efficiency in existing technologies. Therefore, overcoming this technical bottleneck in visual speech recognition is an urgent technical challenge. Summary of the Invention

[0005] This specification describes one or more embodiments of a method and apparatus for visual speech recognition, which addresses one or more problems mentioned in the background art.

[0006] According to a first aspect, a method for visual speech recognition is provided, the method comprising: extracting a first visual feature vector of a first video frame in a video to be recognized through a visual feature extraction network; selecting a predetermined number of first target basis vectors for the first video frame from a number of candidate basis vectors based on the similarity with the first visual feature vector, wherein each candidate basis vector is an embedding vector of a word in a large language model vocabulary; converting the first visual feature vector into a linear combination of the first target basis vectors to obtain a first reconstructed visual feature vector; and recognizing speech information in the video to be recognized using a large language model based on the first reconstructed visual feature vector.

[0007] In one embodiment, the visual feature extraction network includes a pre-trained encoding network and a linear transformation network. Extracting the first visual feature vector of the first video frame in the video to be identified via the visual feature extraction network includes: using the encoding network to extract deep features containing phoneme information frame by frame from the video to be identified; converting the deep features of each frame into alignment vectors with the same dimension as the word embedding vectors of the large language model using the linear transformation network; and determining the first visual feature vector corresponding to the first video frame based on each alignment vector.

[0008] In a further embodiment, the encoding network is an AV-HuBERT model, and the deep features are the output results of a preset number of Transformer output layers in the AV-HuBERT model.

[0009] In one embodiment, the step of extracting the first visual feature vector of the first video frame in the video to be identified through a visual feature extraction network further includes: mapping the visual feature vectors of each frame to the nearest visual speech coding vector in the visual speech codebook to obtain a discretized coding identifier sequence; wherein, the visual speech codebook contains multiple visual speech coding vectors corresponding to multiple cluster centers determined in advance by the visual feature vectors of multiple video frames, each visual speech coding vector being a representation vector of a single cluster center, determined by each visual feature vector in the corresponding cluster, and each cluster center uniquely corresponding to each coding identifier; counting the number of consecutive frames with the same coding identifier according to the coding identifier sequence; merging the visual feature vectors of multiple video frames corresponding to consecutive identical coding identifiers as the deduplicated visual feature vectors of the corresponding multiple video frames, and updating the first visual feature vector of the first video frame according to the deduplication result.

[0010] In one embodiment, merging the visual feature vectors of multiple video frames corresponding to consecutive identical encoding identifiers to obtain the deduplicated visual feature vectors of the corresponding multiple video frames includes: performing an average pooling operation on the visual feature vectors of the multiple video frames corresponding to consecutive identical encoding identifiers to obtain the deduplicated visual feature vectors of the multiple video frames.

[0011] In one embodiment, selecting a predetermined number of first target basis vectors for the first video frame from each candidate basis vector based on the similarity with the first visual feature vector includes: calculating the similarity between the first visual feature vector and each candidate basis vector respectively; sorting the candidate basis vectors in descending order of the similarity and selecting the top K' candidate basis vectors with the highest scores as each first target basis vector, where K' is a predetermined positive integer.

[0012] In one embodiment, the step of converting the first visual feature vector into a linear combination of each first target basis vector to obtain a first reconstructed visual feature vector includes: determining the weights of each first target basis vector according to a predetermined method; using the weights as linear coefficients for the linear combination of each target basis vector to obtain the first reconstructed visual feature vector represented by the linear combination of each first target basis vector.

[0013] The predetermined method is as follows: the first visual feature vector is used as the query vector, each first target basis vector is used as the key vector, and the contribution of the first visual feature vector to each first target basis vector is calculated through the attention mechanism, which is used as the corresponding weight; or, based on the Softmax function, the first visual feature vector is projected onto each first target basis vector to obtain the normalized projection coefficients corresponding to each first target basis vector as the weights.

[0014] In one embodiment, the step of recognizing speech information in a video based on the first reconstructed visual feature vector using a large language model includes: inputting the reconstructed visual feature vectors of each video frame, including the first reconstructed visual feature vector, as cue information into the large language model, and having the large language model predict the next word in an autoregressive manner to obtain the recognized speech information.

[0015] In one embodiment, the method further includes a training step of contrastive learning on the visual feature extraction network, comprising: acquiring each training sample of the current batch, wherein a single training sample includes a single sample video and a corresponding label text; determining a contrastive learning loss based on each sample video, wherein the contrastive learning loss is the sum of the single-frame losses of each video frame of each sample video, the single-frame loss corresponding to a single video frame t of the i-th sample video in the current batch, and is determined based on the similarity between the visual feature vector of video frame t and the embedding vectors of each word in the label text corresponding to the i-th sample video; and adjusting the undetermined parameters in the visual feature extraction network with the goal of minimizing the contrastive learning loss.

[0016] In a further embodiment, adjusting the undetermined parameters in the visual feature extraction network with the goal of minimizing the contrastive learning loss further includes: determining the generation loss of the large language model for the autoregressive generation task of the current batch of training samples; determining the current model loss based on the contrastive learning loss and the generation loss; and adjusting at least the undetermined parameters in the visual feature vector extraction network with the goal of minimizing the model loss.

[0017] In a further embodiment, the step of adjusting the undetermined parameters in the visual feature extraction network with the goal of minimizing the model loss further includes: adjusting the undetermined parameters corresponding to the large language model with the goal of minimizing the model loss, wherein the undetermined parameters corresponding to the large language model are the parameters in the parallel low-rank matrix of the large language model.

[0018] According to a second aspect, a visual speech recognition device is provided, comprising:

[0019] The extraction unit is configured to extract the first visual feature vector of the first video frame in the video to be identified through a visual feature extraction network.

[0020] The selection unit is configured to select a predetermined number of first target basis vectors for the first video frame from each candidate basis vector based on the similarity with the first visual feature vector, wherein each candidate basis vector is the embedding vector of each word in the vocabulary of the large language model.

[0021] The reconstruction unit is configured to convert the first visual feature vector into a linear combination of each first target basis vector to obtain the first reconstructed visual feature vector;

[0022] The recognition unit is configured to recognize speech information in the video to be recognized based on the first reconstructed visual feature vector and using a large language model.

[0023] According to a third aspect, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method described in the first aspect.

[0024] According to a fourth aspect, a computing device is provided, including a memory and a processor, characterized in that the memory stores executable code, and when the processor executes the executable code, it implements the method described in the first aspect.

[0025] The methods and apparatus provided in the embodiments of this specification, during the visual-speech recognition process using a large language model, reconstruct visual feature vectors from the video to be recognized, obtaining reconstructed visual feature vectors. The reconstructed visual feature vectors can be determined based on the embedding vectors of vocabulary elements as basis vectors. Specifically, the visual feature vectors in the video to be recognized can be converted into a linear combination of basis vectors to serve as the reconstructed visual feature vectors. These reconstructed visual feature vectors are then provided as prompts to the large language model, which outputs the speech information from the video. Because the visual feature vectors are represented using embedding vectors of vocabulary elements when constructing the prompts for the large language model, visual-speech features can be effectively aligned with text features, thereby significantly improving the accuracy of the recognition results.

[0026] In addition, using LLM vocabulary vectors as basis vectors to approximate visual features can also solve the problem of validation set loss divergence caused by distribution drift between the visual feature space and the LLM text embedding space, thereby improving the training efficiency during model training. Attached Figure Description

[0027] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 A schematic diagram illustrating the principle of text generation using a large language model is shown.

[0029] Figure 2 This document illustrates a specific application scenario.

[0030] Figure 3 This diagram illustrates the model architecture based on the technical concept described in this specification.

[0031] Figure 4 A schematic diagram of a visual speech recognition process according to one embodiment is shown;

[0032] Figure 5 This example demonstrates video compression.

[0033] Figure 6 A schematic block diagram of a visual speech recognition device according to one embodiment is shown. Detailed Implementation

[0034] First, we will explain some of the terms or concepts that may be used in this article.

[0035] Large Language Models (LLMs), also known simply as large models, are natural language processing models based on deep learning techniques. Their parameter count typically ranges from billions to hundreds of billions or even higher, possessing powerful language understanding and generation capabilities. LLMs can employ the Transformer architecture or its variants (such as GPT and BERT), which utilizes an attention mechanism to globally model sequential data, efficiently handling long-distance dependencies and thus performing exceptionally well in natural language tasks. By pre-training on large-scale corpora, LLMs learn the statistical features and semantic relationships of language, enabling them to generalize effectively. The core capabilities of LLMs include, but are not limited to: understanding contextual semantics, generating coherent and grammatically correct text, performing logical reasoning, and handling multi-task scenarios. Their usage typically includes two modes: direct inference and fine-tuning. In direct inference mode, users design prompts to guide the LLM in generating specific outputs. These prompts can be textual descriptions of the task or instructions used to stimulate the LLM's semantic understanding and generation capabilities. In fine-tuning mode, large language models are further trained on small-scale datasets within a specific domain to optimize their performance on specific tasks. The powerful generalization capabilities and flexibility of large language models make them an important tool in the field of artificial intelligence, providing efficient and accurate solutions for automated text generation and understanding.

[0036] The generation principle of large language models is as follows Figure 1 As shown. Reference Figure 1 A large language model can contain at least multiple decoding layers (and may include encoding layers before the decoding layers). For the input prompt, `Prompt`, after discretization encoding (tokenizer), it can be divided into `m` "prompt tokens". Each "prompt token" represents the discretized prompt information (such as words) within the prompt word. Here, the prompt information can be at least one of text, image, or audio. The token corresponding to the prompt word is translated as "token" in this specification; it can be understood as a feature or a unit of prompt information, such as multiple data points in words, characters, audio, or sub-images segmented from an image, etc. After embedding, each prompt token corresponds to a corresponding embedding vector, which is further processed by the large language model. Each decoding layer can contain encoded representations obtained by encoding (or decoding) the `m` prompt tokens, as well as hidden representations for predicting the next element (a single element, such as a single word). Figure 3As shown, in a single decoding layer a (a=1, 2...N), the corresponding encoding vectors h of m tokens can be obtained. a 1. h a 2……h a m These encoded vectors reflect the ability of large language models to abstractly represent cue information; in other words, a single encoded information carries the content of the corresponding cue token. The hidden representation for predicting the next element is h. a t (a = 1, 2, ..., N). After encoding in the last decoding layer N, the prediction layer maps the hidden representation of the next element to a probability distribution p of each word (a word list character, which is a token used as a candidate element). The word probability distribution p selects the words with the highest probabilities as the predicted next word. For example, the single word with the highest probability can be selected as the predicted next word. The predicted next word can be added to the predicted word sequence, which, along with the original prompt information, can be used as input to the large language model to predict subsequent words. This process is iterated over multiple prediction cycles until the end-of-line character is predicted, and the final word sequence is the generated text.

[0037] Cross-modal refers to the processing and generation of data that falls into multiple modalities, such as visual modal data, audio modal data, and text modal data. For example, images and videos are visual modal data, audio is audio modal data, and text is text modal data. Visual speech recognition is a cross-modal recognition process that processes visual modal data (video) to obtain speech recognition results from text modal data or audio modal data.

[0038] Cross-modal alignment: Aligning data from different modalities in the feature space. For example, aligning the visual feature vectors output by the visual encoder with the semantic space of text, which Large Language Models (LLMs) excel at processing.

[0039] Distribution shift refers to the significant difference in distribution between the visual feature vector space and the LLM text embedding space. This difference can lead to training instability issues such as attention mechanism failure and gradient explosion when unaligned visual feature vectors are directly input into the LLM.

[0040] The technical solutions provided in this specification are described below with reference to the accompanying drawings.

[0041] Figure 2 An example application scenario of an embodiment of this specification is shown. For example... Figure 2As shown, this application scenario involves a system primarily comprised of several user terminals and a server. The user terminals can install and run various applications, such as intelligent agent payment applications, payment platform applications, social platform applications, and so on. The server can provide payment service support for the various applications running on the user terminals (such as intelligent agent payment applications).

[0042] Multiple users interact with the system through their respective smart devices (such as smartphones). Figure 2 In the example, the application running on the user terminal may include an application capable of performing visual speech recognition. The user terminal, as the interaction carrier, runs corresponding applications or web pages. On the visual speech recognition service page, the user can perform lip-reading operations. For example, the user inputs "Please help me recognize the speech in the video below" and provides a video clip. The server receives the video data uploaded by the user and performs speech information recognition through a computing platform. Here, the computing platform can be located on the server or on other devices connected to the server in various ways. The computing platform processes the input video through a feature extraction module, extracting key visual feature vectors from the video frame sequence. These features typically include visual information related to pronunciation actions, such as lip shape, movement trajectory, and texture changes. Then, an LLM (Large Language Model / Visual Language Model) is used to process the extracted visual feature vectors and output the speech information recognition result. The LLM can be seen as the core decoder or inference engine; based on the input visual feature vector sequence, it uses its powerful sequence modeling and semantic understanding capabilities to infer the speech information corresponding to the visual content in the video. The speech information can be in speech or text form. The server will send the recognition results back to the user's terminal and display them in text or voice format as "The voice information recognized for you is: *******".

[0043] It is worth noting that, Figure 2 This is merely a specific scenario example. In practice, the number of user terminals can be any reasonable number. The LLM in the computing platform can be a large multimodal model finely tuned with data and speech data, or it can be a general vision-language model. The specific implementation method of the feature extraction module is not limited, and it can be 3D-CNN, Transformer, etc. Furthermore, this architecture is not only applicable to the mobile terminal shown in the illustration, but can also be applied to any electronic device with video acquisition / storage and display functions, such as tablet computers, smart monitoring devices, and in-vehicle systems.

[0044] In conventional techniques, in order to align visual feature vectors with text features, the following methods are used: (1) A two-stage mapping scheme based on phonemes (the smallest unit of speech divided according to the pronunciation action), which first maps lip video to phonemes and then maps them to the LLM vocabulary; In this two-stage mediation scheme of vision-phoneme-text, phonemes can be predefined and have a predetermined number (e.g., 260). The alignment granularity mismatch may be due to the lack of explicit phoneme units in the LLM vocabulary, which forces the introduction of a phoneme layer to form a double lossy compression, resulting in the loss of subtle semantic information in the intermediate layer and reducing recognition accuracy; (2) A cross-modal direct connection scheme based on linear projection / hard mapping, which directly maps visual continuous features to the semantic space of the LLM. Its deterministic dimensionality reduction processing leads to the premature collapse and loss of fine-grained semantic information when facing homophenes; and because the visual feature vector space and the LLM There is a huge distribution drift between text embedding spaces, and there is a lack of explicit alignment constraints. It is impossible to overcome cross-modal distribution drift, resulting in divergence of training loss and insufficient generalization ability of the model; (3) Based on fixed compression rate video frame downsampling schemes, such as Llama-AVSR, considering that lip reading video data has high temporal redundancy (25fps), a brute-force downsampling strategy with a fixed compression rate (such as K=3) is adopted. One frame is taken every K frames to reduce the length of the input sequence. It is impossible to perceive the actual boundary of the pronunciation action in the video. Important boundary information may be lost at key pronunciation transition points. At the same time, too much redundant information is retained in the static frame area, which leads to LLM attention being scattered and difficulty in effectively reconstructing sentences; etc.

[0045] In view of this, this specification provides a visual speech recognition scheme that aligns visual feature vectors with the basis vectors of words in a large language model vocabulary, thereby effectively solving the problem of aligning visual feature vectors with text features.

[0046] To facilitate understanding of the technical solutions in this specification, through... Figure 3 This illustrates a specific model architecture based on the technical concept of this specification, and describes the technical concept of this specification. For example... Figure 3 As shown, the model architecture includes: a visual feature extraction network (including a visual encoding module and a feature alignment module) and a Large Language Model (LLM). The feature alignment module is implemented through a linear transformation network, such as a multilayer perceptron, and can be an optional module. The input video is a lip movement video. For the lip movement video to be recognized, high-dimensional visual feature vectors can be extracted through visual encoding. The linear transformation network can then align the visual feature vectors with the word embedding vectors of the LLM, for example, transforming them both to 64 dimensions. If the visual encoding result has already undergone dimensionality alignment, this module can be omitted.

[0047] The innovations presented in this specification include aligning visual feature vectors to the feature space of text features using embedding vectors of LLM terminators. Specifically, the text feature space is described by the embedding vectors corresponding to each LLM terminator. These embedding vectors serve as basis vectors, and the visual feature vectors are encoded using these basis vectors. Each embedding vector is then used as a candidate vector to represent the visual feature vector, denoted as a candidate basis vector. Subsequently, the visual feature vectors described by the basis vectors are processed using LLM to generate speech recognition results from the video.

[0048] In further embodiments, clustering can be introduced to merge redundant video information, and contrastive learning can be used to solve fine-grained semantic alignment problems, thereby providing a more effective and accurate visual speech recognition solution.

[0049] Figure 4 A flow diagram of visual speech recognition according to one embodiment is shown. The entity executing this flow can be any computer, device, or server with a certain computing power. More specifically, as... Figure 2 The computing platform in the middle.

[0050] like Figure 2 As shown, the visual speech recognition process may include the following steps: Step 401, extracting the first visual feature vector of the first video frame in the video to be recognized through a visual feature extraction network; Step 402, selecting a predetermined number of first target basis vectors for the first video frame from each candidate basis vector based on the similarity with the first visual feature vector, where each candidate basis vector is the embedding vector of each word in the vocabulary of the large language model; Step 403, converting the first visual feature vector into a linear combination of each first target basis vector to obtain the first reconstructed visual feature vector; Step 404, recognizing the speech information in the video to be recognized using the large language model based on the first reconstructed visual feature vector.

[0051] The following details each step.

[0052] First, in step 401, the first visual feature vector of the first video frame in the video to be identified is extracted by a visual feature extraction network.

[0053] Here, the video to be recognized can be a multimedia data stream containing a sequence of lip movements of the speaking target (usually a person). It can serve as the initial input source for lip reading (i.e., visual speech recognition) tasks, aiming to carry and convey speech content information implicit in the visual modality. The video to be recognized can be any video in which speech information needs to be recognized; it can contain images of the entire face or more parts, or it can contain only lip images.

[0054] Visual feature vector extraction aims to transform pixel data from a video into a machine-understandable semantic representation. Feature extraction extracts various visual morphological features from images in the video, including features highly correlated with the morphology of vocal organs, motion trajectories, and temporal dynamics. The visual feature extraction network starts with image values ​​(such as R, G, and B color values) from the video to extract high-dimensional feature representations, resulting in the final visual feature vector. Optionally, irrelevant interference factors such as illumination variations, background noise, and individual appearance differences can be removed during the visual feature vector extraction process.

[0055] Image feature extraction networks (or encoding networks) for videos are mainly categorized into 2D CNNs (2D Convolutional Neural Networks), 3D CNNs (3D Convolutional Neural Networks), and Vision Transformers (a data extraction network based on the Transformer architecture, also known as ViT). Examples of 2D CNNs include ResNet, VGG, EfficientNet, and MobileNet. They can extract spatial features frame by frame and typically require the use of RNNs (LSTM / GRU) or TCNs to process temporal information. They are suitable for scenarios requiring high detail in individual frames, such as lip reading. 3D CNNs are spatiotemporal joint feature extraction networks, with representative networks including C3D, I3D (Inception-3D), R(2+1)D, and SlowFast. The convolutional kernels of 3D CNNs operate simultaneously in both temporal and spatial dimensions, directly capturing motion patterns and spatiotemporal dynamic features in videos, making them a mainstream solution for action recognition. ViT is a long-range dependency modeling paradigm, with representative networks including TimeSformer, VideoMAE, ViViT, MViT (Multi-scale ViT), AV-HuBERT, and others. ViT utilizes self-attention to capture global spatiotemporal dependencies, exhibiting excellent performance in handling long video sequences and understanding complex actions, and is often used as the foundational architecture for current state-of-the-art (SOTA) models. Currently, lightweight, dedicated networks such as TSM (Temporal Shift Module) and X3D have also emerged, significantly reducing computational cost while maintaining performance, making them suitable for mobile devices or real-time video processing scenarios.

[0056] Visual feature extraction networks include at least an encoding network and may also include a linear transformation network. The encoding network extracts deep features containing phoneme information frame by frame from the video to be recognized. The linear transformation network transforms the deep features of each frame into aligned vectors consistent with the dimensionality of the word embedding vectors in a large language model (e.g., 64 dimensions). It can be understood that the linear transformation network involves not only dimensionality alignment but also feature space alignment, such as numerical mapping, mapping a single dimension's value from 0-255 to 0-1, etc.

[0057] As a concrete example, the process of extracting visual feature vectors through an encoding network includes: First, a preprocessing module is used to traverse the video to be recognized frame by frame, and the Region of Interest (ROI) of the lips in each frame is located and cropped to focus on the core pronunciation region; then, the cropped lip image sequence is input into a pre-trained deep convolutional neural network (such as ResNet, 3D-CNN) or a visual transformer (ViT) encoder; then, through the hierarchical abstraction capability of the network, the pixel matrix in the spatial domain is encoded and mapped into a low-dimensional, dense continuous feature vector or feature map sequence.

[0058] For example, when the original RGB frame sequence of the video to be identified is input into the AV-HuBERT model, the shallow network mainly captures low-level visual information such as edges and textures, while the deep network (such as layer 12), through multiple layers of self-attention mechanisms, maps continuous pixel changes into high-order phoneme-level semantic representations. The feature vectors abstracted by these multiple attention mechanisms can naturally group lip movements of the same phoneme (such as / a / , / b / , / m / , etc.) together in the embedding space, even if the speakers are different or the lighting is slightly different; the feature distance is significantly smaller than the distance between different phonemes. Therefore, the output of layer 12 can be used as a deep feature containing rich phoneme information. .

[0059] The final output deep features can describe the spatiotemporal evolution of lip movements, and can include feature representations corresponding to each frame sequentially. For example, if the extracted result is a continuous feature vector (e.g., 2048 dimensions), the dimension corresponding to a single frame can be obtained according to a predetermined dimension (e.g., 32 dimensions). If the extracted result is a sequence of feature maps (e.g., 64 feature maps), the number of feature maps corresponding to each frame can be determined according to a predetermined number of channels (e.g., 4). It is worth noting that the feature maps can be flattened into 1-dimensional tensors, i.e., vectors.

[0060] Deep features extracted through the encoding network can be used to determine the visual feature vectors of each frame, for example, directly as the visual feature vectors of each frame. Optionally, the deep features extracted through the encoding network can also be aligned in the feature space with the word embedding vectors received by the large language model through a linear transformation, and the resulting aligned vector is denoted as the alignment vector. Figure 3 As shown, the function of a linear transformation network is to map the visual feature vectors of each frame to a predetermined dimension (such as 64 dimensions) through linear transformation, and the values ​​are also aligned with the feature space of the word embedding vector. A linear transformation network is, for example, a fully connected network.

[0061] The first video frame can be any frame in the video to be identified, and the alignment vector corresponding to the first video frame can be denoted as the first alignment vector. In some embodiments, the alignment vector can be directly used as a visual feature vector for subsequent processing. For example, the first alignment vector can be the first visual feature vector.

[0062] However, considering that the high redundancy of lip-reading video frames (25fps) may lead to excessive processing burden, frame compression can also be performed on lip-reading videos. In addition to traditional fixed compression rate schemes (such as Llama-AVSR), this specification also provides an innovative adaptive deduplication strategy based on visual-speech unit clustering as an optional deduplication compression scheme, according to a possible design.

[0063] Specifically, this scheme pre-clusters visual semantic units based on visual feature vectors from multiple visual speech recognition videos, obtaining a predetermined number K cluster centers. The representation vector of each cluster center serves as the standard visual speech coding vector corresponding to the respective cluster, thus constructing a visual speech codebook. During video frame compression, feature merging is performed on video frames based on the visual speech codebook to achieve frame compression.

[0064] First, the process of constructing the visual-speech codebook is described. A predetermined number or proportion of sample videos are randomly selected for constructing the visual-speech codebook. This is because the types of basic articulatory units (phonemes / syllables) in lip-reading are limited (human language phonemes typically range from tens to hundreds; considering coarticulation variants, a few hundred are sufficient for coverage). Full-data clustering is not only computationally expensive but also easily introduces a large number of noisy samples that contribute little to the stability of the cluster centers. The embodiments in this specification can construct the visual-speech codebook through clustering. Taking the K-means method as an example, the number of cluster centers K can be preset (e.g., 200). The sample videos used to construct the codebook are processed by the visual feature extraction network described above, such as a visual feature extraction network containing an AV-HuBERT encoding network, to extract the visual feature vectors of each frame and cluster them into K cluster centers.

[0065] The number of cluster centers, K, can be determined empirically. A value that is too small (e.g., 5) may force the merging of multiple different phonemes, causing semantic confusion (under-segmentation). A value that is too large (e.g., 1000) may classify subtle tremors of the same phoneme (e.g., slight lip tremors) into different categories, leading to deduplication failure and ineffective sequence compression. In this specification, values ​​such as 150, 180, and 200 are used. During K-means clustering, K visual feature vectors are randomly selected as cluster centers. The distance between the visual feature vectors of each frame and these K centers is calculated, and each frame's visual feature vector is assigned to the cluster closest to it. The mean of each cluster is then used to update the cluster centers. This process is iterated until clustering ends (the number of iterations reaches a predetermined period, or the distance between the changed cluster centers is less than a predetermined threshold). At this point, the resulting K cluster centers, each corresponding to a single representation vector, can serve as a benchmark for visual factors, forming a visual-speech codebook. Taking K=180 as an example, the visual speech codebook can be represented as follows: It is understandable that the codebook construction process can be completed independently before video compression (such as before model training), and is fixed as prior knowledge for preprocessing.

[0066] In video frame compression using a visual-speech codebook, the distance from the visual feature vector of each frame to each cluster center can be calculated, and a single frame of video can be mapped to its nearest cluster center (i.e., the corresponding visual-speech encoding vector in the codebook). For example, the mapping result for frame t is denoted as: The sequence of visual feature vectors from the original video. This can be transformed into a discrete coded identifier sequence. When K=180, a single .

[0067] The number of consecutive frames with the same encoded identifier is counted based on the encoded identifier sequence. For example, traversing discrete sequences. It identifies all consecutive segments with the same ID (the idea of ​​Run-Length Encoding). For example, the sequence [A, A, A, B, B, C, C, C, C] is identified as three segments (A, 3), (B, 2), and (C, 4).

[0068] For several consecutive identical frames in the mapping result, these frames can be merged, and a merged visual feature vector can be used to describe these identical frames, serving as the deduplicated visual feature vector of the corresponding multiple video frames. The merging method can be, for example, randomly selecting one frame or following a predetermined rule (such as selecting the first frame), etc. In one embodiment, the number of consecutive identical frames in the mapping result can be counted, and the frames can be merged using methods such as mean pooling. For example, in segments A, B, and C mentioned above, the visual feature vectors of all frames within a single segment can be merged... Perform average pooling to generate an enhanced representative feature. and associate it with its duration count. .

[0069] Figure 5 A specific example of video compression is shown. For example... Figure 5 As shown, in a visual speech video segment containing 6 video frames, the encoding identifiers mapped to the visual speech codebook are 7, 7, 7, 16, 9, and 9, respectively. The first three frames corresponding to 7 are represented by merging them into a visual feature vector through average pooling. The fourth frame corresponding to 16 is completely preserved. The fifth and sixth frames corresponding to 9 are represented by merging them into a visual feature vector through average pooling.

[0070] For the first video frame, the first visual feature vector of the first video frame can be updated based on the deduplication result. If the first video frame and its adjacent video frames are mapped to the same coded identifier, the visual feature vectors of the adjacent video frames with the same coded identifier can be merged and used as the first visual feature vector of the first video frame.

[0071] Experimental data shows that after compression using the above codebook construction method, the video sequence length can be compressed to 47%-53% of the original length. This means that nearly half of the redundant calculations (mainly repeated static lip-sync frames) are eliminated. In addition, since the deduplication logic is strictly based on semantic unit changes, all key pronunciation action start and end points (Onset / Offset) boundaries are completely and losslessly preserved, without motion blur caused by traditional downsampling.

[0072] Then, in step 402, a predetermined number of first target basis vectors are selected for the first video frame from each candidate basis vector based on the similarity with the first visual feature vector.

[0073] A basis vector can be understood as the basic unit vector of the feature space. Typically, in the embedding layer of a Large Language Model (LLM), each discrete token in the vocabulary uniquely corresponds to an embedding vector. Here, the embedding vectors corresponding to the tokens in the LLM vocabulary can be used as candidate basis vectors to construct the semantic vector space defined by the full vocabulary of the LLM.

[0074] Specifically, assuming the vocabulary size of the large language model is... Then it exists candidate basis vectors A single basis vector Represents the first word in the vocabulary. The baseline coordinates of each lexical unit (such as a specific character, word, subword, or special symbol) in the model's semantic space constitute the underlying dictionary foundation for the model's understanding and text generation.

[0075] To better align visual feature vectors to the semantic vector space of a large language model and establish preliminary semantic connections between visual and textual modalities, target basis vectors with high relevance can be selected from various candidate basis vectors based on their similarity to the visual feature vectors. Here, a predetermined number of candidate basis vectors can be selected as target basis vectors; in other words, the K' (a predetermined number, distinguished from the K in the previous clustering) candidate basis vectors with the highest similarity are selected, i.e., the top-K' similarity candidate basis vectors.

[0076] Specifically, the current visual feature vector obtained in the previous step can be retrieved. This vector represents the spatiotemporal semantic information of lip movements in the video clip. Next, this visual feature vector is calculated. With each candidate basis vector The vector similarity between the visual signals and the text words is used as the semantic similarity score. Similarity can be measured using methods such as cosine similarity, the inverse of Euclidean distance, or dot product operations to quantify the proximity of visual signals to each text word in the semantic space. For example, calculating visual feature vectors... With semantic basis vectors cosine similarity, Then, based on the calculated similarity scores, the top-ranked items are truncated from the sorted list in descending order. The candidate basis vector with the highest similarity is selected as the target basis vector.

[0077] It is understandable that when calculating similarity using dot product, vector similarity can be calculated using visual feature vectors. With each candidate basis vector Vector multiplication can be performed one-by-one, or tensor multiplication can be used for batch calculation; neither is specified here. Taking tensor multiplication for batch calculation as an example, let's assume the vocabulary size is... (For example, corresponding to 50,000 tokens), the vector dimension is (For example, 64-dimensional), then all basis vectors can correspond to one The matrix is ​​a similarity vector, where each row represents a basis vector. In one embodiment, this matrix is ​​multiplied by the visual feature vector of a frame to obtain a similarity vector, where the values ​​in each dimension of the vector represent the similarity between the visual feature vector of that frame and the corresponding basis vector. In another embodiment, the matrix is ​​multiplied by the visual feature vectors of each frame to obtain a similarity tensor, where each value in the tensor corresponds to the similarity between a single basis vector and the visual feature vector of a single frame.

[0078] A single video frame corresponds to The ' target basis vectors constitute a dynamic semantic candidate pool, denoted as ' The physical significance lies in the fact that these are the top-K text hypotheses in the large model vocabulary that are most likely to correspond to the meaning of the lip movements in the current frame. Through this filtering mechanism, the originally huge full vocabulary search space can be effectively compressed into a high-confidence local subspace, which not only significantly reduces the computational complexity of the subsequent decoding stage, but also suppresses the interference of noisy words that are irrelevant to the current visual input by introducing visual prior knowledge, thus laying a semantic foundation for the subsequent generation of accurate text sequences.

[0079] The first video frame can select a target basis vector based on its similarity to the first visual feature vector, and this target basis vector is denoted as the first target basis vector.

[0080] Next, in step 403, the first visual feature vector is converted into a linear combination of the first target basis vectors to obtain the first reconstructed visual feature vector.

[0081] It is understandable that each target basis vector A linear combination of target basis vectors can represent the semantic information in the semantic feature space of an LLM. Transforming visual feature vectors into linear combinations of target basis vectors is equivalent to transforming them into a weighted sum of target basis vectors. The linear coefficients of this linear combination are the weights. This step is equivalent to semantic reconstruction and noise filtering of the original visual signal through the semantic space of an LLM. The weights of each target basis vector can be determined using various reasonable methods.

[0082] In one embodiment, visual feature vectors can be projected onto various target basis vectors to obtain corresponding projection coefficients as weights. The projection method can be predefined. For example, a projection method can be determined using a Softmax function, such as:

[0083] Where T is a preset temperature coefficient. Let i be the target basis vector corresponding to frame t. This represents the set of K target basis vectors corresponding to frame t. The visual feature vector of frame t. It can be represented as: .

[0084] In one embodiment, attention coefficients are obtained as weights through an attention mechanism. Specifically, these weights are based on the visual feature vector of the t-th frame obtained from the preceding steps. The selected set of target basis vectors , to convert visual feature vectors As the query vector, it is associated with each target basis vector. Similarity is calculated using the key vector (as the key). For example, a scaled dot-product attention mechanism can be used to calculate the similarity between each target basis vector and the current visual feature vector. Contribution weight These weights This forms a probability distribution that satisfies Its physical meaning lies in quantifying the confidence or relevance of each candidate lexical unit in explaining the current lip movement. Thus: .

[0085] In other embodiments, the visual feature vectors of each frame can also be processed in other reasonable ways. The transformation is a linear combination of the corresponding target basis vectors. Since semantic reconstruction is performed during the transformation, the transformed visual feature vector is called the reconstructed visual feature vector. The first visual feature vector can be transformed into a linear combination of each first target basis vector, and the resulting reconstructed visual feature vector is denoted as the first reconstructed visual feature vector.

[0086] The significance of reconstructing visual feature vectors lies in two aspects: First, it enables visual-text semantic alignment. It is no longer merely the original pixel-level or low-level convolutional features, nor a simple phoneme mapping, but is forcibly projected into the LLM's text embedding space, giving the visual feature vectors explicit linguistic interpretability and providing stable semantic input for subsequent LLM inference. This means that the reconstructed visual feature vectors are highly consistent with the input data of subsequent LLM processing layers in terms of numerical distribution, eliminating distribution shift across modal data. Second, it facilitates denoising and enhancement. Since the weighting process is based only on high-confidence target basis vectors, lighting noise, background interference, or non-phonological facial micro-movements contained in the original visual feature vectors are effectively suppressed, while key semantic information (i.e., the most likely corresponding word meanings) is significantly enhanced.

[0087] Furthermore, in step 404, based on the first reconstructed visual feature vector, a large language model is used to identify the speech information in the video to be identified.

[0088] The reconstructed visual feature vectors of each video frame can serve as prompt information for a large language model, such as discrete encoding of the prompt (see reference). Figure 1 Each embedding vector following the Tokenizer (see reference) Figure 1 The embeddings of the video frames are input into the large language model. Optionally, when the visual feature vectors of video frames are merged, the number of consecutive frames corresponding to a single visual feature vector is also provided to the large language model as a cue for its reference. The large language model can then generate speech recognition results that are fused with the natural language context through the decoder.

[0089] Specifically, since the reconstructed visual feature vectors are aligned in the feature space to the word embedding feature space of the LLM, the system can directly use them as a sequence of "virtual visual tokens" and concatenate them after a specific start prompt, or convert them into several consecutive soft prompt vectors through a projection layer to construct an initial input sequence containing visual semantic information. At this point, the reconstructed visual feature vector is no longer regarded as external image data, but is equivalently understood by the internal mechanism of LLM as a special language context rich in lip movement semantics.

[0090] Then, the large language model is based on the input sequence LLM utilizes its pre-trained multi-head self-attention mechanism to capture the long-range dependencies between reconstructed visual feature vectors and existing textual context, performing autoregressive decoding inference. LLM predicts the next most probable token in an autoregressive manner. When predicting a single token, the probability distribution on the vocabulary is generated by linearly transforming and Softmax normalizing the hidden layer output in the current state. In this process, visual feature vectors are reconstructed. As a strong semantic prior, it can dynamically influence the probability distribution, giving higher confidence to words that highly match lip movements (such as words corresponding to pronunciation mouth shapes), while suppressing interference from homonyms or context-irrelevant words. Based on a pre-defined decoding strategy (such as Greedy Search, Beam Search, or Nucleus Sampling), optimal lexical units are selected from the probability distribution. It is added to the output sequence and used as context for the next step of reasoning.

[0091] The above decoding process iterates until the LLM outputs a specific terminator (such as an EOS token) or reaches the preset maximum generation length. The system then concatenates and detokenizes the discrete word sequences generated sequentially during the decoding process, ultimately outputting the text content of the speech information, such as "I am Chinese".

[0092] Optionally, the text output by the large language model can also be converted into audio information output using text-to-speech (TTS) tools, such as Coqui TTS, StyleTTS 2, ElevenLabs, etc.

[0093] By introducing reconstructed visual feature vectors, true end-to-end semantic reasoning can be achieved. Large language models not only utilize local features of visual signals, but also combine their massive prior knowledge and linguistic logic to effectively correct visual ambiguities caused by occlusion and blurring, significantly improving the accuracy and robustness of lip reading in complex scenarios.

[0094] Based on some possible designs, and considering the potential fine-grained semantic loss that may result when using vocabulary embedding vectors as basis vectors to describe visual feature vectors within the above visual speech recognition architecture, this specification innovatively proposes comparative learning of the visual feature extraction network. It is understandable that, referring to... Figure 3As shown, considering that most encoding networks are existing modular networks, a linear transformation network is usually needed for feature alignment. When the visual feature extraction network includes a linear transformation network, contrastive learning can be performed only on the linear transformation network, i.e., only adjusting the undetermined parameters of the linear transformation network. In an optional embodiment, the encoding network can be custom-designed, in which case contrastive learning can adjust the parameters of both the encoding network and the linear transformation network. Here, this is collectively referred to as adjusting the visual feature extraction network through contrastive learning.

[0095] Unlike the coarse-grained alignment of video-to-sentence (Video-to-Sentence) in conventional technologies, this specification proposes a finer-grained "Frame-to-Global-Token" alignment paradigm. Specifically, the visual feature vectors of a single frame of the sample video (such as...) Figure 3 The output of the linear transformation network in the sentence is aligned with the individual tag text tokens in the final target's speech information (label). In other words, the original embedding vector of each discrete token in the sentence is preserved. No dimensionality reduction or compression is performed. This maintains a high-resolution semantic space, where each token vector represents a specific character, word, or semantic unit. The core of this embodiment lies in establishing a dense mapping relationship between the temporal dimension of the video and the discrete dimension of the text. Specifically, instead of requiring a frame to align with only a specific word, it is stipulated that the visual feature vector of each frame in the video must interact and align with the embedding vectors of each token in the target sentence. This is because, although a frame mainly corresponds to a certain pronunciation moment, the contextual information of that frame (such as co-articulation) actually contains the semantic constraints of the entire sentence. By allowing each frame to align with all tokens of the entire sentence, the model can learn richer global contextual dependencies, thereby using the semantic logic of the entire sentence to correct visual ambiguities in a single frame when reconstructing visual feature vectors.

[0096] For a sample i, it can correspond to sample video X. i , and the tag text y corresponding to the voice information used as a tag i In the contrastive learning process, within a single batch of training samples, for video X... i and label text y i The audio information of any other sample video can be denoted as a positive sample, and the labeled text of the audio information of any other sample video can be denoted as a negative sample.

[0097] Video X i The visual feature vector corresponding to the t-th frame is denoted as follows: The embedding vector corresponding to a single text token in any speech information is denoted as . Then video X i The similarity between the t-th frame and the text token can be measured by methods such as cosine similarity, distance, and dot product, for example, as follows: In one embodiment, a preset temperature coefficient is used. Adjusting the smoothness of the similarity distribution yields the corresponding unnormalized log-odds ratio, Logist:

[0098]

[0099] Based on this similarity, video X can be identified. i The feature representation loss (or encoding loss, alignment loss, etc.) of frame t relative to a single token for video X i In other words, the label text y i Let P be the set of tokens in the dataset. i Let N be the set of tokens for the audio information of other videos. i The label text y can be... i Any text token in the list is denoted as This corresponds to a positive sample. Thus, for video X... i In frame t, through all corresponding The sum of these sums determines the corresponding contrastive learning loss. Typically, the contrastive learning loss for a single frame can be summed with the sum of all frames. corresponding The sum of them is negatively correlated.

[0100] In one embodiment, video X i The contrastive learning loss for frame t can be any corresponding The opposite of the sum.

[0101] In one embodiment, video X i The contrastive learning loss for frame t can be applied to all other frames. of Add and subtract all corresponding sum.

[0102] In some alternative embodiments, video X i The contrastive learning loss for frame t can be obtained from all The mean negative log-likelihood is determined, denoted as:

[0103]

[0104] in, for quantity, for The log probability of being correctly identified is, for example:

[0105] This represents the embedding vector of all text tokens in the current batch, with respect to the embedding vector and video X. i The visual feature vector of frame t The similarity is summed, where exp() represents the exponent of the natural logarithm e, and the value in parentheses is the power of the exponent.

[0106] The sum of the contrast losses of each frame can be used as the video X. i The corresponding contrastive learning loss, the sum of the contrastive losses for each video in the current batch, can be used as the contrastive learning loss for the current batch, denoted as L. NCE .

[0107] For example, the average contrastive learning loss within the current batch. It can be:

[0108]

[0109] Among them, T i Let B represent the frame number of the i-th video, and B be the number of videos in the current batch.

[0110] In other embodiments, the contrastive learning loss can also be determined by other reasonable means, which will not be elaborated here.

[0111] In an alternative implementation, model training can be completed by adjusting the undetermined parameters in the visual extraction network in the direction of minimizing the contrastive learning loss.

[0112] It is understandable that LLMs are typically pre-trained models using massive corpora, which can be used for direct inference or fine-tuned under supervision to be more suitable for vertical domains. To improve the speech recognition performance of LLMs, supervised fine-tuning of large language models can be performed beforehand. In some optional implementations, to introduce semantic constraints while ensuring the generative capabilities of LLMs, the innovative solution in this specification can further optimize the autoregressive generation task (Cross-Entropy Loss) of LLMs together with the contrastive learning task.

[0113] In the autoregressive generation task of LLM, the sequence generated by the large language model can be compared with the text sequence (splitting into words) used as labels. The single-step loss is determined based on the words generated at each time step, and the single-step losses of all time steps are fused together using methods such as summation, averaging, and weighted averaging to form the autoregressive loss, denoted as L. LLM For a length of target sequence Given the input context (As indicated by the prompt message), taking the average value as an example, the total loss of the autoregressive model for:

[0114]

[0115] in, Indicates the current time step (here used) This indicates that the time step is different from t in the t-th frame described above, and its value ranges from 1 to... ), Indicates the first The true target token of the step (Ground Truth, i.e., the first one in the label) (word element) Indicates the first in the label text The token sequence of all steps before step 1 As part of the input context of the current step, This indicates that the current model parameters are In this case, based on the context (such as including X and the first...), Predict the real token from the token sequence of all previous steps. The probability, It takes the logarithmic sign, usually using the natural logarithm. .

[0116] In practical calculations, large language models often output unnormalized log odds (Logits), denoted as... It needs to be converted into probabilities using the Softmax function. Let the vocabulary size be... , No. The Logits vector of the step is The true target word is calculated using Softmax probability. The corresponding predicted probability can be:

[0117] in It is the score of the actual token in the Logits vector.

[0118] Cross-entropy loss (or other loss functions) is commonly used in training large language models, along with single-step loss. For example:

[0119]

[0120] Using the properties of logarithms Simplified to:

[0121]

[0122] The total loss of a large language model for a single sample (if denoted as generation loss) can be expressed as:

[0123]

[0124] Therefore, when jointly optimizing the LLM's autoregressive generative task (Cross-Entropy Loss) with the contrastive learning task, we have:

[0125]

[0126] in, and These are the preset balance coefficients for the two loss functions (e.g., set to...). =1.0, =0.1, etc.). Based on the model loss, at least the undetermined parameters of the visual feature extraction module can be adjusted, and optionally, the parameters in the large language model can also be adjusted. In an optional embodiment, the parameter adjustment of the large language model can be optimized using methods such as LoRa to improve training efficiency. In the LoRa method, a low-rank matrix is ​​externally connected in parallel to the large language model, and the undetermined parameters contained in the low-rank matrix are actually adjusted during the parameter adjustment process.

[0127] During model training, the gradient of the model loss with respect to each undetermined parameter can be calculated, and the undetermined parameters in the model can be updated through backpropagation of the gradient in the model and gradient descent, etc., which will not be elaborated here.

[0128] This joint optimization design ensures that the model passes... While learning sentence grammar structure, utilize Continuously calibrate the semantic orientation of visual feature vectors.

[0129] Reviewing the above process, the visual speech recognition technical solution provided in this specification can reconstruct the visual feature vectors in the video to be recognized during the visual speech recognition process using a large language model, obtaining reconstructed visual feature vectors. The reconstructed visual feature vectors can be determined based on the embedding vectors of vocabulary units as basis vectors. Specifically, the visual feature vectors in the video to be recognized can be converted into a linear combination of basis vectors as the reconstructed visual feature vectors. The reconstructed visual feature vectors are then provided as prompts to the large language model, which outputs the speech information from the video. Because the visual feature vectors are aligned with the embedding vectors of vocabulary units during the construction of the prompts for the large language model, a collaborative alignment strategy of "first narrowing the modal distance through contrastive learning, then reconstructing the basis vectors" is employed. Using frame-level global contrastive learning, the distribution of visual features is successfully calibrated by explicitly narrowing the geometric distance between modalities. This allows the visual features to activate the correct semantic basis vectors with higher confidence during the basis reconstruction stage, forming a virtuous cycle and significantly reducing the word error rate (WER), thereby greatly improving the accuracy of the recognition results. In addition, using LLM vocabulary vectors as basis vectors to approximate visual features can also solve the problem of validation set loss divergence caused by distribution drift between the visual feature space and the LLM text embedding space, thereby improving training efficiency.

[0130] According to another embodiment, a visual speech recognition device is also provided. This device can be installed in any computer, device, or server with a certain computing power, for example... Figure 2 The computing platform, etc. Figure 6 A visual speech recognition apparatus 600 according to one embodiment of this specification is shown.

[0131] like Figure 6 As shown, the visual speech recognition device 600 may include: an extraction unit 601 configured to extract a first visual feature vector of a first video frame in a video to be recognized through a visual feature extraction network; a selection unit 602 configured to select a predetermined number of first target basis vectors for the first video frame from various candidate basis vectors based on the similarity with the first visual feature vector, wherein each candidate basis vector is an embedding vector of each word in the vocabulary of a large language model; a reconstruction unit 603 configured to convert the first visual feature vector into a linear combination of the various first target basis vectors to obtain a first reconstructed visual feature vector; and a recognition unit 604 configured to recognize speech information in the video to be recognized using a large language model based on the first reconstructed visual feature vector.

[0132] It is worth noting that, Figure 6 The device 600 shown and Figure 4 The methods described correspond to, Figure 4The corresponding descriptions in the illustrated method embodiments also apply to apparatus 600, and will not be repeated here.

[0133] According to another embodiment, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed in a computer, causes the computer to perform a combination Figure 4 The methods described above.

[0134] According to another embodiment, a computing device is also provided, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it implements a combination... Figure 4 The methods described above.

[0135] Those skilled in the art will recognize that the functions described in the embodiments of this specification in one or more of the above examples can be implemented using hardware, software, firmware, or any combination thereof. When implemented in software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium.

[0136] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the technical concept in this specification. It should be understood that the above description is only a specific embodiment of the technical concept in this specification and is not intended to limit the scope of protection of the technical concept in this specification. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solutions of the embodiments in this specification should be included within the scope of protection of the technical concept in this specification.

Claims

1. A method for visual speech recognition, the method comprising: The first visual feature vector of the first video frame in the video to be identified is extracted using a visual feature extraction network. Based on the similarity with the first visual feature vector, a predetermined number of first target basis vectors are selected for the first video frame from each candidate basis vector, and each candidate basis vector is the embedding vector of each word in the vocabulary of the large language model. The first visual feature vector is transformed into a linear combination of each first target basis vector to obtain the first reconstructed visual feature vector; Based on the first reconstructed visual feature vector, a large language model is used to identify the speech information in the video to be identified.

2. The method as described in claim 1, wherein, The visual feature extraction network includes a pre-trained encoding network and a linear transform network. The step of extracting the first visual feature vector from the first video frame of the video to be identified through the visual feature extraction network includes: Using the coding network, deep features containing phoneme information are extracted frame by frame from the video to be identified; The deep features of each frame are transformed into aligned vectors with the same dimension as the word embedding vectors of the large language model through a linear transformation network. The first visual feature vector corresponding to the first video frame is determined based on each alignment vector.

3. The method as described in claim 2, wherein, The encoding network is the AV-HuBERT model, and the deep features are the output results of the Transformer output layer with a preset number of layers in the AV-HuBERT model.

4. The method of claim 1, wherein, The step of extracting the first visual feature vector of the first video frame in the video to be identified through a visual feature extraction network further includes: The visual feature vectors of each frame are mapped to the nearest visual speech coding vector in the visual speech codebook to obtain a discrete coding identifier sequence. The visual speech codebook contains multiple visual speech coding vectors corresponding to multiple cluster centers determined in advance by the visual feature vectors of multiple video frames. Each visual speech coding vector is a representation vector of a single cluster center, determined by each visual feature vector in the corresponding cluster. Each cluster center uniquely corresponds to each coding identifier. The number of consecutive frames with the same encoded identifier is counted based on the encoded identifier sequence; The visual feature vectors of multiple video frames corresponding to consecutive identical encoded identifiers are merged to form the deduplicated visual feature vectors of the corresponding multiple video frames. The first visual feature vector of the first video frame is determined based on the deduplication result.

5. The method of claim 4, wherein, The step of merging the visual feature vectors of multiple video frames corresponding to consecutive videos with the same encoded identifier, as the deduplicated visual feature vectors of the corresponding multiple video frames, includes: By performing an average pooling operation on the visual feature vectors of multiple video frames corresponding to the consecutive identical encoded identifiers, the combined and deduplicated visual feature vectors of the multiple video frames are obtained.

6. The method of claim 1, wherein, The step of selecting a predetermined number of first target basis vectors for the first video frame from among the candidate basis vectors based on the similarity with the first visual feature vector includes: Calculate the similarity between the first visual feature vector and each candidate basis vector; Arrange the similarity scores in descending order, and select the top K' candidate basis vectors with the highest scores as the first target basis vectors, where K' is a predetermined positive integer.

7. The method of claim 1, wherein, The step of converting the first visual feature vector into a linear combination of each first target basis vector to obtain the first reconstructed visual feature vector includes: The weights of each first target basis vector are determined according to a predetermined method; The weights are used as linear coefficients for the linear combination of each target basis vector to obtain the first reconstructed visual feature vector represented by the linear combination of each first target basis vector. The reservation method is as follows: The first visual feature vector is used as the query vector, and each first target basis vector is used as the key vector. The contribution of the first visual feature vector to each first target basis vector is calculated through the attention mechanism and used as the corresponding weight. or, Based on the Softmax function, the first visual feature vector is projected onto each first target basis vector to obtain the normalized projection coefficients corresponding to each first target basis vector as weights.

8. The method of claim 1, wherein, The step of recognizing speech information in the video to be recognized based on the first reconstructed visual feature vector using a large language model includes: The reconstructed visual feature vectors of each video frame, including the first reconstructed visual feature vector, are used as prompt information and input into the large language model. The large language model then predicts the next word in an autoregressive manner to obtain the recognized speech information.

9. The method of claim 1, wherein, The method further includes a training step of contrastive learning on the visual feature extraction network, including: Obtain each training sample in the current batch. Each training sample includes a single sample video and its corresponding label text. The contrastive learning loss is determined based on each sample video, wherein the contrastive learning loss is the sum of the single-frame losses of each video frame of each sample video, the single-frame loss corresponding to a single video frame t of the i-th sample video in the current batch, and is determined based on the similarity between the visual feature vector of video frame t and the embedding vector of each word in the tag text corresponding to the i-th sample video. The parameters to be determined in the visual feature extraction network are adjusted with the goal of minimizing the contrastive learning loss.

10. The method of claim 9, wherein, The step of adjusting the undetermined parameters in the visual feature extraction network with the goal of minimizing the contrastive learning loss further includes: Determine the generation loss of the large language model for the autoregressive generation task of the current batch of training samples; Based on the contrastive learning loss and the generative loss, the current model loss is determined; With the goal of minimizing the model loss, at least the undetermined parameters in the visual feature extraction network are adjusted.

11. The method of claim 10, wherein, The step of adjusting at least the undetermined parameters in the visual feature extraction network with the objective of minimizing the model loss also includes: With the goal of minimizing the model loss, the undetermined parameters corresponding to the large language model are adjusted. The undetermined parameters corresponding to the large language model are the parameters in the parallel low-rank matrix of the large language model.

12. A visual speech recognition device, comprising: The extraction unit is configured to extract the first visual feature vector of the first video frame in the video to be identified through a visual feature extraction network. The selection unit is configured to select a predetermined number of first target basis vectors for the first video frame from each candidate basis vector based on the similarity with the first visual feature vector, wherein each candidate basis vector is the embedding vector of each word in the vocabulary of the large language model. The reconstruction unit is configured to convert the first visual feature vector into a linear combination of each first target basis vector to obtain the first reconstructed visual feature vector; The recognition unit is configured to recognize speech information in the video to be recognized based on the first reconstructed visual feature vector and using a large language model.

13. A computing device, comprising a memory and a processor, characterized in that, The memory stores executable code, and when the processor executes the executable code, it implements the method of any one of claims 1-11.

Images