Lip language recognition method fusing external information

Abstract

The invention provides a lip language recognition model construction method fusing external information, and the method comprises the steps: constructing an initial lip language recognition model which comprises a visual modal data processing module, a pre-trained visual encoder, a visual projector, a text embedding module and a first large language model; a data set is constructed, and the initial data set takes each visual sequence image and corresponding external information as samples and takes a real speaking text and a correct reasoning process description text as labels; and based on the constructed training set, taking the visual sequence image and the corresponding external information as input, and taking a predicted speaking text and an actual reasoning process description text as output, carrying out iterative training on the initial lip language recognition model until convergence so as to obtain a final lip language recognition model, in the training process, a preset cross entropy loss function is adopted to calculate loss, and parameters of the first large language model, the visual projector and the pre-trained visual encoder are updated based on the calculated loss.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision, in particular to visual speech recognition technology and natural language processing technology, and more particularly to a lip reading model construction method fusing external information. BACKGROUND

[0002] In the field of human-computer interaction, as a key technology for information transmission, the performance of lip reading is of great concern, and recognition accuracy and environmental adaptability are important indicators for measuring its performance. Lip reading refers to a technology that interprets the speech content of a speaker by watching the dynamic features of the speaker's face and lips when speaking. This technology can complement audio-based speech recognition, making up for the shortcomings of audio-based speech recognition models in high-noise environments, and can also be used independently for speech content recognition in silent environments, thereby achieving efficient transmission of speech content. At the same time, this technology has great application value in human-computer interaction systems and speech disorder assistance. In recent years, with the emergence of large-scale lip reading datasets and the rapid development and effective application of deep learning technology in the fields of computer vision and natural language processing, lip reading has gradually received widespread attention and gradually played a role in practical scenarios.

[0003] Currently, lip reading faces multiple technical challenges. At the visual level, due to the subtlety and continuity of mouth shape changes, visual features are difficult to accurately extract, and the time sequence relationship of the speaking process is complex and needs to be accurately modeled for continuous mouth shape changes. In addition, the speaking posture and expression of the speaker are diverse, and the lighting conditions of the real environment are complex and changeable, all of which bring significant challenges to feature extraction. At the recognition level, the recognition difficulty is further intensified, specifically manifested in that the lip shapes of many words have high similarity and are easily confused, short words have limited information due to mouth shape changes, making their accurate recognition particularly difficult, and in addition, the pronunciation habits and mouth shape characteristics of different speakers are significantly different, which puts high requirements on the generalization ability of the system. The mutual superposition of the above challenges makes lip reading a very difficult task.

[0004] Although some progress has been made in the field of lip-reading in recent years, the existing technology mainly focuses on optimizing the extraction and processing of visual information in the lip area. Specifically, most studies focus on the local area centered on the lips, capturing mouth shape changes through fine modeling; some studies also attempt to expand the field of view to a larger facial range to obtain more rich expression features. Although these methods have made significant progress in the technical level, if only limited to the optimization of visual features, it is difficult to completely solve the lip-reading problem in actual scenarios. By observing the recognition process of professional lip readers, it can be found that humans use a more comprehensive strategy when lip-reading. Humans not only rely on visual dynamic information, but also use knowledge in multiple dimensions, including understanding of the speaker's language habits, long-term accumulated experience and skills, and understanding of the context of the conversation. For example, knowing that the speaker is a professional lecturer and is discussing a specific topic, people can better predict and understand the lip movements, thereby generating more accurate transcribed text. The existing lip-reading system has not effectively simulated the use of external information, resulting in poor performance when processing complex or diverse contextual information. Breaking through the limitations of single visual features and using multi-modal information fusion strategies is the key to improving the accuracy of lip-reading.

[0005] In summary, the existing lip-reading scheme is limited to the optimization of visual dynamic information and fails to effectively combine external information, resulting in poor recognition performance when processing complex or diverse contextual information. Therefore, there is an urgent need for a lip-reading scheme that integrates external information.

[0006] It should be noted that the background art is only used to introduce the relevant information of the present application, so as to help understand the technical solutions of the present application, but does not mean that the relevant information must be prior art. In the absence of evidence that the relevant information has been disclosed before the filing date of the present application, the relevant information should not be considered as prior art. SUMMARY

[0007] Therefore, the purpose of the present application is to overcome the defects of the prior art and provide a lip-reading model construction method that integrates external information.

[0008] The purpose of the present application is achieved by the following technical solutions:

[0009] According to a first aspect of the present application, a method for constructing a lip reading model fusing external information is provided, which is used to construct a lip reading model for recognizing the speech text corresponding to a visual sequence image based on the visual sequence image and external information. The external information includes scene information, experience information and interference information. The method comprises the following steps: S1, constructing an initial lip reading model, which comprises a visual modal data processing module, a pre-trained visual encoder, a visual projector, a text embedding module and a first large language model. The visual modal data processing module is used to pre-process the visual sequence image to obtain a lip visual sequence image. The pre-trained visual encoder is used to encode the lip visual sequence image into a visual feature sequence. The visual projector is used to convert the visual feature sequence into a visual feature representation adapted to the first large language model. The text embedding module is used to encode the external information into an external information feature representation adapted to the first large language model. The first large language model is used to predict the speech text corresponding to the lip visual sequence image and the actual reasoning process description text based on the visual feature representation and the external information feature representation of the lip visual sequence image; S2, obtaining an initial data set, which comprises a plurality of visual sequence images, each of which is configured with a corresponding real speech text and external information; using a preset method to obtain the correct reasoning process description text of the first large language model based on the external information of each visual sequence image; using each visual sequence image and its corresponding external information as a sample, and using the real speech text and the correct reasoning process description text as labels, a training set is constructed; S3, based on the constructed training set, using the visual sequence image and its corresponding external information as input, and using the predicted speech text and the actual reasoning process description text as output, the initial lip reading model is iteratively trained until convergence to obtain a final lip reading model. In the training process, a preset cross-entropy loss function is used to calculate the loss, and the parameters of the first large language model, the visual projector and the pre-trained visual encoder are updated based on the calculated loss.

[0010] Preferably, the external information is a text composed of scene information, experience information and interference information. The scene information is a text describing the scene information of the speech. The experience information is a text composed of experience knowledge related to the understanding of the lip reading task by the first language model and the lip reading task. The interference information is a randomly generated text.

[0011] Preferably, in step S1, the visual sequence image is pre-processed by the following method: identifying the lip region of the speaker in each frame of the visual sequence image, and cropping the image of the lip region to obtain an initial lip visual sequence image; and performing grayscale processing on each frame of the initial lip visual sequence image to obtain a final lip visual sequence image.

[0012] Preferably, in step S1, the pre-trained visual encoder is a visual encoder constructed based on a Transformer architecture.

[0013] Preferably, in step S2, the preset method comprises: obtaining a second large language model, wherein the second large language model is a large language model with a larger scale than the first large language model and is trained; taking the external information of each visual sequence image and the corresponding spoken text as the input of the second large language model, so that the second large language model generates the correct reasoning process description text of the first large language model in the lip language recognition model to infer the lip visual sequence.

[0014] Preferably, in step S3, the preset cross-entropy loss function is:

[0015]

[0016] wherein, denotes the total cross-entropy loss, denotes the reasoning process cross-entropy loss between the actual reasoning process description text and the correct reasoning process description text calculated based on the preset reasoning process cross-entropy loss function, denotes the text cross-entropy loss between the predicted spoken text and the real spoken text calculated based on the preset text cross-entropy loss function, denotes a parameter for adjusting the weight ratio between the reasoning process cross-entropy loss and the text cross-entropy loss.

[0017] Preferably, the preset reasoning process cross-entropy loss function is:

[0018]

[0019] wherein, denotes the reasoning process cross-entropy loss, denotes the length of the correct reasoning process description text, denotes the i-th word in the actual reasoning process description text, denotes the i-th word in the correct reasoning process description text, is an external information feature representation, is a visual feature representation, denotes the probability distribution of the i-th word in the actual reasoning process description text.

[0020] Preferably, the preset text cross-entropy loss function is:

[0021]

[0022] wherein,​​​ represents a text cross-entropy loss, represents a length of a real speech text, represents a first word in a predicted speech text, represents a first word in a real speech text, represents a correct reasoning process description text, is an external information feature representation, is a visual feature representation, represents a probability distribution of a first word in a predicted speech text.

[0023] Preferably, in step S3, the parameters of the first large language model, the visual projector and the pre-trained visual encoder are updated based on the calculated loss using a low-rank adaptive technique or a full-parameter fine-tuning technique.

[0024] According to a second aspect of the present application, a lip-reading method is provided, which comprises: acquiring visual sequence images and external information thereof to be recognized; and recognizing speech text corresponding to the visual sequence images based on the visual sequence images and the external information thereof to be recognized by using a lip-reading model constructed by any one of the methods according to the first aspect of the present application.

[0025] Compared with the prior art, the present application has the following advantages:

[0026] The present application provides a lip-reading model construction scheme fusing external information, which significantly reduces the error rate of lip-reading by fusing visual features and rich external scene information, human priori knowledge information and perturbed semantic information, and combining visual information to jointly infer speech content, and has wide application prospects and value. BRIEF DESCRIPTION OF DRAWINGS

[0027] The embodiments of the present application will be further described below with reference to the accompanying drawings, in which:

[0028] Figure 1 FIG. 1 is a schematic diagram of a lip-reading model construction method according to an embodiment of the present application;

[0029] Figure 2 FIG. 2 is a schematic diagram of a lip-reading model training process according to an embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application. ​​​

[0031] As mentioned in the background section, existing lip reading schemes mainly rely on visual features of lip movements for speech transcription, but ignore external available information, such as the environment can provide rich semantic information (such as the identity of the speaker, the speaking scene, etc.), human prior knowledge and understanding of the lip reading task, and the enhancement effect of the disturbance semantic information. These information has potential great value for improving the accuracy of recognition.

[0032] In order to effectively solve the problem of low recognition accuracy caused by lack of effective use of external information in lip reading, the present application proposes a multi-modal learning framework, constructs a lip reading model based on a large language model, and infers the speaking content by fusing visual features and rich external information, thereby effectively improving the accuracy of lip reading.

[0033] According to an embodiment of the present application, the present application proposes a lip reading model construction scheme fusing external information, see Figure 1The main steps of the lip reading model construction scheme fusing external information are shown in the figure. In summary, the scheme includes steps S1, S2 and S3. In step S1, an initial lip reading model is constructed, which includes a visual modal data processing module, a pre-trained visual encoder, a visual projector, a text embedding module and a first large language model. The visual modal data processing module is used to preprocess visual sequence images to obtain lip visual sequence images. The pre-trained visual encoder is used to encode the lip visual sequence images into visual feature sequences. The visual projector is used to convert the visual feature sequences into visual feature representations adapted to the first large language model. The text embedding module is used to convert external information into external information feature representations adapted to the first large language model. The first large language model is used to predict the speaking text corresponding to the lip visual sequence images and the actual reasoning process description text based on the visual feature representations and the external information feature representations of the lip visual sequence images. In step S2, an initial data set is obtained, which includes a plurality of visual sequence images, each visual sequence image being configured with corresponding real speaking text and external information. A pre-set method is used to obtain the correct reasoning process description text of the first large language model based on the external information of each visual sequence image. A training set is constructed using each visual sequence image and its corresponding external information as a sample, and using the real speaking text and the correct reasoning process description text as labels. In step S3, based on the constructed training set, the initial lip reading model is iteratively trained until convergence using the visual sequence images and their corresponding external information as input and the predicted speaking text and the actual reasoning process description text as output, to obtain the final lip reading model. In the training process, a pre-set cross-entropy loss function is used to calculate the loss, and the parameters of the first large language model, the visual projector and the pre-trained visual encoder are updated based on the calculated loss.

[0034] In order to better understand the present application, the present application will be described in detail below in conjunction with specific examples and drawings.

[0035] According to one embodiment of the present application, in step S1 of the present application, the definition and preparation of the external information is a crucial step in the lip-reading task, which covers auxiliary information irrelevant to the content or image of the speech, and is generated through an explicit derivation and thinking process rather than a simple prompt word form, aiming to fully simulate the cognitive process and coping strategy of human beings in the lip-reading task. Specifically, the external information is a text composed of speaking context information, experience information and interference information, wherein the context information includes but is not limited to the speaker's role, the speaking scene and the speaking topic, and the context information should at least contain certain context-related information. For example, assuming that the text describing the speaking context information is "This is a video from a TED talk, the title of the talk is "Rethinking the way we sit down", the speaker is XX, and the speaker's role is XX". The experience information is a text composed of the experience knowledge related to the understanding of the first language model for the lip-reading task and the lip-reading task, wherein the experience knowledge includes but is not limited to a list of confusing words, and the list of confusing words contains a plurality of text pairs, each text pair contains a plurality of words with the same lip movement, for example, the two words "sit" and "seat" have very similar lip movements when speaking, and it is easy to be confused by relying only on visual information for identification. The interference information is a randomly generated text.

[0036] According to one embodiment of the present application, in step S1 of the present application, the visual sequence image is preprocessed by: identifying the lip region of the speaker in each frame of the visual sequence image, and cropping the image of the lip region to obtain an initial lip visual sequence image; and performing grayscale processing on each frame of the initial lip visual sequence image to obtain a final lip visual sequence image. Specifically, first, a pre-set face detector is used to detect the face region of the speaker in each frame of the visual sequence image, and then the key points (key points include nose tip, two corners of the mouth, chin, etc.) of the face region in each frame are detected, and the lip region is cropped based on the detected key points to obtain the initial lip visual sequence image, wherein each frame of the initial lip sequence image is a complete and valid lip image, and finally each frame of the initial lip sequence image is subjected to grayscale processing to obtain the final lip visual sequence image. By accurately cropping the lip region, the redundant information of the background and other parts of the face can be removed, so that the model focuses on the dynamic changes of the lips directly related to the pronunciation, and in addition, grayscale processing of the image not only significantly reduces the data dimension, but also eliminates the potential interference caused by color differences, thereby improving the accuracy and efficiency of the subsequent lip-reading task.

[0037] According to an embodiment of the present application, in step S1 of the present application, the pre-trained visual encoder is a visual encoder constructed based on a Transformer architecture. Illustratively, the present application employs AV-HuBERT as the visual encoder, which is pre-trained on large-scale multi-modal data in a self-supervised manner based on a Transformer architecture. The core idea is to utilize the alignment relationship between audio and video, mask part of the input and predict the semantic information of the masked part, so as to learn robust visual and audio feature representations. In the present application, AV-HuBERT takes lip visual sequence images as input, models the images in the spatial and temporal domains, and finally outputs high-level visual features. It should be understood that, in addition to the AV-HuBERT visual encoder, the present application can also employ other visual encoders based on the Transformer architecture, such as the self-supervised pre-trained RAVEn model, or the visual encoder in the supervised training model Auto-AVSR, which can also effectively capture the spatio-temporal information in the visual sequence.

[0038] According to an embodiment of the present application, in the present application, the visual projector is used to convert the visual feature sequence encoded by the visual encoder into a visual feature representation adapted to the first large language model so as to be mapped to the embedding space of the first large language model. Specifically, the visual projector is a linear layer, the input dimension of which is equal to the dimension of the visual feature sequence output by the visual encoder, and the output dimension of which is equal to the embedding dimension of the first large language model.

[0039] According to an embodiment of the present application, in the present application, the training set used to train the initial lip reading model takes each visual sequence image and its corresponding external information as a sample, and takes the real speech text and the correct reasoning process description text as labels, wherein the correct reasoning process description text is a textual description of the reasoning process of the first large language model in the lip reading model to perform the lip reading task (the lip reading task is to recognize the speech text corresponding to each visual sequence image based on the visual sequence image and its corresponding external information).

[0040] According to one embodiment of the present application, the way to obtain the correct inference process description text corresponding to the visual sequence image in the present application is as follows: a second large language model is obtained, wherein the second large language model is a large language model with larger scale and stronger capability than the first large language model in the lip reading model, such as the large language model trained by GPT, DeepSeek, etc.; the external information of each visual sequence image is input into the second large language model as the input of the second large language model, so that the second large language model generates the correct inference process description text of the first large language model in the lip reading model for the inference of the lip visual sequence. Since the second large language model consumes more computing resources, the present application preferentially calls the second large language model in the form of API, and it should be understood that the second large language model can also be deployed locally to directly call the second large language model under the premise of sufficient computing resources.

[0041] In order to more clearly describe the way to obtain the correct inference process description text, the following will be described in conjunction with examples. Specifically, the external information and the corresponding instructions are first input into the second large language model. The input instructions can be open questions, analysis tasks or inference requirements in specific scenarios. For example, the instructions can be to require the model to generate a detailed analysis text according to the input external information (such as video title, context description or possible interference text), simulate the thinking logic of the first large language model in the inference process, and the instructions can be, for example, “please generate a first-person analysis text according to the following input information, describe how to combine the context and visual features to infer the speaking content”, or “please simulate the inference process of a model when processing the lip reading task, and focus on analyzing the possible confusion points and solving strategies”. It should be noted that the instructions and the second large language model used here are not limited to specific words or specific model architectures. As long as the model has strong text generation and inference capability, the instructions can be flexibly adjusted to adapt to different task requirements and scenarios.

[0042] For example, the following is a correct reasoning process description text generated by the second large language model based on external information and corresponding instructions: "I am analyzing a video clip from a TED talk titled 'Rethinking the way we sit down.' Based on the title, I infer that the content of the speech may involve ergonomics, lifestyle changes, or health-related topics. While observing the speaker's lip movements, I notice some potential difficulties. For instance, the words'sit' and'seat' are visually very similar and can be easily confused. Additionally, 'down' and 'town' can also be misidentified due to their similar pronunciation actions. To improve accuracy, I need to reason in context. For example, if the previous text mentions 'comfort' or 'posture,' then'sit' is more likely; if it mentions 'city' or 'location,' then 'town' is more likely. Furthermore, I also notice that there may be some random noise or irrelevant words in the input, such as 'random noise' or 'irrelevant words.' These disturbances can mislead my judgment, so I need to be more cautious in analyzing visual features and excluding irrelevant information. Finally, I will integrate visual cues, contextual reasoning, and handling of disturbances to generate the most likely speech content." This example demonstrates a description text (i.e., a correct reasoning process description text) that a large prediction model obtains using a training model to understand and utilize external information. The present invention guides the first large language model in the lip-reading model to simulate the complex cognitive process of humans in the lip-reading task, in which the model combines background descriptions, difficulty analysis, and disturbance handling strategies in external information to conduct comprehensive analysis and reasoning.

[0043] According to an embodiment of the present invention, based on the constructed training set, the initial lip-reading model is iteratively trained to convergence to obtain the final lip-reading model, with the visual sequence images and their corresponding external information as input, and the predicted speech text and the actual reasoning process description text as output. During the training process, a preset cross-entropy loss function is used to calculate the loss, and the parameters of the first large language model, the visual projector, and the pre-trained visual encoder are updated based on the calculated loss. It should be understood that when updating the parameters, the implementer can flexibly choose to freeze or fine-tune any one, any combination, or all of the first large language model, the visual projector, and the pre-trained visual encoder according to precision, efficiency, or training objectives to achieve targeted performance adjustment of the lip-reading model.

[0044] According to an embodiment of the present invention, the preset cross-entropy loss function in the present invention is:

[0045]

[0046]

[0047]

[0048] in, This represents the total cross-entropy loss. This represents the cross-entropy loss between the actual inference process description text and the correct inference process description text, calculated based on a preset inference process cross-entropy loss function. This represents the text cross-entropy loss between the predicted spoken text and the actual spoken text, calculated based on a preset text cross-entropy loss function. This parameter represents the weighting ratio between the cross-entropy loss during the inference process and the cross-entropy loss of the text. The length of the text describing the correct reasoning process. The first character in the text represents the actual reasoning process. word, The first character in the text represents the correct reasoning process. word, As a representation of external information features, Represented by visual features, The first character in the text represents the actual reasoning process. Probability distribution of words, Indicates the length of the actual spoken text. Indicates the prediction of the first word in the spoken text. One word, The first in the actual spoken text One word, As a representation of external information features, Represented by visual features, The text describes the correct reasoning process. Indicates the prediction of the first word in the spoken text. The probability distribution of each word.

[0049] According to one embodiment of the present application, in the present application, the parameters of the first large language model, the visual projector and the pre-trained visual encoder are adjusted based on the calculated loss by using a low-rank adaptation technology (LoRA) or a full model fine-tuning technology, wherein the core idea of the low-rank adaptation technology is to add a small number of trainable low-rank parameters in the key part of the model or module while keeping the original parameters frozen, so as to realize task adaptation while reducing the computational overhead, and the full model fine-tuning technology is to adjust the original parameters in the model or module in full amount. Taking the first large language model as an example, the way of adjusting the parameters of the first large language model by using the low-rank adaptation technology is as follows: the query vector and the value vector of the attention mechanism part of the first large language model are adjusted in low rank, and in each layer of the multi-head attention module, the calculation formulas of the query vector and the value vector are respectively:

[0050]

[0051]

[0052] wherein, and are the weight matrices of the query vector and the value vector, is the input feature. In LoRA, the present application performs low-rank decomposition on and , and represents them as the product of two low-rank matrices:

[0053]

[0054]

[0055] wherein, and are the original weight matrices, the parameters of the two matrices remain frozen, and , and is a newly added low-rank matrix, whose rank is much smaller than the dimension of the original weight matrix (for example, the rank is r = 8, the selection of the rank is not limited, and can be adjusted according to the actual training situation), and the values of the parameters in the low-rank matrix can be adjusted. Therefore, when optimizing the parameters of the first large language model, the present application only needs to update a small number of newly added low-rank parameters (parameters of the low-rank matrix), so as to realize efficient fine-tuning of the first large language model, while avoiding modification of the original large number of parameters. This strategy not only significantly reduces the training cost, but also enhances the adaptation ability of the first large language model to the lip-reading task while maintaining the generality of the first large language model. It should be understood that the way of adjusting the parameters of the visual projector and the pre-trained visual encoder using the low-rank adaptive technology is similar to the way of adjusting the first large language model, which will not be described in detail here.

[0056] In order to more clearly describe the training process of the lip-reading model in the present application, the following will combine the accompanying drawings to Figure 2 be described in detail.

[0057] Referring to the accompanying Figure 2 , the data flow process between each module in each round of training of the lip-reading model is shown: a visual sequence image preprocessing step, in which the visual sequence image is preprocessed by the visual modality data processing module to obtain a lip visual sequence image; a visual feature coding step, in which the lip visual sequence image is coded into a visual feature sequence by the visual coding module; a visual feature embedding step, in which the visual feature sequence is converted into a feature representation suitable for the first large language model by the visual projection module, and embedded into the feature space of the first large language model; an external information text embedding step, in which the external information is converted into a feature representation suitable for the first large language model by the text embedding module; a prediction step, in which the first large language model concatenates the visual feature representation of each visual sequence image with the external information feature representation to form the final feature representation of the visual sequence image, and the first large language model obtains the predicted speaking text corresponding to the visual sequence image based on the final feature representation, and outputs the inference process description text in the first large language model recognition process; a model optimization step, in which a preset cross-entropy loss function is used to calculate the loss, and the parameters of the first large language model, the visual projector and the pre-trained visual encoder are updated based on the calculated loss.

[0058] According to an embodiment of the present application, a lip-reading method is provided for predicting speaking text based on visual sequence images and their external information. In general, the method comprises: obtaining visual sequence images and their external information that need to be recognized, wherein the external information includes speaking context information and a list of confusing words; and using a lip-reading model constructed by the above-mentioned lip-reading model construction method to recognize the speaking text corresponding to the visual sequence images based on the visual sequence images and their external information that need to be recognized.

[0059] In summary, the present application breaks through the dilemma of traditional lip-reading system relying on single visual feature and being difficult to cope with complex environment and ambiguous lip movement by introducing rich external information into the lip-reading process. The present application significantly reduces the error rate of lip-reading by fusing visual feature and various external information, including context information, experience information and disturbance information, and has wide application prospect and value.

[0060] It should be noted that although the above describes each step in a specific order, it does not mean that each step must be performed in the above specific order, in fact, some of the steps can be performed concurrently or even in a changed order, as long as the required function can be realized.

[0061] The present application can be a system, a method, and / or a computer program product. The computer program product can include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present application.

[0062] The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium include the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or punched tape, a magneto-optical or other optical medium, and / or any suitable combination of the foregoing. A non-transitory, non-temperable computer-readable storage medium having instructions stored therein for execution by and to program a processor or controller device is also described herein.

[0063] Embodiments of the application have been described above, and the description is intended to be illustrative, and not restrictive, of the disclosed embodiments. Many modifications and variations of the disclosed embodiments are possible in light of the above teachings without departing from the scope and spirit of the disclosed embodiments. It is therefore to be understood that within the scope of the appended claims and their equivalents, the disclosed embodiments can be practiced otherwise than as specifically described. The description herein is intended for purposes of illustration only and is not necessary intended to limit the scope of the disclosed embodiments.

Claims

1. A method for constructing a lip-reading recognition model that integrates external information, used to construct a lip-reading recognition model based on visual sequence images and external information to identify the spoken text corresponding to the visual sequence images, wherein, External information includes contextual information, experiential information, and interference information. The method is characterized by comprising: Step S1: Construct an initial lip-reading recognition model. The initial lip-reading recognition model includes a visual modality data processing module, a pre-trained visual encoder, a visual projector, a text embedding module, and a first language model. The visual modality data processing module preprocesses visual sequence images to obtain lip visual sequence images. The pre-trained visual encoder encodes the lip visual sequence images into visual feature sequences. The visual projector converts the visual feature sequences into visual feature representations adapted to the first language model. The text embedding module encodes external information into external information feature representations adapted to the first language model. The first language model predicts the spoken text corresponding to the lip visual sequence image and the actual reasoning process description text based on the visual feature representations and external information feature representations of the lip visual sequence image. Step S2: Obtain the initial dataset, which includes multiple visual sequence images, each visual sequence image being configured with corresponding real spoken text and external information; use a preset method to obtain the correct reasoning process description text of the first language model based on the external information of each visual sequence image; use each visual sequence image and its corresponding external information as samples, and use the real spoken text and the correct reasoning process description text as labels to construct a training set; Step S3: Based on the constructed training set, the initial lip-reading model is iteratively trained until convergence, using visual sequence images and their corresponding external information as input, and predicted speaking text and actual reasoning process description text as output, to obtain the final lip-reading model. During the training process, a preset cross-entropy loss function is used to calculate the loss, and the parameters of the first language model, visual projector, and pre-trained visual encoder are updated based on the calculated loss.

2. The method according to claim 1, characterized in that, The external information is text composed of speaking context information, experiential information, and interference information, wherein... The contextual information is text describing the speaking context; The experiential information is text composed of the first language model's understanding of the lip-reading task and experiential knowledge related to the lip-reading task; The interference information is randomly generated text.

3. The method according to claim 1, characterized in that, In step S1, the visual sequence images are preprocessed as follows: Identify the speaker's lip region in each frame of the visual sequence image and crop out the lip region image to obtain the initial lip visual sequence image; Each frame of the initial lip visual sequence image is converted to grayscale to obtain the final lip visual sequence image.

4. The method according to claim 1, characterized in that, In step S1, the pre-trained visual encoder is a visual encoder built based on the Transformer architecture.

5. The method according to claim 1, characterized in that, In step S2, the preset method includes: Obtain a second largest language model, where the second largest language model is a large language model that is larger in scale than the first largest language model and is well trained; The external information of each visual sequence image and its corresponding spoken text are used as input to the second language model, which generates a text describing the correct reasoning process of the first language model in the lip reading recognition model for the lip visual sequence.

6. The method according to claim 1, characterized in that, In step S3, the preset cross-entropy loss function is: in, This represents the total cross-entropy loss. This represents the cross-entropy loss between the actual inference process description text and the correct inference process description text, calculated based on a preset inference process cross-entropy loss function. This represents the text cross-entropy loss between the predicted spoken text and the actual spoken text, calculated based on a preset text cross-entropy loss function. This parameter represents the weighting ratio between the cross-entropy loss in the inference process and the cross-entropy loss in the text.

7. The method according to claim 6, characterized in that, The preset cross-entropy loss function for the inference process is: in, This represents the cross-entropy loss during the inference process. The length of the text describing the correct reasoning process. The first character in the text represents the actual reasoning process. word, The first character in the text represents the correct reasoning process. word, As a representation of external information features, Represented by visual features, The first character in the text represents the actual reasoning process. Probability distribution of words.

8. The method according to claim 6, characterized in that, The preset text cross-entropy loss function is: in, Represents the text cross-entropy loss. Indicates the length of the actual spoken text. Indicates the prediction of the first word in the spoken text. One word, The first in the actual spoken text One word, The text describes the correct reasoning process. As a representation of external information features, Represented by visual features, Indicates the prediction of the first word in the spoken text. The probability distribution of each word.

9. The method according to claim 1, characterized in that, In step S3, the parameters of the first language model, the visual projector, and the pre-trained visual encoder are updated using low-rank adaptive techniques or full-parameter fine-tuning techniques based on the calculated loss.

10. A lip-reading recognition method, characterized in that, The method includes: Acquire the visual sequence images to be identified and their external information; The lip-reading recognition model constructed using the method described in any one of claims 1-9 identifies the spoken text corresponding to the visual sequence image to be recognized based on the visual sequence image to be recognized and its external information.

Images