Chinese lip language recognition method based on face dynamic capture
Through the depth camera to acquire three-dimensional facial key points and combine specific models, the problem of poor recognition effect in low-light environments is solved, and the robustness and efficiency of Chinese lip recognition is improved, which is suitable for the application of smart devices.
Patent Information
- Application Number
- CN202411114776.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-14
- Publication Date
- 2024-11-26
AI Technical Summary
The existing Chinese lip recognition technology has poor recognition effect in non-standardized environments, especially in low light conditions, low accuracy, high computing resources consumption, and lack of depth information, resulting in a degradation of recognition performance in complex environments.
The depth camera is used to collect three-dimensional facial key points, and by calculating the dynamic features of the three-dimensional facial, combining the Conformer encoder and the Transformer-Decoder/CTC-Decoder model to perform lip recognition, providing stable depth data and fast and efficient processing methods.
It significantly improves the robustness and recognition accuracy of the model in low-light environments, reduces the computing complexity, is suitable for smartphones and smart mobile devices, and adapts to the application needs of diverse life scenarios.
Smart Images

Figure CN119028348A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of Chinese lip reading recognition, and in particular to a Chinese lip reading recognition method based on facial dynamic capture. Background Art
[0002] Lip reading recognition has received increasing attention in recent years. It determines the content of a person's speech by capturing the movement of the speaker's lips. It provides an important communication tool for people who have lost their voices due to vocal cord, larynx and tongue injuries. Lip reading recognition technology has shown its indispensable value in improving the quality of life and social participation of these groups. In addition, lip reading recognition can also improve speech recognition in noisy environments and fill in the sound of silent surveillance, etc., and has broad application prospects.
[0003] There have been many studies on lip reading recognition tasks, and the rapid development of deep learning has also promoted the research of lip reading recognition technology. Most of the current lip reading recognition algorithms use 3D convolutional networks to extract features from lip images, and then use models such as attention mechanisms to perform lip reading recognition. However, the existing technology still has limitations: (1) Database monotony: Most of the current mainstream Chinese lip reading recognition models rely on data sets from news broadcasts. These data environments are highly standardized and are not suitable for the changing scenes in daily life. For example, fluctuations in lighting and environmental noise can significantly affect the recognition effect of the model. (2) Lack of depth information: Traditional lip reading recognition technology does not collect depth information, but instead uses algorithms for inference, resulting in low accuracy, high consumption of computing resources, slow inference, and other problems, which greatly reduces the recognition performance of lip reading recognition in complex environments. Summary of the invention
[0004] In view of the defects of the above-mentioned prior art, the present invention proposes a Chinese lip reading recognition method based on facial dynamic capture, which innovatively introduces a depth camera for data collection and calculates three-dimensional facial key points, effectively enhancing the performance of the model in low-light environments. This strategy significantly improves the robustness and practicality of the model in non-standardized environments. At the same time, facial dynamic features are calculated based on the coordinates of three-dimensional facial key points with low complexity, and linear calculations are used instead of complex deep learning preprocessing networks, so that the model can process data quickly and efficiently. Therefore, this technology has the potential to be widely used in smartphones / smart mobile devices.
[0005] A Chinese lip reading recognition method based on facial dynamic capture comprises the following steps:
[0006] (1) Use a depth camera to collect continuous images of the face to be recognized;
[0007] (2) Estimating the three-dimensional facial key points of the collected continuous facial images and calculating the facial dynamic features based on the key points;
[0008] (3) Input the facial dynamic features into the lip reading recognition model to obtain the probability distribution of pinyin categories;
[0009] (4) Use a large language model to predict text sequences based on the probability distribution of pinyin categories and generate Chinese lip reading recognition results.
[0010] In step (1), a depth camera is used to collect continuous facial images to be recognized, which specifically includes:
[0011] A depth camera is used to collect continuous facial images, which include a depth map and a color map.
[0012] Depth cameras can capture three-dimensional depth information of the surface of objects, helping the model to more accurately separate the foreground (such as lips) and background, and distinguish the shape and movement of the lips, thereby improving recognition accuracy. Depth cameras can provide stable depth data under different lighting conditions, which enables depth cameras to provide consistent input data under various environmental conditions, improving the robustness and reliability of the model.
[0013] In step (2), three-dimensional facial key points are estimated for the collected continuous facial images, and facial dynamic features are calculated based on the key points, which specifically includes:
[0014] (2.1) Adjust the collected color image and depth image to a fixed size and normalize the pixel values and depth values;
[0015] (2.2) Input the color image into the MediaPipe Face Mesh function to estimate the two-dimensional facial key points and obtain the x and y coordinates of 468 facial key points;
[0016] (2.2) obtaining the depth information corresponding to the facial key points according to the depth map and the x and y coordinates of the 468 facial key points, i.e., 468 z coordinates. The x, y coordinates and the 468 z coordinates of the 468 facial key points form three-dimensional facial key points, i.e., 468 x, y, z coordinates;
[0017] (2.3) using a filtering and smoothing algorithm to adjust the 468 x, y, and z coordinates to obtain 468 adjusted x, y, and z coordinates;
[0018] (2.4) Use the adjusted 468 x, y, z coordinates to calculate the dynamic features of the face.
[0019] The above facial dynamic features effectively reduce the redundant information in the original image and focus on meaningful facial movements. At the same time, the linear computational complexity is low, which improves the inference speed of the algorithm.
[0020] In step (3), the lip reading recognition model is composed of an encoder and a decoder connected in sequence;
[0021] The encoder is composed of a Conformer, which is used to capture the temporal and spatial characteristics of facial dynamic data;
[0022] The decoder is composed of a Transformer-decoder and a CTC-decoder connected in parallel;
[0023] The feature vector output by the encoder is copied twice and input into the Transformer-decoder and CTC-decoder at the same time, and the two obtained probability distributions are linearly combined as the final output of the probability distribution of the pinyin category.
[0024] The above-mentioned Conformer encoder is good at feature extraction and focuses on lip features that are more effective for lip reading recognition. Transformer-Decoder and CTC-Decoder are respectively good at context modeling and time alignment. The combination of the networks can achieve functional complementarity and improve overall performance.
[0025] In step (3), the pinyin categories refer to dividing 21 initial consonants and 36 finals into 11 pinyin categories according to the lip shape changes when pronouncing Chinese pinyin;
[0026] The lip reading recognition model identifies the probability that the currently pronounced phoneme belongs to each pinyin category.
[0027] The above pinyin classification is in line with the natural laws of Chinese pronunciation. This classification can better capture and distinguish the lip movement characteristics of different pronunciations. Simplifying 57 pinyin (21 initials and 36 finals) into 11 categories helps reduce the complexity of the model, reduces the difficulty of recognition, and effectively reduces the confusion of similar pronunciations while maintaining recognition accuracy. Fewer categories can also reduce training data, making model training more efficient. The probability distribution is combined with the subsequent large language model to further improve the prediction accuracy and naturalness of text sequences.
[0028] In step (3), the lip reading recognition model needs to be trained on a pre-constructed data set. During the training phase, the output of the Transformer-decoder uses a cross-entropy loss function to calculate the loss, and the output of the CTC-decoder uses a CTC (Connectionist Temporal Classification) loss function to calculate the loss.
[0029] In step (4), a large language model is used to predict the text sequence according to the probability distribution of the pinyin category to generate the Chinese lip reading recognition result, which specifically includes:
[0030] The probability distribution of the pinyin categories obtained in step (3) is input into a pre-trained large language model. The model uses context information to predict the most likely text sequence based on the probability distribution of the pinyin categories and generates Chinese lip reading recognition results.
[0031] Further preferably, a Chinese lip reading recognition method based on facial dynamic capture comprises the following steps:
[0032] (1) Use a depth camera to collect continuous images of the face to be recognized. The collected data includes depth maps and color maps. The depth camera provides fast and accurate depth measurement by emitting light signals (usually infrared light) into the scene and then measuring the time required for the light wave to be reflected back to the sensor by the object. Therefore, this method is still effective in dim light and is robust to fluctuations in lighting and environmental noise.
[0033] (2) Estimating the three-dimensional facial key points of the collected facial images and calculating the facial dynamic features based on the key points:
[0034] (2.1) adjusting the color image and depth image collected in step (1) to a fixed size and normalizing the pixel values and depth values;
[0035] (2.2) Input the color image into the MediaPipe Face Mesh function to estimate the two-dimensional facial key points and obtain the x and y coordinates of 468 facial key points;
[0036] (2.2) According to the depth map and the coordinates of the 468 two-dimensional facial key points, the depth information corresponding to the facial key points and the three-dimensional facial key points, i.e., the 468 x, y, and z coordinates, are obtained;
[0037] (2.3) Use filtering and smoothing algorithms to adjust coordinate points to reduce key point jitter and other unnatural movements;
[0038] (2.4) Use the coordinates of 3D facial key points to calculate the dynamic features of the face. These features accurately describe the tiny movements of the face (especially the lips), providing rich raw data for subsequent lip reading decoding;
[0039] (3) Input the facial dynamic features into the lip reading recognition model to obtain the probability distribution of pinyin categories;
[0040] The lip reading recognition model consists of an encoder and a decoder, where the encoder is composed of a Conformer network that combines the local feature extraction capabilities of CNN and the global dependency modeling capabilities of Transformer, and can effectively capture complex relationships in time and space; the decoder is composed of a Transformer-decoder and a CTC (Connectionist temporal classification)-decoder connected in parallel, where the Transformer model further enhances the model's ability to understand time series data and provides an attention mechanism for the CTC-decoder, which focuses on converting these continuous features into independent phonetic category probability distributions.
[0041] (3.1) The Conformer encoder extracts features from the input vector and obtains the output feature vector.
[0042] (3.2) Make two copies of the feature vector output by the encoder and input them into the Transformer-decoder and CTC-decoder modules respectively. The linear combination of the two probability distribution outputs is used as the final output of the pinyin category probability distribution.
[0043] Preferably, the 21 initial consonants and x finals are divided into n pinyin categories according to the lip shape changes during the pronunciation of Chinese pinyin, and the lip reading recognition model recognizes the probability that the currently pronounced phoneme belongs to each category.
[0044] Preferably, the Conformer model consists of a feedforward module, a self-attention module, a convolution module and a second feedforward module, and the modules are connected by residual means; the feedforward module consists of a normalization layer, two linear layers, an activation function layer and two Dropout layers; the self-attention module consists of a normalization layer, a multi-head attention layer and a Dropout layer; the convolution module consists of a normalization layer, a point-by-point convolution layer, a GELU activation layer, a depth convolution layer, a batch normalization layer, an activation function layer, a point-by-point convolution layer and a Dropout layer.
[0045] Transformer-decoder is composed of a self-attention module, a multi-head attention module, and a feedforward module connected in sequence, and the modules are connected in a residual manner; the self-attention module is composed of a self-attention layer and a Dropout layer; the multi-head attention module is composed of a multi-head attention layer and a Dropout layer; the feedforward module is composed of a linear layer, an activation function layer, and a Dropout layer; and the CTC-decoder is composed of a linear layer.
[0046] (4) Use a large language model to predict the text sequence based on the probability distribution of pinyin categories and generate the final recognition result. This model performs deep learning based on the pinyin probability distribution and uses contextual information to predict the most likely text sequence. This step not only improves the accuracy of the overall prediction, but also enhances the model's grasp of the natural fluency of the language.
[0047] In addition to the above steps, the lip reading recognition model described in step (3) needs to be trained on a pre-constructed dataset. During the training phase, the output of the Transformer-decoder uses the cross entropy loss function to calculate the loss, and the output of the CTC-decoder uses the CTC loss function to calculate the loss.
[0048] Compared with the prior art, the present invention has the following beneficial effects:
[0049] The present invention has developed a Chinese lip reading recognition method based on facial dynamic capture. In terms of acquisition method, the present invention innovatively introduces a depth camera for data acquisition, provides depth information as input, and effectively enhances the performance of the model in low-light environments; in terms of training data, the training data of the present invention covers more diverse life scenes and is more in line with actual application needs; in terms of model construction, because the facial dynamic features are directly calculated using three-dimensional facial coordinate points, the complex 3D convolution preprocessing network is omitted, so that the model can process data quickly and efficiently. Therefore, this technology has the potential to be widely used in smart phones / smart mobile devices. BRIEF DESCRIPTION OF THE DRAWINGS
[0050] Figure 1 It is the overall flow chart of the present invention;
[0051] Figure 2 It is a schematic diagram of two-dimensional coordinates of key points in the present invention;
[0052] Figure 3 Constructing a flow chart for the database of the present invention;
[0053] Figure 4 is a structural diagram of the lip reading recognition model in the present invention;
[0054] Figure 5 This is a diagram showing the categories to which each phoneme in the present invention belongs. DETAILED DESCRIPTION
[0055] In order to describe the present invention more specifically, the technical solution of the present invention is described in detail below in conjunction with the accompanying drawings and specific implementation methods.
[0056] The present invention discloses a Chinese lip reading recognition method based on facial dynamic capture, such as Figure 1 The following steps are shown:
[0057] (1) Using a depth camera to collect continuous images of the face to be recognized, the collected data includes a depth map and a color map;
[0058] (2) Estimating the three-dimensional facial key points of the collected facial images and calculating the facial dynamic features based on the key points:
[0059] (2.1) adjusting the color image and depth image collected in step (1) to a fixed size and normalizing the pixel values and depth values;
[0060] (2.2) Input the color image into the MediaPipe Face Mesh function to estimate the two-dimensional facial key points and obtain the x and y coordinates of 468 facial key points;
[0061] (2.2) According to the depth map and the coordinates of 468 two-dimensional facial key points, the depth information corresponding to the facial key points and the three-dimensional facial key points, i.e., 468 x, y, and z coordinates, are obtained.
[0062] (2.3) Use filtering and smoothing algorithms to adjust coordinate points to reduce key point jitter and other unnatural movements.
[0063] (2.4) Using the coordinates of the three-dimensional facial key points to calculate the dynamic features of the face, the specific steps are as follows: Figure 2 As shown, according to the index, the coordinates of 18 key points are obtained: the center point of the upper lip upper_lip\the center point of the lower lip lower_lip\the lowest point of the lower lip lowest_lip\the outer point of the upper lip upper_outer_lip\the upper point of the upper lip over_upper_lip\the left corner of the mouth mouth_left_corner\the right corner of the mouth mouth_right_corner\the center point of the forehead upper_head\the lowest point of the chin lowest_ccin\the upper point of the outer side of the chin upper_outer_chin\the tip of the nose nose_tip\the right corner of the mouth mouth_right _corner\mouth left deformation point mouth_left_frown\mouth right deformation point mouth_right_frown\mouth left stretch point mouth_left_stretch\mouth right stretch point mouth_right_stretch\left lower lip pressing point left_lower_press\right lower lip pressing point right_lower_press\left upper lip pressing point left_upper_press\right upper lip pressing point right_upper_press, the zeroth, first, and second elements of each coordinate point correspond to the x, y, and z coordinate values respectively;
[0064] According to the coordinates of the above key points, the facial dynamic features of mouth width, mouth center, opening distance, distance from mouth to nose, distance from chin to nose, overall facial height, chin opening degree, left smile amplitude, right smile amplitude, mouth corner pull-down amplitude, mouth corner pull-down amplitude, mouth corner horizontal right stretch degree, mouth corner horizontal left stretch degree, mouth center horizontal right stretch degree, mouth center horizontal left stretch degree, distance between lips and lowest lips, vertical distance between upper lip and uppermost point of chin, vertical distance between nose tip and uppermost part of upper lip, expression tension, upper left pressure point, lower left pressure point, left pressure state, upper right pressure point, lower right pressure point, right pressure state. The formula is as follows:
[0065]
[0066] smile_left=upper_lip[1]-mouth_left_corner[1]
[0067] smile_right=upper_lip[1]-mouth_right_corner[1]
[0068] mouth_frown_left=mouth_corner_left-mouth_left_frown
[0069] mouth_frown_right=mouth_corner_right-mouth_right_frown
[0070] mouth_stretch_left=mouth_left_corner[0]-mouth_left_stretch[0]
[0071] mouth_stretch_right=mouth_right_stretch[0]-mouth_right_corner[0]
[0072] mouth_stretch_center_left=mouth_center[0]-mouth_left_stretch[0]
[0073] mouth_stretch_center_right=mouth_center[0]-mouth_right_stretch[0]
[0074]
[0075] upper_lip_nose_dist=nose_tip[1]-uppest_lip[1]
[0076]
[0077]
[0078] Finally, each feature is normalized to represent the magnitude of the feature. Therefore, the input at each moment is represented as a 25-dimensional vector denoted by x t as input for subsequent models.
[0079] (3) Input the processed facial dynamic features into the lip reading recognition model to obtain the pinyin category probability distribution:
[0080] like Figure 4 The lip reading recognition model shown is composed of an encoder and a decoder, wherein: the encoder is composed of a Conformer, which is used to capture the temporal and spatial features of facial dynamic data; the decoder is composed of a Transformer-decoder and a CTC connected in parallel; that is, two copies of the feature vector output by the encoder are made and input into the Transformer-decoder and CTC-decoder modules at the same time, and the linear combination of the two probability distribution outputs is used as the final output of the pinyin category probability distribution.
[0081] (3.1) Conformer encoder takes input sequence X = [x 1 , ..., x T ] for feature extraction, T represents the length of the time step, x t As the 31-dimensional facial dynamic input at time t, the output feature vector Z = [z 1 , ..., Z T ],z t is the eigenvector at time t.
[0082] Specifically, the Conformer model consists of a feedforward module FFN1(x), a self-attention module Self_Attn(x), a convolution module Conv(x), and a second feedforward module FFN2(x). The modules are connected by residuals:
[0083] X 1 =0.5*FFN1(X)+X
[0084] X 2 =Self_Attn(X)+X 1
[0085] X 3 =Conv(X 2)
[0086] Z=0.5*FNN2(X)+X 3
[0087] Among them, the feedforward module FNN2(x) is composed of a normalization layer LayerNorm(x), two linear layers Linear(x), a SiLU activation layer SiLU(x) and two Dropout layers Dropout(x) connected in sequence:
[0088] FNN1(x)=Dropout(Linear2(Dropout(SiLU(Linear1(LayerNorm(x))))))
[0089] The self-attention module Self_Attn(x) consists of a normalization layer LayerNorm(x), a multi-head attention layer MHA(x) and a Dropout layer connected in sequence to form Dropout(x):
[0090] Self_Attn(x)=Dropout(MHA(LayerNorm(x)))
[0091] The convolution module Conv(x) is composed of a normalization layer LayerNorm(x), a point-by-point convolution layer Conv1d(x), a GELU activation layer GLU(x), a depth convolution layer Conv1d(x), a batch normalization layer BatchNorm(x), a SiLU activation layer SiLU(x), a point-by-point convolution layer Conv1d(x) and a Dropout layer Dropout(x) connected in sequence:
[0092] x′=Conv1d(GLU(Conv1d(LayerNorm(x))))
[0093] Conv(x)=Dropout(Conv1d(SiLU(BatchNorm(x′))))
[0094] (3.2) Make two copies of the feature vector Z output by the encoder and input them into the Transformer-decoder and CTC-decoder modules respectively, denoted as Transformer(x) and CTC(x) respectively. The linear combination of the two probability distribution outputs is used as the pinyin category probability distribution D T×n The final output, n is the number of pinyin categories:
[0095] D = Transformer(Z) + CTC(Z)
[0096] Among them, Transformer-decoder is composed of a self-attention module, a multi-head attention module, and a feedforward module connected in sequence, and the modules are connected by residual means; the self-attention module is composed of a self-attention layer and a Dropout layer; the multi-head attention module is composed of a multi-head attention layer and a Dropout layer; the feedforward module is composed of a linear layer, an activation function layer, and a Dropout layer; and the CTC-decoder is composed of a linear layer.
[0097] The final output of the network is the probability distribution of pinyin categories. The specific categories are to classify all Chinese pinyin according to the lip shape during pronunciation, such as Figure 5 As shown in the figure, the specific classification is as follows: (a) initials are divided into 7 categories, including bilabial consonants b\p\m, labiodental consonants f, alveolar fricatives z\c\s, other alveolar consonants d\t\n\l, retroflex consonants zh\ch\sh\r, alveolar-palatal consonants j\q\x, and velar consonants g\k\h; (b) finals are divided into 11 categories, among which open-mouth finals are divided into 4 subcategories, including a\ai\an\ao\ang, o\ou\ong, e\ei\en\eng\er; dental finals are divided into 3 subcategories, including iou\iong, i\in\ing, ia\ie\ian\iang, iao; closed-mouth finals are divided into 3 subcategories, including uei\ueng, u\uo\uai\uei\uan\uen\uang; and pursed-lip finals are divided into 1 subcategory, including ü\üe\üan\ün.
[0098] It should be noted that this classification is only one type of classification based on lip shape. Similar classifications can improve the effect of the lip reading recognition model and are not limited to this specific classification.
[0099] (4) Using a large language model LM ) performs text prediction on the probability distribution of pinyin (D) and generates recognition result Y:
[0100] Y=Model LM (D)
[0101] Where Y is the output text sequence, Y = [y 1 , ..., y L ],y l is the lth character of the output sequence, where L represents the length of the text sequence.
[0102] The lip reading recognition model used in step (3) needs to be trained. The training process is described in detail below:
[0103] (a) Videos of subjects reading designated texts were collected. A total of 21 subjects were included, all of whom had normal language expression abilities. The data collection scenes were selected by the subjects themselves without restriction. The total text read by all subjects was 1 million words.
[0104] (b) Processing the collected data, such as Figure 3 The data set is constructed as follows: First, the same preprocessing method as step (2) is used to obtain processed data. Next, the audio file is separated from the video, and the video content is recognized using an open source speech recognition model to obtain the text corresponding to the video. Finally, the text-to-speech alignment model is used to obtain the time label of each pronunciation phoneme as the label of the data set.
[0105] (c) Input the data set into the network constructed in step (3) and use the true value label to calculate the loss function. The output of Transformer-decoder uses the cross entropy loss function to calculate the loss, and the objective function can be expressed as:
[0106]
[0107] Z refers to the facial dynamic features processed by the Conformer encoder; D is the target phonetic category sequence, L is the length of the sequence; d l is the lth element in the target sequence. <l Represents all elements before the lth element in the target sequence. p(d l |d <l , Z) represents the element sequence d before the given input feature sequence Z and the target element <l Under the condition of , Transformer-decoder infers the lth element d l probability.
[0108] The output of CTC-decoder uses the CTC loss function to calculate the loss, and the objective function can be expressed as:
[0109]
[0110] The overall objective function of the model is a linear combination of two objective functions:
[0111] Loss = α log P CTC (D|Z)+(1-α)P cE (D|Z)
[0112] Among them, α is the relative weight of controlling CTC and attention mechanism.
[0113] (d) Perform back propagation and use GPU to train the model until the model converges.
Claims
1. A Chinese lip reading recognition method based on facial dynamic capture, characterized in that: The following steps are involved: (1) Use a depth camera to collect continuous images of the face to be recognized; (2) Estimating the three-dimensional facial key points of the collected continuous facial images and calculating the facial dynamic features based on the key points; (3) Input the facial dynamic features into the lip reading recognition model to obtain the probability distribution of pinyin categories; (4) Use a large language model to predict text sequences based on the probability distribution of pinyin categories and generate Chinese lip reading recognition results.
2. The Chinese lip reading recognition method based on facial dynamic capture according to claim 1, characterized in that: In step (1), a depth camera is used to collect continuous facial images to be recognized, which specifically includes: A depth camera is used to collect continuous facial images, which include a depth map and a color map.
3. The Chinese lip reading recognition method based on facial dynamic capture according to claim 1, characterized in that: In step (2), three-dimensional facial key points are estimated for the collected continuous facial images, and facial dynamic features are calculated based on the key points, which specifically includes: (2.1) Adjust the collected color image and depth image to a fixed size and normalize the pixel values and depth values; (2.2) Input the color image into the MediaPipe Face Mesh function to estimate the two-dimensional facial key points and obtain the x and y coordinates of 468 facial key points; (2.2) obtaining the depth information corresponding to the facial key points according to the depth map and the x and y coordinates of the 468 facial key points, i.e., 468 z coordinates. The x, y coordinates and the 468 z coordinates of the 468 facial key points form three-dimensional facial key points, i.e., 468 x, y, z coordinates; (2.3) using a filtering and smoothing algorithm to adjust the 468 x, y, and z coordinates to obtain 468 adjusted x, y, and z coordinates; (2.4) Use the adjusted 468 x, y, z coordinates to calculate the dynamic features of the face.
4. The Chinese lip reading recognition method based on facial dynamic capture according to claim 1, characterized in that: In step (3), the lip reading recognition model is composed of an encoder and a decoder connected in sequence; The encoder is composed of a Conformer, which is used to capture the temporal and spatial characteristics of facial dynamic data; The decoder is composed of a Transformer-decoder and a CTC-decoder connected in parallel; The feature vector output by the encoder is copied twice and input into the Transformer-decoder and CTC-decoder at the same time, and the two obtained probability distributions are linearly combined as the final output of the probability distribution of the pinyin category.
5. The Chinese lip reading recognition method based on facial dynamic capture according to claim 1, characterized in that: In step (3), the pinyin categories refer to dividing 21 initial consonants and 36 finals into 11 pinyin categories according to the lip shape changes when pronouncing Chinese pinyin; The lip reading recognition model identifies the probability that the currently pronounced phoneme belongs to each pinyin category.
6. The Chinese lip reading recognition method based on facial dynamic capture according to claim 1, characterized in that: In step (3), the lip reading recognition model needs to be trained on a pre-constructed data set. During the training phase, the output of the Transformer-decoder uses the cross entropy loss function to calculate the loss, and the output of the CTC-decoder uses the CTC loss function to calculate the loss.
7. The Chinese lip reading recognition method based on facial dynamic capture according to claim 1, characterized in that: In step (4), a large language model is used to predict the text sequence according to the probability distribution of the pinyin category to generate the Chinese lip reading recognition result, which specifically includes: The probability distribution of the pinyin categories obtained in step (3) is input into a pre-trained large language model. The model uses context information to predict the most likely text sequence based on the probability distribution of the pinyin categories and generates Chinese lip reading recognition results.
Citation Information
Patent Citations
Lip language recognition method and device, storage medium and mobile terminal
CN108537207A
Multi-mode lip reading method based on facial physiological information
CN110110603A
Speech synthesis system based on face grid
CN116825083A
Lip language recognition method and device
CN117612255A
Image recognition apparatus
JP2005276230A
