Lip language recognition method and device based on space-time network model, medium and product

Abstract

The invention discloses a lip language recognition method and device based on a space-time network model, a medium and a product, and relates to the technical field of lip language recognition. The method comprises the following steps: acquiring a to-be-recognized lip language video image; identifying a lip position in each frame of the to-be-identified lip language video image, and segmenting the corresponding lip image according to the lip position to obtain a plurality of target lip images arranged according to a time sequence; inputting the plurality of target lip images into a lip language recognition model to obtain a lip language prediction result; the lip language recognition model comprises a space-time network model, a gating circulation unit, an attention mechanism and a softmax function; the lip language prediction result is a character expressed by the to-be-recognized lip language video image. The method can improve the recognition precision and detection speed of the lip language.

Description

Technical Field

[0001] The present invention relates to the field of lip reading recognition, and in particular to a lip reading recognition method, device, medium and product based on a spatiotemporal network model. Background Art

[0002] Lip reading recognition technology has gradually become an important means of assisting the deaf and mute to communicate. Existing lip reading recognition technology mainly relies on audio signal processing. This method does not work well in environments with severe noise interference or completely silent scenes. With the widespread application of artificial intelligence technology in the field of computer vision, a new technical path has been provided for lip reading recognition. Current research is gradually turning to video-based lip reading recognition, that is, speech understanding by analyzing lip movements.

[0003] In video-based lip reading recognition, information such as lip shape, texture, and movement are all contained in the spatiotemporal features of the video frames. Although the existing CNN-based lip reading recognition model performs well in extracting the spatial features of lip images, it is still insufficient in processing temporal features. Existing technologies can often only effectively process static images or short time series clips, and it is difficult to capture the complex dynamic changes of lip movements in long time series, especially the limited ability to capture the dynamic changes of continuous lip movements. At the same time, how to efficiently integrate spatial features and temporal features to improve recognition accuracy is still a challenge facing current lip reading recognition technology. In addition, with the increase in the complexity of deep learning models, the training cost increases and the inference speed decreases, which causes large computational overhead and recognition delay in real-time applications of auxiliary communication equipment for the deaf and mute. Summary of the invention

[0004] The purpose of the present invention is to provide a lip reading recognition method, device, medium and product based on a spatiotemporal network model, which can improve the recognition accuracy and detection speed of lip reading.

[0005] To achieve the above object, the present invention provides the following solutions:

[0006] A lip reading recognition method based on a spatiotemporal network model, the method comprising:

[0007] Obtain the lip reading video image to be recognized;

[0008] Identify the lip position in each frame of the lip reading video image to be identified, and segment the corresponding lip image according to the lip position to obtain a plurality of target lip images arranged in chronological order;

[0009] The multiple target lip images are input into a lip reading recognition model to obtain a lip reading prediction result; the lip reading recognition model includes a time shift module, a gated recurrent unit, an attention mechanism and a softmax function; the lip reading prediction result is the text expressed by the lip reading video image to be recognized.

[0010] Optionally, identifying the lip position in each frame of the lip reading video image to be identified, and segmenting the corresponding lip image according to the lip position to obtain a plurality of target lip images arranged in chronological order, specifically includes:

[0011] Performing frame cutting processing on the lip reading video image to be recognized to obtain a plurality of frame images arranged in chronological order;

[0012] The lip position in the frame image is identified, and the corresponding lip image is cut to obtain a plurality of target lip images arranged in time sequence.

[0013] Optionally, the YOLOv7 algorithm is applied to identify the lip position in the frame image, and the corresponding lip image is cut to obtain a plurality of target lip images arranged in chronological order.

[0014] Optionally, the training process of the spatiotemporal network model specifically includes:

[0015] Identify the lip position in each frame of the sample lip reading video image, and segment the corresponding lip image according to the lip position to obtain a plurality of sample target lip images arranged in time sequence for each of the sample characters;

[0016] The lip reading recognition model is trained with the multiple sample target lip images as input and the corresponding sample texts as output to obtain the lip reading recognition model.

[0017] Optionally, the time shift module is obtained by inserting the time shift module into the residual block of ResNet50.

[0018] Optionally, the time shift module includes an initial convolution layer, a maximum pooling layer, a ResNet50 first residual block, a TSM first module, a ResNet50 second residual block, a TSM second module, a ResNet50 third residual block, a TSM third module and a global average pooling layer connected in sequence.

[0019] Optionally, the output end of the time shift module is connected to the input end of the gated recurrent unit; the output end of the gated recurrent unit is connected to the input end of the attention mechanism; and the output end of the attention mechanism is connected to the softmax function.

[0020] A computer device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement any one of the above-mentioned lip reading recognition methods based on a spatiotemporal network model.

[0021] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements any of the above-mentioned lip reading recognition methods based on a spatiotemporal network model.

[0022] A computer program product comprises a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned lip reading recognition methods based on a spatiotemporal network model.

[0023] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0024] The present invention discloses a lip reading recognition method, device, medium and product based on a spatiotemporal network model. The method obtains and identifies the lip position in each frame of a lip reading video image to be recognized, segments the corresponding lip image, obtains a plurality of target lip images arranged in chronological order, and then inputs the plurality of target lip images into a lip reading recognition model to obtain a lip reading prediction result, wherein the lip reading recognition model includes a time shift module, a gated recurrent unit, an attention mechanism and a softmax function; and the lip reading prediction result is the text expressed by the lip reading video image to be recognized. The present invention adds a time shift module to the lip reading recognition model, and introduces GRU and AM on the basis of the time shift module, so that the spatiotemporal network model has a stronger ability to capture the spatiotemporal features of dynamic lip readings. In addition, the Softmax function is used to normalize the feature vector and accurately classify it into the corresponding lip reading category, so that the spatiotemporal network model performs well in multi-category lip reading recognition tasks, and improves the recognition accuracy and detection speed of lip readings. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0026] Figure 1 This is a schematic diagram of the workflow of lip reading recognition based on the spatiotemporal network model;

[0027] Figure 2 This is a schematic diagram of the workflow for lip detection and segmentation based on YOLOv7;

[0028] Figure 3 This is a schematic diagram of the workflow for spatiotemporal feature extraction based on TSM-ResNet50;

[0029] Figure 4 This is a schematic diagram of the workflow for identifying continuous lip movements based on GRU;

[0030] Figure 5 The schematic diagram of the workflow for identifying key lip movements based on AM;

[0031] Figure 6 This is a diagram of the visualization effect of lip reading recognition based on Flask;

[0032] Figure 7 This is a schematic diagram of the lip reading recognition effect of a video stream file;

[0033] Figure 8 This is a schematic diagram of the lip reading recognition effect of a real-time video stream file;

[0034] Fig. 9 This is a schematic diagram of the effect of real-time lip reading recognition under noise interference;

[0035] Fig.10 It is a flowchart of a lip reading recognition method based on a spatiotemporal network model;

[0036] Fig.11 This is a diagram of the internal structure of a computer device. DETAILED DESCRIPTION

[0037] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0038] The purpose of the present invention is to provide a lip reading recognition method, device, medium and product based on a spatiotemporal network model, aiming to improve the recognition accuracy and detection speed of lip reading.

[0039] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0040] In order to solve the problem that spatiotemporal features are difficult to effectively fuse and long-term dependent features are insufficiently extracted in lip reading recognition, the present invention proposes a lip reading recognition method based on the TSM-ResNet50 spatiotemporal network model. The temporal features of the lip image are effectively fused through the TSM time displacement module, and the spatial features extracted by ResNet50 are combined to improve the spatiotemporal representation ability of lip movement. The long-term and short-term dependencies of lip movements in the time series are captured through the GRU gating mechanism to avoid the problem of gradient disappearance in traditional RNN and LSTM models. The AM attention mechanism is used to strengthen the model's attention to key frames, and the feature selection ability of the model in long time series is improved, so as to more accurately capture the subtle changes of lip movements, and can efficiently and accurately capture the dynamic changes of the lips, effectively solving the technical deficiencies of the existing lip reading recognition method. In the case of complex background and large interference, it still has high recognition accuracy and fast detection speed for dynamic lip reading recognition in real-time video. In addition, the lip reading recognition method based on the spatiotemporal network model provided by the present invention adopts a B / S architecture and uses a lightweight Flask framework to realize the visualization of lip reading recognition. Through the above solution, the APP or recognition system obtained by applying the present invention can effectively support lip reading recognition in offline and online real-time videos, and convert lip reading into text output, which greatly improves the user experience and will have broad application prospects in the fields of human-computer interaction, special education, social welfare, etc.

[0041] Example 1

[0042] like Fig.10 As shown, the lip reading recognition method based on the spatiotemporal network model in this embodiment includes:

[0043] S1: Obtain the lip reading video image to be recognized.

[0044] S2: Identify the lip position in each frame of the lip reading video image to be identified, and segment the corresponding lip image according to the lip position to obtain a plurality of target lip images arranged in chronological order.

[0045] S2 specifically includes:

[0046] S21: performing frame cutting processing on the lip reading video image to be recognized to obtain a plurality of frame images arranged in chronological order.

[0047] S22: Identify the lip position in the frame image, and cut the corresponding lip image to obtain a plurality of target lip images arranged in chronological order.

[0048] Furthermore, the YOLOv7 algorithm is applied to identify the lip position in the frame image, and the corresponding lip image is cut to obtain a plurality of target lip images arranged in chronological order.

[0049] In practical applications, lip detection and segmentation are implemented based on YOLOv7. YOLOv7 is an efficient target detection algorithm that performs well in real-time and detection accuracy and is suitable for lip detection and segmentation tasks. In the present invention, YOLOv7 is used to accurately detect the lip area in the video frame and efficiently segment the lips to extract effective lip features. The workflow of implementing lip detection and segmentation based on YOLOv7 is as follows: Figure 2 The specific instructions are as follows:

[0050] 1. Perform preliminary feature extraction on the input frame image through the convolution layer (Conv) to generate a feature map, and generate the corresponding bounding box by detecting the lip area.

[0051] 2. Through multiple efficient layer aggregation networks (ELAN, Efficient Layer Aggregation Networks) and multi-path convolution (MPConv, Multi-Path Convolution) modules, YOLOv7 is able to perform deeper and more diverse feature extraction, which helps to capture the detailed features of the lips and ensure the diversity and depth of features.

[0052] 3. The extracted lip features are upsampled and downsampled respectively, and the feature maps of different resolutions are spliced ​​and fused through the Cat operation. This multi-scale feature fusion process can effectively capture the detailed features of the lips at different scales, ensuring that both large-scale lip movements and subtle lip movements can be accurately detected.

[0053] 4. Spatial Pyramid Pooling (SPP) is used to further improve the recognition ability of lip movements of different scales, which helps YOLOv7 capture global information and avoid the loss of important information, thereby enhancing the understanding of the overall lip shape and improving the perception of lip movements.

[0054] 5. After fusing the feature maps extracted and processed by different modules, the Recurrent Expanded Pyramid (REP) module is used to optimize the features, and the convolutional layer is used to further refine the features, and finally generate the feature map for prediction.

[0055] S3: Input the multiple target lip images into a lip reading recognition model to obtain a lip reading prediction result; the lip reading recognition model includes a time shift module, a gated recurrent unit, an attention mechanism and a softmax function; the lip reading prediction result is the text expressed by the lip reading video image to be recognized.

[0056] The training process of the spatiotemporal network model specifically includes:

[0057] The lip position in each frame of the sample lip reading video image is identified, and the corresponding lip image is segmented according to the lip position to obtain a plurality of sample target lip images arranged in chronological order for each of the sample characters.

[0058] The lip reading recognition model is trained with the multiple sample target lip images as input and the corresponding sample texts as output to obtain the lip reading recognition model.

[0059] During training, the YOLOv7 and TSM-ResNet50 models are trained separately. The YOLOv7 model is trained to detect and segment the lip position. The TSM-ResNet50 spatiotemporal network model with GRU and AM is trained to perform lip reading recognition. The trained models are connected and then applied directly.

[0060] As a specific implementation, the output end of the time shift module is connected to the input end of the gated recurrent unit; the output end of the gated recurrent unit is connected to the input end of the attention mechanism; and the output end of the attention mechanism is connected to the softmax function.

[0061] The time shift module is obtained by inserting the time shift module into the residual block of ResNet50.

[0062] Specifically, the time shift module includes an initial convolution layer, a maximum pooling layer, a ResNet50 first residual block, a TSM first module, a ResNet50 second residual block, a TSM second module, a ResNet50 third residual block, a TSM third module and a global average pooling layer connected in sequence.

[0063] In practical applications, first of all, spatiotemporal feature extraction is performed based on TSM-ResNet50. As a powerful deep convolutional neural network, ResNet50 can effectively capture the spatial features of lip images. In order to achieve spatiotemporal fusion of lip features, the TSM module is embedded in ResNet50 to capture temporal features while extracting lip spatial features, giving it richer spatiotemporal expression capabilities. The workflow for spatiotemporal feature extraction of lip images based on TSM-ResNet50 is as follows: Figure 3 (a), the specific description is as follows:

[0064] 1. Input the lip frame image output by the YOLOv7 model into the ResNet50 model and resize the image to 224×224 to meet the input requirements of ResNet50.

[0065] 2. ResNet50 first captures low-level features such as lip edges and textures through the initial convolution layer (Conv), and then downsamples through the maximum pooling layer (Max Pooling) to reduce the amount of calculation and aggregate the initial features.

[0066] 3. ResNet50 consists of multiple residual blocks, which extract advanced features such as lip motion and complex changes in mouth shape layer by layer. By embedding the time shift module TSM, spatial features and temporal features are gradually integrated in the entire network.

[0067] 4. At the end of feature extraction, the spatiotemporal information of each feature map is compressed into a global feature vector through the global average pooling layer (GAP), and the high-dimensional feature map is converted into a spatiotemporal feature vector of fixed dimension.

[0068] like Figure 3 As shown in (b), the ResNet50 residual block consists of three convolutional layers (Conv), with convolution kernel sizes of 1×1, 3×3, and 1×1, respectively, and the number of output channels is 64, and the dimension of the final output feature is 256. The features output by the maximum pooling layer (Max Pooling) are first extracted by the convolutional layer (Conv), and then nonlinearly transformed by the Relu activation function to enhance the feature representation capability. At the same time, the features output by the maximum pooling layer (Max Pooling) are bypassed by the convolutional layer through the skip connection (Skip Connection) and directly transmitted to the output end to avoid information loss. By adding the output result of the convolutional layer to the output feature of the maximum pooling layer (Max Pooling), a residual connection is formed, which not only retains the information of the original input, but also increases the features extracted by the convolution. Finally, the added features are passed to the time shift module TSM of the next layer to further process the timing information.

[0069] like Figure 3As shown in (c), the TSM module contains three dimensions: time T, space H, W, and channel C. Among them, the time dimension T represents the time step in the lip frame sequence. Different time steps correspond to different frames. This is the key dimension to capture the temporal changes of lip movements; the height H of the spatial dimension represents the vertical size of the lip area in each frame image, that is, the height of the feature map; the width W of the spatial dimension represents the horizontal size of the lip area in each frame image, that is, the width of the feature map. When a part of the features in the channel is shifted in time, "information exchange" will occur on the time dimension T, so that each frame not only contains the spatial information of the current moment, but also integrates the temporal information of the previous and next moments. For the shifted features, the range beyond the time series is filled with zeros. When moving forward or backward, some features will exceed the boundary of the current time series and "cross the border". These excess parts will be truncated to ensure that the dimension and order of the data remain unchanged. This simple operation introduces temporal dependency with minimal computational overhead and improves the model's ability to capture dynamic temporal features. These dimensions together describe the structure of the feature map in time and space. Combined with the time shift and zero padding mechanism, the TSM module can capture the time and space features and perform feature shift and fusion. Compared with traditional 3D convolution, TSM does not increase additional computational costs or parameters, and can perform efficient and accurate lip time feature extraction.

[0070] Next, continuous lip movement recognition is performed based on GRU. Since TSM mainly captures the short-term temporal relationship of lip frame images in video sequences, in order to avoid information loss in long-term sequences, the gated recurrent unit GRU is introduced to capture the long-term dependency of lip movement temporal information. GRU mainly introduces a reset gate r t and update gate z t Two gating mechanisms realize the selective memory and forgetting of input information, making TSM-ResNet50 more robust in lip reading recognition. Compared with the more complex long short-term memory network (LSTM, Long Short-term Memory), GRU has a simpler structure and fewer parameters, which can reduce the computational cost while still maintaining excellent performance. Figure 4 The specific instructions are as follows:

[0071] 1. The spatiotemporal feature vector x output by the TSM-ResNet50 model t Input into GRU to represent the spatiotemporal feature information of the lips in the current frame. Among them, ResNet50 outputs spatial features and TSM outputs temporal features. The TSM-ResNet50 model outputs a spatiotemporal feature vector that combines temporal and spatial features. The spatiotemporal feature vector x t It is composed of temporal features and spatial features. The hidden state h of the previous time step of GRU t-1, used to represent GRU’s memory of previous lip movements, h t-1 The spatiotemporal feature information of the lips in the past frame is retained, and then combined with the current input frame x t Updating helps the GRU model learn the dependencies between lip movements in time series.

[0072] 2. Reset gate r t Used to control whether to ignore the hidden state h of the previous frame t-1 When r t When it is close to 0, it means that the reset gate tends to ignore the hidden state of the previous time step, that is, the GRU model does not need to consider the historical lip motion information of the previous frame at the current time step, and only relies on the input feature x of the current frame t Vector. The calculation formula for the reset gate is:

[0073] r t =σ(W r ·[h t-1 ,x t ]) (1)

[0074] In formula (1), σ represents σ(x)=1 / (1+e -x ) function, which compresses the input to between 0 and 1. r Represents the weight matrix of the reset gate, which is used to reset the current input x t and the previous hidden state h t-1 Perform a linear transformation to generate the activation value of the reset gate. t-1 ,x t ] means concatenating the hidden state of the previous time step and the input features of the current time step. By combining the two, the reset gate can decide how to deal with the relationship between the new input and the previous hidden state.

[0075] 3. Update gate z t Determines how much hidden state information h of the previous time step should be retained by GRU t-1 , and how much input feature information x of the current frame is fused t For continuous lip action sequences, the update gate can help the GRU model retain important information from previous frames for a long time. The calculation formula of the update gate is:

[0076] z t =σ(W z ·[h t-1 ,x t ]) (2)

[0077] In formula (2), W z Represents the weight matrix of the update gate, which is used to update the current input x t and the previous hidden state h t-1Perform a linear transformation to generate the activation value of the update gate.

[0078] 4. The candidate hidden state is a key step for GRU to understand and characterize the dynamics of continuous lip movements, helping the model to retain or update action features in a long time series, thereby improving recognition accuracy. t The hidden state h t-1 and the current input x t Working together, the candidate hidden state is calculated It can be expressed as a combination of the lip motion features of the current frame and the information of the previous frame. The candidate hidden state calculation formula is:

[0079]

[0080] In formula (3), [r t *h t-1 ,x t ] means to reset the previous hidden state and the current input x t Splice to form a new input, and then pass through the weight matrix W h Perform linear combination to generate candidate hidden states. h Represents the weight matrix for generating candidate hidden states for the current input x t and the previous hidden state h t-1 Perform linear transformation. Tanh means tanh(x)=(e x -e -x ) / (e x +e -x ), which is used to map the linearly combined values ​​to the interval [-1,1]; by increasing the nonlinear ability, it helps GRU capture more complex lip relationships.

[0081] 5. Hidden state h t is the candidate hidden state of the current frame and the hidden state h at the previous time step t-1 The weighted sum of t Control the ratio of the two. This process allows GRU to dynamically adjust the degree of dependence of each frame on the previous and next frames when processing the continuous movement of the lips. The hidden state calculation formula is:

[0082] h t =z t *h t-1 +(1-z t )*h t (4)

[0083] 6. Hidden state h at each time step t tIt contains the lip movement information of the current frame and the previous frame, and retains the long-term dependencies in the time series. These hidden states will be passed to the next time step, gradually forming the spatiotemporal dynamic features of the entire lip sequence.

[0084] Then, key lip movement recognition is performed based on AM. The AM attention mechanism can improve the performance of the model in long sequence tasks by weighting the importance of different frames, making it easier to capture the key lip movement features in lip movements. t As query Q, the hidden states of all previous time steps are used as keys K i , the hidden state corresponding to the key is taken as the value V i , where i = 1, 2, 3...n. Through this mechanism, the TSM-ResNet50 model can find the correlation between the current moment and other moments in the entire time series and capture the key lip movement features. The AM mechanism indirectly optimizes the spatiotemporal features generated by TSM-ResNet50, making the final lip reading recognition more accurate and effective. Although AM is not directly connected to TSM-ResNet50, the weights of key lip movement features are further improved by optimizing the GRU output, making the final classification stage more robust and accurate. The final classification result is based on the features processed by AM, and these features originally come from the spatiotemporal features of TSM-ResNet50. Therefore, the AM mechanism indirectly improves the feature expression effect of TSM-ResNet50. The workflow for identifying key lip movements based on AM is as follows: Figure 5 The specific instructions are as follows:

[0085] Step 1: Similarity score S i Measures the correlation between the current moment and the previous moment's lip movement F(Q,K i ), the calculation formula is:

[0086] S i =F(Q,K i )=QK i T (5)

[0087] In formula (5), is the key vector K i The transpose operation is to make Q and K i The dimensions of are matched so that matrix multiplication can be performed.

[0088] Step 2: Set the similarity score S i The Softmax operation is used to normalize the value range to between 0 and 1, and a iRepresents the different weight coefficients after normalization, that is, the degree to which the query Q focuses on a given time step. Normalizing the similarity scores to values ​​between 0 and 1 allows the TSM-ResNet50 model to more clearly determine which lip frames or lip spatiotemporal features should be focused on. The weight coefficient calculation formula is:

[0089]

[0090] Step 3: According to the weight coefficient a i V i Perform weighted summation to obtain the final attention value Attention(a i ,V i ), which is the output spatiotemporal feature vector after the attention mechanism calculation, integrating the information of different lip positions and reflecting the part that the TSM-ResNet50 model pays most attention to. The calculation formula is:

[0091]

[0092] Finally, lip reading classification and recognition is performed based on Softmax. After AM outputs the final spatiotemporal feature vector, the complex spatiotemporal features need to be mapped to lip reading categories and the lip reading classification and recognition are performed through the Softmax function. The specific instructions are as follows:

[0093] 1. Obtain the spatiotemporal feature vector output by AM, which contains weighted information of different time steps and spatial positions and represents the most important lip movement features in the video frame.

[0094] 2. Input the spatiotemporal feature vector into the fully connected layer to map the high-dimensional spatiotemporal feature vector to the dimensions related to the classification task. Each dimension represents the score of a category. The score calculation formula is as follows:

[0095] Z=W·A+b (8)

[0096] In formula (8), A represents the spatiotemporal feature vector output by AM, W represents the weight matrix of the fully connected layer, and b represents the bias vector of the fully connected layer.

[0097] 3. Normalize the output of the fully connected layer through the Softmax function to obtain the probability distribution of each category. The Softmax function converts the score into a probability ranging from 0 to 1, and the sum of the probabilities of all categories is 1.

[0098] 4. Output the probability of each lip reading category through the Softmax function, and select the category with the highest probability as the final lip reading recognition result.

[0099] In addition, the present invention also realizes the visualization of lip reading recognition results based on Flask, which can ensure the real-time interaction between the front-end user and the back-end lip reading recognition system, and conveniently and clearly view the results of lip reading recognition, thereby forming a closed loop from video input to visualization result display. Figure 6 The specific instructions are as follows:

[0100] 1. The Flask backend defines routes and Uniform Resource Locators (URLs). When a user uploads a video through the front-end page or opens the camera to initiate a lip reading recognition request, the URL will process the request.

[0101] 2. Perform frame preprocessing on the video files uploaded by users or the real-time videos initiated by the backend, call YOLOv7 for lip detection and segmentation, pass the segmented lip frame images into the TSM-ResNet50, GRU, and AM models for lip reading recognition, and output the lip reading recognition results.

[0102] 3. After the lip reading recognition results generated by the backend are passed to the front-end page through Flask's template rendering function, they will be dynamically displayed in the designated area in the form of text for users to view.

[0103] In summary, in the lip reading recognition scenario, time information refers to the temporal features of the lips in continuous video frames, and spatial information refers to the spatial features of the lips in a single video frame. The present invention constructs a spatiotemporal network model of the Temporal Shift Module-ResidualNetwork50 (TSM-ResNet50) architecture, and comprehensively utilizes the feature information of both time and space dimensions to understand lip movements, thereby achieving more accurate and efficient recognition of dynamic lip readings in real-time videos.

[0104] Workflow such as Figure 1As shown in the figure, after the video is frame-cut, the YOLOv7 algorithm is used to accurately detect and quickly segment the lip position in the frame image. The powerful feature extraction and stable training performance of ResNet50 are used to extract more abstract and high-level lip spatial features. By inserting TSM into the residual block of ResNet50, the short-term temporal relationship in the video sequence is captured to enhance the ability of the ResNet50 network to extract temporal features. The gated recurrent unit (GRU) is introduced to process the extracted spatiotemporal features. By capturing the long-term dependencies in the video sequence, the network's long-term temporal modeling ability is improved, which helps the TSM-ResNet50 model dynamically adjust the internal state according to the context of the sequence, thereby more accurately identifying continuous lip movements. The application of attention mechanism (AM) focuses on the key frames with the most information in the sequence, so that the TSM-ResNet50 model can focus on the lip movements that are most critical to understanding the current word or phrase, and adjust the attention to different frames according to the current context state, so as to more effectively capture the semantically coherent lip movement patterns, and can accurately and efficiently deeply understand and recognize lip movements. Finally, the classification verification is performed through the Softmax function, the lip reading prediction result with the maximum probability value is output, and the visualization of the lip reading recognition results is realized based on the lightweight Flask framework.

[0105] The present invention is based on machine vision and artificial intelligence technology. It can overcome the influence of adverse factors such as image angle, light intensity and facial expression under conditions of complex background, large interference and strong noise. It has higher recognition accuracy, faster detection speed and stronger adaptability for dynamic lip reading in real-time video, and will have broad application prospects in human-computer interaction, special education, social welfare and other fields.

[0106] For dynamic lip reading recognition in real-time video, this method has the advantages of higher recognition accuracy, faster detection speed and stronger adaptability. The specific description is as follows:

[0107] 1. In order to accurately detect the lip area in the video frame and efficiently segment the lips, the Xinwang Bank lip reading dataset is used to compare and analyze the current mainstream target detection algorithms YOLOv7, Faster-RCNN, YOLOv3-spp, and SSD300. As shown in Table 1, considering the performance in four aspects, namely, inference speed, model parameter quantity, confidence, and accuracy, YOLOv7 has the fastest inference speed of 2.4ms, which is at least 1.9ms faster than other algorithms; the number of parameters of YOLOv7 is only 7.5M, which is significantly reduced compared with other algorithms; although the confidence and accuracy are 0.9 and 56.2 respectively, for the detection and segmentation tasks of the lip area in real-time video frames, the inference speed determines whether the model can keep up with the frame rate requirements, and the number of model parameters determines the complexity of the model and the computing resources required during operation. Therefore, in real-time lip detection and segmentation tasks, inference speed and model parameter quantity are more critical.

[0108] Table 1. Comparison of target detection algorithm performance

[0109]

[0110] Through multi-level feature extraction and multi-scale fusion, YOLOv7 can ensure that lip movements of different scales and complexities are accurately captured and analyzed, thereby accurately detecting the lip area and generating a clear bounding box. In addition, based on a lightweight network structure, efficient computing methods, and an optimized inference engine, YOLOv7 can quickly segment the lip area and provide rich and high-quality lip features for subsequent action recognition. Therefore, choosing YOLOv7 for lip detection and segmentation has higher efficiency while maintaining high accuracy.

[0111] 2. Higher lip reading recognition accuracy.

[0112] The feature extraction network is crucial to the accuracy of lip reading recognition. As shown in Table 2, the TSM-ResNet50 spatiotemporal network model proposed in this method is compared with the current mainstream feature extraction networks VGGM, SE-ResNet-18, and SE-DenseNet-121. It can be seen that VGGM has a relatively weak ability to extract visual features of continuous lip frame images, and its recognition accuracy is only 62.976%; SE-ResNet-18 and SE-DenseNet-121 have recognition accuracies of 84.424% and 85.572% respectively; TSM-ResNet50 efficiently uses feature information in both time and space dimensions to understand lip movements, which can increase the recognition accuracy to 88.467%.

[0113] Table 2 Comparison of recognition accuracy of different feature extraction networks

[0114]

[0115] TSM effectively captures dynamic timing information through lightweight time shift operations, effectively improving efficiency without adding additional computational overhead; ResNet50 extracts rich spatial features through a deep residual network. The combination of the two can simultaneously capture the temporal changes and spatial details of lip movements without adding additional computational overhead, and achieve deep fusion of spatiotemporal information. At the same time, the GRU gating mechanism is introduced to capture the long-term dependencies of lip movements in time series, ensuring that TSM-ResNet50 comprehensively considers long-term and short-term time features, so that it can effectively process time series data of continuous lip movements; AM dynamically adjusts the degree of attention paid to different parts when TSM-ResNet50 processes the sequence, thereby effectively capturing the key moment features in lip movements. In summary, the method proposed in the present invention has significant advantages in lip reading recognition accuracy.

[0116] 3. Stronger model adaptability. The model adaptability in lip reading recognition tasks is mainly reflected in the following aspects:

[0117] (1) Adaptability to diverse lip movements. In lip reading recognition, different people have different facial structures and speaking habits. The model needs to adapt to a variety of lip shapes, movement amplitudes, and lip reading pronunciation methods. The video stream file and the real-time video stream file were tested for lip reading recognition of girls and boys. The test cases were: "It will rain at noon today" and "Remember to bring an umbrella". The lip reading recognition results are as follows: Figure 7 , Figure 8 As shown. Figure 7 , Figure 8 It can be seen that the TSM-ResNet50 model can effectively adapt to the differences in facial structure and speaking habits of different subjects, and has high-accuracy lip reading recognition for various lip movements. The main reason is that the present invention introduces GRU and AM on the basis of the TSM-ResNet50 model, which makes the TSM-ResNet50 model more capable of capturing the spatiotemporal characteristics of dynamic lip readings. In addition, the Softmax function is used to normalize the feature vectors and accurately classify them into corresponding lip reading categories, so that the TSM-ResNet50 model performs well in multi-category lip reading recognition tasks.

[0118] (2) Noise resistance in multi-modal input. In the case of video and audio input with background noise interference, a real-time video test was conducted on the question "How will the weather be tomorrow?". The recognition results are as follows: Fig. 9 As shown, it can be seen that this method can effectively combine visual and audio multimodal information, supplement it with lip visual features, and continue to maintain a high recognition accuracy.

[0119] In summary, the TSM-ResNet50 model can adapt to a variety of environments and individual differences, and can still maintain high accuracy and stability under multimodal input. Therefore, this method can cope with complex real-world scenarios and maintain efficient and accurate recognition capabilities in dynamically changing lip reading recognition scenarios.

[0120] Example 2

[0121] A computer device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the lip reading recognition method based on a spatiotemporal network model in Example 1.

[0122] Example 3

[0123] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the lip reading recognition method based on a spatiotemporal network model in Example 1.

[0124] Example 4

[0125] A computer program product includes a computer program, which implements the lip reading recognition method based on a spatiotemporal network model in embodiment 1 when executed by a processor.

[0126] Example 5

[0127] A computer device, which may be a database, may have an internal structure as shown in Fig.11 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store pending transactions. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the lip reading recognition method based on the spatiotemporal network model in Example 1 is implemented.

[0128] It should be noted that the object information (including but not limited to object device information, object personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the present invention are all information and data authorized by the object or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0129] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided by the present invention can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided by the present invention may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited thereto. The processor involved in each embodiment provided by the present invention may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited thereto.

[0130] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0131] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A lip reading recognition method based on a spatiotemporal network model, characterized in that: The method comprises: Obtain the lip reading video image to be recognized; Identify the lip position in each frame of the lip reading video image to be identified, and segment the corresponding lip image according to the lip position to obtain a plurality of target lip images arranged in chronological order; The multiple target lip images are input into a lip reading recognition model to obtain a lip reading prediction result; the lip reading recognition model is obtained by training a spatiotemporal network model using a training data set; the spatiotemporal network model includes a time shift module, a gated recurrent unit, an attention mechanism and a softmax function; the lip reading prediction result is the text expressed by the lip reading video image to be recognized; the training data set is a sample text data set and a sample lip reading video image of each sample text in the sample text data set.

2. The lip reading recognition method based on the spatiotemporal network model according to claim 1, characterized in that: Identifying the lip position in each frame of the lip reading video image to be identified, and segmenting the corresponding lip image according to the lip position to obtain a plurality of target lip images arranged in chronological order, specifically comprising: Performing frame cutting processing on the lip reading video image to be recognized to obtain a plurality of frame images arranged in chronological order; The lip position in the frame image is identified, and the corresponding lip image is cut to obtain a plurality of target lip images arranged in time sequence.

3. The lip reading recognition method based on the spatiotemporal network model according to claim 2, characterized in that: The YOLOv7 algorithm is applied to identify the lip position in the frame image, and the corresponding lip image is cut to obtain a plurality of target lip images arranged in chronological order.

4. The lip reading recognition method based on the spatiotemporal network model according to claim 1, characterized in that: The training process of the spatiotemporal network model specifically includes: Identify the lip position in each frame of the sample lip reading video image, and segment the corresponding lip image according to the lip position to obtain a plurality of sample target lip images arranged in time sequence for each of the sample characters; The lip reading recognition model is trained with the multiple sample target lip images as input and the corresponding sample texts as output to obtain the lip reading recognition model.

5. The lip reading recognition method based on the spatiotemporal network model according to claim 1, characterized in that: The time shift module is obtained by inserting the time shift module into the residual block of ResNet50.

6. The lip reading recognition method based on the spatiotemporal network model according to claim 1, characterized in that: The time shift module includes an initial convolutional layer, a maximum pooling layer, a ResNet50 first residual block, a TSM first module, a ResNet50 second residual block, a TSM second module, a ResNet50 third residual block, a TSM third module and a global average pooling layer connected in sequence.

7. The lip reading recognition method based on the spatiotemporal network model according to claim 1, characterized in that: The output end of the time shift module is connected to the input end of the gated recurrent unit; the output end of the gated recurrent unit is connected to the input end of the attention mechanism; and the output end of the attention mechanism is connected to the softmax function.

8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the lip reading recognition method based on the spatiotemporal network model as described in any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the lip reading recognition method based on the spatiotemporal network model described in any one of claims 1 to 7 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the lip reading recognition method based on the spatiotemporal network model described in any one of claims 1 to 7 is implemented.

Images