Efficient dense prediction visual language recognition algorithm based on multi-scale Transform

Abstract

The invention discloses an efficient dense prediction visual speech recognition algorithm based on a multi-scale Transform. According to the algorithm, an innovative MTF-TCN network architecture is provided for solving the problems that an existing visual speech recognition technology is insufficient in the aspect of capturing a long-range dependency relationship and local high-frequency features and the calculation complexity is high. By combining parallel convolution of an Inception mix structure, a maximum pooling path and a self-attention path, high-frequency and low-frequency information in visual data is effectively learned, and spectrum components on different layers are balanced through a frequency slope structure. Moreover, the introduced sparse attention mechanism and time convolution technology reduce the calculation complexity of the model, maintain the sensitivity to key visual features, and introduce inductive deviation of local connectivity for the model, thereby enhancing the extraction capability of high locality features, and further optimizing the performance and calculation efficiency of the model. The method has the beneficial effects that the accuracy and robustness of visual speech recognition are improved, the calculation cost of the algorithm is reduced, and the method is suitable for speech recognition tasks in various noise environments.

Description

Technical Field

[0001] The present invention relates to the fields of visual speech recognition and audio-visual language recognition, and in particular to an algorithm for analyzing and understanding human lip movements in video clips. Background Art

[0002] Visual speech recognition, also known as visual speech reading or lip reading technology, is a technology that recognizes speech information by analyzing the speaker's lip movements and other facial features. This technology is particularly important in high-noise environments or when the audio signal quality is degraded, as it provides an alternative means of understanding speech that does not rely on audio input.

[0003] Traditional visual speech recognition systems are usually based on manual feature extraction and traditional machine learning algorithms such as Hidden Markov Model (HMM). However, with the rise of deep learning, methods based on deep neural networks have become mainstream, which can automatically learn complex feature representations in data.

[0004] Currently, most deep learning methods use temporal convolutional networks (TCNs) as the backbone network to handle visual speech recognition tasks. TCNs capture the temporal relationship between frames through temporal convolution, especially the 1D dilated convolution technique, which expands the receptive field by increasing the dilation factor layer by layer to capture long-term dependencies. However, these methods have limitations in extracting global features and constructing long-range dependencies, and their generalization ability is limited.

[0005] Furthermore, although the Transformer architecture has attracted much attention due to its success in natural language processing, its application in visual tasks, especially in visual speech recognition, remains challenging. The original Transformer model relies on a self-attention mechanism to capture global dependencies, but this mechanism is insufficient in terms of computational efficiency and capturing high-frequency local features when processing visual data.

[0006] To solve these problems, the present invention proposes an efficient visual speech recognition algorithm based on multi-scale Transformer, which can not only capture high-frequency and low-frequency information in visual data, but also significantly improve the computational efficiency and the ability to extract local features by introducing sparse attention mechanism and temporal convolution, thus achieving a breakthrough in visual speech recognition tasks. Summary of the invention

[0007] 1. Technical issues to be solved The purpose of the present invention is to solve the deficiencies in existing visual speech recognition technology in capturing long-range dependencies, extracting and comprehensively utilizing global low-frequency features and local high-frequency features, and computational efficiency, and to propose a dense prediction visual speech recognition algorithm based on a multi-scale Transformer.

[0008] 2. Technical solution

[0009] In order to achieve the above object, the present invention adopts the following technical solutions: Preferably, data preprocessing is performed: the present invention first preprocesses the acquired lip reading data, including cleaning, normalization and enhancement of video data and audio data, to improve the accuracy and robustness of subsequent feature extraction. Face detection and face alignment are performed for each video sequence in the video data set, each frame is aligned with the reference average face shape, a fixed 96×96 pixel wide region of interest (ROI) is cropped from the aligned face image, so that the mouth area is always roughly centered in the image cropping area, and the cropped image is converted to grayscale.

[0010] Preferably, the Neural Architecture Search (NAS) technology of EfficientNetV2 is used to automatically search for the optimal backbone network architecture, including determining hyperparameters such as network depth, bandwidth, and resolution, to achieve the optimal balance of feature extraction, and can significantly reduce the amount of model parameters and improve the model inference speed.

[0011] Preferably, the present invention adopts a multi-scale Transformer structure, and processes convolution, maximum pooling path and self-attention path in parallel through Transformer modules of different scales to effectively learn high-frequency and low-frequency information in visual data.

[0012] Preferably, a sparse attention mechanism is introduced to reduce the computational complexity of the model by reducing unnecessary calculations while maintaining the ability to capture key visual features.

[0013] Preferably, extended temporal convolution is used as the feed-forward layer in the Transformer block, replacing the traditional fully connected layer, introducing the inductive bias of local connectivity and enhancing the model's extraction of local high-frequency features.

[0014] Preferably, the preprocessed data is input into the backbone network of 3D convolution + EfficientNetV2 to extract visual features, then input into MTF-TCN to extract temporal features, and finally input into the fully connected layer for classification to obtain the final detection result.

[0015] 3. Beneficial effects

[0016] Compared with the prior art, the advantages of the present invention are: Through the multi-scale Transformer architecture, the present invention can effectively capture high-frequency and low-frequency information in visual data, significantly improving the accuracy of visual speech recognition, especially in the presence of background noise or poor audio quality.

[0017] The introduced sparse attention mechanism greatly reduces the number of operations required by the model in self-attention calculation, thereby reducing the computational complexity and resource consumption of the algorithm, making the model more suitable for real-time or resource-constrained application scenarios.

[0018] The introduction of temporal convolution brings an inductive bias of local connectivity to the Transformer model, enhancing the model's ability to capture the details of lip movements, thereby optimizing the feature extraction process. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a schematic diagram of an efficient multi-scale Transformer-based dense prediction visual speech recognition algorithm pipeline proposed by the present invention; Figure 2 This is the structural diagram of the MTF-TCN model proposed in the present invention; Figure 3 This is the multi-scale self-attention fusion module Inception Mixer used in the present invention. DETAILED DESCRIPTION

[0020] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Example 1

[0021] Reference Figure 1 , Figure 2 and Figure 3 , an efficient dense prediction visual speech recognition algorithm based on multi-scale Transformer, including: data preprocessing, data enhancement, front-end part, back-end part, and classification head.

[0022] Data preprocessing: Face detection and face alignment are performed for each video sequence in the video dataset. Each frame is aligned with the reference average face shape. A fixed 96×96 pixel wide region of interest (ROI) is cropped from the aligned face image so that the mouth area is always roughly centered in the image cropping area. The cropped image is converted to grayscale.

[0023] Data Augmentation: Mixup creates new augmented training samples by linearly combining two input video sequences and their corresponding targets. We set the linear combination weight λ to 0.4. In addition, we also use time masking to mask N consecutive frames in each training sequence, where N is uniformly sampled between 0 and Nmax. Each masked frame is replaced by the average frame of the sequence it belongs to.

[0024] Front-end: We first use a 3D convolutional layer to obtain spatial-temporal features with a shape of T × H1 × W1 × C1, where C1 is the number of feature channels. On top of this layer, an EfficientNetV2 is applied to generate features with a shape of T × H2 × W2 × C2. And in the next layer, average pooling is used to extract spatial information and reduce the dimension to T × C2. We use the NAS technology of EfficientNetV2 to automatically search for the optimal backbone network architecture. Determine the hyperparameters of the network such as depth, bandwidth, and resolution.

[0025] Backend part: The visual sequence feature information extracted by the front end is sent to the MTF-TCN module, which extracts and analyzes the time series information. Figure 2 As shown in the figure, a multi-scale Transformer structure is used to process convolution, maximum pooling path and self-attention path in parallel through Transformer modules of different scales, effectively learning high-frequency and low-frequency information in visual data. It consists of multiple Encoder blocks, which are similar to the Encoder in Transformer. The difference is that we use Inception Mixer to replace the original multi-head self-attention layer and use TCN temporal convolutional network to replace the fully connected layer of the feedforward network.

[0026] like Figure 3 It is an Inception Mixer structure, which aims to integrate the advantages of convolutional neural networks (CNNs) in capturing high-frequency information into the Transformer architecture to enhance the model's ability to learn the comprehensive features of visual data.

[0027] 1. Input feature segmentation: First, Inception Mixer divides the input feature map X into high-frequency and low-frequency parts along the channel.

[0028] 2. High-frequency mixer: The high-frequency mixer consists of a maximum pooling operation and a parallel convolution operation. The input is further divided into two parts, each of which is convolved with a different convolution kernel size to capture high-frequency information of different scales. For example, one part is channel mixed through 1x1 convolution, and the other part is extracted through 3x3 depth convolution.

[0029] 3. Low-frequency mixer: The low-frequency mixer is processed using the standard Multi-Head Self-Attention (MSA) mechanism. Before the self-attention operation, the average pooling layer is used to reduce the spatial scale of the feature map to reduce the computational cost, and the upsampling layer is used to restore it to the original spatial dimension to maintain the integrity of the global information.

[0030] 4. Feature fusion: The outputs of the high-frequency and low-frequency mixers are concatenated in the channel dimension to achieve the integration of different frequency information.

[0031] 5. Frequency ramp structure: Considering the characteristics of the bottom layer capturing high-frequency details and the high layer capturing low-frequency global information, a frequency ramp structure is designed. From the low layer to the high layer, the channel dimension sent to the low-frequency mixer is gradually increased, and the channel dimension sent to the high-frequency mixer is reduced, so as to effectively balance the high-frequency and low-frequency components at different levels.

[0032] The above are only preferred specific implementation modes of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical solutions and inventive concepts of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. An efficient multi-scale Transformer-based visual speech recognition algorithm, characterized by: include: Obtaining corresponding lip reading data and performing preprocessing, the preprocessed data including video data and audio data; Use EfficientNetV2's Neural Architecture Search technology to automatically search for the backbone network architecture and determine the backbone network's depth, bandwidth, and resolution hyperparameters; Using parallel processing of convolution, max pooling path and self-attention path to implement multi-scale Transformer to learn high-frequency and low-frequency information in visual data; Use sparse attention mechanism to reduce computational complexity; Through temporal convolution, the inductive bias of local connectivity is introduced to Transformer to enhance local high-frequency feature extraction.

2. The algorithm according to claim 1, characterized in that This includes using parallel processing of convolutions, maximum pooling paths, and self-attention paths to enable multi-scale Transformers to learn high-frequency and low-frequency information in visual data.

3. An algorithm according to claim 1 or 2, characterized in that A sparse attention mechanism is used to reduce computational complexity.

4. An algorithm according to any one of claims 1 to 3, characterized in that Through temporal convolution, an inductive bias of local connectivity is introduced to the Transformer to enhance the extraction of local high-frequency features.

5. The algorithm according to claim 4, characterized in that The temporal convolution includes setting different step sizes and padding parameters to adapt to the extraction of different frequency features.

6. The algorithm according to claim 1, characterized in that A visual speech recognition system, comprising: a) Input module, used to receive and pre-process lip reading data; b) Front-end feature extraction, using 3D convolution and EfficientNetV2 to process the pre-processed input data; c) The back-end sequence data processing module processes the feature sequence data through the MTF-TCN module; d) Output module: The output result of the backend passes through the fully connected layer to obtain the recognition result.

Images