Lip language recognition method and system, terminal equipment and medium

Abstract

The invention provides a lip language recognition method and system, terminal equipment and a medium, and the method comprises the steps: collecting a lip reading video through an event camera, and converting the lip reading video into an event sequence; voxelization is carried out on the event sequence, and data representation is carried out on the event sequence after voxelization in a voxel grid mode; and inputting the event sequence in the voxel grid form into a binary lip language recognition network model for lip language recognition to obtain a lip language recognition result corresponding to the lip reading video. According to the invention, the efficiency and accuracy of lip language recognition can be improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of lip reading recognition, and in particular relates to a lip reading recognition method, system, terminal equipment and medium. Background Art

[0002] Lip reading, also known as visual speech recognition, infers the content of a speaker's speech based on his or her lip movements. This technology not only helps the hearing-impaired communicate, but also shows wide application potential in many fields such as biometric verification. In addition, lip reading can improve the accuracy of speech recognition in noisy environments, and improve the quality of speaker facial generation by integrating lip reading expert models, correcting inaccurate lip movements, thereby enhancing the accuracy and comprehensibility of generated images. This technology is also suitable for DeepFake detection, which can effectively distinguish between real and fake lip movements by learning the characteristics of natural mouth movements through pre-training models.

[0003] Traditional optical cameras are subject to environmental interference when capturing lip movements, and suffer from data redundancy, low temporal resolution, and limited dynamic range. In contrast, event cameras record movements by detecting brightness changes at the pixel level, showing superior performance. They have made significant progress in time series modeling tasks such as action recognition and gesture recognition. The microsecond temporal resolution of event cameras enables them to accurately capture subtle changes in the lips, greatly improving the accuracy and robustness of lip reading recognition. In addition, due to their high dynamic range, they can accurately record information even in extreme lighting conditions, and their low power consumption makes them very suitable for wearable and mobile devices that operate for long periods of time.

[0004] The practical application of lip reading recognition requires the system to be highly lightweight to adapt to the limitations of edge devices. Event cameras have demonstrated significant advantages in lightweight data input through their efficient data capture methods. However, in order to further achieve the lightweight of the entire system, optimization is also required in the model design. Minimizing the number of model parameters and computational complexity without losing performance is the key to achieving efficient operation of lip reading recognition systems on edge devices. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a lip reading recognition method, system, terminal device and medium to improve the efficiency and accuracy of lip reading recognition.

[0006] In a first aspect, the present invention provides a lip reading recognition method, the method comprising the following steps: Using an event camera to collect lip-reading videos, the lip-reading videos are converted into event sequences; voxelize the event sequence, and represent the voxelized event sequence in a voxel grid manner to obtain the event sequence in a voxel grid form; An event sequence in the form of a voxel grid is input into a binary lip reading recognition network model for lip reading recognition to obtain a lip reading recognition result corresponding to the lip reading video; the binary lip reading recognition network model includes a first feature extraction module for extracting global spatiotemporal features of the event sequence corresponding to the lip reading video, a second feature extraction module for extracting local spatiotemporal features of the event sequence corresponding to the lip reading video from the global spatiotemporal features, an MSTCN module for extracting context-dependent characteristics of the event sequence corresponding to the lip reading video from the local spatiotemporal features, and a recognition module for outputting the lip reading recognition result according to the context-dependent characteristics. The network weights of the binary lip reading recognition network model are binary weights.

[0007] Optionally, convert the lip-reading video into a sequence of events, including: An image frame sequence of a lip reading video is obtained, and divided according to millisecond timestamps of the image frame sequence to obtain a plurality of frame sequence segments; the frame sequence segment includes a plurality of image frames; The pixel position information and the pixel brightness information of the image frame in each frame sequence segment are collected, and an event is generated according to the pixel position information and the pixel brightness information to obtain an event sequence.

[0008] Optionally, the event sequence expression is in, represents a sequence of events, Indicates events, , Indicates the total number of frame sequence segments, Indicates The pixel position information of the image frame in each event, Indicates The timestamp corresponding to the event, Indicates The brightness change of the image frame in an event, when When When , it means the brightness is reduced. Represents pixels In time interval The brightness changes within Represents pixels In timestamp brightness.

[0009] Optionally, the event sequence is voxelized, and the voxelized event sequence is represented as data in a voxel grid manner to obtain an event sequence in a voxel grid form, including: By calculating the formula Get the event sequence in voxel grid form ;in, Indicates The intensity of an event, represents the normalized time, Indicates the frame number. represents the total number of events, Indicates the time of the first event.

[0010] Optionally, extract global spatiotemporal features of the event sequence corresponding to the lip reading video, including: By calculating the formula Get global spatiotemporal features ;in, , express Convolution processing, represents batch normalization processing, express Activation processing, Represents a sequence of events.

[0011] Optionally, extracting local spatiotemporal features of the event sequence corresponding to the lip reading video from the global spatiotemporal features includes: By calculating the formula

[0012] Get local spatiotemporal features Among them, when hour, ,when hour, Indicates passing The feature map after layer residual block processing, , Indicates residual blocks, represents residual block processing, express Convolution processing.

[0013] Optionally, context-dependent features of the event sequence corresponding to the lip-reading video are extracted from the local spatiotemporal features, including: By calculating the formula Get context-dependent features ; represents the concatenation of feature dimensions, Indicates expansion coefficient Multi-scale convolution, , the scale set is , Indicates The stride of the convolution, Indicates Features of layer convolution.

[0014] In a second aspect, the present invention provides a lip reading recognition system, comprising: A video conversion module, used to collect lip-reading videos using an event camera and convert the lip-reading videos into event sequences; A voxelization module is used to voxelize the event sequence and represent the voxelized event sequence in a voxel grid manner to obtain the event sequence in a voxel grid form; The recognition module is used to input the event sequence in the form of voxel grid into the binary lip reading recognition network model for lip reading recognition, and obtain the lip reading recognition result corresponding to the lip reading video; the binary lip reading recognition network model includes a first feature extraction module for extracting the global spatiotemporal features of the event sequence corresponding to the lip reading video, a second feature extraction module for extracting the local spatiotemporal features of the event sequence corresponding to the lip reading video from the global spatiotemporal features, an MSTCN module for extracting the context-dependent characteristics of the event sequence corresponding to the lip reading video from the local spatiotemporal features, and a recognition module for outputting the lip reading recognition result according to the context-dependent characteristics. The network weights of the binary lip reading recognition network model are binary weights.

[0015] In a third aspect, the present invention provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned lip reading recognition method when executing the computer program.

[0016] In a fourth aspect, the present invention provides a computer-readable storage medium storing a computer program, which implements the above-mentioned lip reading recognition method when executed by a processor.

[0017] The beneficial effects of the present invention are: The lip reading recognition method provided by the present invention can effectively improve the universality of the lip reading recognition method, effectively reduce the interference of environmental factors on lip reading recognition, and improve the accuracy of lip reading recognition by using an event camera to collect lip reading videos, convert the lip reading videos into event sequences, voxelize the event sequences, and represent the voxelized event sequences in a voxel grid manner. The event camera is used to collect lip reading videos, which can effectively reduce the amount of data, reduce the calculation complexity, and improve the efficiency of lip reading recognition. By voxelizing the event sequence, redundant features unrelated to lip movements can be filtered out, thereby improving the efficiency of lip reading recognition. At the same time, it also helps to improve the accuracy of lip reading recognition; the network weights of the binary lip reading recognition network model used are binary weights, so that the convolution operation of the binary lip reading recognition network model in the reasoning process does not need to perform multiplication calculations, but only needs to perform addition operations, which can significantly improve the efficiency of lip reading recognition; at the same time, the second feature extraction module can extract the local spatiotemporal features of the event sequence, so as to accurately capture the short-term changes of the lips, and the MSTCN module can capture the context-dependent characteristics of the event sequence, and the combination of the short-term changes of the lips and the context-dependent characteristics of the event sequence can further improve the accuracy of lip reading recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is a flow chart of a lip reading recognition method in one embodiment of the present application; Figure 2 This is a structural diagram of a binary lip reading recognition network model in one embodiment of the present application; Figure 3a This is a diagram of the calculation process when all convolution kernels share a scaling factor in one implementation manner of the present application; Figure 3b A diagram showing a calculation process when each convolution kernel calculates a scaling factor separately in one implementation manner of the present application; Figure 4 This is a structural diagram of a lip reading recognition system in one embodiment of the present application; Figure 5 This is a structural diagram of a terminal device in one of the implementation modes of the present application. DETAILED DESCRIPTION

[0019] In view of the technical problems of low efficiency and poor accuracy of traditional lip reading recognition methods, the present invention provides a lip reading recognition method, system, terminal device and medium; wherein, the method uses an event camera to collect lip reading videos, converts the lip reading videos into event sequences, voxelizes the event sequences, and represents the voxelized event sequences as data in a voxel grid manner, which can effectively improve the universality of the lip reading recognition method, effectively reduce the interference of environmental factors on lip reading recognition, and improve the accuracy of lip reading recognition; using an event camera to collect lip reading videos can effectively reduce the amount of data, reduce the computational complexity, and improve the efficiency of lip reading recognition; by voxelizing the event sequences, it is possible to filter out Redundant features that are not related to lip movements not only improve the efficiency of lip reading recognition, but also help improve the accuracy of lip reading recognition. The network weights of the binary lip reading recognition network model used are binary weights, so that the convolution operation of the binary lip reading recognition network model in the reasoning process does not require multiplication calculations, but only needs to perform addition operations, which can significantly improve the efficiency of lip reading recognition. At the same time, the second feature extraction module can extract the local spatiotemporal features of the event sequence, so as to accurately capture the short-term changes of the lips. The MSTCN module can capture the context-dependent characteristics of the event sequence, and the combination of the short-term changes of the lips and the context-dependent characteristics of the event sequence can further improve the accuracy of lip reading recognition.

[0020] The lip reading recognition method provided by the present invention is described below.

[0021] like Figure 1 As shown, the lip reading recognition method includes steps 11 to 13.

[0022] Step 11: Use an event camera to collect lip-reading videos and convert the lip-reading videos into event sequences.

[0023] Exemplarily, in some embodiments of the present invention, a DAVIS346 event camera may be used to capture lip reading videos.

[0024] The process of converting the lip reading video into an event sequence in step 11 is described below, specifically including steps 11.1 to 11.2.

[0025] Step 11.1, obtain the image frame sequence of the lip reading video, and divide it according to the millisecond timestamp of the image frame sequence to obtain multiple frame sequence segments.

[0026] Specifically, the lip reading video can be decomposed into multiple image frames, and 25 frames per second are extracted to obtain a video frame sequence. After obtaining the video frame sequence, it can be divided according to the timestamp of each frame; since the timestamps of the video frames are continuous, the timestamps can be used directly to label or group each frame. For example, assuming that the video frame rate is 25 frames per second, that is, the time interval between each frame is 40ms, then every 40 milliseconds will correspond to a new image frame.

[0027] In an embodiment of the present invention, each frame sequence segment includes multiple image frames, each image frame represents image information at a moment, and the image information is specifically a 128*128 pixel image centered on the mouth, which can be extracted from the image output by the event camera using existing face detection tools.

[0028] Step 11.2, collecting pixel position information and pixel brightness information of the image frames in each frame sequence segment, and generating an event according to the pixel position information and the pixel brightness information to obtain an event sequence.

[0029] Specifically, the expression of the event sequence is in, represents a sequence of events, Indicates events, , Indicates the total number of frame sequence segments, Indicates The pixel position information of the image frame in each event, Indicates The timestamp corresponding to the event, Indicates The brightness change of the image frame in an event, when When When , it means the brightness is reduced. Represents pixels In time interval The brightness changes within Represents pixels In timestamp brightness.

[0030] It should be noted that, in the embodiment of the present invention, each event is represented as a four-tuple, which are the pixel position where the event occurs, and time , and brightness increase or decrease, denoted as .

[0031] Step 12, voxelize the event sequence, and represent the voxelized event sequence in a voxel grid manner to obtain the event sequence in the form of a voxel grid.

[0032] In order to facilitate the extraction of features of event sequences, in an embodiment of the present invention, the event sequences are voxelized for alignment, projection and aggregation, and the voxel grid in the embodiment of the present invention has three dimensions: T, W and H. T represents the number of video frames, and W and H represent the width and height of the video, respectively.

[0033] Specifically, through the calculation formula: Get the event sequence in voxel grid form ;in, Indicates The intensity of an event, represents the normalized time, Indicates the frame number. represents the total number of events, Indicates the time of the first event.

[0034] It should be noted that the time information of events in the event sequence is normalized to , a single grid in the voxel grid Only the event information within the adjacent range will be merged as the aggregate representation of the grid. The event representation after voxel grid representation is similar to that of a traditional video camera. The data format of the event after voxel grid representation is , and the video data format is , where 1 represents the event aggregation of the pixel and 3 represents the RGB channel. The difference between events and videos is that events can filter out most of the features that are not related to lip movements, while videos often contain a lot of irrelevant background information and motion blur. Therefore, by using an event camera to collect lip-reading videos, converting the lip-reading videos into event sequences, voxelizing the event sequences, and representing the voxelized event sequences in a voxel grid manner, the universality of the lip reading recognition method can be effectively improved, the interference of environmental factors on lip reading recognition can be effectively reduced, and the accuracy of lip reading recognition can be improved. Using an event camera to collect lip-reading videos can effectively reduce the amount of data, reduce the computational complexity, and improve the efficiency of lip reading recognition; by voxelizing the event sequence, redundant features that are not related to lip movements can be filtered out, which improves the efficiency of lip reading recognition while also helping to improve the accuracy of lip reading recognition.

[0035] Step 13, inputting the event sequence in the form of voxel grid into a binary lip reading recognition network model for lip reading recognition, and obtaining the lip reading recognition result corresponding to the lip reading video.

[0036] In an embodiment of the present invention, a binary lip reading recognition network model includes a first feature extraction module for extracting global spatiotemporal features of an event sequence corresponding to a lip reading video, a second feature extraction module for extracting local spatiotemporal features of an event sequence corresponding to a lip reading video from the global spatiotemporal features, an MSTCN module for extracting context-dependent characteristics of an event sequence corresponding to a lip reading video from the local spatiotemporal features, and a recognition module for outputting lip reading recognition results according to the context-dependent characteristics. The network weights of the binary lip reading recognition network model are binary weights.

[0037] like Figure 2 As shown, the first feature extraction module 201, the second feature extraction module 202, the MSTCN module 203 and the recognition module 204 are connected in sequence.

[0038] The first feature extraction module is described below.

[0039] The first feature extraction module uses a 3D convolutional neural network, and the process of extracting the global spatiotemporal features of the event sequence corresponding to the lip reading video can be formulated as: in, represents the global spatiotemporal characteristics, , express Convolution processing, represents batch normalization processing, represents pooling processing, express Activation processing, Represents a sequence of events.

[0040] The second feature extraction module is described below.

[0041] The second feature extraction module includes a plurality of sub-extraction modules connected in sequence, each of which is composed of a 2D convolution and a residual block. The process of extracting the local spatiotemporal features of the event sequence corresponding to the lip reading video from the global spatiotemporal features can be formulated as follows: in, represents the local spatiotemporal characteristics, when hour, ,when hour, Indicates passing The feature map after layer residual block processing, , Indicates residual blocks, represents residual block processing, express Convolution processing.

[0042] The feature map obtained after being processed by each sub-extraction module The feature map can be input into the MSTCN module to extract the context-dependent characteristics of the event sequence corresponding to the lip reading video by using average pooling in the spatial dimension.

[0043] It should be noted that, in other embodiments of the present invention, the number of residual blocks may be other numbers, and the number of residual blocks is not limited here. As long as it belongs to the same concept as the present invention, it belongs to the protection scope of the present invention.

[0044] The MSTCN module is described below.

[0045] In an embodiment of the present invention, the MSTCN module is stacked by multiple multi-scale residual TCN blocks, which contain multi-scale and dilated convolutions to extract long-term context dependencies.

[0046] Specifically, by calculating the formula Get context-dependent features ; represents the concatenation of feature dimensions, Indicates expansion coefficient Multi-scale convolution, , the scale set is , Indicates The stride of the convolution, Indicates Features of layer convolution.

[0047] The identification module is described below.

[0048] In an embodiment of the present invention, the recognition module is composed of a linear layer and a SoftMax layer connected in sequence. After receiving the context-dependent characteristics, it can calculate the classification probability of each event and finally use the classification result corresponding to the maximum probability as the lip reading recognition result.

[0049] In some embodiments of the present invention, the type of event corresponds to a word one-to-one, the classification probability refers to the probability of a word, different types correspond to different words, and the lip reading recognition result is the word combination corresponding to the lip reading video.

[0050] It should be noted that the network weights of the binary lip reading recognition network model provided by the present invention are binary weights.

[0051] It should be understood that the binarization strategy is the most extreme quantization strategy, which usually directly quantizes the full-precision weights to 1 and -1 through the sign function. Although this method has a great compression ratio, its weak expression ability also greatly affects the performance of the neural network.

[0052] In an embodiment of the present invention, in order to reduce the error between the full-precision weight and the binary weight, a scaling factor is introduced Assumptions , They are full-precision weights and binary weights, and their dimensions are , represents the convolution kernel size. In order to find The optimal estimate of , defines the following optimization problem: It can be inferred that when When , the error between full-precision weights and binary weights is minimal.

[0053] For one-dimensional convolution, since the number of convolution kernel parameters is small, The value of will be smaller as the convolution kernel dimension increases. The calculation method of has not changed, but its error generally increases exponentially with the increase of dimension. Therefore, in an embodiment of the present invention, each convolution kernel calculates a scaling factor separately to reduce the overall error. For example, given the convolution input , convolution kernel , the original calculation process (all convolution kernels share a scaling factor) is as follows Figure 3a As shown in , the optimal scaling factor of Cout convolution kernels is calculated and then multiplied by the result after convolution. After the improvement (each convolution kernel calculates a scaling factor separately), each convolution kernel is calculated separately and has its own scaling factor. After the convolution operation, the results are concatenated together, as shown in Figure 3b As shown. After the improvement, only parameters.

[0054] In order to enhance the expressive power of the binary lip reading recognition network model, the present invention uses the RPReLU function, which adds three learnable variables: , , (dimension is c) to enhance the expressive power of the PReLU function, and its formula is as follows: in, is the input of the RPReLU function, Representative channels. and is a learnable offset parameter used to change the distribution of input parameters, is a learnable scaling factor to control the slope of the negative part.

[0055] The following is an explanation of the training process of the binary lip reading recognition network model in the embodiment of the present invention.

[0056] Specifically, the training of the binary lip reading recognition network model adopts a quantization perception strategy, that is, the inaccuracy of 1 bit has been modeled during the training process. Specifically, during the model reasoning process, the present invention directly binarizes the weights, that is: in, Represents the weight of the convolution kernel. Since the sign function is not differentiable, the following clip function is used to replace the sign function:

[0057] During the back propagation process, the derivative is calculated as follows: .

[0058] In order to further verify the accuracy of the lip reading recognition method provided by the present invention, a comparative experiment was conducted, as follows.

[0059] Comparative Experiment 1 A lip reading recognition model is constructed using the first feature extraction module, the second feature extraction module, the MSTCN module, and the recognition module, but its network weights are not binarized. It is denoted as STCNET and trained and tested on the DVS-Lip dataset. The comparison objects are conventional lip reading recognition models (Event Clouds, EV-Gait-3DGraph, EV-Gait-IMG, EST, I3D, TANet, ACTION-Net, MSTP, SSGRU).

[0060] Comparative Experiment 2 The lip reading recognition model is constructed using the first feature extraction module, the second feature extraction module, the MSTCN module, and the recognition module, and its network weights are binarized to obtain a binarized lip reading recognition network model, denoted as B-STCNET. During the binarization process, the original weights of the first convolution layer and the last fully connected layer are retained, while the weights of all other convolution layers are converted to binary form, that is, only containing values ​​of 1 and -1. The training and testing are also carried out on the DVS-Lip dataset, and the comparison object is the conventional lip reading recognition model.

[0061] The training parameters of the two comparative experiments are shown in Table 1.

[0062] Table 1

[0063] The results of the two comparative experiments are shown in Table 2.

[0064] Table 2

[0065] The results of the two comparative experiments are shown in Table 3.

[0066] Table 3

[0067] As can be seen from Table 3, the binary lip reading recognition network model provided by the present invention shows excellent performance on the DVS-Lip dataset. The model achieves the current best (state-of-the-art, SOTA) results in both test parts. Specifically, in the first test part, STCNet achieved an accuracy of 65.86%; while in the second test part, the accuracy reached a higher 87.43%, making the average accuracy reach 76.62%. This result significantly exceeds the optimal result (75.3%) in previous studies, fully demonstrating the powerful ability of the binary lip reading recognition network model in processing event-driven lip reading data. The binarized model (B-STCNet) has almost no loss in performance and still maintains an accuracy comparable to the full-precision model. In the first test part, B-STCNet achieved an accuracy of 66.03%, and in the second part it reached 86.26%. This makes the average accuracy of B-STCNet 76.15%, close to the original model without binarization. These results not only verify the effectiveness of the binary lip reading recognition network model in understanding and processing event-based visual data, but also demonstrate the possibility of maintaining high accuracy while greatly reducing computing resource consumption through carefully designed network architecture and binary technology. In addition, this finding has important implications for the deployment of efficient visual recognition systems on resource-constrained devices in the future.

[0068] In summary, the lip reading recognition method provided by the present invention can effectively improve the universality of the lip reading recognition method, effectively reduce the interference of environmental factors on lip reading recognition, and improve the accuracy of lip reading recognition by using an event camera to collect lip reading videos, convert the lip reading videos into event sequences, voxelize the event sequences, and represent the voxelized event sequences in a voxel grid manner. The event camera is used to collect lip reading videos, which can effectively reduce the amount of data, reduce the computational complexity, and improve the efficiency of lip reading recognition. By voxelizing the event sequences, redundant features unrelated to lip movements can be filtered out, thereby improving the accuracy of lip reading recognition. While improving efficiency, it also helps to improve the accuracy of lip reading recognition; the network weights of the binary lip reading recognition network model used are binary weights, so that the convolution operation of the binary lip reading recognition network model in the reasoning process does not need to perform multiplication calculations, but only needs to perform addition operations, which can significantly improve the efficiency of lip reading recognition; at the same time, the second feature extraction module can extract the local spatiotemporal features of the event sequence, so as to accurately capture the short-term changes of the lips, and the MSTCN module can capture the context-dependent characteristics of the event sequence, and the comprehensive short-term changes of the lips and the context-dependent characteristics of the event sequence can further improve the accuracy of lip reading recognition.

[0069] The lip reading recognition system provided by the present invention is described below.

[0070] like Figure 4 As shown, the lip reading recognition system 400 includes: A video conversion module 401 is used to collect lip reading videos using an event camera and convert the lip reading videos into event sequences; A voxelization module 402 is used to voxelize the event sequence and represent the voxelized event sequence in a voxel grid manner to obtain the event sequence in a voxel grid form; The recognition module 403 is used to input the event sequence in the form of voxel grid into the binary lip reading recognition network model for lip reading recognition, and obtain the lip reading recognition result corresponding to the lip reading video; the binary lip reading recognition network model includes a first feature extraction module for extracting the global spatiotemporal features of the event sequence corresponding to the lip reading video, a second feature extraction module for extracting the local spatiotemporal features of the event sequence corresponding to the lip reading video from the global spatiotemporal features, an MSTCN module for extracting the context-dependent characteristics of the event sequence corresponding to the lip reading video from the local spatiotemporal features, and a recognition module for outputting the lip reading recognition result according to the context-dependent characteristics. The network weights of the binary lip reading recognition network model are binary weights.

[0071] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are not related to the implementation method of the present application. The examples are based on the same concept, and their specific functions and technical effects can be specifically referred to in the method embodiment section, which will not be repeated here. Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit, and the above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0072] like Figure 5 As shown, an embodiment of the present invention provides a terminal device, such as Figure 5 As shown, the terminal device D10 of this embodiment includes: at least one processor D100 ( Figure 5 Only one processor is shown in the figure), a memory D101, and a computer program D102 stored in the memory D101 and executable on the at least one processor D100, wherein the processor D100 implements the steps of any of the above-mentioned method embodiments when executing the computer program D102.

[0073] Specifically, when the processor D100 executes the computer program D102, it uses an event camera to collect lip-reading videos, converts the lip-reading videos into event sequences, voxelizes the event sequences, and represents the voxelized event sequences as data in a voxel grid manner; the event sequences in the form of voxel grids are input into a binary lip-reading recognition network model for lip reading recognition, and the lip-reading recognition results corresponding to the lip-reading videos are obtained; wherein, by using an event camera to collect lip-reading videos, converting the lip-reading videos into event sequences, voxelizing the event sequences, and representing the voxelized event sequences as data in a voxel grid manner, the universality of the lip-reading recognition method can be effectively improved, the interference of environmental factors on lip-reading recognition can be effectively reduced, and the accuracy of lip-reading recognition can be improved; using an event camera to collect lip-reading videos can The method can effectively reduce the amount of data, reduce the computational complexity, and improve the efficiency of lip reading recognition. By voxelizing the event sequence, redundant features irrelevant to lip movements can be filtered out, which improves the efficiency of lip reading recognition while also helping to improve the accuracy of lip reading recognition. The network weights of the binary lip reading recognition network model used are binary weights, so that the convolution operation of the binary lip reading recognition network model during the inference process does not require multiplication calculations, but only requires addition operations, which can significantly improve the efficiency of lip reading recognition. At the same time, the second feature extraction module can extract the local spatiotemporal features of the event sequence, thereby accurately capturing the short-term changes of the lips. The MSTCN module can capture the context-dependent characteristics of the event sequence, and the short-term changes of the lips and the context-dependent characteristics of the event sequence can be combined to further improve the accuracy of lip reading recognition.

[0074] The processor D100 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0075] In some embodiments, the memory D101 may be an internal storage unit of the terminal device D10, such as a hard disk or memory of the terminal device D10. In other embodiments, the memory D101 may also be an external storage device of the terminal device D10, such as a plug-in hard disk, a smart memory card (SMC, SmartMedia Card), a secure digital (SD, Secure Digital) card, a flash card (Flash Card), etc. equipped on the terminal device D10. Further, the memory D101 may also include both an internal storage unit of the terminal device D10 and an external storage device. The memory D101 is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of the computer program, etc. The memory D101 may also be used to temporarily store data that has been output or is to be output.

[0076] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0077] An embodiment of the present application provides a computer program product. When the computer program product is run on a terminal device, the terminal device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0078] A person skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of protection of the present application is limited to these examples. In line with the concept of the present application, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of different aspects of one or more embodiments of the present application as described above, which are not provided in detail for the sake of simplicity.

[0079] One or more embodiments of the present application are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the present application. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present application should be included in the protection scope of the present application.

Claims

1. A lip reading recognition method, characterized in that: include: Using an event camera to collect lip-reading videos, and converting the lip-reading videos into event sequences; voxelize the event sequence, and represent the voxelized event sequence in a voxel grid manner to obtain an event sequence in a voxel grid form; An event sequence in the form of a voxel grid is input into a binary lip reading recognition network model for lip reading recognition to obtain a lip reading recognition result corresponding to the lip reading video; the binary lip reading recognition network model includes a first feature extraction module for extracting global spatiotemporal features of the event sequence corresponding to the lip reading video, a second feature extraction module for extracting local spatiotemporal features of the event sequence corresponding to the lip reading video from the global spatiotemporal features, an MSTCN module for extracting context-dependent characteristics of the event sequence corresponding to the lip reading video from the local spatiotemporal features, and a recognition module for outputting the lip reading recognition result according to the context-dependent characteristics. The network weights of the binary lip reading recognition network model are binary weights.

2. The lip reading recognition method according to claim 1, characterized in that: The step of converting the lip reading video into an event sequence comprises: Acquire an image frame sequence of the lip-reading video, and divide the image frame sequence according to millisecond-level timestamps of the image frame sequence to obtain a plurality of frame sequence segments; the frame sequence segment includes a plurality of image frames; The pixel position information and the pixel brightness information of the image frames in each of the frame sequence segments are collected, and an event is generated according to the pixel position information and the pixel brightness information to obtain the event sequence.

3. The lip reading recognition method according to claim 2, characterized in that: The expression of the event sequence is: in, represents the sequence of events, Indicates events, , Indicates the total number of frame sequence segments, Indicates The pixel position information of the image frame in each event, Indicates The timestamp corresponding to the event, Indicates The brightness change of the image frame in an event, when When When , it means the brightness is reduced. Represents pixels In time interval The brightness changes within Represents pixels In timestamp brightness.

4. The lip reading recognition method according to claim 3, characterized in that: The voxelization of the event sequence and data representation of the voxelized event sequence in a voxel grid manner to obtain the event sequence in a voxel grid form includes: By calculating the formula Get the event sequence in voxel grid form ;in, Indicates The intensity of an event, represents the normalized time, Indicates the frame number. represents the total number of events, Indicates the time of the first event.

5. The lip reading recognition method according to claim 4, characterized in that: The step of extracting the global spatiotemporal features of the event sequence corresponding to the lip reading video comprises: By calculating the formula Get the global spatiotemporal features ;in, , express Convolution processing, represents batch normalization processing, express Activation processing, Represents a sequence of events.

6. The lip reading recognition method according to claim 5, characterized in that: The step of extracting local spatiotemporal features of the event sequence corresponding to the lip-reading video from the global spatiotemporal features comprises: By calculating the formula Get the local spatiotemporal features Among them, when hour, ,when hour, Indicates passing The feature map after layer residual block processing, , Indicates residual blocks, represents residual block processing, express Convolution processing.

7. The lip reading recognition method according to claim 6, characterized in that: The extracting the context-dependent characteristics of the event sequence corresponding to the lip-reading video from the local spatiotemporal features includes: By calculating the formula Get the context-dependent properties ; represents the concatenation of feature dimensions, Indicates expansion coefficient Multi-scale convolution, , the scale set is , Indicates The stride of the convolution, Indicates Features of layer convolution.

8. A lip reading recognition system, characterized in that: include: A video conversion module, used for collecting lip-reading videos using an event camera and converting the lip-reading videos into event sequences; A voxelization module, used for voxelizing the event sequence, and representing the voxelized event sequence in a voxel grid manner to obtain an event sequence in a voxel grid form; A recognition module is used to input an event sequence in the form of a voxel grid into a binary lip reading recognition network model for lip reading recognition to obtain a lip reading recognition result corresponding to the lip reading video; the binary lip reading recognition network model includes a first feature extraction module for extracting global spatiotemporal features of the event sequence corresponding to the lip reading video, a second feature extraction module for extracting local spatiotemporal features of the event sequence corresponding to the lip reading video from the global spatiotemporal features, an MSTCN module for extracting context-dependent characteristics of the event sequence corresponding to the lip reading video from the local spatiotemporal features, and a recognition module for outputting a lip reading recognition result according to the context-dependent characteristics. The network weights of the binary lip reading recognition network model are binary weights.

9. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the lip reading recognition method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the lip reading recognition method according to any one of claims 1 to 7 is implemented.

Images