A Cantonese lip reading recognition method, device and storage medium

Abstract

The present invention discloses a Cantonese lip reading recognition method, device and storage medium. The method comprises obtaining a first Cantonese video clip; cutting out useless clips in the first Cantonese video clip to obtain a second Cantonese video clip; dividing the video sequence and audio sequence in the second Cantonese video clip, segmenting the audio sequence and generating a segmentation timestamp, and generating a label according to the segmentation and the segmentation timestamp; extracting a face image in the video sequence, filtering incomplete face images, and generating a sample image according to the filtered face image and the label; training a preset Cantonese lip reading recognition model according to the sample image to obtain a trained Cantonese lip reading recognition model; and recognizing a target video sequence according to the trained Cantonese lip reading recognition model to obtain a recognition result. The method can collect a Cantonese word-level lip reading sample image data set, and can improve the recognition accuracy of the trained model because useless sequences in the video sequence are eliminated.

Description

Technical Field

[0001] The present invention relates to the technical field of lip reading recognition, and in particular to a Cantonese lip reading recognition method, device and storage medium. Background Art

[0002] Lip reading recognition is to identify the corresponding text information by observing the speaker's mouth movement sequence, which is commonly known as "lip reading". In recent years, with the development of deep learning, lip reading recognition technology has also made great breakthroughs, and English lip reading recognition at the word level and sentence level has achieved a very high recognition accuracy.

[0003] As the only Chinese dialect that has been publicly studied at home and abroad, Cantonese has a complete writing system that can be expressed entirely in Chinese characters. It is widely used in Guangdong, Guangxi, Hong Kong, Macau, and Chinese communities around the world. Cantonese has nearly 120 million speakers worldwide, which makes the research prospects of Cantonese lip reading tasks huge. Unlike the English lip reading task, Chinese is a syllable-based language that distinguishes words or sentences by tone and pinyin, which undoubtedly brings great challenges to the entire Chinese lip reading task. In addition, the pronunciation, tone, and length of Cantonese are different when the content of the speech is the same as that of Mandarin. Therefore, the Chinese Mandarin lip reading model cannot be directly migrated to the Cantonese lip reading task.

[0004] There are very few studies on Cantonese lip reading recognition in the existing lip reading tasks. Combined with the research progress in the field of lip reading at home and abroad, since there is no organization or individual that has released a large-scale Cantonese lip reading dataset, in the relevant schemes for Cantonese lip reading recognition, the model is often trained by extracting features from the video sequence. However, there is often a small section of useless sequence in the video sequence. If the model is trained using a useless sequence, the recognition accuracy of the model will be reduced. Summary of the invention

[0005] The present invention aims to at least solve the technical problems existing in the prior art. To this end, the present invention proposes a Cantonese lip reading recognition method, device and storage medium, which can improve the recognition accuracy of the trained model.

[0006] A first aspect of the present invention provides a Cantonese lip reading recognition method, comprising the following steps:

[0007] Get the first Cantonese video clip;

[0008] Cutting out useless segments from the first Cantonese video segment to obtain a second Cantonese video segment, wherein the useless segments include segments with human voice but no image and / or segments where the human voice and the image do not match;

[0009] Dividing the video sequence and the audio sequence in the second Cantonese video clip, segmenting the audio sequence and generating a segmentation timestamp, and generating a label according to the segmentation and the segmentation timestamp;

[0010] Extracting face images from the video sequence, filtering incomplete face images, and generating a sample sequence according to the filtered face images and the labels;

[0011] Training a preset Cantonese lip reading recognition model according to the sample sequence to obtain the trained Cantonese lip reading recognition model;

[0012] The target video sequence is recognized according to the trained Cantonese lip reading recognition model to obtain a recognition result.

[0013] According to the embodiments of the present invention, there are at least the following technical effects:

[0014] After obtaining the first Cantonese video clip, the method removes the clips without human voice and image and / or the clips with mismatched human voice and image in the first Cantonese video clip to obtain the second Cantonese video clip, and uses the second Cantonese video clip to generate a sample sequence data set containing labels. The method can collect Cantonese word-level lip-reading sample sequence data sets, filling the gap of the current lack of large-scale lip-reading sample sequence data sets, and because useless sequences in the video sequence are removed, the model is trained through the sample sequence, which can improve the recognition accuracy of the trained model.

[0015] According to some embodiments of the present invention, before training a preset Cantonese lip reading recognition model according to the sample sequence, the method further includes:

[0016] Boundary information is added to the sample sequence, and the sample sequence with the added boundary information is encoded according to Libjpeg.

[0017] According to some embodiments of the present invention, the Cantonese lip reading recognition model includes a feature extraction network, an LSTM network, a three-layer BiGRU network, and a mutual information maximization network. The training process of the Cantonese lip reading recognition model includes:

[0018] Extracting features from the sample sequence according to the feature extraction network, and setting mutual information constraints between the features and labels;

[0019] Generate corresponding weights based on different frames of the label according to the LSTM network;

[0020] Classify the features according to the three-layer BiGRU network to obtain an output result;

[0021] Generate a global average feature according to the output result and the weight;

[0022] The mutual information between the global average feature and the label is maximized according to the mutual information maximization network.

[0023] According to some embodiments of the present invention, the feature extraction network includes a 3D CNN network, a spatial maximum pooling layer, a ResNet network, and a global average pooling layer connected in sequence; extracting features in the sample sequence according to the feature extraction network and setting mutual information constraints between the features and labels include:

[0024] Extracting initial features from the sample sequence according to the 3D CNN network;

[0025] compressing the initial features according to the spatial maximum pooling layer;

[0026] Divide the initial features into multiple parts, extract features of each part according to the ResNet network, and add mutual information constraints between the features and labels;

[0027] The features with added mutual information constraints are average pooled according to the global average pooling layer.

[0028] According to some embodiments of the present invention, the ResNet network is a ResNet-34 network.

[0029] According to some embodiments of the present invention, the LSTM network includes an LSTM layer and a linear layer connected in sequence; the calculation formula for generating corresponding weights based on different frames of the label according to the LSTM network includes:

[0030] a t =Relu(w linear ×LSTM(G) t +b linear )

[0031] Wherein, G represents the output result of the spatial maximum pooling layer, and w linear and the b linear represents the parameters of the linear layer, the LSTM (G) t represents the hidden state of the LSTM layer at time step t, Relu() represents the Relu function, and the a t Represents the weight of the frame sequence when the time step is t.

[0032] According to some embodiments of the present invention, the optimization function of the mutual information maximization network includes:

[0033] Loss MI =E p(F,L) [(log(MI (F,L) )]+Ep(F)p(L) [(log(1-MI (F,L) )]

[0034] Among them, the Loss MI represents the optimization function of the global mutual information maximization network, the p(F,L) represents the joint distribution of the sample pair (F,L), the p(F)p(L) represents the marginal distribution of the sample pair (F,L), the MI (F,L) represents the mutual information between F and L, E is used to obtain the mathematical expectation, F represents the global average feature, and L represents the label.

[0035] According to some embodiments of the present invention, the loss function of the Cantonese lip reading recognition model includes:

[0036]

[0037] Among them, the represents the cross entropy loss function, the c represents the total number of classes of the vocabulary in the sample sequence, The label L represents the output of the three-layer BiGRU network i score.

[0038] According to a second aspect of the present invention, an electronic device is provided, comprising at least one control processor and a memory for communicating with the at least one control processor; the memory stores instructions executable by the at least one control processor, and the instructions are executed by the at least one control processor so that the at least one control processor can execute the above-mentioned Cantonese lip reading recognition method.

[0039] According to a third aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the above-mentioned Cantonese lip reading recognition method.

[0040] It should be noted that the beneficial effects between the second aspect and the third aspect of the present invention and the prior art are the same as the beneficial effects between the above-mentioned Cantonese lip reading recognition method and the prior art, and will not be described in detail here.

[0041] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0043] Figure 1A schematic diagram of a flow chart of a Cantonese lip reading recognition method provided by one embodiment of the present invention;

[0044] Figure 2 A schematic flow chart of a Cantonese lip reading recognition method provided by another embodiment of the present invention;

[0045] Figure 3 A schematic flow chart of a Cantonese lip reading recognition method provided by another embodiment of the present invention;

[0046] Figure 4 A schematic flow chart of a Cantonese lip reading recognition method provided by another embodiment of the present invention;

[0047] Figure 5 A flowchart of constructing a sample data set provided by an embodiment of the present invention;

[0048] Figure 6 A flowchart of a Cantonese lip reading recognition model provided by one embodiment of the present invention;

[0049] Figure 7 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention;

[0050] Figure 8 A schematic diagram of the structure of a storage medium provided for one embodiment of the present invention. DETAILED DESCRIPTION

[0051] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0052] Reference Figures 1 to 3 One embodiment of the present invention provides a Cantonese lip reading recognition method, comprising the following steps:

[0053] Step S110: Obtain a first Cantonese video clip.

[0054] For example: use the you-get tool (the you-get tool is a video, picture and music download tool based on Python3) to crawl Cantonese TV programs from the Internet, such as Cantonese news broadcasts, Cantonese variety shows, Cantonese character interviews and talk shows, etc., to get the first Cantonese video clip.

[0055] Step S120, cutting out useless segments in the first Cantonese video segment to obtain a second Cantonese video segment, wherein the useless segments include segments with human voice but no image and / or segments with mismatched human voice and image. Here, the useless segments in the first Cantonese video segment can be cut out manually.

[0056] Step S130: divide the video sequence and audio sequence in the second Cantonese video clip, segment the audio sequence and generate segmentation timestamps, and generate tags according to the segmentations and segmentation timestamps.

[0057] For example: Use iFlytek's speech transcription tool to segment the audio sequence and generate segmentation timestamps, and name the video sequence and audio sequence in the same way, and name the video sequence and its corresponding audio sequence the same name for easy pairing later. Extend all segmentation timestamps by 0.02s on each side, and generate labels in the order of video sequence name, segmentation timestamp, segmentation pinyin, and segmentation. The audio sequence in this step is to mark the sample data set obtained later. Use iFlytek's speech transcription tool to process the audio to generate corresponding text information. The text information is the text content of the corresponding video sequence, which is equivalent to a label.

[0058] Step S140: extracting face images from the video sequence, filtering incomplete face images, and generating a sample sequence according to the filtered face images and labels.

[0059] For example, use the MediaPipe tool to extract faces from video sequences, and train a filter to filter out all faces that are not recognized normally (face cropping is incomplete), and finally obtain a sample sequence dataset consisting of multiple face images.

[0060] Step S150: training a preset Cantonese lip reading recognition model according to the sample sequence to obtain a trained Cantonese lip reading recognition model.

[0061] Step S160: Recognize the target video sequence according to the trained Cantonese lip reading recognition model to obtain a recognition result.

[0062] After obtaining the first Cantonese video clip, the method removes the clips without human voice and image and / or the clips with mismatched human voice and image in the first Cantonese video clip to obtain the second Cantonese video clip, and uses the second Cantonese video clip to generate a sample image dataset containing labels. The method can collect Cantonese word-level lip-reading sample datasets, filling the gap of the current lack of large-scale lip-reading sample datasets, and because useless sequences in the video sequence are removed, the model is trained through the sample sequence, which can improve the recognition accuracy of the trained model.

[0063] In the training process of the model, related solutions have the problem of unclear boundary information and difficulty in distinguishing key frames and useless frames. When the boundary information is unclear, it will make it difficult to eliminate useless frames in the boundary information. When the key frames and useless frames are difficult to distinguish, the model cannot well screen the key frames when extracting features from the video sequence, which often makes the lip reading speed slower. In addition, affected by non-key frames (useless frames), the recognition accuracy of the model will also be low.

[0064] Therefore, based on the above embodiment, the method further includes the following steps before step S150:

[0065] Step S1401, adding boundary information to the sample sequence, and encoding the sample sequence with added boundary information according to Libjpeg. Step S1401 can solve the problem that useless frames at the boundary cannot be removed by adding boundary information. Libjpeg is used to encode the training data, and the purpose of encoding is to compress the data, thereby speeding up the subsequent training process.

[0066] In existing solutions, a recognition model composed of a backbone network of ResNet-18 and GRU is usually used. Different from the existing solutions, the preset Cantonese lip reading recognition model includes a feature extraction network, an LSTM network, a three-layer BiGRU network, and a mutual information maximization network. The training process of the Cantonese lip reading recognition model includes the following steps:

[0067] Step S1501: extract features from a sample sequence according to a feature extraction network, and set mutual information constraints between features and labels.

[0068] In one embodiment, the feature extraction network includes a ResNet-34 network and a global average pooling layer. Compared with the general ResNet-18 network, this embodiment uses a ResNet-34 network that is deeper than the ResNet-18 network and can extract deeper features. The purpose of using the global average pooling layer instead of the fully connected layer is to prevent overfitting by regularizing the structure of the entire network.

[0069] In another embodiment, the feature extraction network includes a 3D CNN network, a spatial maximum pooling layer, a ResNet-34 network, and a global average pooling layer connected in sequence. Step S1501 specifically includes the following steps:

[0070] Step S15011: extract initial features from the sample sequence according to the 3D CNN network.

[0071] Step S15012: compress the initial features according to the spatial maximum pooling layer.

[0072] Step S15013: Divide the initial features into multiple parts, extract the features of each part according to the ResNet-34 network, and add mutual information constraints between the features and the labels.

[0073] Step S15014: average pooling the features with added mutual information constraints according to the global average pooling layer.

[0074] In steps S15011 and S15012, a 3D CNN network and a spatial maximum pooling layer are set at the front end of the ResNet-34 network. The 3D CNN network first extracts features from the initial frame to achieve the purpose of preliminary time alignment, and then uses the spatial maximum pooling to compress the features of the spatial domain. This processing can achieve a better recognition effect. In steps S15013 to S15014, according to the number of frames of the input sequence (for example, T frames), the sequence features are evenly divided into T parts, and the features of each part are extracted using the ResNet-34 network. In order to improve the ability to capture the corresponding labels of fine-grained lip movements and further improve the recognition accuracy of the model, mutual information constraints are imposed between the output of ResNet-34 and the label. It should be noted that mutual information is mostly used for the degree of association between feature items and categories during feature extraction. In this embodiment, the mutual information between the feature and its corresponding label is maximized in order to better correspond them one by one during training. The acquired features are then sent to the global average pooling layer, and the entire network is structurally regularized to prevent overfitting.

[0075] Step S1502: Generate corresponding weights based on different frames of the label according to the LSTM network.

[0076] An LSTM network is added to the output end of ResNet-34. The LSTM network includes an LSTM layer and a linear layer. In this embodiment, the purpose of the added LSTM network is to filter out key frames, that is, to assign different weights to different frames of labels to distinguish key frames from non-key frames. In this embodiment, the weight can be any positive number, but the weight of useless frames should be as close to 0 as possible. Therefore, the Relu function can be introduced to obtain the weight.

[0077] Step S1503: classify the features according to the three-layer BiGRU network to obtain output results.

[0078] The output result of the above feature extraction network is used as input to the three-layer BiGRU network, which performs feature classification to obtain the output result of the three-layer BiGRU network.

[0079] Step S1504: Generate global average features based on the output results and weights.

[0080] The global average feature is obtained by weighted processing of the output and weights of the three-layer BiGRU.

[0081] Step S1505: Maximize the mutual information between the global average feature and the label according to the mutual information maximization network.

[0082] The one-hot vectors of the global average feature and the label are concatenated, and the concatenation result is used as the input of the global mutual information maximization network. In this embodiment, the mutual information maximization network consists of two linear layers and a sigmoid activation layer. The mutual information between the global average feature and the given label is maximized through the mutual information maximization network. If the global average feature and the label are from the same sample, the output of the global mutual information maximization network is close to 1 (positive sample). If the global average feature and the label are not paired samples, the output of the global mutual information maximization network is close to 0 (negative sample).

[0083] The above method embodiment has the following beneficial effects:

[0084] (1) In obtaining a sample data set for training a model, after obtaining a first Cantonese video clip, remove the clips with human voice but no image and / or the clips with mismatched human voice and image in the first Cantonese video clip to obtain a second Cantonese video clip, and use the second Cantonese video clip to generate a sample sequence containing a label. Removing useless sequences from the video sequence can improve the recognition accuracy of the trained model.

[0085] (2) The traditional lip reading recognition model uses the ResNet-18+GRU structure. This method uses the ResNet-34 network, which is deeper than the ResNet-18 network, and can extract deeper features.

[0086] (3) Since the boundary information is unclear and key frames and useless frames are difficult to distinguish in the lip reading task, adding boundary information to the sample sequence before inputting the samples in the sample data set into the model can solve the problem that useless boundary frames cannot be eliminated. An LSTM network consisting of an LSTM and a linear layer is added to the back end of the feature extraction network. The LSTM network is used to filter key frames by generating weights, so that key frames and non-key frames can be correctly distinguished, which can effectively improve the lip reading recognition accuracy of the model.

[0087] (4) In order to improve the ability to capture fine-grained lip movements and corresponding labels and further improve the accuracy of model recognition, this method also designs a mutual information maximization network consisting of two linear layers and a sigmoid activation layer. The features output by the weights and three-layer BiGRU network are weighted to obtain the global average features. The global average features and labels are concatenated as the input of the global mutual information maximization network. The mutual information maximization network is used to maximize the mutual information of the global average features and labels. If it is a paired sample, the output of the global mutual information maximization network is close to 1, otherwise it is close to 0, which can effectively improve the lip reading recognition accuracy of the model.

[0088] Reference Figures 4 to 6 In order to facilitate the understanding of those skilled in the art, an embodiment of the present invention provides a Cantonese lip reading recognition method, the method comprising the following steps:

[0089] Step S210: construct a sample data set for Cantonese lip reading.

[0090] Step S2101: crawl Cantonese TV programs from the Internet through the you-get tool to obtain video segments. The Cantonese TV programs here include but are not limited to Cantonese news broadcasts, Cantonese variety shows, Cantonese character interviews and talk shows.

[0091] Step S2102: Remove useless segments from the video segment. The useless segments here refer to segments with human voice but no image, or segments with mismatched human voice and image.

[0092] Step S2103, separate the audio sequence and video sequence of the video segment, use a speech processing tool (such as the transcription function of iFLYTEK speech) to segment the audio sequence and generate a timestamp, and name the video sequence and audio in the same way, giving the video sequence and its corresponding audio sequence the same name for easy pairing later.

[0093] Step S2104: Extend all word segmentation timestamps by 0.02s on the left and right, and generate labels in the order of video sequence name, word segmentation timestamp, word segmentation pinyin, and word segmentation.

[0094] Step S2105: Use the MediaPipe tool to extract faces from the video sequence, and train a filter to filter out all images that are incompletely cropped for face recognition.

[0095] The above steps S2101 to S2105 can more effectively eliminate useless fragments and ensure that the collected data has environmental diversity and is closer to life scenes.

[0096] Step S220: construct a Cantonese lip reading recognition model based on maximizing global mutual information.

[0097] The Cantonese lip reading recognition model is mainly composed of a backbone network combining a ResNet-34 network and a three-layer BiGRU network, as well as a global mutual information maximization network.

[0098] First, add a 3D CNN layer and a spatial maximum pooling layer before the ResNet-34 network. By using the 3D CNN layer to extract features from the initial frame of the input video sequence, the purpose of preliminary time alignment is achieved. Then, spatial maximum pooling is used to compress the features in the spatial domain, which can reduce the training time without affecting the recognition effect. Adding a 3D CNN layer and a spatial maximum pooling layer before the ResNet-34 network can make the Cantonese lip reading network model achieve better recognition results.

[0099] Assume that the number of frames of the input video sequence is T. According to the number of frames of the input sequence T, the sequence features are evenly divided into T parts, and then the features of each part are extracted using the ResNet-34 network. In order to improve the ability to capture fine-grained lip movements and the corresponding label L, a mutual information constraint is imposed between the output of the ResNet-34 network and the label L. After the mutual information constraint is imposed, the acquired features are input into the global average pooling layer. The purpose of using global average pooling instead of the fully connected layer here is to prevent overfitting by regularizing the structure of the entire network.

[0100] At the output of the ResNet-34 network, add an LSTM and a linear layer. An LSTM and a linear layer form an LSTM network. The added LSTM network will assign different weights a to different frames of the label L. In order to filter key frames, the weight a can be any positive number, but for the weight a of useless frames, it should be as close to 0 as possible. So Relu is introduced to obtain the weight a:

[0101] a t =Relu(w linear ×LSTM(G) t +b linear )

[0102] In the above formula, G represents the output of the global average pooling layer, w linear and b linear Represents the parameters of the linear layer, LSTM(G) t represents the hidden state of the LSTM layer at time step t, a t Represents the weight of the frame sequence at time step t.

[0103] The final global average feature F consists of the output P and weight a of the three-layer BiGRU network.

[0104]

[0105] Where T represents the length of the entire video sequence, that is, the total number of frames.

[0106] The global average feature F and the one-hot vector of the label L are concatenated together as the input of the global mutual information maximization network.

[0107] The global mutual information maximization network consists of two linear layers and a sigmoid activation layer. The global mutual information maximization network can maximize the mutual information between the global average feature and the given label. If the global average feature F and the label L come from the same sample, the output of the global mutual information maximization network should be as close to 1 as possible (positive sample). If the global average feature F and the label L are not paired samples, the output of the global mutual information maximization network should be as close to 0 as possible (negative sample). Therefore, the global mutual information maximization network can be expressed as:

[0108] Loss MI =E p(F,L) [(log(MI (F，L) )]+E p(F)p(L) [(log(1-MI (F，L) )]

[0109] Among them, Loss MI represents the optimization function for maximizing global mutual information, p(F,L) represents the joint distribution of sample pairs (F,L), p(F)p(L) represents the marginal distribution of sample pairs (F,L), and MI (F，L) It represents the mutual information between the two variables F and L, and E is used to obtain the mathematical expectation.

[0110] The output P of the three-layer BiGRU network in the backbone network t After the linear layer, the dimension becomes C, and the final classification score is shown below:

[0111]

[0112] Where c represents the total number of classes in the vocabulary in the dataset. Indicates label L i score.

[0113] The loss function of the entire model is as follows:

[0114]

[0115] Among them, L irepresents the i-th label, Loss total Represents the total loss, the first term of the loss function is the cross entropy loss function, the second term Loss MI is the optimization function of the network that maximizes the global mutual information.

[0116] Step S230: adding boundary information to the data in the dataset and training a Cantonese lip reading recognition model.

[0117] Step S2301, perform boundary information addition processing. The video sequence is cropped into 88×88 size, and then a boundary information is added to the video sequence according to the timestamp information. The processing method for adding boundary information here is: select the integer of OP (start timestamp)×25 (video frame rate) as the start frame, select the next 40 frames as the input video sequence and filter out data with a length of less than 40 frames. By adding boundary information, the problem that useless boundary frames cannot be removed can be solved.

[0118] Step S2302: Encode the video sequence through Libjpeg, set the batch-size and epoch training parameters, input the encoded data into the Cantonese lip reading recognition model for training, obtain the training weight, and obtain the trained Cantonese lip reading recognition model. The purpose of encoding here is to compress the data after adding boundary information processing, so as to speed up the subsequent training.

[0119] Step S240: Recognize the target video sequence using the trained Cantonese lip reading recognition model.

[0120] Step S2401, use the pyside2 toolkit to design the UI interface of the system, the UI interface is divided into three parts: the prompt area, the face display area and the result display area.

[0121] Step S2402: load the training weights, load the Cantonese lip reading recognition model, and match the buttons on the UI interface with the functions in the code.

[0122] Step S2403, start recognition, collect the target video sequence through the face display area, and use MediaPipe to extract the face in the target video sequence.

[0123] Step S2404: load the network and process the target video sequence using the Cantonese lip reading recognition model.

[0124] Step S2405: Display the lip-reading recognition result obtained by the Cantonese lip-reading recognition model in the display area.

[0125] The research on lip reading recognition is of great significance to the hearing-impaired. Cantonese is widely used in Guangdong and Guangxi regions of my country, as well as in Hong Kong and Macao, as well as in Chinese communities around the world. Therefore, the research on Cantonese lip reading is urgent. At present, no company or large research institution has launched a large-scale Cantonese lip reading dataset. This application can obtain word-level sample data by downloading online video resources, manually trimming and screening useless segments. Since the boundary information is unclear and the key frames and useless frames are difficult to distinguish in the lip reading task, in order to solve these problems, this method first pre-processes the data in the sample dataset, and by adding boundary information, the problem that useless boundary frames cannot be removed can be solved. After that, the processed data is put into the Cantonese lip reading recognition model (composed of ResNet-34 network and three-layer BiGRU network as the backbone network and global mutual information maximization network) for training. The existing model is a ResNet-18+GRU structure. This method makes improvements on the existing model, replacing the ResNet-18 network with the ResNet-34 network to extract deeper features, adding an LSTM network (composed of an LSTM and a linear layer) at the output of the ResNet-34 network, and using the LSTM network to filter key frames by generating weights a, and using Relu to generate a, so that the weights of useless frames can be reduced to 0. Then, the weights a and the features output by the three-layer BiGRU network are weighted to obtain the global average feature F. In order to improve the ability to capture fine-grained lip movements and corresponding labels and further improve the accuracy of model recognition, the global average feature F and label L are concatenated as the input of the global mutual information maximization network, and then the mutual information of the global average feature F and label L is maximized. If it is a paired sample, the output of the global mutual information maximization network is close to 1, otherwise, it is close to 0.

[0126] This method fills the gap in the field of Cantonese lip reading, which is the lack of a large-scale lip reading sample dataset. In the Cantonese lip reading recognition model used, the accuracy of lip reading recognition can be effectively improved by proposing a method of adding boundary information and maximizing global mutual information.

[0127] Reference Figure 7 The present application also provides a computer device 301, comprising: a memory 310, a processor 320, and a computer program 311 stored in the memory 310 and executable on the processor. When the processor 320 executes the computer program 311, it implements: the Cantonese lip reading recognition method as described above.

[0128] The processor 320 and the memory 310 may be connected via a bus or in other ways.

[0129] The memory 310, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory 310 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory 310 may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0130] The non-transitory software program and instructions required to implement the Cantonese lip reading recognition method of the above embodiment are stored in the memory. When executed by the processor, the Cantonese lip reading recognition method of the above embodiment is executed, for example, the above described Figure 1 Steps S110 to S160 of the method or Figure 4 The method comprises steps S210 to S240.

[0131] Reference Figure 8 The present application also provides a computer-readable storage medium 401 storing computer-executable instructions 410, and the computer-executable instructions 410 are used to execute: the Cantonese lip reading recognition method as described above.

[0132] The computer-readable storage medium 401 stores computer-executable instructions 410. The computer-executable instructions 410 are executed by a processor or a controller, for example, by a processor in the above electronic device embodiment, so that the above processor can execute the Cantonese lip reading recognition method in the above embodiment, for example, execute the above described Figure 1 Steps S110 to S160 of the method or Figure 4 The method comprises steps S210 to S240.

[0133] It will be appreciated by those skilled in the art that all or some of the steps and systems in the disclosed method above may be implemented as software, firmware, hardware and appropriate combinations thereof. Some physical components or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor or a microprocessor, or may be implemented as hardware, or may be implemented as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or a non-transitory medium) and a communication medium (or a temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing data (such as computer-readable instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage or other magnetic storage devices, or any other medium that may be used to store desired data and may be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any data delivery media.

[0134] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0135] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

Claims

1. A Cantonese lip reading recognition method, characterized in that: The steps include: Get the first Cantonese video clip; Cutting out useless segments from the first Cantonese video segment to obtain a second Cantonese video segment, wherein the useless segments include segments with human voice but no image and / or segments where the human voice and the image do not match; Dividing the video sequence and the audio sequence in the second Cantonese video clip, segmenting the audio sequence and generating a segmentation timestamp, and generating a label according to the segmentation and the segmentation timestamp; Extracting face images from the video sequence, filtering incomplete face images, and generating a sample sequence according to the filtered face images and the labels; Training a preset Cantonese lip reading recognition model according to the sample sequence to obtain the trained Cantonese lip reading recognition model; The target video sequence is recognized according to the trained Cantonese lip reading recognition model to obtain a recognition result.

2. The Cantonese lip reading recognition method according to claim 1, characterized in that: Before training the preset Cantonese lip reading recognition model according to the sample sequence, the method further includes: Boundary information is added to the sample sequence, and the sample sequence with the added boundary information is encoded according to Libjpeg.

3. The Cantonese lip reading recognition method according to claim 2, characterized in that: The Cantonese lip reading recognition model includes a feature extraction network, an LSTM network, a three-layer BiGRU network and a mutual information maximization network. The training process of the Cantonese lip reading recognition model includes: Extracting features from the sample sequence according to the feature extraction network, and setting mutual information constraints between the features and labels; Generate corresponding weights based on different frames of the label according to the LSTM network; Classify the features according to the three-layer BiGRU network to obtain an output result; Generate a global average feature according to the output result and the weight; The mutual information between the global average feature and the label is maximized according to the mutual information maximization network.

4. The Cantonese lip reading recognition method according to claim 3, characterized in that: The feature extraction network includes a 3D CNN network, a spatial maximum pooling layer, a ResNet network and a global average pooling layer connected in sequence; extracting features in the sample sequence according to the feature extraction network and setting mutual information constraints between the features and labels include: Extracting initial features from the sample sequence according to the 3D CNN network; compressing the initial features according to the spatial maximum pooling layer; Divide the initial features into multiple parts, extract features of each part according to the ResNet network, and add mutual information constraints between the features and labels; The features with added mutual information constraints are average pooled according to the global average pooling layer.

5. The Cantonese lip reading recognition method according to claim 4, characterized in that: The ResNet network is a ResNet-34 network.

6. The Cantonese lip reading recognition method according to claim 4, characterized in that: The LSTM network includes LSTM layers and linear layers connected in sequence; the calculation formula for generating corresponding weights based on different frames of the label according to the LSTM network include: am t (Relu(w linear ×LSTM(G) t +b linear ) Wherein, G represents the output result of the spatial maximum pooling layer, and w linear and the b linear represents the parameters of the linear layer, the LSTM (G) t represents the hidden state of the LSTM layer at time step t, Relu() represents the Relu function, and the a t Represents the weight of the frame sequence when the time step is t.

7. The Cantonese lip reading recognition method according to claim 6, characterized in that: The optimization function of the mutual information maximization network includes: Loess MI =E p(F，L) [(log(MI (F，L) )]+E p(F)p(L) [(log(1-MI (F，L) )] Among them, the Loss MI represents the optimization function of the global mutual information maximization network, the p(F, L) represents the joint distribution of the sample pair (F, L), the p(F)p(L) represents the marginal distribution of the sample pair (F, L), the MI (F，L) represents the mutual information between F and L, E is used to obtain the mathematical expectation, F represents the global average feature, and L represents the label.

8. The Cantonese lip reading recognition method according to claim 7, characterized in that: The loss function of the Cantonese lip reading recognition model includes: Among them, the represents the cross entropy loss function, the c represents the total number of classes of the vocabulary in the sample sequence, The label L represents the output of the three-layer BiGRU network i score.

9. An electronic device, characterized in that: It comprises at least one control processor and a memory for communicating with the at least one control processor; the memory stores instructions executable by the at least one control processor, and the instructions are executed by the at least one control processor so that the at least one control processor can execute the Cantonese lip reading recognition method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the Cantonese lip reading recognition method according to any one of claims 1 to 8.

Images