A lip reading method and device based on an edge computing terminal

Abstract

The application relates to a lip language recognition method and device based on an edge computing terminal, and comprises the following steps: S1, a lip language recognition model is constructed, a pre-training model is extracted through model pre-training of a Chinese lip language data set; S2, model compression is performed on the pre-training model; S3, video data of the edge computing terminal is continuously collected and saved as a target data set; S4, the target data set is preprocessed; S5, according to the target data set, fine-tuning training is performed on the pre-training model to obtain a light-weight model; S6, the light-weight model is optimized, the hardware platform of the edge computing terminal is adapted, and the light-weight model is deployed to the edge computing terminal; video data of a user is collected, lip language information of the user is recognized, and a recognition result of a user instruction is output. Compared with the prior art, the application has the advantages of improving the recognition effect and accuracy of speech recognition, improving the robustness of the lip language recognition model, and being beneficial to deployment on an edge computing terminal with limited computing capability.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of lip reading, in particular to a lip reading method and device based on an edge computing terminal. BACKGROUND

[0002] In recent years, with the in-depth popularization of Internet information technology and the rapid development of artificial intelligence, the field of human-computer interaction of edge computing terminals has ushered in broad development prospects. Human-computer interaction refers to the user inputting instructions, and the machine recognizing and understanding the instructions and making the expected response. Today, the depth and breadth of human-computer interaction research are expanding, including speech, gestures, actions, head tracking, and visual tracking in various forms. Among them, the most natural and convenient for people is speech interaction, also known as speech recognition (ASR). Speech recognition refers to inputting the audio of the user's speech into a computer after processing, and the computer converts the speech signal into the corresponding speech content through recognition and understanding. With the development of technology, in a quiet environment using Mandarin and common vocabulary, the accuracy of speech recognition has exceeded 95%, and speech recognition in complex scenarios such as dialects and slight noise can also be relatively accurate. However, speech recognition also has many shortcomings, such as for strong noise, super far field, strong interference, service robots in shopping malls, vehicles during travel, and roadside ATMs, the accuracy of speech recognition will be greatly reduced.

[0003] Lip reading is through visual information, such as facial expressions, body movements, and changes in lip shape. The dynamic changes in lip shape are an important visual carrier of speech content. Therefore, interpreting lip language can provide an auxiliary means for speech recognition, supplement visual information, and improve recognition results in some restricted scenarios.

[0004] Currently, the related work of lip reading based on deep learning at home and abroad is based on open source data sets AVLetters, GRID, OuluVS, and LRW, in addition to research work on small data sets such as IBMIIH and MOBIO. In order to obtain a high recognition accuracy on the data set, the model complexity also increases, which is not convenient for application personnel to optimize, and large-scale models have high requirements for the computing power of edge computing terminals. In addition, for edge computing terminals, the user's usage scenario is relatively fixed and cannot provide enough training samples, and the lack of training samples will lead to underfitting of the model. Therefore, related technologies are needed to solve the problem of insufficient labeled data in small sample learning. SUMMARY

[0005] The purpose of the present application is to overcome the defects of the prior art, and provide a lip reading method and device based on an edge computing terminal, which starts from an existing large open-source Chinese data set, pre-trains a model, and then fine-tunes and trains the model on a data set collected in another specific use scenario through model pruning and parameter migration, to obtain a final model.

[0006] The purpose of the present application can be achieved by the following technical solutions:

[0007] A lip reading method based on an edge computing terminal, specifically comprising the following steps:

[0008] S1, a lip reading model based on deep learning is constructed, a pre-trained model that performs well on a data set is extracted through model pre-training of a Chinese lip reading data set;

[0009] S2, the pre-trained model is compressed through knowledge distillation, model pruning and parameter migration;

[0010] S3, video data used by a user in a use scenario of an edge computing terminal is continuously collected and saved as a target data set for transfer learning;

[0011] S4, the target data set is pre-processed, and the audio data obtained after pre-processing is processed through speech recognition technology and audio-video timeline alignment to generate Chinese labels corresponding to the video frames;

[0012] S5, the pre-trained model after model compression is fine-tuned and trained according to the target data set after pre-processing, to obtain a lightweight model;

[0013] S6, the calculation speed of the lightweight model is optimized, the hardware platform of the edge computing terminal is adapted, and the lightweight model is deployed to the corresponding edge computing terminal, video data of a user is collected, lip information of the user is recognized, and the recognition result of the user's command is output.

[0014] The lip reading model includes a front-end network structure for processing image features, a middle-end network structure for deeply processing image information, and a back-end network structure for processing pre-context and post-context information.

[0015] The content of the Chinese lip reading data set includes various phrases, speakers, speaking angles and lighting environment data, and the form of the Chinese lip reading data set includes video data and corresponding Chinese character label files.

[0016] The model pruning process in step S2 is specifically automatic sparse neuron weight, and the network structure size of the model is reduced according to the L1 / L2 norm of the neuron weight.

[0017] The parameter migration process in step S2 is specifically to extract the multi-layer model parameters of the lip reading model into the pre-trained model, and the specific formula is as follows:

[0018]

[0019] Wherein, is the model parameter of the pre-trained model, w i is the network parameter set of one layer of the lip reading model, i∈n, the model parameter W of the lip reading model is [w1, w2, …, w n ], m<n.

[0020] The video data used by the user in step S3 includes a video of user interaction instructions and clear interaction instruction audio.

[0021] The process of preprocessing the target data set in step S4 includes data cleaning and noise reduction processing of image data and audio data in the target data set by audio and video separation to obtain video data and audio data.

[0022] The step S4 specifically includes converting the audio data into Chinese character information data by speech recognition technology, and the Chinese character information data includes instruction Chinese characters and timestamp information.

[0023] Further, the step S5 specifically includes the following steps:

[0024] S51, load the model parameters of the pre-trained model extracted by the parameter migration method into the to-be-trained lightweight lip reading model;

[0025] S52, train the to-be-trained lightweight lip reading model according to the video data and the Chinese character information data to obtain the final lightweight model;

[0026] The formula of the to-be-trained lightweight lip reading model is as follows:

[0027]

[0028] Wherein, W init is the new model parameter which is not trained and not loaded from the pre-trained model.

[0029] The target data set does not have a fixed image input resolution size, and the image is enlarged or scaled to a specific resolution when the image is cut and compressed.

[0030] The edge computing terminal can carry an AI acceleration card / FPGA that can perform deep learning computation in computing capability.

[0031] The video face recognition and cutting performed in the step S4 are based on a deep learning model algorithm.

[0032] The lip reading device based on the edge computing terminal comprises a memory, a processor, a camera and an AI acceleration card, the memory stores a computer program executable by the processor, and the processor implements the lip reading method when executing the computer program.

[0033] Compared with the prior art, the present application has the following beneficial effects:

[0034] 1. The present application starts from lip reading, in-depth analysis, realizes the silent man-machine interaction mode of the edge computing terminal, provides an auxiliary means for speech recognition, supplements visual information, and improves the recognition effect and accuracy of speech recognition in some restricted scenes.

[0035] 2. The present application starts from the existing large Chinese lip reading data set, and there is sufficient data to learn the knowledge of lip reading in the lip reading model pre-training stage, and the robustness of the lip reading model is improved.

[0036] 3. The present application adopts the model pruning method, prunes the large-scale model trained by large data, compresses the model complexity and model size, and is more conducive to deployment on actual edge computing terminals.

[0037] 4. The present application adopts the speech recognition technology, constructs the lip reading video data set used by the transfer learning as the target data set, reduces the tedious work of manually labeling Chinese characters and aligning the time axis, and is convenient for constructing a small specific data set for edge computing terminals.

[0038] 5. The present application adopts the transfer learning method, matches the pre-trained lip reading model with the target lip reading task, and since the specific task uses scene is not much, a more lightweight model can be designed, and on this basis, the real-time performance and efficiency of the model are improved, and the model is convenient for deployment to the edge computing terminal. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 The present application is a flowchart. DETAILED DESCRIPTION

[0040] The present application will be described in detail below in combination with the drawings and specific embodiments. The present embodiment is implemented on the premise of the technical scheme of the present application, and gives a detailed implementation manner and specific operation process, but the protection scope of the present application is not limited to the following embodiments.

[0041] As Figure 1 shown, a lip reading method based on an edge computing terminal, specifically comprising the following steps:

[0042] S1, a lip reading model based on deep learning is constructed, a pre-trained model with good performance on a data set is extracted by pre-training the model on a Chinese lip data set;

[0043] S2, the pre-trained model is compressed by knowledge distillation, model pruning and parameter migration;

[0044] S3, continuously collect video data used by users in the use scenario of the edge computing terminal, and save it as a target data set for transfer learning;

[0045] S4, pre-process the target data set, and generate Chinese labels corresponding to the video frames by aligning the audio data obtained after pre-processing with the video time axis through speech recognition technology and audio-video time axis alignment;

[0046] S5, according to the target data set after pre-processing, fine-tune the pre-trained model after model compression to obtain a lightweight model;

[0047] S6, optimize the calculation speed of the lightweight model, adapt to the hardware platform of the edge computing terminal and deploy it to the corresponding edge computing terminal, collect video data of users, recognize lip information of users, and output recognition results of user instructions.

[0048] The lip reading model includes a front-end network structure for processing image features, a middle-end network structure for deep processing of image information, and a back-end network structure for processing context information.

[0049] In this embodiment, the lip reading model adopts a 3DConv+ResNet-18+MS-TCN model structure.

[0050] The content of the Chinese lip data set includes various phrases, speakers, speaking angles and lighting environment data, and the form of the Chinese lip data set includes video data and corresponding Chinese character label files.

[0051] The process of model pruning in step S2 is to automatically sparsify neuron weights, and sort them according to the L1 / L2 norm of the neuron weights, to reduce the network structure size of the model.

[0052] In this embodiment, the neuron weight matrix W n×m is as follows:

[0053]

[0054] A new neuron weight matrix is obtained after deleting positions with small weight contribution As shown below:

[0055]

[0056] wherein w nm is a neuron weight, a≤n, b≤m, so the matrix size satisfies

[0057] In this embodiment, the pre-trained model obtained in step S1 is sorted by weight, and the weight of 3DConv is reduced by 20%.

[0058] The process of parameter migration in step S2 is to migrate the multi-layer model parameters of the lip speech recognition model to the pre-trained model, and the specific formula is as shown below:

[0059]

[0060] wherein, is a model parameter of the pre-trained model, w i is a network parameter set of one layer of the lip speech recognition model, i∈n, the model parameter W of the lip speech recognition model is [w1, w2, …, w n ], and m<n.

[0061] In this embodiment, all parameters of the 3DConv layer are migrated, the ResNet-18 layer migrates the parameters of the first 6 layers, and all parameters of the MS-TCN layer are migrated.

[0062] The video data used by the user in step S3 includes a video of a user interaction instruction and clear interaction instruction audio. In this embodiment, the edge computing terminal is a notebook computer, the notebook computer collects the target data set TD by using the front camera of the notebook computer to collect the target data set TD, and the voice data of the command in this link is required to be clearly recorded.

[0063] The process of pre-processing the target data set in step S4 includes data cleaning and noise reduction processing of image data and audio data in the target data set by audio and video separation, to obtain video data and audio data.

[0064] In this embodiment, OpenCV is used to pre-process the target data set.

[0065] Step S4 specifically includes converting the audio data into Chinese character information data by using a voice recognition technology, and the Chinese character information data Y=(str, timestep) includes the Chinese character str of the instruction and the timestamp information timestep.

[0066] In this embodiment, the speech recognition technology API provided by the company of the University of Science and Technology of China is adopted.

[0067] Step S5 specifically includes the following steps:

[0068] S51, load the model parameters of the pre-training model extracted by the parameter migration method to the to-be-trained lightweight lip reading model;

[0069] S52, train the to-be-trained lightweight lip reading model according to the video data and Chinese character information data, and obtain the final lightweight model;

[0070] The formula of the to-be-trained lightweight lip reading model is as follows:

[0071]

[0072] Wherein, W init is the initialization parameter of ResNet-18 without migration parameters.

[0073] There is no fixed image input resolution size in the target data set, and the image is enlarged or scaled to a specific resolution when cutting and compressing the image.

[0074] The edge computing terminal can carry an AI acceleration card / FPGA for deep learning calculation in computing capability.

[0075] The video face recognition and cutting and the speech recognition algorithm performed in step S4 are all based on deep learning model algorithm.

[0076] A lip reading device based on an edge computing terminal, comprising a memory, a processor, a camera and an AI acceleration card, the memory stores a computer program executable by the processor, and the processor implements the lip reading method when executing the computer program.

[0077] In specific implementation, the Chinese lip data set adopted in the embodiment is LRW-1000, which includes 1000 categories, and about 718018 video samples are collected from more than 2000 single speakers. There are more than 100000 Chinese character instances in total. Each lesson corresponds to a syllable of a Chinese word composed of one or more Chinese characters. This data set aims to cover the natural variability of different speech patterns and imaging conditions to include the challenges encountered in practical applications. It shows great differences in several aspects, including the number of samples per category, the resolution of the video, the lighting conditions, and the attributes of the speaker, such as posture, age, gender and makeup, etc.

[0078] The pre-training stage model experiment verification result is shown in the following table:

[0079] Model Accuracy (%) 3D Conv+ResNet-18+Bi-LSTM 32.6 3D Conv+ResNet-18+Bi-GRU 38.9 3D Conv+ResNet-18+MS-TCN 41.4

[0080] When the lightweight model of step S5 is obtained, the lightweight model is adapted to the notebook computer, and testing is performed. In the experiment, all test instructions can accurately identify 86% of the instructions, and the effect is good.

Claims

1. A lip reading method based on an edge computing terminal, characterized in that, Specifically comprising the following steps: S1, a lip reading model based on deep learning is constructed, a pre-training model with good performance on a data set is extracted by pre-training the model on a Chinese lip reading data set; the lip reading model includes a front-end network structure for processing image features, a middle-end network structure for deeply processing image information, and a back-end network structure for processing context information, and adopts a 3DConv+ResNet-18+MS-TCN model structure; S2, the pre-training model is compressed by knowledge distillation, model pruning and parameter migration; wherein the parameters of the 3DConv layer are all migrated, the parameters of the front 6 layers of the ResNet-18 layer are migrated, and the parameters of the MS-TCN layer are all migrated; S3, continuously collect video data used by users in the use scenario of the edge computing terminal as a target data set for migration learning and save; S4, pre-process the target data set, and generate Chinese labels corresponding to the video frames by speech recognition technology and audio-video timeline alignment for the pre-processed audio data; S5, fine-tune the pre-training model compressed in the model according to the pre-processed target data set to obtain a lightweight model; S6, optimize the calculation speed of the lightweight model, adapt the hardware platform of the edge computing terminal and deploy it to the corresponding edge computing terminal, collect video data of users, recognize lip information of users, and output recognition results of user instructions. 2.The lip reading method based on edge computing terminal according to claim 1, characterized in that, The content of the Chinese lip reading data set includes various phrases, speakers, speaking angles and lighting environment data, and the form of the Chinese lip reading data set includes video data and corresponding Chinese character label files. 3.The lip reading method based on edge computing terminal according to claim 1, characterized in that, The process of model pruning in step S2 is specifically to automatically sparsify neuron weights, and sort according to the L1 / L2 norm of the neuron weights to reduce the network structure size of the model. 4.The lip reading method based on edge computing terminal according to claim 1, characterized in that, The process of parameter migration in step S2 is specifically to extract the multi-layer model parameters of the lip reading model and migrate them to the pre-training model, and the specific formula is as follows: in, w represents the model parameters of the pre-trained model. i Let W be the set of network parameters for one layer of the lip-reading model, i∈n, and let W = [w1, w2, ..., wn]. n ], m <n。 5.The lip reading method based on edge computing terminal according to claim 1, characterized in that, The video data used by the user in step S3 includes video of user interaction instructions and clear interaction instruction audio. 6.The lip reading method based on edge computing terminal according to claim 4, characterized in that, The process of pre-processing the target data set in step S4 includes data cleaning and noise reduction processing of image data and audio data in the target data set by audio-video separation to obtain video data and audio data. 7.The lip reading method based on edge computing terminal according to claim 6, characterized in that, The process of step S4 specifically includes converting the audio data into Chinese character information data by speech recognition technology, and the Chinese character information data includes Chinese characters of instructions and timestamp information. 8.The lip reading method based on the edge computing terminal according to claim 7, characterized in that, The step S5 specifically comprises the following steps: S51, load the model parameters of the pre-training model extracted by the parameter migration method to the lightweight lip reading model to be trained; S52, train the lightweight lip reading model to be trained according to the video data and the Chinese character information data to obtain the final lightweight model; The formula of the lightweight lip reading model to be trained is as follows: wherein W init are new model parameters that are untrained and not loaded from a pre-trained model.

9. An edge computing terminal-based lip reading device, comprising a memory, a processor, a camera and an AI acceleration card, characterized in that, The memory stores a computer program executable by the processor, and the processor executes the computer program to implement the lip-reading method according to any one of claims 1-8.

Images