Lip movement recognition method based on deep learning

Through the deep learning-based lip motion recognition method, the lip motion parameters in the face image are extracted and feature analysis is performed, which solves the problems of low accuracy and environmental sensitivity of traditional methods, and achieves more accurate and robust mouth motion recognition .

CN118675232BActive Publication Date: 2024-11-05HANGZHOU YOUHANG INFORMATION TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411157382.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-22
Publication Date
2024-11-05
Estimated Expiration
2044-08-22

AI Technical Summary

Technical Problem

The traditional lip motion state recognition method has limited accuracy, is difficult to capture complex lip motion characteristics, is sensitive to light and head posture changes, and is susceptible to environmental factors and noise.

Method used

Using the deep learning-based lip motion recognition method, the time series of face images is collected through the camera, face recognition and key point detection are performed, the set of lip motion parameters is extracted, the embedding encoding of lip motion parameters and the extraction of temporal context features are performed, and the feature distribution is used. The reverse mask clustering analysis network generates a lip feature timing aggregation representation vector to identify whether the mouth is moving.

Benefits of technology

It improves the accuracy and robustness of mouth motion recognition, can capture lip motion characteristics more effectively, reduces the impact on light and head posture changes, and enhances the generalization ability and prediction performance of lip motion recognition.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118675232B_ABST
    Figure CN118675232B_ABST
Patent Text Reader

Abstract

The present application relates to a lip movement recognition method based on deep learning. The method includes: performing face recognition and key point detection on the time series of face images to obtain the time series of face key point parameter sets; extracting the time series of lip movement parameter sets from the time series of face key point parameter sets; performing lip movement parameter embedding coding and temporal context feature extraction on the time series of lip movement parameter sets to obtain the time series of contextual lip movement parameter semantic coding feature vectors; inputting the time series of contextual lip movement parameter semantic coding feature vectors into a feature distribution reverse mask clustering analysis network to obtain a lip movement feature temporal aggregation representation vector, and then determining the recognition result, which is used to indicate whether the mouth is moving. In this way, it is possible to identify and detect whether the mouth is moving to help determine when to perform speech analysis.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of lip movement recognition, and in particular, to a lip movement recognition method based on deep learning. Background Art

[0002] Lip movement recognition refers to the technology of understanding the semantic information of the speaker by analyzing and identifying human mouth movements. Lip movement has important application value in speech recognition, emotion recognition, lip shape recognition and other fields. By identifying and analyzing mouth movements, it is possible to understand and infer the content of speech, emotional state and lip shape information. Therefore, lip movement recognition can be used as a hearing aid to help hearing-impaired people better understand the content of the conversation.

[0003] Lip movement recognition can be used as a pre-step for speech recognition to help determine when to perform speech analysis. However, traditional lip movement recognition schemes usually rely on the subjective judgment of the listener or simple processing and analysis of the speaker's facial image to identify the lip movements of the face, thereby inferring the spoken language and content information. These traditional methods have difficulty capturing complex lip movement features, resulting in limited recognition accuracy. In addition, traditional lip movement recognition methods that rely on image processing are sensitive to lighting conditions and changes in head posture, and are easily affected by environmental factors and noise.

[0004] Therefore, an optimized lip movement recognition solution is desired. Summary of the invention

[0005] This summary is provided to introduce concepts in a brief form that will be described in detail in the detailed description below. This summary is not intended to identify key features or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0006] In a first aspect, the present application provides a lip movement recognition method based on deep learning, the method comprising:

[0007] Obtain a time series of face images captured by a camera;

[0008] Performing face recognition and key point detection on the time series of face images to obtain a time series of face key point parameter sets;

[0009] Extracting a time series of a lip movement parameter set from the time series of the facial key point parameter set, wherein the lip movement parameter set includes a lip opening and closing score, a pitch angle, a yaw angle, and a roll angle;

[0010] Performing lip movement parameter embedding coding and temporal context feature extraction on the time series of the lip movement parameter set to obtain a time series of contextual lip movement parameter semantic coding feature vectors;

[0011] Inputting the time series of the contextual lip movement parameter semantic encoding feature vector into a feature distribution reverse mask clustering analysis network to obtain a lip movement feature time series aggregation representation vector;

[0012] Based on the temporal aggregation representation vector of the lip movement features, a recognition result is determined, and the recognition result is used to indicate whether the mouth is moving.

[0013] Optionally, face recognition and key point detection are performed on the time series of face images to obtain a time series of face key point parameter sets, including: inputting the time series of face images into a face recognizer based on a Yolo-v5 model to obtain a time series of face frame area images; and performing face key point detection on each face frame area image in the time series of face frame area images to obtain a time series of face key point parameter sets.

[0014] Optionally, lip movement parameter embedding encoding and temporal context feature extraction are performed on the time series of the lip movement parameter sets to obtain a time series of contextual lip movement parameter semantic encoding feature vectors, including: inputting each lip movement parameter set in the time series of the lip movement parameter sets into a lip movement parameter embedding encoder based on a multi-layer perceptron to obtain a time series of lip movement parameter embedding encoding vectors; inputting the time series of the lip movement parameter embedding encoding vectors into a lip movement parameter temporal context encoder based on a Transformer structure to obtain a time series of contextual lip movement parameter semantic encoding feature vectors.

[0015] Optionally, the time series of the contextual lip movement parameter semantic encoding feature vector is input into a feature distribution reverse mask clustering analysis network to obtain a lip movement feature temporal aggregation representation vector, including: clustering the time series of the contextual lip movement parameter semantic encoding feature vector to obtain a contextual lip movement parameter semantic clustering center feature vector; calculating the semantic correlation coefficient between each contextual lip movement parameter semantic encoding feature vector in the sequence of the contextual lip movement parameter semantic encoding feature vector and the contextual lip movement parameter semantic clustering center feature vector to obtain a sequence of contextual lip movement parameter clustering center semantic correlation coefficients; taking the inverse of each contextual lip movement parameter clustering center semantic correlation coefficient in the sequence of the contextual lip movement parameter clustering center semantic correlation coefficients to obtain a sequence of contextual lip movement parameter clustering center semantic anti-correlation coefficients; using a Sigmoid function to cluster the contextual lip movement parameter semantic encoding feature vector. The sequence of semantic anti-correlation coefficients of lip movement parameter cluster centers is normalized to obtain a sequence of semantic anti-correlation weight coefficients of context lip movement parameter cluster centers; each context lip movement parameter semantic coding feature vector in the sequence of context lip movement parameter semantic coding feature vectors is weighted using each context lip movement parameter cluster center semantic anti-correlation weight coefficient in the sequence of context lip movement parameter cluster centers as a weight to obtain a sequence of suppressed context lip movement parameter semantic coding feature vectors; the positional difference between the sequence of context lip movement parameter semantic coding feature vectors and the sequence of suppressed context lip movement parameter semantic coding feature vectors is calculated to obtain a sequence of enhanced context lip movement parameter semantic coding feature vectors; the positional mean vector of the sequence of enhanced context lip movement parameter semantic coding feature vectors is calculated to obtain a fused feature vector.

[0016] Optionally, the semantic correlation coefficients between each context lip movement parameter semantic encoding feature vector in the sequence of the context lip movement parameter semantic encoding feature vectors and the context lip movement parameter semantic cluster center feature vector are calculated to obtain a sequence of context lip movement parameter cluster center semantic correlation coefficients, including: respectively calculating the norms of the context lip movement parameter semantic encoding feature vector and the context lip movement parameter semantic cluster center feature vector to obtain a context lip movement parameter semantic encoding norm representation value and a context lip movement parameter semantic cluster center norm representation value; calculating the context lip movement parameter semantic encoding norm representation value and the context lip movement parameter semantic cluster center norm representation value. The product of the context lip movement parameter semantic cluster center-norm representation values ​​is calculated to obtain the context lip movement parameter semantic-cluster center semantic correlation-norm representation value; the multiplication between the transposed vector of the context lip movement parameter semantic encoding feature vector and the context lip movement parameter semantic cluster center feature vector is calculated to obtain the context lip movement parameter semantic-cluster center semantic correlation feature value; the division between the context lip movement parameter semantic-cluster center semantic correlation feature value and the context lip movement parameter semantic-cluster center semantic correlation-norm representation value is calculated to obtain the context lip movement parameter cluster center semantic correlation coefficient.

[0017] Optionally, based on the lip movement feature time-series aggregate representation vector, a recognition result is determined, and the recognition result is used to indicate whether the mouth is moving, including: inputting the lip movement feature time-series aggregate representation vector into a classifier-based lip movement state recognizer to obtain the recognition result, and the recognition result is used to indicate whether the mouth is moving.

[0018] Optionally, it also includes a training step: for training the face recognizer based on the Yolo-v5 model, the lip movement parameter embedding encoder based on the multi-layer perceptron, the lip movement parameter temporal context encoder based on the Transformer structure, the feature distribution reverse mask clustering analysis network and the classifier-based lip movement state recognizer.

[0019] Optionally, the training step includes: obtaining a time series of training face images captured by a camera, and a true value of whether the mouth is moving; inputting the time series of training face images into the face recognizer based on the Yolo-v5 model to obtain a time series of training face frame area images; performing face key point detection on each training face frame area image in the time series of training face frame area images to obtain a time series of training face key point parameter sets; extracting a time series of training lip movement parameter sets from the time series of training face key point parameter sets, wherein the training lip movement parameter sets include training lip opening and closing scores, training pitch angles, training yaw angles and training roll angles; inputting each training lip movement parameter set in the time series of the training lip movement parameter sets into the lip movement parameter embedding encoder based on the multilayer perceptron to obtain a time series of training lip movement parameter embedding coding vectors; and The time series of the vector is input into the lip movement parameter temporal context encoder based on the Transformer structure to obtain the time series of the training context lip movement parameter semantic encoding feature vector; the time series of the training context lip movement parameter semantic encoding feature vector is input into the feature distribution reverse mask clustering analysis network to obtain the training lip movement feature temporal aggregation representation vector; the training lip movement feature temporal aggregation representation vector is passed through the classifier-based lip movement state recognizer to obtain a recognition result; the cross entropy loss function value between the recognition result and the true value is calculated to obtain a classification loss function value; based on the classification loss function value, the Yolo-v5 model-based face recognizer, the multi-layer perceptron-based lip movement parameter embedding encoder, the Transformer-based lip movement parameter temporal context encoder, the feature distribution reverse mask clustering analysis network and the classifier-based lip movement state recognizer are trained.

[0020] The above technical solution is adopted to obtain the time series of face key point parameter set by performing face recognition and key point detection on the time series of face images; extracting the time series of lip movement parameter set from the time series of face key point parameter set; performing lip movement parameter embedding coding and temporal context feature extraction on the time series of lip movement parameter set to obtain the time series of contextual lip movement parameter semantic coding feature vector; inputting the time series of contextual lip movement parameter semantic coding feature vector into the feature distribution reverse mask clustering analysis network to obtain the lip movement feature temporal aggregation representation vector, and then determining the recognition result, which is used to indicate whether the mouth is moving. In this way, it is possible to identify and detect whether the mouth is moving to help determine when to perform speech analysis.

[0021] Other features and advantages of the present application will be described in detail in the subsequent detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The above and other features, advantages and aspects of the embodiments of the present application will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale. In the drawings:

[0023] Figure 1 The present invention is a flowchart of a lip movement recognition method based on deep learning according to an exemplary embodiment.

[0024] Figure 2 is a flowchart of another lip movement recognition method based on deep learning according to an exemplary embodiment.

[0025] Figure 3 It is a block diagram of a lip movement recognition system based on deep learning according to an exemplary embodiment.

[0026] Figure 4 It is a block diagram of an electronic device according to an exemplary embodiment.

[0027] Figure 5 This is an application scenario diagram of a lip movement recognition method based on deep learning according to an exemplary embodiment. DETAILED DESCRIPTION

[0028] The embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be construed as being limited to the embodiments described herein. Instead, these embodiments are provided to provide a more thorough and complete understanding of the present application. It should be understood that the drawings and embodiments of the present application are only for exemplary purposes and are not intended to limit the scope of protection of the present application.

[0029] It should be understood that the various steps described in the method implementation of the present application can be performed in different orders and / or performed in parallel. In addition, the method implementation may include additional steps and / or omit the steps shown. The scope of the present application is not limited in this respect.

[0030] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0031] It should be noted that the concepts such as "first" and "second" mentioned in this application are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0032] It should be noted that the modifications of "one" and "plurality" mentioned in the present application are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0033] The names of the messages or information exchanged between multiple devices in the embodiments of the present application are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0034] In order to solve the above problems, the present application provides a lip movement recognition method based on deep learning, which performs face recognition and key point detection on the time series of face images to obtain the time series of face key point parameter sets; extracts the time series of lip movement parameter sets from the time series of face key point parameter sets; performs lip movement parameter embedding coding and temporal context feature extraction on the time series of lip movement parameter sets to obtain the time series of contextual lip movement parameter semantic coding feature vectors; inputs the time series of contextual lip movement parameter semantic coding feature vectors into the feature distribution reverse mask clustering analysis network to obtain the lip movement feature temporal aggregation representation vector, and then determines the recognition result, which is used to indicate whether the mouth is moving. In this way, it is possible to identify and detect whether the mouth is moving to help determine when to perform speech analysis.

[0035] The specific implementation of the present application is described in detail below with reference to the accompanying drawings.

[0036] Figure 1 is a flowchart of a lip movement recognition method based on deep learning according to an exemplary embodiment. Figure 1 As shown, the method includes:

[0037] Step 101, obtaining a time series of facial images captured by a camera;

[0038] Step 102: performing face recognition and key point detection on the time series of the face images to obtain a time series of face key point parameter sets;

[0039] Step 103: extracting a time series of a lip movement parameter set from the time series of the facial key point parameter set, wherein the lip movement parameter set includes a lip opening and closing score, a pitch angle, a yaw angle, and a roll angle;

[0040] Step 104: performing lip movement parameter embedding coding and temporal context feature extraction on the time series of the lip movement parameter set to obtain a time series of contextual lip movement parameter semantic coding feature vectors;

[0041] Step 105: input the time series of the contextual lip movement parameter semantic encoding feature vector into a feature distribution reverse mask clustering analysis network to obtain a lip movement feature time series aggregation representation vector;

[0042] Step 106: Determine a recognition result based on the temporal aggregation representation vector of the lip movement features, wherein the recognition result is used to indicate whether the mouth is moving.

[0043] In response to the above technical problems, in the technical solution of the present application, a lip movement recognition method based on deep learning is proposed, which can collect time series data of facial images through a camera, and introduce image processing and analysis algorithms based on artificial intelligence and deep learning in the back end to analyze and understand the time series data of the facial images, so as to learn and capture the temporal features and aggregate representation information about the semantics of lip movement states in facial images, thereby identifying and detecting whether the mouth is moving, to help determine when to perform speech analysis.

[0044] Specifically, in the technical solution of the present application, first, a time series of facial images captured by a camera is obtained. Then, in order to quickly and accurately locate the face in the image, so as to facilitate the subsequent timing analysis and lip movement recognition monitoring of the lip semantics of the face, in the technical solution of the present application, the time series of the facial images is input into a face recognizer based on the Yolo-v5 model to obtain a time series of facial frame area images. Through the processing of the face recognizer based on the Yolo-v5 model, the face-related areas in each facial image can be effectively extracted and framed, which helps to focus attention on the facial area in the future, so as to reduce the amount of calculation and complexity of subsequent processing, while improving the accuracy of lip movement parameter extraction.

[0045] In one embodiment of the present application, face recognition and key point detection are performed on the time series of face images to obtain a time series of face key point parameter sets, including: inputting the time series of face images into a face recognizer based on the Yolo-v5 model to obtain a time series of face frame area images; and performing face key point detection on each face frame area image in the time series of face frame area images to obtain a time series of face key point parameter sets.

[0046] Then, in order to analyze the lips in the face area more finely and comprehensively, so as to perform lip movement recognition and detection more effectively, in the technical solution of the present application, face key point detection is further performed on each face frame area image in the time series of the face frame area image to obtain a time series of face key point parameter sets, and then, the time series of lip movement parameter sets is extracted from the time series of the face key point parameter sets, wherein the lip movement parameter sets include lip opening and closing scores, pitch angles, yaw angles, and roll angles. It should be understood that face key point detection can provide accurate position and parameter semantic information of facial features such as mouth, eyes, and nose, so as to further extract a set of lip movement parameters, so as to perform targeted analysis of the lip movement information and semantics of the face part to perform lip movement recognition tasks. In particular, the key point detection step can be implemented based on a heat map regression method. By extracting the lip opening and closing score, pitch angle, yaw angle and roll angle, it can help capture the movement characteristics of the lips. The lip opening and closing score reflects the degree of opening and closing of the lips, providing intuitive dynamic information for lip movement recognition. The pitch angle, yaw angle and roll angle describe the posture and rotation state of the lips in three-dimensional space, which helps to capture subtle movements of the lips and provides a more comprehensive perspective for understanding complex lip movement patterns.

[0047] Furthermore, considering the synergistic semantics and mutual influence of correlation between different lip movement parameters at each time point, the accuracy of the lip movement recognition task is jointly determined. Based on this, in order to capture the semantic association features between different lip movement parameters at each time point, in the technical solution of the present application, each lip movement parameter set in the time series of the lip movement parameter set is input into the lip movement parameter embedding encoder based on the multi-layer perceptron to obtain the time series of the lip movement parameter embedding coding vector. Through the processing of the lip movement parameter embedding encoder based on the multi-layer perceptron, lip movement parameters of different dimensions and ranges can be embedded and encoded to be mapped to a common high-dimensional space, and the semantic association relationship and coding features between these different lip movement parameters can be captured.

[0048] It should be understood that since each lip movement parameter embedded coding vector in the time series of the lip movement parameter embedded coding vector contains the lip movement parameter embedded semantic association coding feature information at different time points, and these lip movement semantics at different time points have temporal dynamic association and interactive information, these association information is of great significance for the recognition and detection of lip movements. Based on this, in the technical solution of the present application, the time series of the lip movement parameter embedded coding vector is further input into the lip movement parameter temporal context encoder based on the Transformer structure for encoding, so as to extract the contextual semantic association feature information of the lip movement parameter embedded coding features at each time point in the time dimension based on the full time domain, thereby obtaining the time series of contextual lip movement parameter semantic coding feature vectors.

[0049] In one embodiment of the present application, lip movement parameter embedding encoding and temporal context feature extraction are performed on the time series of the lip movement parameter sets to obtain a time series of contextual lip movement parameter semantic encoding feature vectors, including: inputting each lip movement parameter set in the time series of the lip movement parameter sets into a lip movement parameter embedding encoder based on a multi-layer perceptron to obtain a time series of lip movement parameter embedding encoding vectors; inputting the time series of the lip movement parameter embedding encoding vectors into a lip movement parameter temporal context encoder based on a Transformer structure to obtain a time series of contextual lip movement parameter semantic encoding feature vectors.

[0050] Furthermore, in order to extract and strengthen the lip movement features, and to effectively fuse and aggregate these lip movement context features at different time points based on time series, so as to enhance the generalization ability and prediction performance of the model for lip movement recognition and detection, in the technical solution of the present application, the time series of the semantic encoding feature vector of the contextual lip movement parameters is further input into the feature distribution reverse mask clustering analysis network to obtain the lip movement feature time series aggregate representation vector. It is worth mentioning that the feature distribution reverse mask clustering analysis network can identify the internal structure between the lip movement context features at different time points, so as to better understand the dynamic distribution and distribution pattern of the lip movement features. Specifically, by calculating the semantic correlation coefficient between each lip movement time series semantic and the representative cluster center vector, the similarity between them can be quantified, and the semantic anti-correlation coefficient can be obtained by taking the inverse of these correlation coefficients, so as to emphasize the features with large differences from the cluster center, which may reveal abnormalities or key information in the data, thereby enhancing the distinguishability and importance of the lip movement features. Subsequently, the Sigmoid function is used to normalize these anti-correlation coefficients to ensure that the weight coefficients are within a reasonable range, providing a balanced weight distribution for subsequent feature weighting. Then, by generating a suppression feature vector, the features that may be interference components or noise components are filtered out, thereby improving the purity and effectiveness of the lip movement features. In this way, the temporal semantic information of the lip movement context after feature enhancement at each time point can be integrated to form a key feature vector that can represent the entire data set, providing a richer and more discriminative lip movement feature representation for deep learning models for lip movement recognition, and enhancing the model's generalization ability for different lip movement recognition tasks.

[0051] In one embodiment of the present application, the time series of the contextual lip movement parameter semantic encoding feature vector is input into a feature distribution inverse mask clustering analysis network to obtain a lip movement feature temporal aggregation representation vector, including: clustering the time series of the contextual lip movement parameter semantic encoding feature vector to obtain a contextual lip movement parameter semantic clustering center feature vector; calculating the semantic correlation coefficient between each contextual lip movement parameter semantic encoding feature vector in the sequence of the contextual lip movement parameter semantic encoding feature vector and the contextual lip movement parameter semantic clustering center feature vector to obtain a sequence of contextual lip movement parameter clustering center semantic correlation coefficients; taking the inverse of each contextual lip movement parameter clustering center semantic correlation coefficient in the sequence of the contextual lip movement parameter clustering center semantic correlation coefficients to obtain a sequence of contextual lip movement parameter clustering center semantic anti-correlation coefficients; using a Sigmoid function to The sequence of semantic anti-correlation coefficients of the context lip movement parameter cluster centers is normalized to obtain a sequence of semantic anti-correlation weight coefficients of the context lip movement parameter cluster centers; each context lip movement parameter semantic coding feature vector in the sequence of context lip movement parameter semantic coding feature vectors is weighted using each context lip movement parameter cluster center semantic anti-correlation weight coefficient in the sequence of context lip movement parameter cluster centers as a weight to obtain a sequence of suppressed context lip movement parameter semantic coding feature vectors; the positional difference between the sequence of context lip movement parameter semantic coding feature vectors and the sequence of suppressed context lip movement parameter semantic coding feature vectors is calculated to obtain a sequence of enhanced context lip movement parameter semantic coding feature vectors; the positional mean vector of the sequence of enhanced context lip movement parameter semantic coding feature vectors is calculated to obtain a fused feature vector.

[0052] Furthermore, in one embodiment of the present application, the semantic correlation coefficients between each context lip movement parameter semantic encoding feature vector in the sequence of the context lip movement parameter semantic encoding feature vectors and the context lip movement parameter semantic cluster center feature vector are calculated to obtain a sequence of context lip movement parameter cluster center semantic correlation coefficients, including: respectively calculating the norms of the context lip movement parameter semantic encoding feature vectors and the context lip movement parameter semantic cluster center feature vectors to obtain context lip movement parameter semantic encoding a norm representation value and context lip movement parameter semantic cluster center a norm representation value; calculating the context lip movement parameter semantic encoding a norm representation value The product of the numerical representation value and the norm representation value of the semantic cluster center of the context lip movement parameter is calculated to obtain the context lip movement parameter semantic-cluster center semantic correlation-norm representation value; the multiplication between the transposed vector of the context lip movement parameter semantic encoding feature vector and the feature vector of the context lip movement parameter semantic cluster center is calculated to obtain the context lip movement parameter semantic-cluster center semantic correlation feature value; the division between the context lip movement parameter semantic-cluster center semantic correlation feature value and the context lip movement parameter semantic-cluster center semantic correlation-norm representation value is calculated to obtain the context lip movement parameter cluster center semantic correlation coefficient.

[0053] Specifically, the time series of the contextual lip movement parameter semantic encoding feature vector is input into the feature distribution reverse mask clustering analysis network to be processed by the following clustering analysis formula to obtain the lip movement feature time series aggregation representation vector;

[0054] Wherein, the cluster analysis formula is:

[0055]

[0056] in, is a time series of semantic encoding feature vectors of the contextual lip movement parameters, The first contextual lip movement parameter semantic encoding feature vector, is the number of vectors in the time series of the semantic encoding feature vector of the contextual lip movement parameter, is the semantic clustering center feature vector of contextual lip movement parameters, is the one-norm of the vector, For the said The semantic correlation coefficient between the contextual lip movement parameter semantic encoding feature vector and the contextual lip movement parameter semantic cluster center feature vector, For the said The semantic anti-correlation coefficient corresponding to the semantic encoding feature vector of the contextual lip movement parameters, For the said The semantic anti-correlation weight coefficient corresponding to the semantic encoding feature vector of the contextual lip movement parameters, For the said The suppression context lip movement parameter semantic encoding feature vector corresponding to the context lip movement parameter semantic encoding feature vector, To suppress the sequence of semantic encoding feature vectors of contextual lip movement parameters, To strengthen the semantic encoding feature vector of contextual lip movement parameters, To strengthen the sequence of semantic encoding feature vectors of contextual lip movement parameters, is a temporal aggregation representation vector of the lip movement features.

[0057] Then, the lip movement feature time series aggregation representation vector is input into the lip movement state recognizer based on the classifier to obtain a recognition result, and the recognition result is used to indicate whether the mouth is moving. In other words, the lip movement state is effectively recognized by using the time series aggregation representation information of the lip movement features for classification processing to detect whether the mouth is moving. This can serve as a pre-step for speech recognition, helping to determine when to perform speech analysis, and providing a basis for subsequent analysis and understanding of the conversation content.

[0058] In the technical solution of the present application, a face recognition device based on the Yolo-v5 model is used to focus on the face part in the face image, and then a face key point detection algorithm is used to identify the face key points in the face frame area image and extract the time series of lip movement parameter sets, wherein the lip movement parameter set includes lip opening and closing scores, pitch angles, yaw angles and roll angles. Considering that each lip movement parameter set in the time series of the lip movement parameter set represents the lip movement state at each predetermined time point, and the time series of the lip movement parameter set is used to represent the temporal change pattern of the lip movement state, therefore, in the technical solution of the present application, each lip movement parameter set in the time series of the lip movement parameter set is further subjected to lip movement parameter embedding encoding based on a multilayer perceptron to capture the lip movement state semantic information contained in the lip movement parameter set at each predetermined time point, and the obtained time series of the lip movement parameter embedding encoding vector is input into a lip movement parameter temporal context encoder based on a Transformer structure to capture the temporal correlation information of the lip movement state. However, when the Transformer-based lip movement parameter temporal context encoder is used to perform context encoding based on the transformer structure on the time series of the lip movement parameter embedded coding vector, considering that the coding structure characteristics of the transformer structure will ignore the influence of temporal order information on feature expression, this will cause the time series of the contextual lip movement parameter semantic coding feature vector to have feature distribution structure temporal ambiguity. In this way, the lip movement feature temporal aggregation representation vector obtained by inputting the time series of the contextual lip movement parameter semantic coding feature vector into the feature distribution reverse mask clustering analysis network will have a technical problem of insufficient feature distribution aggregation coverage, thereby causing class regression reasoning mapping deviation, affecting the accuracy of the recognition result obtained by inputting the lip movement feature temporal aggregation representation vector into the classifier-based lip movement state recognizer.

[0059] Preferably, in a preferred example of the present application, when the lip movement feature temporal aggregation representation vector is input into a classifier-based lip movement state identifier to obtain a recognition result, the lip movement feature temporal aggregation representation vector is optimized, and the optimization process specifically includes:

[0060] Calculate the feature mean of the lip movement feature time series aggregation representation vector, and divide the feature mean by the difference between the maximum eigenvalue and the minimum eigenvalue of the lip movement feature time series aggregation representation vector to obtain the lip movement feature time series aggregation distribution representation value;

[0061] Subtract the lip movement feature time series aggregate distribution representation value from one and divide it by the lip movement feature time series aggregate distribution representation value to obtain a lip movement feature time series aggregate distribution modulation value;

[0062] Activating the lip movement feature time series aggregation representation vector through a probabilistic function to obtain a probabilistic lip movement feature time series aggregation representation vector;

[0063] After performing point subtraction between the probabilistic lip movement feature time series aggregate representation vector and the lip movement feature time series aggregate distribution modulation value, taking the absolute value and calculating the negative of the logarithmic value with base 2 to obtain the probabilistic lip movement feature time series aggregate distribution modulation information vector;

[0064] After dividing the lip movement feature time series aggregate distribution representation value by one minus the difference of each eigenvalue of the probabilistic lip movement feature time series aggregate representation vector, summing all eigenvalues ​​of the probabilistic lip movement feature time series aggregate representation vector and dividing by the length of the lip movement feature time series aggregate representation vector to obtain a probabilistic lip movement feature time series aggregate distribution modulation bias value;

[0065] The product of the probabilistic lip movement feature timing aggregate distribution modulation information vector and the probabilistic lip movement feature timing aggregate distribution modulation bias value and the weight as a hyperparameter is multiplied to obtain an optimized lip movement feature timing aggregate representation vector.

[0066] The optimized representation of the temporal aggregation representation vector of the lip movement feature is:

[0067]

[0068] in, is the probabilistic lip movement feature temporal aggregation representation vector, The middle one representing the probabilistic lip movement feature temporal aggregation representation vector eigenvalues, is the temporal aggregate distribution representation value of the lip movement feature, is the length of the temporal aggregation representation vector of the lip movement feature, is the weight hyperparameter, It means to make a difference by position. It means adding by position. is the optimized lip movement feature temporal aggregation representation vector

[0069] And among them:

[0070]

[0071] in, represents the feature mean of the temporal aggregation representation vector of the lip movement feature, and represent the maximum eigenvalue and the minimum eigenvalue of the temporal aggregation representation vector of the lip movement feature, respectively. It is the temporal aggregate distribution representation value of the lip movement feature.

[0072] That is, in the above preferred example, the eigenvalue-based probability information distribution planning of the lip movement feature temporal aggregate representation vector is performed through the Bernoulli probability modulation distribution of the lip movement feature temporal aggregate representation vector relative to the eigenvalue distribution, and the probability reverse mapping of the probability features of the lip movement feature temporal aggregate representation vector as a whole is used as the extended coverage of the set mapping space of the lip movement feature temporal aggregate representation vector to independently understand the interaction path between the intuitive probability information distribution and the abstract probability space mapping of the lip movement feature temporal aggregate representation vector, so as to improve the accuracy of the recognition result obtained by the optimized lip movement feature temporal aggregate representation vector input to the classifier-based lip movement state recognizer by avoiding the counterfactual reasoning mapping of the outlier feature distribution of the lip movement feature temporal aggregate representation vector to the class regression probability.

[0073] In one embodiment of the present application, a recognition result is determined based on the lip movement feature time-series aggregate representation vector, and the recognition result is used to indicate whether the mouth is moving, including: inputting the lip movement feature time-series aggregate representation vector into a classifier-based lip movement state recognizer to obtain the recognition result, and the recognition result is used to indicate whether the mouth is moving.

[0074] Furthermore, in one embodiment of the present application, the deep learning-based lip movement recognition method also includes a training step: for training the Yolo-v5 model-based face recognizer, the multi-layer perceptron-based lip movement parameter embedding encoder, the Transformer-based lip movement parameter temporal context encoder, the feature distribution inverse mask clustering analysis network and the classifier-based lip movement state recognizer. The training step includes: obtaining a time series of training face images collected by a camera, and a true value of whether the mouth is moving; inputting the time series of training face images into the face recognizer based on the Yolo-v5 model to obtain a time series of training face frame area images; performing face key point detection on each training face frame area image in the time series of training face frame area images to obtain a time series of training face key point parameter sets; extracting a time series of training lip movement parameter sets from the time series of training face key point parameter sets, wherein the training lip movement parameter sets include training lip opening and closing scores, training pitch angles, training yaw angles, and training roll angles; inputting each training lip movement parameter set in the time series of the training lip movement parameter set into the lip movement parameter embedding encoder based on a multilayer perceptron to obtain a time series of training lip movement parameter embedding coding vectors; and embedding the training lip movement parameter embedding coding vectors. The time series is input into the Transformer-based lip movement parameter temporal context encoder to obtain the time series of the training context lip movement parameter semantic encoding feature vector; the time series of the training context lip movement parameter semantic encoding feature vector is input into the feature distribution reverse mask clustering analysis network to obtain the training lip movement feature temporal aggregation representation vector; the training lip movement feature temporal aggregation representation vector is passed through the classifier-based lip movement state recognizer to obtain a recognition result; the cross entropy loss function value between the recognition result and the true value is calculated to obtain a classification loss function value; based on the classification loss function value, the Yolo-v5 model-based face recognizer, the multi-layer perceptron-based lip movement parameter embedding encoder, the Transformer-based lip movement parameter temporal context encoder, the feature distribution reverse mask clustering analysis network and the classifier-based lip movement state recognizer are trained.

[0075] Preferably, in a preferred example of the present application, the training lip movement feature time series aggregation representation vector is passed through a classifier-based lip movement state recognizer to obtain a recognition result, which includes:

[0076] Performing a clustering operation on the training lip movement feature time series aggregation representation vector, for example, performing a clustering operation based on the distance between feature values, and determining the number of in-class features and the number of out-of-class features;

[0077] Dividing the total number of feature values ​​of the training lip movement feature time series aggregation representation vector by the number of in-class features and the number of out-class features respectively to obtain a class importance value and an out-class proportion value, and calculating the inverse of the class importance value to obtain a class constraint value;

[0078] Calculate each eigenvalue of the training lip movement feature time series aggregation representation vector as a power function with the class importance value as an exponent, add it to an exponential value with the natural function as a base and the class constraint value as an exponent, and then multiply it by the out-of-class ratio value to obtain a lip movement feature time series aggregation modulation vector;

[0079] Performing a dot multiplication of the training lip movement feature temporal aggregation representation vector and the class importance value to obtain a lip movement feature temporal aggregation ontology vector;

[0080] Calculating the weighted sum of the lip movement feature temporal aggregation modulation vector and the lip movement feature temporal aggregation ontology vector using a weighted hyperparameter to obtain an optimized lip movement feature temporal aggregation representation vector; and

[0081] The optimized lip movement feature temporal aggregation representation vector is input into the classifier-based lip movement state recognizer to obtain the recognition result.

[0082] That is, in the technical solution of the present application, the training lip movement feature time series aggregation representation vector is passed through the classifier-based lip movement state recognizer to obtain the recognition result, including:

[0083] The training lip movement feature temporal aggregation representation vector is clustered and optimized to obtain the optimized lip movement feature temporal aggregation representation vector. The process is expressed by the formula:

[0084]

[0085] in, The total number of eigenvalues ​​representing the temporal aggregation representation vector of lip movement features, represents the number of features within a class, represents the temporal aggregation representation vector of lip movement features, represents the weight hyperparameter, It means point multiplication by position. Add by position, It means to perform exponential operation on the eigenvalues ​​at each position in the vector. A temporal aggregation representation vector for the optimized lip movement features;

[0086] The optimized lip movement feature temporal aggregation representation vector is input into the classifier-based lip movement state recognizer to obtain the recognition result.

[0087] Accordingly, in this preferred embodiment, while performing clustering operation on the training lip movement feature time series aggregate representation vector, the applicant of the present application further performs responsive modulation with the out-of-class scale factor by combining the clustering importance measure of the feature value of the training lip movement feature time series aggregate representation vector with the class constraint measure of the clustering operation, and establishes the simplicity and effectiveness of the overall feature purification of the feature set of the training lip movement feature time series aggregate representation vector based on the clustering scaling of the feature ontology representation of the training lip movement feature time series aggregate representation vector, and suppresses the logical inconsistency of the probability density distribution convergence caused by the clustering operation, so as to improve the classification iteration effect of the lip movement state recognizer based on the classifier input of the feature set of the training lip movement feature time series aggregate representation vector, that is, to improve the speed of classification training and the accuracy of classification results. In this way, the lip movement state can be more accurately identified as a pre-step of speech recognition, helping to determine when to perform speech analysis, and providing a basis for understanding the subsequent conversation content.

[0088] In summary, the above solution is adopted to collect time series data of facial images through the camera, and introduce image processing and analysis algorithms based on artificial intelligence and deep learning in the back end to analyze and understand the time series data of the facial images, so as to learn and capture the temporal features and aggregate representation information about the semantics of lip movement status in the facial images, so as to identify and detect whether the mouth is moving, so as to help determine when to perform speech analysis.

[0089] In another embodiment of the present application, Figure 2 As shown, a lip movement algorithm recognition method is provided, comprising the steps of:

[0090] 1. Use the yolo-v5 model for face recognition and obtain the coordinate information of all face frames.

[0091] 2. Select the main face, with the face in the center of the picture as the standard when the distance between the face and the camera is between 0.8m and 1.7m.

[0092] 3. Use the trained model to detect facial key points and obtain 98 facial key points. Key point detection is based on the heat map regression method, and the network is mainly implemented using mobilenet-v3 and unet.

[0093] 4. Suppress the jitter of key points through weighted filtering of previous and next frame images.

[0094] 5. Face pose estimation: calculate the pitch, yaw and roll angles of the face.

[0095] 6. Perform lip movement recognition, which mainly includes the following steps:

[0096] Set up four 15-bit bidirectional queues to record the lip opening and closing score (score), pitch angle (Pitch), yaw angle (Yaw) and roll angle (Roll) of the current frame, and fill them with empirical values ​​by default.

[0097] Calculate the three-dimensional attitude angles, the mean and variance of the three-dimensional angles within 15 frames, recorded as pitchAverage, yawAverage, rollAverage, pitchVariance, yawVariance, rollVariance. When the mean yaw angle (yawAverage) is greater than 45 or less than -45, it is considered to be out of interaction and the microphone is turned off.

[0098] Correct the key points of the face in the lip area in three dimensions and convert the points into the coordinates when looking straight ahead to overcome the inaccuracy when looking down.

[0099] The lip movement calculation score is defined, and the Euclidean distance between point 89 and point 95 is defined as a, the Euclidean distance between point 90 and point 94 is defined as b, the Euclidean distance between point 91 and point 93 is defined as c, and the Euclidean distance between point 88 and point 92 is defined as d. The lip movement score = (a+b+c) / (3*d).

[0100] Add the current score to the end of the queue, pop the first data in the queue, calculate the overall mean and variance, and record them as scoreAverage and scoreVariance. If the current state is silent, determine the sum of the three-dimensional angle variances. If pitchVariance+yawVariance+rollVariance>100, then

[0101] scoreVariance = (pitchVariance+yawVariance+rollVariance) / 50, so as to suppress the influence of face shaking on the result. If the current score variance scoreVariance is greater than the set threshold, the state is set to the open mouth state, the current score mean scoreAvarage is recorded, and the current score threshold scoreThreshold = scoreAvarage + 0.012 is set.

[0102] If the current state is open mouth, define the state transition counter sToCCount. If the current score variance is less than variance threshold 1, then sToCCount + 5, if the variance is less than variance threshold 2 and the current score is less than score threshold scoreThreshold, then sToCCount + 2, if sToCCount > score queue size (default 15), set the state to silent state, sToCCount is set to 0, otherwise when sToCCount > 0, sToCCount -1.

[0103] When the interactive object disappears or changes the interactive object, reset the 4 queues.

[0104] Figure 3 FIG. 1 is a block diagram of a lip movement recognition system based on deep learning according to an exemplary embodiment. Figure 3 As shown, the lip movement recognition system 200 includes:

[0105] The face image acquisition module 201 is used to acquire the time series of face images collected by the camera;

[0106] A face recognition and key point detection module 202, configured to perform face recognition and key point detection on the time series of the face images to obtain a time series of face key point parameter sets;

[0107] A lip movement parameter extraction module 203 is used to extract a time series of a lip movement parameter set from the time series of the facial key point parameter set, wherein the lip movement parameter set includes a lip opening and closing score, a pitch angle, a yaw angle, and a roll angle;

[0108] The lip movement parameter embedding coding and temporal context feature extraction module 204 is used to perform lip movement parameter embedding coding and temporal context feature extraction on the time series of the lip movement parameter set to obtain a time series of contextual lip movement parameter semantic coding feature vectors;

[0109] A reverse mask clustering analysis module 205 is used to input the time series of the contextual lip movement parameter semantic encoding feature vector into a feature distribution reverse mask clustering analysis network to obtain a lip movement feature time series aggregation representation vector;

[0110] The mouth movement determination module 206 is used to determine a recognition result based on the lip movement feature time series aggregation representation vector, and the recognition result is used to indicate whether the mouth is moving.

[0111] Reference below Figure 4, which shows a schematic diagram of the structure of an electronic device 600 suitable for implementing the embodiment of the present application. The terminal device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 4 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0112] like Figure 4 As shown, the electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 are also stored. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0113] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 4 The electronic device 600 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0114] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 609, or installed from the storage device 608, or installed from the ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of the embodiment of the present application are executed.

[0115] It should be noted that the computer-readable medium mentioned above in the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0116] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0117] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0118] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0119] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0120] The modules involved in the embodiments described in this application can be implemented by software or hardware. The name of the module does not limit the module itself in some cases. For example, the test parameter acquisition module can also be described as a "module for acquiring device test parameters corresponding to the target device".

[0121] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0122] In the context of the present application, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0123] Figure 5 FIG. 1 is an application scenario diagram of a lip movement recognition method based on deep learning according to an exemplary embodiment. Figure 5 As shown, in this application scenario, first, a time series of face images captured by a camera is obtained (for example, Figure 5 Then, the acquired time series of face images is input to a server (for example, Figure 5 In S) shown in , the server is capable of processing the time series of the facial images based on a deep learning lip movement recognition algorithm to determine a recognition result, wherein the recognition result is used to indicate whether the mouth is moving.

[0124] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.

[0125] In addition, although each operation is described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or to be performed in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be construed as limiting the scope of the application. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0126] Although the subject matter has been described in language specific to structural features and / or method logic actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims. Regarding the device in the above embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.

Claims

1. A lip movement recognition method based on deep learning, characterized in that: include: Obtain a time series of face images captured by a camera; Performing face recognition and key point detection on the time series of face images to obtain a time series of face key point parameter sets; Extracting a time series of a lip movement parameter set from the time series of the facial key point parameter set, wherein the lip movement parameter set includes a lip opening and closing score, a pitch angle, a yaw angle, and a roll angle; Performing lip movement parameter embedding coding and temporal context feature extraction on the time series of the lip movement parameter set to obtain a time series of contextual lip movement parameter semantic coding feature vectors; Inputting the time series of the contextual lip movement parameter semantic encoding feature vector into a feature distribution reverse mask clustering analysis network to obtain a lip movement feature time series aggregation representation vector; Determine a recognition result based on the lip movement feature time series aggregation representation vector, wherein the recognition result is used to indicate whether the mouth is moving; Inputting the time series of the contextual lip movement parameter semantic encoding feature vector into a feature distribution reverse mask clustering analysis network to obtain a lip movement feature time series aggregation representation vector, including: Clustering the time series of the contextual lip movement parameter semantic encoding feature vector to obtain a contextual lip movement parameter semantic clustering center feature vector; Calculating the semantic correlation coefficient between each contextual lip movement parameter semantic encoding feature vector in the sequence of contextual lip movement parameter semantic encoding feature vectors and the contextual lip movement parameter semantic clustering center feature vector to obtain a sequence of contextual lip movement parameter clustering center semantic correlation coefficients; Taking the reciprocal of each contextual lip movement parameter cluster center semantic correlation coefficient in the sequence of contextual lip movement parameter cluster center semantic correlation coefficients to obtain a sequence of contextual lip movement parameter cluster center semantic anti-correlation coefficients; Using a Sigmoid function, normalizing the sequence of semantic anti-correlation coefficients of the contextual lip movement parameter clustering centers to obtain a sequence of semantic anti-correlation weight coefficients of the contextual lip movement parameter clustering centers; Using each context lip movement parameter cluster center semantic anti-correlation weight coefficient in the sequence of context lip movement parameter cluster center semantic anti-correlation weight coefficients as a weight, weighting each context lip movement parameter semantic coding feature vector in the sequence of context lip movement parameter semantic coding feature vectors to obtain a sequence of suppressed context lip movement parameter semantic coding feature vectors; Calculating the positional difference between the sequence of the contextual lip movement parameter semantic encoding feature vectors and the sequence of the suppressed contextual lip movement parameter semantic encoding feature vectors to obtain a sequence of enhanced contextual lip movement parameter semantic encoding feature vectors; The position-wise mean vector of the sequence of the enhanced context lip movement parameter semantic encoding feature vectors is calculated to obtain a fused feature vector.

2. The lip movement recognition method based on deep learning according to claim 1, characterized in that: Performing face recognition and key point detection on the time series of the face images to obtain a time series of face key point parameter sets, including: Input the time series of the face images into a face recognizer based on the Yolo-v5 model to obtain a time series of face frame area images; Performing facial key point detection on each face frame area image in the time series of the face frame area images to obtain the time series of the face key point parameter set.

3. The lip movement recognition method based on deep learning according to claim 2, characterized in that: Performing lip movement parameter embedding coding and temporal context feature extraction on the time series of the lip movement parameter set to obtain a time series of contextual lip movement parameter semantic coding feature vectors, including: Inputting each lip movement parameter set in the time series of the lip movement parameter set into a lip movement parameter embedding encoder based on a multi-layer perceptron to obtain a time series of lip movement parameter embedding encoding vectors; The time series of the lip movement parameter embedding coding vector is input into the lip movement parameter temporal context encoder based on the Transformer structure to obtain the time series of the context lip movement parameter semantic coding feature vector.

4. The lip movement recognition method based on deep learning according to claim 3 is characterized in that: Calculating the semantic correlation coefficient between each contextual lip movement parameter semantic encoding feature vector in the sequence of contextual lip movement parameter semantic encoding feature vectors and the contextual lip movement parameter semantic cluster center feature vector to obtain a sequence of contextual lip movement parameter cluster center semantic correlation coefficients, including: Calculating the norms of the contextual lip movement parameter semantic coding feature vector and the contextual lip movement parameter semantic cluster center feature vector respectively to obtain the contextual lip movement parameter semantic coding-norm representation value and the contextual lip movement parameter semantic cluster center-norm representation value; Calculate the product of the contextual lip movement parameter semantic encoding-norm representation value and the contextual lip movement parameter semantic cluster center-norm representation value to obtain the contextual lip movement parameter semantic-cluster center semantic correlation-norm representation value; Calculate the multiplication between the transposed vector of the contextual lip movement parameter semantic encoding feature vector and the contextual lip movement parameter semantic cluster center feature vector to obtain the contextual lip movement parameter semantic-cluster center semantic correlation feature value; The division between the contextual lip movement parameter semantics-cluster center semantic correlation feature value and the contextual lip movement parameter semantics-cluster center semantic correlation-norm representation value is calculated to obtain the contextual lip movement parameter cluster center semantic correlation coefficient.

5. The lip movement recognition method based on deep learning according to claim 4 is characterized in that: Based on the lip movement feature time-series aggregate representation vector, a recognition result is determined, and the recognition result is used to indicate whether the mouth is moving, including: inputting the lip movement feature time-series aggregate representation vector into a classifier-based lip movement state identifier to obtain the recognition result, and the recognition result is used to indicate whether the mouth is moving.

6. The lip movement recognition method based on deep learning according to claim 5, characterized in that: It also includes a training step: for training the face recognizer based on the Yolo-v5 model, the lip movement parameter embedding encoder based on the multi-layer perceptron, the lip movement parameter temporal context encoder based on the Transformer structure, the feature distribution reverse mask clustering analysis network and the classifier-based lip movement state recognizer.

7. The lip movement recognition method based on deep learning according to claim 6, characterized in that: The training step comprises: Obtain the time series of training face images captured by the camera, as well as the true value of whether the mouth is moving; Input the time series of the training face images into the face recognizer based on the Yolo-v5 model to obtain the time series of training face frame area images; Performing facial key point detection on each training face frame area image in the time series of the training face frame area images to obtain a time series of training face key point parameter sets; Extracting a time series of a training lip movement parameter set from the time series of the training face key point parameter set, wherein the training lip movement parameter set includes a training lip opening and closing score, a training pitch angle, a training yaw angle, and a training roll angle; Inputting each training lip movement parameter set in the time series of the training lip movement parameter set into the multi-layer perceptron based lip movement parameter embedding encoder to obtain a time series of training lip movement parameter embedding encoding vectors; Input the time series of the training lip movement parameter embedding coding vector into the lip movement parameter temporal context encoder based on the Transformer structure to obtain the time series of the training context lip movement parameter semantic coding feature vector; Inputting the time series of the semantic encoding feature vector of the training context lip movement parameters into the feature distribution reverse mask clustering analysis network to obtain a time series aggregation representation vector of the training lip movement features; Passing the training lip movement feature time series aggregation representation vector through the classifier-based lip movement state recognizer to obtain a recognition result; Calculating a cross entropy loss function value between the recognition result and the true value to obtain a classification loss function value; Based on the classification loss function value, the face recognizer based on the Yolo-v5 model, the lip movement parameter embedding encoder based on the multi-layer perceptron, the lip movement parameter temporal context encoder based on the Transformer structure, the feature distribution reverse mask clustering analysis network and the classifier-based lip movement state recognizer are trained.