A composite visual, auditory multi-modal lip movement detection method and apparatus and device

Abstract

The application discloses a kind of composite vision, hearing multimodal lip detection method and device and equipment.The method comprises: synchronously collecting voice data and face data;Multi-modal neural network model based on the voice data and face data is constructed;The multi-modal neural network model is trained using the way of double decoder joint optimization;According to the trained multi-modal neural network model, the speech of human body and face key point are predicted according to the face lip corresponding to the speech.This application uses key point detection algorithm to extract the face key point of face image and aggregate face features.On the other hand, considering the change of facial features and voice features, the change of mouth shape is represented by extracting more rich features by fusing the two, so that the vivid image is more lively.

Description

Technical Field

[0001] This invention relates to the field of facial feature recognition technology, and in particular to a multimodal method, apparatus, and device for detecting mouth shapes using composite vision and hearing. Background Technology

[0002] The rise of deep learning has triggered transformations across various industries. Among them, the recently popular metaverse concept, benefiting from deep learning technology, has also generated considerable buzz. The metaverse is a possible digital world where people can freely participate and live with digital identities; its core concept is the digital human or virtual human. In the development of digital humans, it's necessary to consider how to map specific facial expressions, including lip movements, and body language from real people onto digital humans, making our digital humans lifelike and more closely resembling real human reactions. However, existing lip-reading detection schemes in the metaverse concept mostly employ single-modal techniques. Single-modal methods suffer from insufficient feature richness; for example, audio- or image-based single-modal methods do not fully consider facial image features and speech features, resulting in poor modeling performance and requiring more data to fit the model. Multimodal solutions, often based on video, require much larger neural network parameters and structures, which is not conducive to deployment on mobile devices. Based on these shortcomings, the applicant proposes a composite visual and auditory multimodal lip-reading detection method, device, and equipment. Summary of the Invention

[0003] In view of this, the purpose of this invention is to propose a composite visual and auditory multimodal lip-shape detection method, device, and equipment. Unlike traditional multimodal methods (which utilize video, i.e., combining images and audio), this invention, on the one hand, uses a keypoint detection algorithm to extract facial keypoints from face images, aggregating facial features to extract more effective facial features. The keypoint information itself includes the contours of the face and facial features such as the mouth. Compared to face images (typically using a 112x112 size), the feature dimensions are smaller, but the feature correlation is greater, allowing the model to be as small as possible and more suitable for mobile deployment. On the other hand, compared to single-modal solutions, this invention considers the changes in facial feature contours, further starting from facial features and speech features, fusing the two to extract richer features to represent changes in lip shape, making the simulated image more vivid.

[0004] According to one aspect of the present invention, a composite visual and auditory multimodal lip-reading detection method is provided, comprising: simultaneously acquiring speech data and face data; constructing a multimodal neural network model based on the speech data and face data; training the multimodal neural network model using a dual-decoder joint optimization method; and predicting the facial lip-reading corresponding to the speech based on the trained multimodal neural network model and facial key points.

[0005] According to another aspect of the present invention, a composite visual and auditory multimodal lip-sync detection device is provided, comprising: an audio data component, a facial landmark extraction component, a neural network encoder component, a dual decoder component, and a prediction component; the audio data component is used to acquire speech data and extract speech features; the facial landmark extraction component is used to acquire face data and extract facial landmarks; the neural network encoder component is used to construct a multimodal neural network model based on the speech data and face data; the dual decoder component is used to train the multimodal neural network model using a dual decoder joint optimization method; and the prediction component is used to predict the lip-sync of a person's speech and facial landmarks corresponding to the speech based on the trained multimodal neural network model.

[0006] According to another aspect of the present invention, a composite visual and auditory multimodal lip-shape detection device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the composite visual and auditory multimodal lip-shape detection method as described in any of the preceding claims.

[0007] According to another aspect of the present invention, a computer-readable storage medium is provided, storing a computer program, characterized in that, when the computer program is executed by a processor, it implements the composite visual and auditory multimodal lip-shape detection method as described in any of the preceding claims.

[0008] It can be observed that the above scheme can simultaneously or stepwise collect speech and face data and extract speech features and facial key point features; input the speech features and facial key point features into a neural network encoder for multimodal fusion encoding to output a first encoded feature; input the first encoded feature into a CTC decoder to perform frame alignment between the speech feature sequence and the facial key point feature sequence and generate at least one first prediction result; and input the at least one first prediction result into an Attention decoder for secondary decoding to generate at least one second prediction result. This scheme differs from traditional multimodal methods (using video, i.e., combining images and audio). This invention, on the one hand, uses a key point detection algorithm to extract facial key points from face images and aggregate facial features, extracting more effective facial features. Key point information itself includes the outline of the face and the outline of facial features such as the mouth. Compared to face images (generally using a size of 112x112), the feature dimension is smaller but the feature correlation is greater, allowing the model to be as small as possible and more suitable for mobile deployment. On the other hand, compared with the single-modal solution, the present invention takes into account the changes in the facial features and further starts from facial features and voice features, and integrates the two to extract richer features to represent the changes in mouth shape, making the simulated image more vivid. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 This is a flowchart illustrating an embodiment of the composite visual and auditory multimodal lip-shape detection method of the present invention;

[0011] Figure 2 This is a flowchart illustrating another embodiment of the composite visual and auditory multimodal lip-shape detection method of the present invention;

[0012] Figure 3 This is a schematic diagram of the structure of an embodiment of the composite visual and auditory multimodal mouth shape detection device of the present invention;

[0013] Figure 4 This is a schematic diagram of another embodiment of the composite visual and auditory multimodal mouth shape detection device of the present invention. Detailed Implementation

[0014] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be particularly noted that the following embodiments are for illustrative purposes only and do not limit the scope of the invention. Similarly, the following embodiments are only some, not all, embodiments of the present invention, and all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0015] This invention provides a composite visual and auditory multimodal lip-shape detection method, which differs from traditional multimodal methods (using video, i.e., combining images and audio). Firstly, this invention uses a keypoint detection algorithm to extract facial keypoints from face images, aggregating facial features to obtain more effective facial characteristics. Keypoint information itself includes the contours of the face and facial features such as the mouth. Compared to face images (typically using a 112x112 pixel size), the feature dimensions are smaller, but the feature correlation is greater, allowing the model to be as small as possible and more suitable for mobile deployment. Secondly, compared to single-modal solutions, this invention considers changes in facial feature contours, further integrating facial and speech features to extract richer features to represent changes in lip shape, resulting in a more vivid and realistic image.

[0016] Please see Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the multimodal lip-shape detection method combining visual and auditory perception according to the present invention. It should be noted that if substantially the same result is obtained, the method of the present invention is not necessarily identical. Figure 1 The illustrated process sequence is limited. For example... Figure 1 As shown, the method includes the following steps:

[0017] S101. Synchronously collect voice data and facial data;

[0018] In this embodiment, voice data and face data can be collected simultaneously and voice features and facial key point features can be extracted. Alternatively, voice data and face data can be collected step by step and voice features and facial key point features can be extracted. This invention does not limit the scope of the invention.

[0019] S102. Construct a multimodal neural network model based on the aforementioned voice data and face data;

[0020] In this embodiment, the multimodal neural network model includes a neural network encoder, which is a transformer neural network improved using conv. It is a conformer encoder component used to fuse audio features (spectral graph) and facial key point coordinates, and uses a multimodal approach to further extract effective features. This is equivalent to encoding the original features, and is also called an encoder.

[0021] S103. The multimodal neural network model is trained using a dual-decoder joint optimization method;

[0022] In this embodiment, the dual decoder employs a CTC decoder and an Attention decoder. A face capture tool is used to label each speech and face data point with corresponding lip shape tags. In this embodiment, LiveLinkFace is used for lip shape labeling, which is more vivid and intuitive than traditional manual labeling or mapping phonemes to lip shapes. Specifically, speech and face data are recorded simultaneously, and LiveLinkFace is used to capture facial expressions to obtain realistic lip shape tags. Speech features and facial features are extracted from the lip-labeled speech and face data. Spectrograms and facial keypoint coordinates are extracted from these features and used as training inputs for the multimodal neural network. The multimodal encoder of the multimodal neural network model performs multimodal fusion of the spectrograms and facial keypoint coordinates. The CTC decoder and Attention decoder, using the fused spectrograms and facial keypoint coordinates as training inputs, jointly train the constructed multimodal neural network model.

[0023] S104. Based on the trained multimodal neural network model, predict the facial lip shape corresponding to the speech and facial key points of the human body.

[0024] In this embodiment, based on the trained multimodal neural network model, spectrograms and facial landmark coordinates associated with the collected speech and face data are extracted. The extracted spectrograms and facial landmark coordinates are input into the trained multimodal neural network model, and the spectrograms and facial landmark coordinates are fused using a multimodal encoder. The fused spectrograms and facial landmark coordinates are then input into the CTC decoder and Attention decoder of the multimodal neural network model to predict the facial lip shape corresponding to the speech and facial landmarks.

[0025] It can be observed that in this embodiment, voice data and face data can be acquired simultaneously, and voice features and facial key point features can be extracted. These voice features and facial key point features can be input into a neural network encoder for multimodal fusion. The purpose of this setting is to consider changes in facial contours, further extracting richer features from both facial and voice features to characterize mouth shape changes, making the simulated image more vivid. The fused spectrogram and facial key point coordinates can be input into a CTC decoder to perform frame alignment of voice features and facial key point coordinates and generate at least one first mouth shape prediction result. Furthermore, at least one first mouth shape prediction result can be input into an Attention decoder for secondary decoding to generate at least one second mouth shape prediction result. This setup aims to acquire voice and face data synchronously in real time, allowing subsequent models to control the components of key frames based on the number of voice frames, thereby controlling the dynamic input nodes of the model. This further makes the model applicable to streaming scenarios, especially online virtual human live streaming scenarios. The results from the CTC decoder are displayed as intermediate results, and at the end of streaming recognition, low-latency secondary decoding (Rescoring) can be used to repair the results, further improving the final recognition rate. Furthermore, when voice data and face data are collected step by step and voice features and facial key point features are extracted, all the data from each step can be input into the subsequent model. This allows for full utilization of the sequential relationship between voice data and facial key point features to obtain the best recognition effect, thereby achieving real-time offline lip detection.

[0026] As can be seen, the above solution differs from traditional multimodal approaches (which utilize video, combining images and audio). This invention, on one hand, uses a keypoint detection algorithm to extract facial keypoints from face images, aggregating facial features to obtain more effective facial characteristics. The keypoint information itself includes the contours of the face and facial features such as the mouth. Compared to face images (typically 112x112 pixels), the feature dimensions are smaller, but the feature correlation is greater, allowing the model to be as small as possible and more suitable for mobile deployment. On the other hand, compared to single-modal solutions, this invention considers changes in facial feature contours, further integrating facial and speech features to extract richer features to represent changes in mouth shape, making the simulated image more vivid.

[0027] Please see Figure 2 , Figure 2 This is a flowchart illustrating another embodiment of the composite visual and auditory multimodal lip-shape detection method of the present invention. In this embodiment, the method includes the following steps:

[0028] S201. Synchronously collect voice data and facial data.

[0029] As described in S101 above, it will not be repeated here.

[0030] S202. Construct a multimodal neural network model based on the aforementioned voice data and face data;

[0031] As described in S102 above, it will not be repeated here.

[0032] S203. The multimodal neural network model is trained using a dual-decoder joint optimization method;

[0033] As described in S103 above, it will not be repeated here.

[0034] S204. Based on the trained multimodal neural network model, predict the facial lip shape corresponding to the speech and facial key points of the human body.

[0035] As described in S104 above, it will not be repeated here.

[0036] S205. Update the parameters of the multimodal neural network using the loss function of the dual decoder, and train and update the dual decoder using the multimodal neural network after parameter update through the number of prediction iterations.

[0037] In this embodiment, the dual decoders are a CTC decoder and an Attention decoder.

[0038] It can be observed that the above approach jointly trains the entire network using frame-level CTC loss and label-level attention-based auto-regression loss. Backpropagation is then used to update and correct the neural network weights using the combined loss of CTC and Attention. As the data is continuously updated, the model's accuracy gradually improves.

[0039] This invention also provides a composite visual and auditory multimodal lip-shape detection device. Unlike traditional multimodal methods (which utilize video, combining images and audio), this invention uses a keypoint detection algorithm to extract facial keypoints from face images, aggregating facial features to obtain more effective facial characteristics. The keypoint information itself includes the contours of the face and facial features such as the mouth. Compared to face images (typically 112x112 pixels), the feature dimensions are smaller, but the feature correlation is greater, allowing the model to be as small as possible and more suitable for mobile deployment. Furthermore, compared to single-modal solutions, this invention considers changes in facial feature contours, further integrating facial and speech features to extract richer features to represent lip-shape changes, resulting in a more vivid and realistic image.

[0040] Please see Figure 3 , Figure 3This is a schematic diagram of an embodiment of the composite visual and auditory multimodal lip-shape detection device of the present invention. In this embodiment, the composite visual and auditory multimodal lip-shape detection device includes: an audio data component M1, a facial key point extraction component M2, a neural network encoder component M3, a dual decoder component (CTC decoder component M4 and Attention decoder component M5), and a prediction component M0.

[0041] The audio data component M1 is used to acquire speech data and extract speech features;

[0042] The facial landmark extraction component M2 is used to collect facial data and extract facial landmarks;

[0043] The neural network encoder component M3 is used to construct a multimodal neural network model based on the aforementioned voice data and face data;

[0044] The dual decoder components (CTC decoder component M4 and Attention decoder component M5) are used to train the multimodal neural network model using a dual decoder joint optimization approach;

[0045] The prediction component M0 is used to predict the facial lip shape corresponding to the human speech and facial key points based on the trained multimodal neural network model.

[0046] Please see Figure 4 , Figure 4 This is a schematic diagram of another embodiment of the composite visual and auditory multimodal lip-shape detection device of the present invention. Unlike the previous embodiment, the composite visual and auditory multimodal lip-shape detection device in this embodiment further includes an updating component M6.

[0047] The update component M6 updates the parameters of the multimodal neural network using the loss function of the dual decoder components (CTC decoder component M4 and Attention decoder component M5), and trains and updates the dual decoder using the parameter-updated multimodal neural network through iterations of prediction.

[0048] Each component of the composite visual and auditory multimodal mouth shape detection device described in the above two embodiments can perform the corresponding steps in the above method embodiments, so the details of each component will not be elaborated here. Please refer to the description of the corresponding steps above for details.

[0049] The present invention also provides a composite visual and auditory multimodal lip-shape detection device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the above-described composite visual and auditory multimodal lip-shape detection method.

[0050] The memory and processor are connected via a bus, which can include any number of interconnecting buses and bridges, connecting various circuits of one or more processors and memories. The bus can also connect various other circuits, such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and will not be described further herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by the processor is transmitted over the wireless medium via an antenna, which further receives data and transmits it to the processor.

[0051] The processor manages the bus and general processing, and also provides various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory is used to store data used by the processor during operation.

[0052] The present invention further provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the above-described method embodiments.

[0053] It can be observed that the above scheme can acquire speech data from people of different ages, construct a Long Short-Term Memory (LSTM) network regression model based on this speech data, train this model using an LTM network, and predict the age of the person speaking based on the trained LTM network regression model. Unlike traditional multimodal methods (using video, i.e., combining images and audio), this invention uses a keypoint detection algorithm to extract facial keypoints from face images, aggregating facial features to extract more effective facial features. Keypoint information itself includes the outline of the face and the outlines of facial features such as the mouth. Compared to face images (typically using a 112x112 size), the feature dimension is smaller but the feature correlation is greater, allowing the model to be as small as possible and more suitable for mobile deployment. Furthermore, compared to single-modal schemes, this invention considers the changes in facial feature outlines, further integrating facial and speech features to extract richer features to represent changes in mouth shape, making the simulated image more vivid.

[0054] In the several embodiments provided by this invention, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of components or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of apparatuses or units may be electrical, mechanical, or other forms.

[0055] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0056] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0057] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0058] The above description is only a part of the embodiments of the present invention and does not limit the scope of protection of the present invention. Any equivalent device or equivalent process transformation made based on the content of the present invention specification and drawings, or direct or indirect application in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A multimodal lip-shape detection method incorporating visual and auditory senses, characterized in that, include: Collect voice and facial data; Construct a multimodal neural network model based on the aforementioned voice data and face data; The multimodal neural network model is trained using a dual-decoder joint optimization method; Based on the trained multimodal neural network model, the facial lip shape corresponding to the speech and facial key points of the human body is predicted. The multimodal neural network model is trained using a dual-decoder joint optimization method, including... A face capture tool is used to label each voice and face data with corresponding mouth shapes. Voice features and face features are extracted from the lip-shaped labeled voice and face data. Spectrograms and facial key point coordinates are extracted from the voice and face features and used as training inputs for a multimodal neural network. The multimodal encoder of the multimodal neural network model performs multimodal fusion of the spectrograms and facial key point coordinates. The CTC decoder and Attention decoder, using the multimodal fused spectrograms and facial key point coordinates as training inputs, are jointly used to train the constructed multimodal neural network model.

2. The composite visual and auditory multimodal lip-shape detection method as described in claim 1, characterized in that, The step of predicting facial lip movements corresponding to human speech and facial key points based on the trained multimodal neural network model includes: Based on the trained multimodal neural network model, spectrograms and facial landmark coordinates associated with the collected speech and face data are extracted. The extracted spectrograms and facial landmark coordinates are input into the trained multimodal neural network model, and the spectrograms and facial landmark coordinates are fused using a multimodal encoder. The fused spectrograms and facial landmark coordinates are then input into the CTC decoder and Attention decoder of the multimodal neural network model to predict the facial lip shape corresponding to the speech and facial landmarks.

3. The composite visual and auditory multimodal lip-shape detection method as described in claim 1, characterized in that, After predicting the facial lip shape corresponding to the speech based on the trained multimodal neural network model and facial key points, the method further includes: The parameters of the multimodal neural network are updated using the loss function of the dual decoder, and the dual decoder is trained and updated using the multimodal neural network with updated parameters through iterations of prediction.

4. A composite visual and auditory multimodal lip-shape detection device, characterized in that, include: Audio data component, facial landmark extraction component, neural network encoder component, dual decoder component, prediction component; This audio data component is used to acquire speech data and extract speech features; This facial landmark extraction component is used to collect facial data and extract facial landmarks; The neural network encoder component is used to construct a multimodal neural network model based on the aforementioned voice data and face data; This dual-decoder component is used to train the multimodal neural network model using a dual-decoder joint optimization approach; The prediction component is used to predict the facial lip shape corresponding to the human speech and facial key points based on the trained multimodal neural network model. The dual decoder component is specifically used for A face capture tool is used to label each voice and face data with corresponding mouth shapes. Voice features and face features are extracted from the lip-shaped labeled voice and face data. Spectrograms and facial key point coordinates are extracted from the voice and face features and used as training inputs for a multimodal neural network. The multimodal encoder of the multimodal neural network model performs multimodal fusion of the spectrograms and facial key point coordinates. The CTC decoder and Attention decoder, using the multimodal fused spectrograms and facial key point coordinates as training inputs, are jointly used to train the constructed multimodal neural network model.

5. The composite visual and auditory multimodal mouth shape detection device as described in claim 4, characterized in that, The prediction component is specifically used for: Based on the trained multimodal neural network model, spectrograms and facial landmark coordinates associated with the collected speech and face data are extracted. The extracted spectrograms and facial landmark coordinates are input into the trained multimodal neural network model, and the spectrograms and facial landmark coordinates are fused using a multimodal encoder. The fused spectrograms and facial landmark coordinates are then input into the CTC decoder and Attention decoder of the multimodal neural network model to predict the facial lip shape corresponding to the speech and facial landmarks.

6. The composite visual and auditory multimodal mouth shape detection device as described in claim 4, characterized in that, The composite visual and auditory multimodal lip-shape detection device also includes: Update component: Update the parameters of the multimodal neural network using the loss function of the dual decoder, and train and update the dual decoder using the multimodal neural network with updated parameters through iterations of prediction.

7. A composite visual and auditory multimodal mouth shape detection device, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the composite visual and auditory multimodal lip detection method as described in any one of claims 1 to 3.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the composite visual and auditory multimodal lip-shape detection method according to any one of claims 1 to 3.

Images