Voice-lip synchronization identification method and apparatus, and method and apparatus for training voice-lip synchronization identification network

Abstract

Provided in the present invention are a voice-lip synchronization identification method and apparatus, and a method and apparatus for training a voice-lip synchronization identification network. The voice-lip synchronization identification method comprises: acquiring a first audio signal of a video to be subjected to identification, wherein the duration of the first audio signal is a first duration; acquiring N video frames corresponding to the first audio signal in said video, wherein N is determined according to the first duration and a frame rate of said video; acquiring lip key point information in the N video frames; inputting the first audio signal into an audio coder of a voice-lip synchronization identification network and performing coding processing on the first audio signal, so as to obtain an audio feature vector; inputting the lip key point information into an image coder of the voice-lip synchronization identification network and performing coding processing on the lip key point information, so as to obtain an image feature vector; and determining the distance between the audio feature vector and the image feature vector by using a loss function of the voice-lip synchronization identification network, and determining, according to the distance, whether voice-lip synchronization is achieved. By means of the present invention, voice-lip synchronization identification can be performed on videos of different frame rates.

Description

Speech-to-mouth synchronization recognition method, speech-to-mouth synchronization recognition network training method and device

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to Chinese patent application No. 202310447732.4 filed in China on April 24, 2023, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The embodiments of the present invention relate to the field of video processing technology, and in particular to a method for recognizing speech and voice synchronization, and a method and device for training a speech and voice synchronization recognition network. Background Art

[0004] When playing some videos, the sound and lip movements often do not align, which affects the viewing experience. Therefore, there is a need for a technology that can effectively detect whether the sound and lip movements in the video are synchronized.

[0005] Summary of the Invention

[0006] Embodiments of the present invention provide a method for speech and voice synchronization recognition, and a method and device for training a speech and voice synchronization recognition network, which can perform speech and voice synchronization recognition on videos with different frame rates.

[0007] In order to solve the above-mentioned technical problems, the present invention is achieved as follows:

[0008] In a first aspect, an embodiment of the present invention provides a method for voice and voice synchronization recognition, comprising:

[0009] Acquire a first audio signal of a video to be identified, where the duration of the first audio signal is a first duration;

[0010] Obtaining N video frames in the video to be identified, wherein the N video frames correspond to the first audio signal, N is determined according to the first duration and the frame rate of the video to be identified, and N is an integer greater than or equal to 1;

[0011] Obtaining lip key point information in the N video frames;

[0012] Inputting the first audio signal into an audio encoder of a speech and mouth synchronization recognition network for encoding processing to obtain an audio feature vector;

[0013] Inputting the lip key point information into the image encoder of the voice-mouth synchronization recognition network for encoding processing to obtain an image feature vector;

[0014] The loss function of the speech-mouth synchronization recognition network is used to determine the distance between the audio feature vector and the image feature vector, and whether the speech is synchronized is determined based on the distance.

[0015] Optionally, obtaining the first audio signal of the video to be identified includes:

[0016] Acquire a second audio signal of the video to be identified, where the duration of the second audio signal is a second duration, and the second duration is M times the first duration, where M is an integer greater than or equal to 1;

[0017] performing sampling processing on the second audio signal to obtain a plurality of sampled audio signals;

[0018] Performing Fourier transform on the sampled audio signal to obtain a Fourier transformed audio signal;

[0019] Extracting the Mel spectrum feature of the audio signal after the Fourier transformation to obtain an extracted audio signal;

[0020] The extracted audio signal is divided into M first audio signals.

[0021] Optionally, before obtaining N video frames corresponding to the first audio signal in the video to be identified, the step further includes:

[0022] Obtaining the frame rate of the video to be identified;

[0023] The value of N is determined according to the frame rate and the first duration.

[0024] Optionally, determining a value of N according to the frame rate and the first duration includes:

[0025] Determine the quotient of the preset duration and the first duration to obtain a first value;

[0026] Determining a quotient of the frame rate and the first value to obtain a second value;

[0027] If the second value is an integer, the second value is used as N;

[0028] If the second value is not an integer, the second value is rounded to obtain a third value as N.

[0029] Optionally, obtaining lip key point information in the N video frames includes:

[0030] Using a face detector to detect face images in the N video frames;

[0031] Extracting facial key points from the facial image using a facial key point extractor;

[0032] Select lip key points from the facial key points.

[0033] Optionally, the audio encoder includes: a plurality of stacked convolution modules and a pooling layer and a fully connected layer at the top layer;

[0034] and / or

[0035] The image encoder includes: multiple stacked LSTM modules and fully connected layers; or, multiple stacked Bi-LSTM modules and fully connected layers.

[0036] Optionally, the loss function is contrastive loss.

[0037] In a second aspect, an embodiment of the present invention provides a method for training a speech lip synchronization recognition network, wherein the speech lip synchronization recognition network includes: an audio encoder, an image encoder, and a loss function, and the method includes:

[0038] Acquire training data, the training data including positive sample data and negative sample data, the positive sample data including: a first audio signal of a video sample and N video frames corresponding to the first audio signal; the negative sample data including: the first audio signal of the video sample and N video frames including at least one video frame located at a different time period from the first audio signal; wherein the duration of the audio signal is a first duration, N is determined based on the first duration and a frame rate of the video sample, and N is an integer greater than or equal to 1;

[0039] Obtaining lip key point information in the N video frames;

[0040] Inputting the first audio signal into the audio encoder for encoding to obtain an audio feature vector;

[0041] Inputting the lip key point information into the image encoder for encoding processing to obtain an image feature vector;

[0042] A loss function of the speech and mouth synchronization recognition network is used to determine a distance between the audio feature vector and the image feature vector, and parameters of the audio encoder and / or the image encoder are adjusted according to the distance.

[0043] Optionally, obtaining training data includes:

[0044] Acquire a second audio signal of the video sample, where the duration of the second audio signal is a second duration, and the second duration is M times the first duration, where M is an integer greater than or equal to 1;

[0045] performing sampling processing on the second audio signal to obtain a plurality of sampled audio signals;

[0046] Performing Fourier transform on the sampled audio signal to obtain a Fourier transformed audio signal;

[0047] Extracting the Mel spectrum feature of the audio signal after the Fourier transformation to obtain an extracted audio signal;

[0048] The first audio signal is extracted from the extracted audio signal.

[0049] Optionally, obtaining training data includes:

[0050] Get the frame rate of the video sample;

[0051] The value of N is determined according to the frame rate and the first duration.

[0052] Optionally, determining a value of N according to the frame rate and the first duration includes:

[0053] Determine the quotient of the preset duration and the first duration to obtain a first value;

[0054] Determining a quotient of the frame rate and the first value to obtain a second value;

[0055] If the second value is an integer, the second value is used as N;

[0056] If the second value is not an integer, the second value is rounded to obtain a third value as N.

[0057] In a third aspect, an embodiment of the present invention provides a device for recognizing speech and voice synchronization, comprising:

[0058] A first acquisition module is configured to acquire a first audio signal of a video to be identified, where the duration of the first audio signal is a first duration;

[0059] a second acquisition module, configured to acquire N video frames from the video to be identified, wherein the N video frames correspond to the first audio signal, N is determined according to the first duration and a frame rate of the video to be identified, and N is an integer greater than or equal to 1;

[0060] A third acquisition module is used to obtain lip key point information in the N video frames;

[0061] an audio encoding module, configured to input the first audio signal into an audio encoder of a speech and mouth synchronization recognition network for encoding processing to obtain an audio feature vector;

[0062] An image encoding module, configured to input the lip key point information into the image encoder of the voice-mouth synchronization recognition network for encoding processing to obtain an image feature vector;

[0063] The synchronization determination module is used to determine the distance between the audio feature vector and the image feature vector using the loss function of the voice-mouth synchronization recognition network, and determine whether the voice-mouth synchronization is achieved based on the distance.

[0064] In a fourth aspect, an embodiment of the present invention provides a training device for a speech lip synchronization recognition network, wherein the speech lip synchronization recognition network includes: an audio encoder, an image encoder, and a loss function, and the training device for the speech lip synchronization recognition network includes:

[0065] A first acquisition module is configured to acquire training data, the training data including positive sample data and negative sample data, the positive sample data including: a first audio signal of a video sample and N video frames corresponding to the first audio signal; the negative sample data including: the first audio signal of a video sample and N video frames including at least one video frame in a different time period from the first audio signal; wherein the duration of the audio signal is a first duration, N is determined based on the first duration and a frame rate of the video sample, and N is an integer greater than or equal to 1;

[0066] A second acquisition module is used to obtain lip key point information in the N video frames;

[0067] an audio encoding module, configured to input the first audio signal into the audio encoder for encoding processing to obtain an audio feature vector;

[0068] An image encoding module, configured to input the lip key point information into the image encoder of the voice-mouth synchronization recognition network for encoding processing to obtain an image feature vector;

[0069] An optimization module is configured to determine a distance between the audio feature vector and the image feature vector using a loss function of the speech-to-mouth synchronization recognition network, and adjust parameters of the audio encoder and / or the image encoder according to the distance.

[0070] In a fifth aspect, an embodiment of the present invention provides an electronic device, comprising: a processor, a memory, and a program stored in the memory and runnable on the processor, wherein when the program is executed by the processor, the steps of the method for speech-to-lip synchronization recognition as described in the first aspect are implemented, or, when the program is executed by the processor, the steps of the method for training a speech-to-lip synchronization recognition network as described in the second aspect are implemented.

[0071] In a sixth aspect, an embodiment of the present invention provides a non-volatile computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the computer program implements the steps of the method for speech-to-lip synchronization recognition as described in the first aspect above; or, when the computer program is executed by a processor, the computer program implements the steps of the method for training a speech-to-lip synchronization recognition network as described in the second aspect above.

[0072] In an embodiment of the present invention, a first audio signal of a fixed duration of a video to be identified is obtained, and N video frames corresponding to the first audio signal are obtained, wherein N is a non-fixed value and is determined according to the duration of the first audio signal and the frame rate of the video to be identified. Then, an audio encoder of a speech-to-mouth synchronization recognition network is used to process the first audio signal to obtain an audio feature vector, and an image encoder of the speech-to-mouth synchronization recognition network is used to process the lip key point information in the N video frames to obtain an image feature vector. Subsequently, a loss function of the speech-to-mouth synchronization recognition network is used to determine the distance between the audio feature vector and the image feature vector, and whether the speech is synchronized is determined based on the distance. Since the value of N is not fixed, it is applicable to speech-to-mouth synchronization recognition of videos with different frame rates, thereby improving the versatility of the speech-to-mouth synchronization recognition network. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Various other advantages and benefits will become apparent to those skilled in the art upon reading the detailed description of the preferred embodiment below. The accompanying drawings are for illustration purposes only and are not to be considered as limiting the present invention. The same reference symbols are used throughout the drawings to represent the same components. In the drawings:

[0074] FIG1 is a flow chart of a method for speech and voice synchronization recognition according to an embodiment of the present invention;

[0075] FIG2 is a schematic diagram of extracted facial key points according to an embodiment of the present invention;

[0076] FIG3 is a schematic diagram of the structure of a speech and voice synchronization recognition network according to an embodiment of the present invention;

[0077] FIG4 is a flow chart of a method for training a speech and voice synchronization recognition network according to an embodiment of the present invention;

[0078] FIG5 is a schematic structural diagram of a device for voice and voice synchronization recognition according to an embodiment of the present invention;

[0079] FIG6 is a schematic structural diagram of a training device for a speech-mouth synchronization recognition network according to an embodiment of the present invention;

[0080] FIG7 is a schematic structural diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0081] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort shall fall within the scope of protection of the present invention.

[0082] The lip sync recognition network in the related art is trained using videos with a fixed frame rate (such as 25fps, i.e., 25 frames per second). During network training, the audio signal input to the lip sync recognition network is usually 200ms long, and the number of frames of the image encoder input to the lip sync recognition network is fixed at 5 frames (also 200ms). However, it is difficult to ensure that the frame rate of current videos, especially videos crawled from the Internet, is 25fps. In other words, if the frame rate is not 25fps, the number of frames corresponding to 200ms may no longer be 5 frames. Therefore, it is difficult for the trained lip sync recognition network to perform lip sync recognition on such videos.

[0083] To solve the above problem, referring to FIG1 , an embodiment of the present invention provides a method for speech and voice synchronization recognition, comprising:

[0084] Step 11: Acquire a first audio signal of the video to be identified, where the duration of the first audio signal is a first duration;

[0085] The first duration may be, for example, 200 milliseconds (ms), although other durations are not excluded.

[0086] Step 12: Obtaining N video frames from the video to be identified, wherein the N video frames correspond to the first audio signal, where N is determined according to the first duration and the frame rate of the video to be identified, and N is an integer greater than or equal to 1;

[0087] The N value corresponding to the video to be identified at different frame rates may be different. For example, the N value corresponding to the video to be identified at a frame rate of 25 fps may be 5, and the N value corresponding to the video to be identified at a frame rate of 30 fps may be 6.

[0088] In the embodiment of the present invention, N video frames correspond to the first audio signal, which means that the N video frames and the corresponding first audio signal are located at the same time of the video to be recognized.

[0089] It should be noted that, in the embodiment of the present invention, the first audio signal may be acquired first, and then the N video frames corresponding to the first audio signal may be acquired. Alternatively, the N video frames may be acquired first, and then the first audio signal corresponding to the N video frames may be acquired. Alternatively, the first audio signal and the corresponding N video frames may be acquired simultaneously.

[0090] Step 13: Obtain lip key point information in the N video frames;

[0091] Step 14: Inputting the first audio signal into an audio encoder of a speech and mouth synchronization recognition network for encoding to obtain an audio feature vector;

[0092] Step 15: Inputting the lip key point information into the image encoder of the voice-mouth synchronization recognition network for encoding processing to obtain an image feature vector;

[0093] Step 16: Using the loss function of the speech-mouth synchronization recognition network to determine the distance between the audio feature vector and the image feature vector, and determining whether the speech is synchronized based on the distance.

[0094] In an embodiment of the present invention, a first audio signal of a fixed duration of a video to be identified is obtained, and N video frames corresponding to the first audio signal are obtained, wherein N is a non-fixed value and is determined according to the duration of the first audio signal and the frame rate of the video to be identified. Then, an audio encoder of a speech-to-mouth synchronization recognition network is used to process the first audio signal to obtain an audio feature vector, and an image encoder of the speech-to-mouth synchronization recognition network is used to process the lip key point information in the N video frames to obtain an image feature vector. Subsequently, a loss function of the speech-to-mouth synchronization recognition network is used to determine the distance between the audio feature vector and the image feature vector, and whether the speech is synchronized is determined based on the distance. Since the value of N is not fixed, it is applicable to speech-to-mouth synchronization recognition of videos with different frame rates, thereby improving the versatility of the speech-to-mouth synchronization recognition network.

[0095] In the embodiment of the present invention, optionally, obtaining the first audio signal of the video to be identified includes:

[0096] Step 111: Acquire a second audio signal of the video to be identified, where the duration of the second audio signal is a second duration, and the second duration is M times the first duration, where M is an integer greater than or equal to 1;

[0097] For the convenience of calculation, for example, the second time length may be 1 second (s), although other values ​​are not excluded.

[0098] Step 112: Sampling the second audio signal to obtain a plurality of sampled audio signals;

[0099] In the embodiment of the present invention, the second audio signal may be sampled at a sampling rate of 16 kHz (16000 Hz) to obtain a one-dimensional array with a length of 16000. Of course, the sampling rate is not limited to 16 kHz, and other values ​​may also be used.

[0100] Step 113: performing Fourier transform on the sampled audio signal to obtain a Fourier transformed audio signal;

[0101] Taking the second audio signal having a duration of 1s and being sampled at a sampling rate of 16kHz (16000Hz) to obtain a one-dimensional array of length 16000 as an example, the padded one-dimensional array can be Fourier transformed with a step size of 200 and a window size of 800 according to a padding method on both sides, and encoded into a one-dimensional array of length 800.

[0102] The padding method on both sides means that when extracting an array of length 800 (i.e. window length 800) every 200 values ​​(i.e. step length 200), when it comes to the 800th group, the one-dimensional array needs to be padded to ensure that 800 numbers are extracted. Therefore, in the first step of Fourier transform, the missing numbers can be padded with the number 0 at the front and end of the array respectively.

[0103] Step 114: extracting the Mel spectrum features of the audio signal after the Fourier transform to obtain an extracted audio signal;

[0104] Still using the example in step 113, after extracting the Mel-spectrogram features, an audio signal of size [400, 16] can be obtained, where 400 represents the number of Mel-spectrogram features corresponding to 1 second of the audio signal, and 16 represents the length of each Mel-spectrogram feature.

[0105] Mel-spectrogram features are a feature extraction method commonly used in audio signal processing. They are designed based on the human ear's auditory perception. The human ear's perception of audio signals is not linear but rather highly complex. Mel-spectrograms are a frequency scale designed based on the human ear's auditory characteristics and differ from conventional linear frequency scales. By filtering the signal at the Mel-spectrogram frequency scale, we can better simulate the human ear's perceptual characteristics and extract more discriminative signal features.

[0106] Step 115: Divide the extracted audio signal into M first audio signals.

[0107] If the first duration is 200ms and the second duration is 1s, then M is 5.

[0108] Still using the example in step 113 , the audio signal of size [400, 16] is divided into five arrays of size [80, 16], thus obtaining five first audio signals, each of which has a length of 200 ms.

[0109] In the embodiment of the present invention, optionally, before obtaining N video frames corresponding to the first audio signal in the video to be recognized, the step further includes:

[0110] Step 121: Obtaining the frame rate of the video to be identified;

[0111] Step 122: Determine the value of N according to the frame rate and the first duration.

[0112] Further optionally, determining the value of N according to the frame rate and the first duration includes:

[0113] Step 1221: Determine the quotient of the preset duration and the first duration to obtain a first value;

[0114] Optionally, the preset duration is 1 second, or other values ​​set as needed.

[0115] Step 1222: Determine the quotient of the frame rate and the first value to obtain a second value;

[0116] Step 1223: If the second value is an integer, use the second value as N;

[0117] Step 1224: If the second value is not an integer, round the second value to obtain a third value as N.

[0118] Assuming the first duration is 200ms, the first value is 1s / 200ms = 5. If the frame rate is 25fps, the second value is 25 / 5 = 5, that is, N = 5. If the frame rate is 30fps, the second value is 30 / 5 = 6, that is, N = 6. If the frame rate is 28fps, the second value is 28 / 5 = 5.6, 5.6 ≈ 6, that is, N = 6.

[0119] In the embodiment of the present invention, optionally, obtaining lip key point information in the N video frames includes:

[0120] Step 131: using a face detector to detect face images in the N video frames;

[0121] The face detector may adopt yolov5 or the like.

[0122] Step 132: extracting facial key points from the facial image using a facial key point extractor;

[0123] 2 , the number of facial key points may be 68. The facial key point extractor may adopt a PFLD (A Practical Facial Landmark Detector) algorithm or the like.

[0124] Step 133: Select lip key points from the facial key points.

[0125] Taking Figure 2 as an example, 20 key points numbered 49-68 can be selected, and the horizontal and vertical coordinates of the selected key points are arranged in sequence to form a one-dimensional array of length 40. The N one-dimensional arrays of length 40 of N frames of video are used as the input of the image encoder.

[0126] In an embodiment of the present invention, optionally, please refer to Figure 3, the audio encoder includes: multiple stacked convolution modules (conv) and a pooling layer (pooling) and a fully connected layer (FC) at the top layer; in an embodiment of the present invention, the pooling layer and the fully connected layer at the top layer can extract an audio feature vector of a fixed length. For example, if the output of the top fully connected layer is set to 512, a 512-dimensional audio feature vector can be obtained.

[0127] In an embodiment of the present invention, the image encoder optionally includes: multiple stacked LSTM (Long Short-Term Memory) modules and fully connected (FC) layers; or multiple stacked Bi-LSTM modules and fully connected layers. The Bi-LSTM modules can help better learn the relationship between consecutive video frames.

[0128] In the embodiment of the present invention, optionally, the loss function is contrastive loss.

[0129] Referring to FIG4 , an embodiment of the present invention further provides a method for training a speech lip synchronization recognition network, wherein the speech lip synchronization recognition network includes an audio encoder, an image encoder, and a loss function. The method includes:

[0130] Step 41: Acquire training data, where the training data includes positive sample data and negative sample data, where the positive sample data includes: a first audio signal of a video sample and N video frames corresponding to the first audio signal; and the negative sample data includes: the first audio signal of a video sample and N video frames including at least one video frame that is located at a different time period from the first audio signal. The duration of the audio signal is a first duration, and N is determined based on the first duration and a frame rate of the video sample, where N is an integer greater than or equal to 1.

[0131] The first duration may be, for example, 200 milliseconds (ms), although other durations are not excluded.

[0132] The N value corresponding to video samples with different frame rates may be different. For example, the N value corresponding to a video sample with a frame rate of 25 fps may be 5, and the N value corresponding to a video sample with a frame rate of 30 fps may be 6.

[0133] In the embodiment of the present invention, the video samples may include video samples of multiple frame rates.

[0134] In the embodiment of the present invention, optionally, the N video frames of the negative sample data and the first audio signal may be located at different times.

[0135] Step 42: Obtain lip key point information in the N video frames;

[0136] Step 43: Inputting the first audio signal into the audio encoder for encoding to obtain an audio feature vector;

[0137] Step 44: Inputting the lip key point information into the image encoder for encoding processing to obtain an image feature vector;

[0138] Step 45: Using the loss function of the speech-to-mouth synchronization recognition network to determine the distance between the audio feature vector and the image feature vector, and adjusting parameters of the audio encoder and / or the image encoder according to the distance.

[0139] In this embodiment of the present invention, because the number of video frames input to the image encoder is not a fixed value, the trained lip sync recognition network is suitable for lip sync recognition in videos with different frame rates, thereby improving the versatility of the lip sync recognition network. Furthermore, by pre-extracting facial lip key points as input to the lip sync recognition network, the present invention reduces the learning difficulty of the network and helps accelerate its convergence.

[0140] In the embodiment of the present invention, optionally, obtaining training data includes:

[0141] Step 411a: Acquire a second audio signal of the video sample, where the duration of the second audio signal is a second duration, and the second duration is M times the first duration, where M is an integer greater than or equal to 1;

[0142] For the convenience of calculation, for example, the second time length may be 1 second (s), although other values ​​are not excluded.

[0143] Step 412a: Sampling the second audio signal to obtain a plurality of sampled audio signals;

[0144] In the embodiment of the present invention, the second audio signal may be sampled at a sampling rate of 16 kHz (16000 Hz) to obtain a one-dimensional array with a length of 16000. Of course, the sampling rate is not limited to 16 kHz, and other values ​​may also be used.

[0145] Step 413a: Performing Fourier transform on the sampled audio signal to obtain a Fourier transformed audio signal;

[0146] Taking the second audio signal having a duration of 1s and being sampled at a sampling rate of 16kHz (16000Hz) to obtain a one-dimensional array of length 16000 as an example, the padded one-dimensional array can be Fourier transformed with a step size of 200 and a window size of 800 according to a padding method on both sides, and encoded into a one-dimensional array of length 800.

[0147] The padding method on both sides means that when extracting an array of length 800 (i.e. window length 800) every 200 values ​​(i.e. step length 200), when it comes to the 800th group, the one-dimensional array needs to be padded to ensure that 800 numbers are extracted. Therefore, in the first step of Fourier transform, the missing numbers can be padded with the number 0 at the front and end of the array respectively.

[0148] Step 414a: extracting the Mel spectrum features of the Fourier transformed audio signal to obtain an extracted audio signal;

[0149] Still using the example in step 413a, after extracting the Mel-spectrogram features, an audio signal of size [400, 16] can be obtained, where 400 represents the number of Mel-spectrogram features corresponding to 1 second of the audio signal, and 16 represents the length of each Mel-spectrogram feature.

[0150] Step 415a: extract the first audio signal from the extracted audio signal.

[0151] In the embodiment of the present invention, optionally, obtaining training data further includes:

[0152] Step 411b: Obtain the frame rate of the video sample;

[0153] Step 412b: Determine the value of N according to the frame rate and the first duration.

[0154] Further optionally, determining the value of N according to the frame rate and the first duration includes:

[0155] Step 412b1: Determine the quotient of the preset duration and the first duration to obtain a first value;

[0156] The preset duration is 1 second, or other values ​​set as needed.

[0157] Step 412b2: Determine the quotient of the frame rate and the first value to obtain a second value;

[0158] Step 412b3: If the second value is an integer, use the second value as N;

[0159] Step 412b4: If the second value is not an integer, round the second value to obtain a third value as N.

[0160] Assuming the first duration is 200ms, the first value is 1s / 200ms = 5. If the frame rate is 25fps, the second value is 25 / 5 = 5, that is, N = 5. If the frame rate is 30fps, the second value is 30 / 5 = 6, that is, N = 6. If the frame rate is 28fps, the second value is 28 / 5 = 5.6, 5.6 ≈ 6, that is, N = 6.

[0161] In the embodiment of the present invention, optionally, obtaining training data further includes:

[0162] Step 411c: using a face detector to detect face images in the N video frames;

[0163] The face detector may adopt yolov5 or the like.

[0164] Step 412c: extracting facial key points from the facial image using a facial key point extractor;

[0165] 2 , the number of facial key points may be 68. The facial key point extractor may adopt a PFLD algorithm or the like.

[0166] Step 413c: Select lip key points from the facial key points.

[0167] Taking Figure 2 as an example, 20 key points numbered 49-68 can be selected, and the horizontal and vertical coordinates of the selected key points are arranged in sequence to form a one-dimensional array of length 40. The N one-dimensional arrays of length 40 of N frames of video are used as the input of the image encoder.

[0168] In the embodiment of the present invention, the above-mentioned face detector and facial key points can be trained using video samples.

[0169] In an embodiment of the present invention, a pre-trained image feature extractor can also be used to extract lip key points of a video frame, such as using resnet50 (or other backbones such as resnet34) pre-trained on the ImageNet dataset to extract lip key points.

[0170] In an embodiment of the present invention, optionally, please refer to Figure 3, the audio encoder includes: multiple stacked convolution modules (conv) and a pooling layer (pooling) and a fully connected layer (FC) at the top layer; in an embodiment of the present invention, the pooling layer and the fully connected layer at the top layer can extract an audio feature vector of a fixed length. For example, if the output of the top fully connected layer is set to 512, a 512-dimensional audio feature vector can be obtained.

[0171] In an embodiment of the present invention, the image encoder optionally includes: a plurality of stacked LSTM (Long Short-Term Memory) modules and fully connected layers (FC); or a plurality of stacked Bi-LSTM modules and fully connected layers. In one embodiment of the present invention, the number of hidden units of the LSTM module can be 400. For N video frames, the input size of the image encoder is [batch_size, N, 40], where batch_size is the number of training data fed into the image encoder at one time (i.e., the number of groups of lip key point information included), N is the number of video frames corresponding to a group of lip key point information, and 40 is the number of horizontal and vertical coordinates of a group of lip key point information. After encoding by the LSTM module, a feature array of size [batch_size, N, 400] is obtained. The one-dimensional feature of length 400 corresponding to the last frame is selected and input into the fully connected layer to obtain the final image feature vector. The LSTM module encodes the lip key point information of each video frame in N video frames, and the feature encoding of each video frame also learns the features of the previous frames, so in the end only the encoding result of the features of the last video frame is taken as the final image feature vector.

[0172] In the embodiment of the present invention, optionally, the loss function is contrastive loss.

[0173] In recent years, the digital human industry has flourished. In this industry, a low-input, high-output generation method is to use a pre-trained lip generation network to drive the static host face to "open" and speak according to a given speech synthesis (Text To Speech, TTS) signal, thereby realizing the interaction between the digital human and the real world. Generally, more effective lip generation networks, such as the wav2lip model, require a speech and mouth synchronization recognition network as a training "expert" during the training process to constrain the learning of the lip generation network. The speech and mouth synchronization recognition network of the embodiment of the present invention can be used as an expert model to train the lip generation model, which has important guiding significance for the production of digital humans. In addition, the speech and mouth synchronization recognition network of the embodiment of the present invention can also be used to help identify whether there is a problem of audio and video being out of sync in videos such as movies, TV series, and news.

[0174] Referring to FIG5 , an embodiment of the present invention further provides a voice and voice synchronization recognition device 50 , comprising:

[0175] A first acquisition module 51 is configured to acquire a first audio signal of a video to be identified, where the duration of the first audio signal is a first duration;

[0176] a second acquisition module 52 configured to acquire N video frames from the video to be identified, wherein the N video frames correspond to the first audio signal, N is determined according to the first duration and the frame rate of the video to be identified, and N is an integer greater than or equal to 1;

[0177] A third acquisition module 53 is used to obtain lip key point information in the N video frames;

[0178] An audio encoding module 54 is configured to input the first audio signal into an audio encoder of a speech and mouth synchronization recognition network for encoding processing to obtain an audio feature vector;

[0179] An image encoding module 55 is configured to input the lip key point information into the image encoder of the voice-mouth synchronization recognition network for encoding processing to obtain an image feature vector;

[0180] The synchronization determination module 56 is configured to determine the distance between the audio feature vector and the image feature vector using the loss function of the speech-mouth synchronization recognition network, and determine whether the speech-mouth synchronization is achieved based on the distance.

[0181] Optionally, the first acquisition module 51 is used to obtain a second audio signal of the video to be identified, where the duration of the second audio signal is a second duration, and the second duration is M times the first duration; sample the second audio signal to obtain multiple sampled audio signals; perform Fourier transform on the sampled audio signal to obtain a Fourier transformed audio signal; extract the Mel spectrum feature of the Fourier transformed audio signal to obtain an extracted audio signal; and divide the extracted audio signal into M first audio signals, where M is an integer greater than or equal to 1.

[0182] Optionally, the voice and mouth synchronization recognition device 50 further includes:

[0183] A fourth acquisition module, configured to acquire a frame rate of the video to be identified;

[0184] A determination module is used to determine the value of N according to the frame rate and the first duration.

[0185] Optionally, the determination module is used to determine the quotient of the preset duration and the first duration to obtain a first value; determine the quotient of the frame rate and the first value to obtain a second value; if the second value is an integer, the second value is used as N; if the second value is not an integer, the second value is rounded to obtain a third value as N.

[0186] Optionally, the third acquisition module 53 is configured to detect facial images in the N video frames using a face detector; extract facial key points in the facial images using a face key point extractor; and select lip key points from the facial key points.

[0187] Optionally, the audio encoder includes: a plurality of stacked convolution modules and a pooling layer and a fully connected layer at the top layer;

[0188] and / or

[0189] The image encoder includes: multiple stacked LSTM modules and fully connected layers; or, multiple stacked Bi-LSTM modules and fully connected layers.

[0190] Optionally, the loss function is contrastive loss.

[0191] 6 , an embodiment of the present invention further provides a training device 60 for a speech lip synchronization recognition network. The speech lip synchronization recognition network includes: an audio encoder, an image encoder, and a loss function. The training device 60 for the speech lip synchronization recognition network includes:

[0192] A first acquisition module 61 is configured to acquire training data, the training data including positive sample data and negative sample data, the positive sample data including: a first audio signal of a video sample and N video frames corresponding to the first audio signal; the negative sample data including: the first audio signal of a video sample and N video frames including at least one video frame in a different time period from the first audio signal; wherein the duration of the audio signal is a first duration, N is determined based on the first duration and a frame rate of the video sample, and N is an integer greater than or equal to 1;

[0193] A second acquisition module 62 is used to obtain lip key point information in the N video frames;

[0194] An audio encoding module 63 is configured to input the first audio signal into the audio encoder for encoding to obtain an audio feature vector;

[0195] An image encoding module 64 is configured to input the lip key point information into the image encoder of the voice-mouth synchronization recognition network for encoding processing to obtain an image feature vector;

[0196] The optimization module 65 is configured to determine the distance between the audio feature vector and the image feature vector using the loss function of the speech-to-lip synchronization recognition network, and adjust parameters of the audio encoder and / or the image encoder according to the distance.

[0197] Optionally, the first acquisition module 61 is used to obtain a second audio signal of the video sample, where the duration of the second audio signal is a second duration, and the second duration is M times the first duration; sample the second audio signal to obtain multiple sampled audio signals; perform Fourier transform on the sampled audio signal to obtain a Fourier transformed audio signal; extract the Mel spectrum feature of the Fourier transformed audio signal to obtain an extracted audio signal; and extract the first audio signal from the extracted audio signal, where M is an integer greater than or equal to 1.

[0198] Optionally, the first acquisition module 61 is further configured to acquire a frame rate of the video sample; and determine a value of N according to the frame rate and the first duration.

[0199] Further optionally, determining the value of N based on the frame rate and the first duration includes: determining the quotient of the preset duration and the first duration to obtain a first value; determining the quotient of the frame rate and the first value to obtain a second value; if the second value is an integer, taking the second value as N; if the second value is not an integer, rounding the second value to obtain a third value as N.

[0200] Optionally, the first acquisition module 61 is further used to detect facial images in the N video frames using a face detector; extract facial key points in the facial images using a face key point extractor; and select lip key points from the facial key points.

[0201] In an embodiment of the present invention, optionally, the audio encoder includes: multiple stacked convolution modules (conv) and a pooling layer (pooling) and a fully connected layer (FC) at the top layer; in an embodiment of the present invention, the pooling layer and the fully connected layer at the top layer can extract audio feature vectors of fixed length. For example, if the output of the top fully connected layer is set to 512, a 512-dimensional audio feature vector can be obtained.

[0202] In an embodiment of the present invention, the image encoder optionally includes: a plurality of stacked LSTM (Long Short-Term Memory) modules and fully connected layers (FC); or a plurality of stacked Bi-LSTM modules and fully connected layers. In one embodiment of the present invention, the number of hidden units of the LSTM module can be 400. For N video frames, the input size of the image encoder is [batch_size, N, 40], where batch_size is the number of training data fed into the image encoder at one time (i.e., the number of groups of lip key point information included), N is the number of video frames corresponding to a group of lip key point information, and 40 is the number of horizontal and vertical coordinates of a group of lip key point information. After encoding by the LSTM module, a feature array of size [batch_size, N, 400] is obtained. The one-dimensional feature of length 400 corresponding to the last frame is selected and input into the fully connected layer to obtain the final image feature vector. The LSTM module encodes the lip key point information of each video frame in N video frames, and the feature encoding of each video frame also learns the features of the previous frames, so in the end only the encoding result of the features of the last video frame is taken as the final image feature vector.

[0203] Please refer to Figure 7. An embodiment of the present invention further provides a ring back tone platform 70, including a processor 71, a memory 72, and a computer program stored in the memory 72 and executable on the processor 71. When the computer program is executed by the processor 71, each process of the above-mentioned speech-mouth synchronization recognition method or speech-mouth synchronization recognition network training method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0204] An embodiment of the present invention further provides a non-transitory computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the various processes of the above-mentioned embodiment of the method for training a speech-to-mouth synchronization recognition network or the method for training a speech-to-mouth synchronization recognition network, and can achieve the same technical effects. To avoid repetition, the description is omitted here. The non-transitory computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0205] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0206] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better embodiment. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0207] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are protected by the present invention.

Claims

1. A method for speech and mouth synchronization recognition, characterized in that: include: Acquire a first audio signal of a video to be identified, where the duration of the first audio signal is a first duration; Obtaining N video frames in the video to be identified, wherein the N video frames correspond to the first audio signal, N is determined according to the first duration and the frame rate of the video to be identified, and N is an integer greater than or equal to 1; Obtaining lip key point information in the N video frames; Inputting the first audio signal into an audio encoder of a speech and mouth synchronization recognition network for encoding processing to obtain an audio feature vector; Inputting the lip key point information into the image encoder of the voice-mouth synchronization recognition network for encoding processing to obtain an image feature vector; The loss function of the speech-mouth synchronization recognition network is used to determine the distance between the audio feature vector and the image feature vector, and whether the speech is synchronized is determined based on the distance.

2. The method according to claim 1, characterized in that The step of obtaining a first audio signal of a video to be identified includes: Acquire a second audio signal of the video to be identified, where the duration of the second audio signal is a second duration, and the second duration is M times the first duration, where M is an integer greater than or equal to 1; performing sampling processing on the second audio signal to obtain a plurality of sampled audio signals; Performing Fourier transform on the sampled audio signal to obtain a Fourier transformed audio signal; Extracting the Mel spectrum feature of the audio signal after the Fourier transformation to obtain an extracted audio signal; The extracted audio signal is divided into M first audio signals.

3. The method according to claim 1, characterized in that Before obtaining N video frames corresponding to the first audio signal in the video to be identified, the method further includes: Obtaining the frame rate of the video to be identified; The value of N is determined according to the frame rate and the first duration.

4. The method according to claim 3, characterized in that The determining a value of N according to the frame rate and the first duration includes: Determine the quotient of the preset duration and the first duration to obtain a first value; determining a quotient of the frame rate and the first value to obtain a second value; If the second value is an integer, the second value is used as N; If the second value is not an integer, the second value is rounded to obtain a third value as N.

5. The method according to claim 1, wherein The obtaining of lip key point information in the N video frames includes: Using a face detector to detect face images in the N video frames; Extracting facial key points from the facial image using a facial key point extractor; Select lip key points from the facial key points.

6. The method according to claim 1, characterized in that The audio encoder includes: a plurality of stacked convolution modules and a pooling layer and a fully connected layer at the top; and / or The image encoder includes: multiple stacked LSTM modules and fully connected layers; or, multiple stacked Bi-LSTM modules and fully connected layers.

7. A method for training a speech and voice synchronization recognition network, characterized in that: The speech and voice synchronization recognition network includes: an audio encoder, an image encoder and a loss function, and the method includes: Acquire training data, the training data including positive sample data and negative sample data, the positive sample data including: a first audio signal of a video sample and N video frames corresponding to the first audio signal; the negative sample data including: the first audio signal of the video sample and N video frames including at least one video frame located at a different time period from the first audio signal; wherein the duration of the audio signal is a first duration, N is determined based on the first duration and a frame rate of the video sample, and N is an integer greater than or equal to 1; Obtaining lip key point information in the N video frames; Inputting the first audio signal into the audio encoder for encoding to obtain an audio feature vector; The lip key point information is input into the image encoder for encoding processing to obtain the image Like eigenvectors; A loss function of the speech and mouth synchronization recognition network is used to determine a distance between the audio feature vector and the image feature vector, and parameters of the audio encoder and / or the image encoder are adjusted according to the distance.

8. The method according to claim 7, characterized in that The obtaining of training data includes: Acquire a second audio signal of the video sample, where the duration of the second audio signal is a second duration, and the second duration is M times the first duration, where M is an integer greater than or equal to 1; performing sampling processing on the second audio signal to obtain a plurality of sampled audio signals; Performing Fourier transform on the sampled audio signal to obtain a Fourier transformed audio signal; Extracting the Mel spectrum feature of the audio signal after the Fourier transformation to obtain an extracted audio signal; The first audio signal is extracted from the extracted audio signal.

9. The method according to claim 7, characterized in that The obtaining of training data includes: Get the frame rate of the video sample; The value of N is determined according to the frame rate and the first duration.

10. The method according to claim 9, characterized in that The determining a value of N according to the frame rate and the first duration includes: Determine the quotient of the preset duration and the first duration to obtain a first value; determining a quotient of the frame rate and the first value to obtain a second value; If the second value is an integer, the second value is used as N; If the second value is not an integer, the second value is rounded to obtain a third value as N.

11. A speech and mouth synchronization recognition device, characterized in that: include: A first acquisition module is configured to acquire a first audio signal of a video to be identified, where the duration of the first audio signal is a first duration; a second acquisition module, configured to acquire N video frames from the video to be identified, wherein the N video frames correspond to the first audio signal, N is determined according to the first duration and a frame rate of the video to be identified, and N is an integer greater than or equal to 1; A third acquisition module is used to obtain lip key point information in the N video frames; an audio encoding module, configured to input the first audio signal into an audio encoder of a speech and mouth synchronization recognition network for encoding processing to obtain an audio feature vector; An image encoding module, configured to input the lip key point information into the image encoder of the voice-mouth synchronization recognition network for encoding processing to obtain an image feature vector; The synchronization determination module is used to determine the distance between the audio feature vector and the image feature vector using the loss function of the voice-mouth synchronization recognition network, and determine whether the voice-mouth synchronization is achieved based on the distance.

12. A training device for a speech and voice synchronization recognition network, characterized in that: The speech lip synchronization recognition network includes: an audio encoder, an image encoder and a loss function, and the training device of the speech lip synchronization recognition network includes: A first acquisition module is configured to acquire training data, the training data including positive sample data and negative sample data, the positive sample data including: a first audio signal of a video sample and N video frames corresponding to the first audio signal; the negative sample data including: the first audio signal of a video sample and N video frames including at least one video frame located at a different time period from the first audio signal; wherein the duration of the audio signal is a first duration, N is determined based on the first duration and a frame rate of the video sample, and N is an integer greater than or equal to 1; A second acquisition module is used to obtain lip key point information in the N video frames; an audio encoding module, configured to input the first audio signal into the audio encoder for encoding processing to obtain an audio feature vector; An image encoding module, configured to input the lip key point information into the image encoder of the voice-mouth synchronization recognition network for encoding processing to obtain an image feature vector; An optimization module is configured to determine a distance between the audio feature vector and the image feature vector using a loss function of the speech-to-mouth synchronization recognition network, and adjust parameters of the audio encoder and / or the image encoder according to the distance.

13. An electronic device, characterized in that: include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, the program implements the steps of the method for speech-to-lip synchronization recognition as described in any one of claims 1 to 6, or when the program is executed by the processor, the program implements the steps of the method for training a speech-to-lip synchronization recognition network as described in any one of claims 7 to 10.

14. A non-transitory computer-readable storage medium, characterized in that The non-transitory computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, it implements the steps of the method for speech synchronization recognition as described in any one of claims 1 to 6; or, when the computer program is executed by the processor, it implements the steps of the method for training a speech synchronization recognition network as described in any one of claims 7 to 10.

Images