A method for lip-tone consistency judgment based on multi-keynote joint score fusion

Abstract

The present application discloses a method for determining lip-tone consistency based on the fusion of joint scores of multiple key sounds. Vowel-like sounds with significant lip shape changes are used as key sounds. A vowel-like detection method combining Hilbert envelope and zero-frequency filtering is used to screen each key audio segment in the audio data to be processed. Then, based on the delay difference, a key video segment is determined from the key audio segment. A multi-layer neural network describing two different modes is trained through deep canonical correlation analysis coupling to obtain a nonlinear representation mapping that maximizes the correlation between the two types of audio and video data, thereby more accurately performing correlation analysis on the lip-tone data of the key audio segment. The joint consistency score of each key audio segment and key video segment is estimated by combining the delay difference and the correlation difference. Finally, a scoring mechanism for the fusion of the joint scores of multiple key audio segments is proposed to fuse the scores of each key audio segment to determine the consistency.

Description

Technical Field

[0001] The present application relates to the field of audio and video processing, and in particular to a method for determining lip-voice consistency based on the fusion of multiple key-note joint scores. Background Art

[0002] The lip-sound consistency judgment, that is, the lip-movement consistency judgment of speech, is to judge whether the audio and video are recorded at the same time, whether they are from the same person, and whether they are from the same sentence through the connection between the lip movement and the audio change during the pronunciation process. Existing lip-sound consistency judgment methods mainly process and analyze the lip movement video data collected from the front angle, and the database used in the experiment is also mainly based on the front shooting data set, but lack consideration of the impact of lip angle changes on consistency judgment.

[0003] In addition, the existing methods for determining lip-concordance have too large data dimensions during training, slow model convergence, and consume a lot of computing resources. They also lack screening for the content in the analyzed sentences, and the results are easily affected by weakly associated fragments such as silence. Summary of the invention

[0004] The purpose of this application is to provide a method for determining lip-tone consistency based on the fusion of multiple key-note joint scores, which can improve the above-mentioned problem.

[0005] The embodiment of the present application is implemented as follows:

[0006] The present application provides a method for determining lip consistency based on the fusion of multiple key-note joint scores, which includes:

[0007] S1. Acquire lip data to be processed, and split the lip data to be processed into audio data to be processed and video data to be processed;

[0008] S2, extracting each key audio segment in the audio data to be processed, and extracting a key video segment matching each key audio segment in the video data to be processed;

[0009] S3, performing deep canonical correlation analysis on each pair of the key audio segment and the key video segment;

[0010] S4. Calculating a joint consistency score for each pair of the key audio segment and the key video segment based on the delay difference and the analysis result of the deep canonical correlation analysis;

[0011] S5. Calculate a joint score of the lip data to be processed according to the joint consistency score of each pair of the key audio segment and the key video segment.

[0012] The present application discloses a method for determining lip-vowel consistency based on the fusion of multiple key-vowel joint scores, wherein quasi-vowels with significant lip shape changes are taken as key sounds, and each key audio segment in the audio data to be processed is screened by a quasi-vowel detection method combining Hilbert envelope and zero-frequency filtering; then, a key video segment is determined from the key audio segment based on the delay difference; a multi-layer neural network describing two different modes is coupled and trained by deep canonical correlation analysis (DCCA) to obtain a nonlinear representation mapping that maximizes the correlation between the two types of audio and video data, thereby more accurately performing correlation analysis on the lip-vowel data of the key audio segment; the consistency joint score of each key audio segment and key video segment is estimated by combining the delay difference and the correlation difference, and finally a scoring mechanism of multi-key audio segment joint score fusion (Multi-key Segment Joint Score Fusion, MS-JSF) is proposed, and the scores of each key audio segment are fused to estimate the final consistency score of the whole sentence.

[0013] It can be understood that the present application uses vowel-like sounds with significant lip shape changes as key sounds to perform lip-sound consistency analysis of key audio and video segments, and uses the lip-sound consistency analysis results of each pair of key audio and video segments to determine the lip-sound consistency of the entire sentence. This solves the problem of too large data dimensions, slow model convergence, and consumption of a large amount of computing resources during training. Moreover, the judgment results are not easily affected by weakly related segments such as lack of content screening and silence, and are more stable.

[0014] In an optional embodiment of the present application, after step S2 and before step S3, the method further includes: generating a corresponding reconstructed frontal lip picture according to the non-frontal lip picture in the key video segment through a generative adversarial network, and replacing the non-frontal lip picture in the key video segment with the reconstructed frontal lip picture.

[0015] Among them, the generative adversarial network is a self-mapping supervised cyclic generative adversarial network, including a frontal lip generator, a non-frontal lip generator, a frontal lip discriminator and a non-frontal lip discriminator; the frontal lip generator is used to generate a corresponding reconstructed frontal lip picture according to the non-frontal lip picture in the key video segment; the frontal lip discriminator is used to judge the true or false of the reconstructed frontal lip picture and the real frontal lip picture in the key video segment; the non-frontal lip generator is used to restore the reconstructed frontal lip picture to a reconstructed non-frontal lip picture corresponding to the non-frontal lip picture in the video data to be processed; the non-frontal lip discriminator is used to judge the true or false of the reconstructed non-frontal lip picture and the real non-frontal lip picture in the key video segment.

[0016] Among them, the total loss of the self-mapping supervised cycle generation adversarial network is:

[0017] L(G X ,G Y ,D X ,D Y )=L GAN (G X ,D X )+L GAN (G Y ,D Y )+λL cyc (G X ,G Y )+μL S (G X ,G Y );

[0018] Among them, G Y is the function corresponding to the front lip generator, G X is the function corresponding to the non-frontal lip generator, D Y is the function corresponding to the frontal lip discriminator, D X is the function corresponding to the non-frontal lip discriminator, L GAN (G X ,D X ) and L GAN (G Y ,D Y ) is to combat the loss, L cyc (G X ,G Y ) is the reconstruction loss, L S (G X ,G Y ) is the self-mapping test loss, λ and μ are the weights of the reconstruction loss and the self-mapping test loss respectively. ;

[0019] The adversarial loss of the self-mapping supervised recurrent generative adversarial network includes:

[0020]

[0021] Among them, y~pdata(y) represents the real front lip picture in the video data to be processed, x~p data (x) represents the real non-frontal lip picture in the video data to be processed, and E(·) represents the expectation;

[0022] The reconstruction loss of the self-mapping supervised cyclic generative adversarial network satisfies the following formula:

[0023]

[0024] The self-mapping test loss of the self-mapping supervised cyclic generative adversarial network satisfies the following formula:

[0025]

[0026] It can be understood that the method for determining the consistency of the sound and lips disclosed in the present application aims at the influence of the change of lip angle on the consistency determination, and proposes a self-mapping supervised cyclic generative adversarial network to generate corresponding reconstructed frontal lip pictures according to the non-frontal lip pictures in the video data to be processed, thereby enriching the number of frontal lip pictures in the video data to be processed, and being more conducive to the accuracy of subsequent sound and lips consistency determination. The self-mapping supervised cyclic generative adversarial network solves the problem that if the traditional generative adversarial network is used to generate pictures without constraints, the images after conversion are prone to blurring and deformation.

[0027] In an optional embodiment of the present application, the step of “extracting each key audio segment from the audio data to be processed” in step S2 includes:

[0028] S21, extracting the Hilbert envelope and zero-frequency filtering signal of the audio data to be processed;

[0029] S22, convolving the Hilbert envelope and the second-order difference of the zero-frequency filter signal with the window function of the first-order Gaussian differentiator respectively, and then summing them, and normalizing the summation result to obtain a first curve;

[0030] S23, confirming the maximum value between two consecutive zero-crossing points in the first curve as the starting point of the audio data to be processed;

[0031] S24, convolving the Hilbert envelope and the first-order difference of the zero-frequency filter signal with a window function respectively and summing them to obtain a second curve;

[0032] S25. Determine the end point of the audio data to be processed from right to left on the second curve by using a peak search strategy;

[0033] S26. Determine each key audio segment according to the starting point and the ending point.

[0034] In an optional embodiment of the present application, in step S2, “extracting a key video segment matching each key audio segment from the video data to be processed” includes:

[0035] S27, according to the audio starting point of the key audio segment and audio end point Determine the time period of the key audio segment Wherein m represents the index of the key audio segment in the audio data to be processed;

[0036] S28. Determine the starting point delay parameter - φ left and end point delay parameter +φ right;

[0037] S29, moving the audio start point forward by the start point delay parameter, and moving the audio end point backward by the end point delay parameter, so as to obtain the extraction range of the key video segment

[0038] S30: Extract the key video segment from the video data to be processed according to the extraction range.

[0039] It is understandable that the audio and lip movements of the lip-congruent data will also have initial delays at corresponding time points, which are mainly distributed within [-50ms, +50ms], while the inconsistent data are mostly distributed outside the above range. The audio and video signals of lip-congruent belong to the same time axis, and the lip movements and audio changes at each key sound position must be highly correlated. At the same time, the audio and video delays at the above positions should also be within the reasonable distribution range of the initial delay.

[0040] In an optional embodiment of the present application, step S3 includes:

[0041] S31, inputting each pair of the audio features of the key audio segment and the video features of the key video segment into a first multi-layer neural network and a second multi-layer neural network respectively, to obtain an audio output result and a video output result;

[0042] S32. Calculate a depth correlation change curve according to the audio output result and the video output result according to the following formula:

[0043]

[0044] in, represents the depth correlation variation curve, h A The mapping vector representing the audio output result, h V represents the mapping vector of the video output result, J is the first J mapping vectors selected, j represents the index of the mapping vector, F AO Represents the audio output result, F VO represents the video output result, corr() is the correlation function,

[0045] τ represents the delay, wherein the delay corresponding to the mth key audio segment and the key video segment is calculated by the following formula:

[0046]

[0047] in, Represents the time delay corresponding to the mth key audio segment and the key video segment.

[0048] Among them, in the depth correlation change curve, |corr(FAO h A,j ,F VO h V,j )| can be expressed as ||Z||;

[0049] in,

[0050]

[0051] and are covariance ∑ AA ,∑ VV and the cross-covariance∑ AV The scattering matrix.

[0052] It can be understood that the deep canonical correlation analysis proposed in the application overcomes the shortcomings of non-parametric models. By coupling training of multi-layer neural networks that describe two different modal data, a nonlinear representation mapping that maximizes the correlation between the two types of audio and video data is obtained, thereby more accurately performing correlation analysis on the lip data of key audio segments, and improving the robustness of the model to changes in sample quantity caused by screening through re-estimation of the covariance matrix.

[0053] Among them, the correlation change curve A peak will appear at the corresponding position of the initial delay, and the values ​​on both sides will gradually decrease; the curve of inconsistent data is relatively flat. Even if a pseudo peak appears, it is mainly outside the reasonable distribution range of the initial delay, and the value will be significantly lower than the consistent data.

[0054] In an optional embodiment of the present application, step S4 includes:

[0055] The consistency joint score of each pair of the key audio segment and the key video segment is calculated by the following formula:

[0056]

[0057] Among them, δ m represents the consistency joint score corresponding to the mth key audio segment and the key video segment;

[0058] in,

[0059]

[0060] and represent the consistency joint score outside and within the reasonable delay range respectively, and ζ represents the correlation threshold between inconsistent data and consistent data obtained by training the first multi-layer neural network and the second multi-layer neural network.

[0061] In an optional embodiment of the present application, step S5 includes:

[0062] The joint score of the lip data to be processed is calculated by the following formula:

[0063]

[0064] in, represents the joint score, M represents the logarithm of the key audio segment and the key video segment in the lip data to be processed, σ m represents the standard deviation of the inconsistent data distribution, Represents the local decision threshold.

[0065] In a second aspect, the present application discloses an electronic device comprising a processor, an input device, an output device and a memory connected to each other, wherein the memory is used to store a computer program, the computer program includes program instructions, and the processor is configured to call the program instructions to execute any method described in the first aspect.

[0066] In a third aspect, the present application further discloses a computer-readable storage medium, wherein the computer storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes the method as described in any one of the first aspects.

[0067] Beneficial effects:

[0068] The present application discloses a method for determining lip-vowel consistency based on the fusion of joint scores of multiple key sounds. Vowel-like sounds with significant lip shape changes are taken as key sounds. Each key audio segment in the audio data to be processed is screened by a vowel-like detection method combining Hilbert envelope and zero-frequency filtering. Then, a key video segment is determined from the key audio segment based on the delay difference. A multi-layer neural network describing two different modes is trained by deep canonical correlation analysis coupling to obtain a nonlinear representation mapping that maximizes the correlation between the two types of audio and video data, thereby more accurately performing correlation analysis on the lip-vowel data of the key audio segment. The joint consistency score of each key audio segment and key video segment is estimated by combining the delay difference and the correlation difference. Finally, a scoring mechanism for the fusion of the joint scores of multiple key audio segments is proposed, and the scores of each key audio segment are fused to estimate the final consistency score of the whole sentence.

[0069] It can be understood that the present application uses vowel-like sounds with significant lip shape changes as key sounds to perform lip-sound consistency analysis of key audio and video segments, and uses the lip-sound consistency analysis results of each pair of key audio and video segments to determine the lip-sound consistency of the entire sentence. This solves the problem of too large data dimensions, slow model convergence, and consumption of a large amount of computing resources during training. Moreover, the judgment results are not easily affected by weakly related segments such as lack of content screening and silence, and are more stable.

[0070] It can be understood that the method for determining the consistency of the sound and lips disclosed in the present application aims at the influence of the change of lip angle on the consistency determination, and proposes a self-mapping supervised cyclic generative adversarial network to generate corresponding reconstructed frontal lip pictures according to the non-frontal lip pictures in the video data to be processed, thereby enriching the number of frontal lip pictures in the video data to be processed, and being more conducive to the accuracy of subsequent sound and lips consistency determination. The self-mapping supervised cyclic generative adversarial network solves the problem that if the traditional generative adversarial network is used to generate pictures without constraints, the images after conversion are prone to blurring and deformation.

[0071] It can be understood that the deep canonical correlation analysis proposed in the application overcomes the shortcomings of non-parametric models. By coupling training of multi-layer neural networks that describe two different modal data, a nonlinear representation mapping that maximizes the correlation between the two types of audio and video data is obtained, thereby more accurately performing correlation analysis on the lip data of key audio segments, and improving the robustness of the model to changes in sample quantity caused by screening through re-estimation of the covariance matrix.

[0072] In order to make the above-mentioned objects, features and advantages of the present application more obvious and understandable, optional embodiments are specifically listed below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0074] Figure 1 This is a flowchart of a method for determining lip-tone consistency based on the fusion of multiple key-note joint scores provided by the present application;

[0075] Figure 2 It is a schematic diagram of the structure of the self-mapping supervised recurrent generative adversarial network provided by this application;

[0076] Figure 3 It is a model diagram of the deep canonical correlation analysis provided by this application. DETAILED DESCRIPTION

[0077] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0078] like Figure 1 As shown, the present application provides a method for determining lip consistency based on the fusion of multiple key-note joint scores, which includes:

[0079] S1. Obtain lip data to be processed, and split the lip data to be processed into audio data to be processed and video data to be processed.

[0080] S2. Extract each key audio segment from the audio data to be processed, and extract a key video segment matching each key audio segment from the video data to be processed.

[0081] The correlation of vowels, diphthongs, semi-vowels and other similar vowels is close to the correlation of the whole sentence. The lip movement rules of their pronunciation are relatively stable and the lip shape visual element is more representative. Therefore, we can try to analyze this type of pronunciation unit instead of the whole sentence to achieve consistency judgment. This application uses similar vowels with significant lip shape changes as key sounds to perform sound-lip consistency analysis of key audio and video segments, and uses the sound-lip consistency analysis results of each pair of key audio and video segments to judge the sound-lip consistency of the whole sentence.

[0082] S3. Perform deep canonical correlation analysis on each pair of key audio segments and key video segments.

[0083] It can be understood that the deep canonical correlation analysis proposed in the application overcomes the shortcomings of non-parametric models. By coupling training of multi-layer neural networks that describe two different modal data, a nonlinear representation mapping that maximizes the correlation between the two types of audio and video data is obtained, thereby more accurately performing correlation analysis on the lip data of key audio segments, and improving the robustness of the model to changes in sample quantity caused by screening through re-estimation of the covariance matrix.

[0084] S4. Based on the analysis results of delay difference and deep canonical correlation analysis, calculate the joint consistency score of each pair of key audio segments and key video segments.

[0085] S5. Calculate the joint score of the lip data to be processed according to the joint consistency score of each pair of key audio segments and key video segments.

[0086] The present application discloses a method for determining lip-vowel consistency based on the fusion of multiple key-vowel joint scores, wherein quasi-vowels with significant lip shape changes are taken as key sounds, and each key audio segment in the audio data to be processed is screened by a quasi-vowel detection method combining Hilbert envelope and zero-frequency filtering; then, a key video segment is determined from the key audio segment based on the delay difference; a multi-layer neural network describing two different modes is coupled and trained by deep canonical correlation analysis (DCCA) to obtain a nonlinear representation mapping that maximizes the correlation between the two types of audio and video data, thereby more accurately performing correlation analysis on the lip-vowel data of the key audio segment; the consistency joint score of each key audio segment and key video segment is estimated by combining the delay difference and the correlation difference, and finally a scoring mechanism of multi-key audio segment joint score fusion (Multi-key Segment Joint Score Fusion, MS-JSF) is proposed, and the scores of each key audio segment are fused to estimate the final consistency score of the whole sentence.

[0087] It can be understood that the present application uses vowel-like sounds with significant lip shape changes as key sounds to perform lip-sound consistency analysis of key audio and video segments, and uses the lip-sound consistency analysis results of each pair of key audio and video segments to determine the lip-sound consistency of the entire sentence. This solves the problem of too large data dimensions, slow model convergence, and consumption of a large amount of computing resources during training. Moreover, the judgment results are not easily affected by weakly related segments such as lack of content screening and silence, and are more stable.

[0088] In an optional embodiment of the present application, after step S2 and before step S3, the method further includes: generating a corresponding reconstructed frontal lip picture according to the non-frontal lip picture in the key video segment through a generative adversarial network, and replacing the non-frontal lip picture in the key video segment with the reconstructed frontal lip picture.

[0089] Among them, Figure 2 As shown, the generative adversarial network is a self-mapping supervised cyclic generative adversarial network, including a frontal lip generator 201, a non-frontal lip generator 202, a frontal lip discriminator 203 and a non-frontal lip discriminator (not shown in the figure); the frontal lip generator 201 is used to generate a corresponding reconstructed frontal lip picture according to the non-frontal lip picture in the key video segment; the frontal lip discriminator 203 is used to judge the true or false of the reconstructed frontal lip picture and the real frontal lip picture in the key video segment; the non-frontal lip generator 202 is used to restore the reconstructed frontal lip picture to the corresponding reconstructed non-frontal lip picture; the non-frontal lip discriminator is used to judge the true or false of the reconstructed non-frontal lip picture and the real non-frontal lip picture in the key video segment.

[0090] Among them, the total loss of the self-mapping supervised cycle generation adversarial network is:

[0091] L(G X ,G Y ,D X ,D Y )=L GAN (G X ,D X )+L GAN (G Y ,D Y )+λL cyc (G X ,G Y )+μL S (G X ,G Y );

[0092] Among them, G Y is the function corresponding to the front lip generator, G X is the function corresponding to the non-frontal lip generator, D Y is the function corresponding to the front lip discriminator, D X is the function corresponding to the non-frontal lip discriminator, L GAN (G X ,D X ) and L GAN (G Y ,D Y ) is to combat the loss, L cyc (G X ,G Y ) is the reconstruction loss, L S (G X ,G Y ) is the self-mapping test loss, λ and μ are the weights of the reconstruction loss and the self-mapping test loss respectively. ;

[0093] The adversarial loss of the self-mapping supervised recurrent generative adversarial network consists of:

[0094]

[0095] Among them, y~pdata(y) represents the real front lip picture in the video data to be processed, x~p data (x) represents the real non-frontal lip picture in the video data to be processed, and E(·) represents the expectation;

[0096] The reconstruction loss of the self-mapping supervised cycle generation adversarial network satisfies the following formula:

[0097]

[0098] The self-mapping test loss of the self-mapping supervised cycle generation adversarial network satisfies the following formula:

[0099]

[0100] It can be understood that the method for determining the consistency of the sound and lips disclosed in the present application aims at the influence of the change of lip angle on the consistency determination, and proposes a self-mapping supervised cyclic generative adversarial network to generate corresponding reconstructed frontal lip pictures according to the non-frontal lip pictures in the video data to be processed, thereby enriching the number of frontal lip pictures in the video data to be processed, and being more conducive to the accuracy of subsequent sound and lips consistency determination. The self-mapping supervised cyclic generative adversarial network solves the problem that if the traditional generative adversarial network is used to generate pictures without constraints, the images after conversion are prone to blurring and deformation.

[0101] In an optional embodiment of the present application, the step S2 of “extracting each key audio segment in the audio data to be processed” includes:

[0102] S21, extracting the Hilbert envelope and zero-frequency filtering signal of the audio data to be processed.

[0103] The Hilbert envelope of the audio data to be processed can be extracted by the following formula:

[0104]

[0105] in, represents the Hilbert envelope, z(n) represents the audio data to be processed, and represent the analytical signal and Hilbert transform of z(n) respectively.

[0106] The zero-frequency filtered signal of the audio data to be processed can be extracted by the following formula:

[0107]

[0108] Among them, 2K+1 is the average fundamental frequency period.

[0109] S22, convolving the Hilbert envelope and the second-order difference of the zero-frequency filter signal with the window function of the first-order Gaussian differentiator respectively, and then summing them, and normalizing the summation result to obtain a first curve.

[0110] S23: Confirm the maximum value between two consecutive zero-crossing points in the first curve as the starting point of the audio data to be processed.

[0111] S24, convolving the Hilbert envelope and the first-order difference of the zero-frequency filtering signal with the window function of FOGD respectively and then summing them to obtain a second curve.

[0112] S25. On the second curve, determine the end point of the audio data to be processed from right to left by using a peak search strategy.

[0113] S26. Determine each key audio segment according to the starting point and the ending point.

[0114] After obtaining each starting point and ending point, it is also necessary to match the starting and ending points of each key pronunciation in the speech segment. First, count the number of potential starting and ending points in the speech segment. If there is no ending point between two adjacent starting points, the potential starting point after the previous starting point is the corresponding ending point; if there are multiple potential ending points between two adjacent starting points, only the ending point with the largest peak value is retained. Finally, if there is no ending point after the last starting point in the speech segment, remove the starting point, otherwise retain the ending point corresponding to the maximum value and remove other potential ending points, and remove all potential ending points before the first starting point.

[0115] In an optional embodiment of the present application, in step S2, “extracting a key video segment matching each key audio segment from the video data to be processed” includes:

[0116] S27, based on the audio starting point of the key audio segment and audio end point Determine the duration of key audio segments Wherein m represents the index of the key audio segment in the audio data to be processed.

[0117] S28. Determine the starting point delay parameter - φ left and end point delay parameter +φ right .

[0118] S29, moving the audio start point forward by the start point delay parameter, and moving the audio end point backward by the end point delay parameter, to obtain the extraction range of the key video segment

[0119] S30: extracting key video segments from the video data to be processed according to the extraction range.

[0120] It is understandable that the audio and lip movements of the lip-congruent data will also have initial delays at corresponding time points, which are mainly distributed within [-50ms, +50ms], while the inconsistent data are mostly distributed outside the above range. The audio and video signals of lip-congruent belong to the same time axis, and the lip movements and audio changes at each key sound position must be highly correlated. At the same time, the audio and video delays at the above positions should also be within the reasonable distribution range of the initial delay.

[0121] In an optional embodiment of the present application, Figure 3 As shown, step S3 includes:

[0122] S31, inputting the audio features of each pair of key audio segments and the video features of each pair of key video segments into the first multi-layer neural network 301 and the second multi-layer neural network 302 respectively to obtain audio output results and video output results.

[0123] S32. According to the audio output result and the video output result, a depth correlation change curve is calculated according to the following formula:

[0124]

[0125] in, Represents the depth correlation change curve, h A The mapping vector representing the audio output result, h V Represents the mapping vector of the video output result, J is the first J mapping vectors selected, j represents the index of the mapping vector, F AO Represents the audio output result, F VO represents the video output result, corr() is the correlation function,

[0126] τ represents the delay, where the delay corresponding to the mth key audio segment and key video segment is calculated by the following formula:

[0127]

[0128] in, Represents the delay corresponding to the mth key audio segment and key video segment.

[0129] Among them, in the depth correlation change curve,

[0130] |corr(F AO h A,j ,F VO h V,j )| can be expressed as ||Z||;

[0131] in,

[0132]

[0133] and are covariance ∑ AA ,∑ VV and the cross-covariance∑ AV The scattering matrix.

[0134] It can be understood that the deep canonical correlation analysis proposed in the application overcomes the shortcomings of non-parametric models. By coupling training of multi-layer neural networks that describe two different modal data, a nonlinear representation mapping that maximizes the correlation between the two types of audio and video data is obtained, thereby more accurately performing correlation analysis on the lip data of key audio segments, and improving the robustness of the model to changes in sample quantity caused by screening through re-estimation of the covariance matrix.

[0135] Among them, the correlation change curve A peak will appear at the corresponding position of the initial delay, and the values ​​on both sides will gradually decrease; the curve of inconsistent data is relatively flat. Even if a pseudo peak appears, it is mainly outside the reasonable distribution range of the initial delay, and the value will be significantly lower than the consistent data.

[0136] In an optional embodiment of the present application, step S4 includes:

[0137] The joint consistency score of each pair of key audio segments and key video segments is calculated by the following formula:

[0138]

[0139] Among them, δ m represents the joint consistency score corresponding to the mth key audio segment and key video segment;

[0140] in,

[0141]

[0142] and represent the joint consistency scores outside and within the reasonable delay range respectively, and ζ represents the correlation threshold between inconsistent data and consistent data obtained by training the first multi-layer neural network and the second multi-layer neural network.

[0143] In an optional embodiment of the present application, step S5 includes:

[0144] The joint score of the lip data to be processed is calculated by the following formula:

[0145]

[0146] in, represents the joint score, M represents the logarithm of the key audio segment and the key video segment in the lip data to be processed, σ m represents the standard deviation of the inconsistent data distribution, Represents the local decision threshold.

[0147] In a second aspect, the present application provides an electronic device. The electronic device includes one or more processors; one or more input devices, one or more output devices and a memory. The processor, input device, output device and memory are connected via a bus. The memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the memory. The processor is configured to call the program instructions to perform the operation of any method of the first aspect:

[0148] It should be understood that in the embodiments of the present invention, the processor referred to may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0149] Input devices may include a touchpad, a fingerprint sensor (for collecting the user's fingerprint information and fingerprint direction information), a microphone, etc., and output devices may include a display (LCD, etc.), a speaker, etc.

[0150] The memory may include a read-only memory and a random access memory, and provide instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the device type.

[0151] In a specific implementation, the processor, input device, and output device described in the embodiments of the present invention may execute the implementation method described in any method of the first aspect, or may execute the implementation method of the terminal device described in the embodiments of the present invention, which will not be repeated here.

[0152] In a third aspect, the present invention provides a computer-readable storage medium storing a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the steps of any method of the first aspect are implemented.

[0153] The computer-readable storage medium may be an internal storage unit of the terminal device of any of the aforementioned embodiments, such as a hard disk or memory of the terminal device. The computer-readable storage medium may also be an external storage device of the terminal device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the terminal device. Furthermore, the computer-readable storage medium may also include both an internal storage unit of the terminal device and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by the terminal device. The computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.

[0154] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0155] In the several embodiments provided in the present application, it should be understood that the disclosed terminal device and method can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, or it can be an electrical, mechanical or other form of connection.

[0156] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of the present invention.

[0157] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0158] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0159] The expressions "first", "second", "the first" or "the second" used in various embodiments of the present disclosure may modify various components regardless of order and / or importance, but these expressions do not limit the corresponding components. The above expressions are only configured for the purpose of distinguishing an element from other elements. For example, a first user device and a second user device represent different user devices, although both are user devices. For example, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element without departing from the scope of the present disclosure.

[0160] When one element (e.g., a first element) is referred to as being "(operably or communicatively) coupled" or "(operably or communicatively) coupled to" or "connected to" another element (e.g., a second element), it is understood that the one element is directly connected to the other element or the one element is indirectly connected to the other element via yet another element (e.g., a third element). Conversely, it is understood that when an element (e.g., a first element) is referred to as being "directly connected" or "directly coupled" to another element (the second element), no element (e.g., a third element) is interposed between the two.

[0161] It should be noted that, in this article, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element. In addition, components, features, and elements with the same name in different embodiments of the present application may have the same meaning or different meanings, and their specific meanings need to be determined by their explanation in the specific embodiment or further combined with the context of the specific embodiment.

[0162] The above description is only an optional embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above invention concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other.

[0163] As used herein, the words "if" and "if" may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0164] The above description is only an optional embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above invention concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other.

[0165] The above description is only an optional embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for determining lip-tone consistency based on the fusion of multiple key-note joint scores, characterized in that: include: Acquire lip data to be processed, and split the lip data to be processed into audio data to be processed and video data to be processed; Extracting each key audio segment from the audio data to be processed, and extracting a key video segment matching each key audio segment from the video data to be processed; Performing deep canonical correlation analysis on each pair of the key audio segment and the key video segment; Calculating a joint consistency score for each pair of the key audio segment and the key video segment based on the delay difference and the analysis result of the deep canonical correlation analysis; Calculating a joint score of the lip data to be processed according to a joint score of consistency of each pair of the key audio segment and the key video segment; The extracting each key audio segment from the audio data to be processed includes: Extracting the Hilbert envelope and zero-frequency filtering signal of the audio data to be processed; Convolving the Hilbert envelope and the second-order difference of the zero-frequency filter signal with the window function of the first-order Gaussian differentiator respectively and then summing them, and normalizing the summation result to obtain a first curve; Confirming the maximum value between two consecutive zero-crossing points in the first curve as the starting point of the audio data to be processed; Convolving the Hilbert envelope and the first-order difference of the zero-frequency filter signal with the window function respectively and then summing them to obtain a second curve; On the second curve, determining the end point of the audio data to be processed from right to left by a peak search strategy; Determine each key audio segment according to the starting point and the ending point; Extracting a key video segment matching each key audio segment from the video data to be processed includes: According to the audio starting point of the key audio segment and audio end point Determine the time period of the key audio segment Wherein m represents the index of the key audio segment in the audio data to be processed; Determine the starting point delay parameter - φ left and end point delay parameter +φ right ; The audio start point is moved forward by the start point delay parameter, and the audio end point is moved backward by the end point delay parameter to obtain the extraction range of the key video segment Extracting the key video segment from the video data to be processed according to the extraction range; The performing deep canonical correlation analysis on each pair of the key audio segment and the key video segment comprises: Inputting each pair of the audio features of the key audio segment and the video features of the key video segment into the first multi-layer neural network and the second multi-layer neural network respectively to obtain an audio output result and a video output result; According to the audio output result and the video output result, a depth correlation change curve is calculated according to the following formula: in, represents the depth correlation variation curve, h A The mapping vector representing the audio output result, h V represents the mapping vector of the video output result, J is the first J mapping vectors selected, j represents the index of the mapping vector, F AO Represents the audio output result, F VO represents the video output result, corr() is the correlation function, τ represents the delay, wherein the delay corresponding to the mth key audio segment and the key video segment is calculated by the following formula: in, Represents the time delay corresponding to the mth key audio segment and the key video segment. The step of calculating the consistency joint score of each pair of the key audio segment and the key video segment based on the analysis result of the time delay difference and the deep canonical correlation analysis includes: The consistency joint score of each pair of the key audio segment and the key video segment is calculated by the following formula: Wherein, δm represents the consistency joint score corresponding to the mth key audio segment and the key video segment; in, and represent the consistency joint score outside and within the reasonable delay range respectively, and ζ represents the correlation threshold of inconsistent data and consistent data obtained by training the first multi-layer neural network and the second multi-layer neural network; The step of calculating the joint score of the lip data to be processed according to the joint score of consistency of each pair of the key audio segment and the key video segment comprises: The joint score of the lip data to be processed is calculated by the following formula: in, represents the joint score, M represents the logarithm of the key audio segment and the key video segment in the lip data to be processed, σm represents the standard deviation of the inconsistent data distribution, Represents the local decision threshold.

2. The method for determining lip-tone consistency based on multi-keynote joint score fusion according to claim 1, characterized in that: After extracting the key video segment matching each key audio segment from the video data to be processed, and before performing deep canonical correlation analysis on each pair of the key audio segment and the key video segment, the method further includes: A corresponding reconstructed frontal lip picture is generated according to the non-frontal lip picture in the key video segment through a generative adversarial network, and the non-frontal lip picture in the key video segment is replaced with the reconstructed frontal lip picture.

3. The method for determining lip-tone consistency based on multi-keynote joint score fusion according to claim 2, characterized in that: The generative adversarial network is a self-mapping supervised cyclic generative adversarial network, including a frontal lip generator, a non-frontal lip generator, a frontal lip discriminator and a non-frontal lip discriminator; The front lip generator is used to generate a corresponding reconstructed front lip picture according to the non-front lip picture in the key video segment; The frontal lip discriminator is used to judge the authenticity of the reconstructed frontal lip picture and the real frontal lip picture in the key video segment; The non-frontal lip generator is used to restore the reconstructed frontal lip picture to a reconstructed non-frontal lip picture corresponding to the non-frontal lip picture in the video data to be processed; the non-frontal lip discriminator is used to judge the authenticity of the reconstructed non-frontal lip picture and the real non-frontal lip picture in the key video segment.

4. The method for determining lip-tone consistency based on multi-keynote joint score fusion according to claim 3, characterized in that: The total loss of the self-mapping supervised recurrent generative adversarial network is: L(G X ,G Y ,D X ,D Y )=L GAN (G X ,D X )+L GAN (G Y ,D Y )+λL cyc (G X ,G Y )+μL S (G X ,G Y ); Among them, G Y is the function corresponding to the front lip generator, G X is the function corresponding to the non-frontal lip generator, D Y is the function corresponding to the frontal lip discriminator, D X is the function corresponding to the non-frontal lip discriminator, L GAN (G X ,D X ) and L GAN (G Y ,D Y ) is to combat the loss, L cyc (G X ,G Y ) is the reconstruction loss, L S (G X ,G Y ) is the self-mapping check loss, λ and μ are the weights of the reconstruction loss and the self-mapping check loss respectively. The adversarial loss of the self-mapping supervised recurrent generative adversarial network includes: Wherein, y~pdata(y) represents the real frontal lip picture in the video data to be processed, x~pdata(x) represents the real non-frontal lip picture in the video data to be processed, and E(·) represents the expectation; The reconstruction loss of the self-mapping supervised cyclic generative adversarial network satisfies the following formula: The self-mapping test loss of the self-mapping supervised cyclic generative adversarial network satisfies the following formula:

Images