A method for simulating head movement in a three-dimensional avatar articulatory process

Abstract

The application provides a three-dimensional image pronunciation process head action simulation method, and belongs to the technical field of three-dimensional virtual images. The three-dimensional image pronunciation process head action simulation method obtains a human face video and corresponding audio from a video library, aligns video frames and audio frames, extracts multiple frames of human face images, head posture parameters and mel spectra as training samples; pre-processes the human face images to generate face images after erasing the mouth; establishes a three-dimensional image head model and trains the three-dimensional image head model by using the training samples. The three-dimensional image head model comprises an audio feature extraction module, a lip shape synchronization module, a mouth generation module, a head posture module and a fusion module. The trained three-dimensional image head model is used to generate a three-dimensional image head model for specific audio. The method greatly reduces the calculation amount, simultaneously enables good linkage between the head posture and pronunciation, and avoids the stiff phenomenon of the three-dimensional image pronunciation process.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of three-dimensional virtual image, and particularly relates to a three-dimensional image pronunciation process head action simulation method. BACKGROUND

[0002] Many people have small head movements when speaking, which people do not pay attention to when speaking. When a camera is used to collect a human face speaking image, the mouth needs to be tracked due to the change of the human face head, thereby bringing a large amount of operation. On the other hand, the small change of the head posture of each person when speaking is not universal, and the small change of the head posture needs to be ignored at the collection time to collect the human face speaking image faster and improve the processing efficiency.

[0003] A Chinese invention with the authorization number CN111081270B (application number CN201911314031.3) discloses a virtual character mouth shape synchronization control method driven by real-time audio. The method comprises the following steps: identifying a viseme probability from a real-time speech stream; filtering the viseme probability; converting the sampling rate of the viseme probability into a sampling rate same as the rendering frame rate of a virtual character; and converting the viseme probability into a standard mouth shape configuration and performing mouth shape rendering. The method can avoid the requirement of synchronously transmitting phoneme sequence or mouth shape sequence information when transmitting an audio stream, can significantly reduce system complexity, coupling degree and implementation difficulty, and is suitable for various application scenarios of rendering a virtual character on a display device.

[0004] The above-mentioned invention and many current three-dimensional images only have simple mouth shape changes in the pronunciation process, and the head posture and pronunciation lack linkage, so that the pronunciation process of the three-dimensional image is stiff. SUMMARY

[0005] Therefore, the present application provides a three-dimensional image pronunciation process head action simulation method, which can solve the technical problem that the three-dimensional image only has simple mouth shape changes in the pronunciation process, and the head posture and pronunciation lack linkage, so that the pronunciation process of the three-dimensional image is stiff.

[0006] The present application is implemented as follows:

[0007] The present application provides a three-dimensional image pronunciation process head action simulation method, which comprises the following steps:

[0008] S10: obtaining a human face video and corresponding audio from a video signal library, aligning a video frame and an audio frame, and extracting multiple frames of human face images, head postures and mel spectra as training samples; preprocessing the human face images to generate face images after erasing the mouth;

[0009] S20: a three-dimensional image head model is established and trained using training samples, the three-dimensional image head model comprises an audio feature extraction module, a lip shape synchronization module, a mouth generation module, a head posture control module, and a fusion module, wherein:

[0010] The audio feature extraction module is used to extract features from the mel spectrum obtained in S10 to generate final audio features.

[0011] The lip shape synchronization module is used to generate multi-level lip shape image features according to the final audio features, and generate a lip shape image according to the last level of lip shape image features, and calculate the lip shape loss between the generated lip shape image and the lip shape image in the human face image sample, the lip shape loss comprises a mean square error loss and a contrast loss.

[0012] The mouth generation module is used to generate multi-level mouth image features according to the multi-level lip shape image features, and generate a mouth image according to the last level of mouth image features, and calculate the mouth loss between the generated mouth image and the mouth image in the human face image sample, the mouth loss uses a mean square error loss.

[0013] The head posture control module is used to generate head image features according to the center point.

[0014] The fusion module is used to fuse the head image features and the multi-level mouth image features into the face image after the mouth is erased in S10, calculate the fusion loss, the fusion loss uses the corresponding fusion loss of the PCONV network, and update the parameters of the three-dimensional image head model according to the weighted loss sum of the lip shape loss, the mouth loss and the fusion loss.

[0015] S30: using the trained three-dimensional image head model, generating a three-dimensional image head model for a specific audio.

[0016] The mouth erasing network adopts a Unet network to generate a mouth mask representing the position of the mouth, and erases the position of the mouth in the human face image according to the mouth mask.

[0017] The audio feature extraction module is composed of a audio down-sampling layer and an LSTM layer, first, the multi-frame mel spectrum is sequentially processed by several audio down-sampling layers to generate multi-level audio features, and then the last level of audio features of the multi-frame mel spectrum is fused by the LSTM layer to generate the final audio features.

[0018] The lip shape synchronization module is composed of b lip shape up-sampling layers in series, b≥3; the final audio features obtained by the audio feature extraction module are taken as input, and the multi-level lip shape image features are generated by the multiple lip shape up-sampling layers in turn, and the last level of lip shape image features is converted into a lip shape image.

[0019] The mouth generation module is composed of c mouth up-sampling layers in series, and c>=3;The first level lip shape image feature generated by the lip shape synchronization module is spliced with the head parameter to serve as the input of the first mouth up-sampling layer, the first level mouth image feature output by the first mouth up-sampling layer is spliced with the second level lip shape image feature to serve as the input of the second mouth up-sampling layer, the second level mouth image feature output by the second mouth up-sampling layer is spliced with the third level lip shape image feature to serve as the input of the third mouth up-sampling layer, and the third level mouth image feature output by the third mouth up-sampling layer serves as the input of the next mouth up-sampling layer, until the last level mouth image feature is generated and converted into a mouth image.

[0020] The fusion module adopts a Unet network, the face image after the mouth is erased is taken as the input of the encoder in the Unet network, and the output of each layer of the encoder is fused into the input of each layer of the decoder together with the multi-level mouth image features generated by the mouth generation module to generate a complete face image after fusion.

[0021] On the basis of the above technical scheme, the three-dimensional image pronunciation process head movement simulation method of the present application can be further improved as follows:

[0022] The video signal library is established by the following method:

[0023] Step one: a small plastic ball with a reflective outer wall is attached to the tip of the experimenter's nose, and a small black paper is attached to the key points of the experimenter's head posture;

[0024] Step two: a video camera is set up in front of the experimenter, signal transmitting ends and receiving ends are set up on both sides of the experimenter's face, the signal transmitting ends and receiving ends are in a straight line with the small plastic ball, and the distance between the signal transmitting ends and receiving ends is 1m;

[0025] Step three: a three-dimensional coordinate system is established with the center point of the video camera as the center, the signal transmitting end is turned on to send signals, the video camera is turned on, and the experimenter reads aloud;

[0026] Step four: after the experimenter finishes reading, the face audio and video recorded by the video camera and the signal data received by the corresponding receiving end are saved to the video signal library.

[0027] Further, the S10 specifically includes:

[0028] The video in the video signal library contains a complete face image and the audio of the person speaking in each frame;

[0029] According to the signal data received by the receiving end corresponding to the video, it is determined whether the head posture of the experimenter has changed.

[0030] If the head posture of the experimental personnel does not change, the face image set is extracted from all frames in the video, and the lip shape part in the face image is intercepted as a sample lip shape image;

[0031] If the head posture of the experimental personnel does not change, the plastic ball image is extracted from all frames in the video, the three-dimensional coordinates of the plastic ball are established in the three-dimensional coordinate system, and the corresponding lip shape in the phoneme mouth shape driving method is used as a sample lip shape image;

[0032] The mouth erasing network is constructed, part of the face images are randomly taken from the face image set and the mouth position is marked, the mouth erasing network is trained, the mouth position of the face image without marking the mouth position is recognized and erased by using the trained mouth erasing network, and the face image is reserved;

[0033] The time-domain audio is converted into the frequency-domain mel spectrum, and the frequency-domain sampling rate is consistent with the video frame sampling rate.

[0034] Further, the step of "judging whether the head posture of the experimental personnel changes according to the signal data received by the receiving end corresponding to the video" specifically comprises:

[0035] Step 1: data processing is performed on the signal received by the receiving end;

[0036] Step 2: the extended Kalman filtering method is used to realize the ball detection;

[0037] Step 3: the likelihood ratio value is calculated by using the obtained multi-path time delay joint estimation value, the obtained likelihood ratio value is compared with the detection threshold, and the detection result of whether the ball position changes is obtained;

[0038] Step 4: if the ball position changes, it is determined that the head posture of the experimental personnel changes; if the ball position does not change, it is determined that the head posture of the experimental personnel does not change.

[0039] Further, the step 1 specifically comprises:

[0040] First step: the transmitting end signal is expressed in the frequency domain, wherein the frequency domain form of the transmitting end signal is S = [S(0), S(1), K, S(K-1)], after underwater propagation, the frequency domain form of the signal received by the receiving end is a matrix X;

[0041] Second step: the binary hypothesis testing method is used to estimate the parameters of the frequency domain form of the specified number of received signals, specifically:

[0042] According to the test method of different hypotheses H0 and H1 in the binary hypothesis testing, the frequency domain form X k(k = 1, 2, 3, L) to perform parameter estimation;

[0043] Third step: using EM time delay estimation algorithm to calculate the direct penetration wave multipath time delay and the small ball scattering wave multipath time delay, specifically:

[0044] Using the EM time delay estimation algorithm, the direct penetration wave multipath time delay is and the small ball scattering wave multipath time delay is The number of sound lines of direct penetration wave and small ball scattering wave is M and N respectively, The time delay estimation value represented by each sound line is and

[0045] Further, the step 2 specifically includes:

[0046] First step: according to the method of extended Kalman filter, the state equation and observation equation of extended Kalman filter are established, specifically:

[0047] According to the method of extended Kalman filter, the state quantity x = [x, v x ,y,v y ] T and observation quantity The state equation and observation equation of extended Kalman filter are established:

[0048] x k =Fx k-1 +w k

[0049] z k =h(x k )+v k

[0050] Wherein: F is the state transition matrix, which is determined by the motion form of the small ball, h(·) is the observation function, w k represents the state noise matrix, which satisfies w k ~N(0,Q) and v k is the observation noise matrix, which satisfies v k ~N(0,R)

[0051] Second step: according to the known information at the specified time, the state prediction equation and the predicted covariance matrix at the next time are obtained by using the extended Kalman filter method, specifically:

[0052] According to the known information at k-1 time, the state prediction equation at k time is obtained by using the extended Kalman filter method And the predicted covariance matrix P k|k-1 :

[0053]

[0054] P k|k-1 = FP k-1|k-1 F T + Q k-1|k-1

[0055] Third step: calculate the function relationship between the small ball motion state and the multipath delay, specifically:

[0056] Since the function relationship h(x k ) between the delay parameter and the small ball motion parameter is nonlinear, according to the processing method of extended Kalman filter, linearization approximation is performed on it using the first-order Taylor formula, the expression of the function relationship h(x k ) needs to be obtained, the function relationship is represented using the virtual source mirror image method, and the function relationship between the small ball motion state x k and the multipath delay is obtained.

[0057]

[0058] The function relationship h(x k ) is linearized and approximated using the virtual source mirror image method, and for the small ball scattering wave, its propagation process is divided into two sections of the transmitting end-small ball and the small ball-receiving end, and the virtual source mirror image method is used to describe the two sections.

[0059] For the transmitting end-small ball (st) section, let the number of sound rays be N st , then the relationship between the sound ray travel and the small ball position is:

[0060]

[0061]

[0062]

[0063] L

[0064] For the small ball-receiving end (tr) section, let the number of sound rays be N tr , then the relationship between the sound ray travel and the small ball position is:

[0065]

[0066]

[0067]

[0068] L

[0069] (x sy s ,z s ), (x t ,y t ,z t ) and (x r ,y r ,z r ) represent the coordinates of the transmitting end, the small ball and the receiving end respectively. For the selection of the number of sound rays, the principle of N st ×N tr =N is followed to ensure the consistency of the dimensions of the matrix. In shallow water, the sound speed gradient does not change much, and when calculating the time delay, the sound speed can be set as a constant c, so the multipath time delay can be expressed as:

[0070]

[0071] Therefore, the relationship between the motion state of the small ball and the multipath time delay is:

[0072]

[0073] The Jacobian matrix of the observation function h(x k ) is calculated, which is the observation matrix H k .

[0074]

[0075] Step 4: Calculate the prediction of the observation value and the Kalman gain, specifically calculate the prediction of the observation value and the Kalman gain K k , the calculation formula is:

[0076]

[0077]

[0078] Step 5: Update the observation value to update the multipath time delay joint estimation value obtained by combining the small ball motion information, specifically:

[0079] After obtaining the observation z k at time k, the state update value x k|k and the error covariance update matrix P k|k are obtained after the update process.

[0080]

[0081] P k|k =P k|k-1 -K k H k P k|k-1

[0082] z k|k = h(x k|k )

[0083] wherein the observation update value represents a joint estimation value of multipath delay obtained by combining the ball movement information.

[0084] Further, the step 3 specifically comprises:

[0085] First step: taking the joint estimation value of multipath delay as the parameter estimation value in the generalized likelihood ratio test, wherein the joint estimation value of multipath delay is represented as

[0086] Second step: using the likelihood function to calculate the likelihood ratio value based on the obtained parameter estimation value, specifically: using the likelihood functions under the hypotheses H0 and H1: to calculate the likelihood ratio value L GLRT , wherein

[0087]

[0088] Third step: comparing the obtained likelihood ratio value with the detection threshold value to obtain the detection result of whether the ball has coordinate change exceeding the threshold value, specifically:

[0089] After simplifying equation (1), the test statistic T(X k ) is obtained, which is compared with the corresponding detection threshold value η * to determine whether the ball has coordinate change exceeding the threshold value.

[0090]

[0091] wherein the matrix is only related to the multipath delay of the direct penetrating wave and the ball scattering wave, and:

[0092]

[0093] is a projection matrix in space,

[0094] wherein the step S30 specifically comprises:

[0095] For the given audio mel spectrum, the multi-frame face image after the ball character erases the mouth and the corresponding head posture parameter are obtained according to the method of step 1, and the mel spectrum in the frequency domain and the multi-frame face image are aligned in time;

[0096] The trained face forgery generation model first extracts features from the mel spectrum of the given audio by the audio feature extraction module to generate final audio features, then generates multi-level lip image features according to the final audio features by the lip synchronization module, and then generates multi-level mouth image features according to the multi-level lip image features and head posture parameters by the mouth generation module, and finally fuses the multi-level mouth image features into the multi-frame face image after the ball character erases the mouth to generate a forged face image for the mouth action under a specific audio.

[0097] Further, the specific steps of the head posture module generating the head posture feature include:

[0098] The head posture of the experimental personnel is determined according to the coordinate variation of the plastic ball and the variation of the black paper sheet with the plastic ball as the center point.

[0099] Further, the head posture key points at least include points of the left corner of the eye, the right corner of the eye, the left corner of the mouth, the right corner of the mouth, the center of the top of the head, the top of the left ear, the lower tip of the left ear, the top of the right ear, and the lower tip of the right ear.

[0100] Compared with the prior art, the beneficial effects of the three-dimensional image pronunciation process head action simulation method provided by the application are: the head action is described by using the head posture key points, the ball at the tip of the nose is used as the signal transmitting end and receiving end ball, when the ball coordinates change, the signal received by the receiving end will change, so as to judge the slight change of the head of the experimental personnel, and the detection threshold is used to set the threshold of the ball coordinate variation, when the ball coordinate variation exceeds the threshold, it is judged that the head posture of the face changes, at this time, the lip shape in the phoneme mouth shape driving method is used to replace the lip shape collected in the video image, which greatly reduces the calculation amount, and makes the head posture and pronunciation have good linkage, and avoids the phenomenon of rigidity in the three-dimensional image pronunciation process. BRIEF DESCRIPTION OF DRAWINGS

[0101] In order to more clearly illustrate the technical solutions of the embodiments of the application, the following will briefly introduce the drawings needed to be used in the description of the embodiments of the application. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0102] Figure 1 The flow chart of the three-dimensional image pronunciation process head action simulation method provided by the application. DETAILED DESCRIPTION

[0103] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall into the scope of protection of the present application.

[0104] Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall into the scope of protection of the present application.

[0105] It should be noted that: similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0106] In the description of the present application, it should be understood that the terms "center", "longitudinal", "transverse", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the indicated device or element must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application.

[0107] In addition, the terms "first", "second" are only for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include one or more of the features. In the description of the present application, the meaning of "multiple" is two or more, unless otherwise specifically limited.

[0108] As Figure 1 shown is a flowchart of a three-dimensional image pronunciation process head movement simulation method provided by the present application, the method comprises the following steps:

[0109] S10: obtaining a face video and corresponding audio from a video signal library, aligning the video frame and the audio frame, extracting multiple frames of face images, head poses and mel spectra as training samples; pre-processing the face images to generate face images with the mouth erased;

[0110] S20: a three-dimensional image head model is established and the three-dimensional image head model is trained using training samples, the three-dimensional image head model comprises an audio feature extraction module, a lip shape synchronization module, a mouth generation module, a head posture control module, and a fusion module, wherein:

[0111] The audio feature extraction module is configured to perform feature extraction on the mel spectrum obtained in S10 to generate final audio features.

[0112] The lip shape synchronization module is configured to generate multi-stage lip shape image features according to the final audio features, and generate a lip shape image according to the last stage of lip shape image features, and calculate a lip shape loss between the generated lip shape image and the lip shape image in the face image sample, wherein the lip shape loss comprises a mean square error loss and a contrast loss.

[0113] The mouth generation module is configured to generate multi-stage mouth image features according to the multi-stage lip shape image features, and generate a mouth image according to the last stage of mouth image features, and calculate a mouth loss between the generated mouth image and the mouth image in the face image sample, wherein the mouth loss uses a mean square error loss.

[0114] The head posture control module is configured to generate a head image feature according to the center point.

[0115] The fusion module is configured to fuse the head image feature and the multi-stage mouth image feature into the face image with the mouth erased in S10, calculate a fusion loss, wherein the fusion loss uses a fusion loss corresponding to the PCONV network, and update the parameters of the three-dimensional image head model according to the weighted loss sum of the lip shape loss, the mouth loss, and the fusion loss.

[0116] S30: using the trained three-dimensional image head model, a three-dimensional image head model for a specific audio is generated.

[0117] The mouth erasing network adopts a Unet network to generate a mouth mask representing the position of the mouth, and erases the position of the mouth in the face image according to the mouth mask.

[0118] The audio feature extraction module is composed of a audio downsampling layer and an LSTM layer, first, the multi-frame mel spectrum is sequentially processed by several audio downsampling layers to generate multi-stage audio features, and then the last stage of audio features of the multi-frame mel spectrum is fused by the LSTM layer to generate the final audio features.

[0119] The lip shape synchronization module is composed of b lip shape upsampling layers connected in series, b≥3; the final audio features obtained by the audio feature extraction module are taken as input, and the multi-stage lip shape image features are generated by the multiple lip shape upsampling layers in turn, and the last stage of lip shape image features is converted into a lip shape image.

[0120] The mouth generation module is composed of c mouth up-sampling layers in series, c>=3; the first-level lip shape image feature generated by the lip shape synchronization module and the head parameter are spliced and taken as the input of the first mouth up-sampling layer, the first-level mouth image feature output by the first mouth up-sampling layer and the second-level lip shape image feature are spliced and taken as the input of the second mouth up-sampling layer, the second-level mouth image feature output by the second mouth up-sampling layer and the third-level lip shape image feature are spliced and taken as the input of the third mouth up-sampling layer, the third-level mouth image feature output by the third mouth up-sampling layer is taken as the input of the next mouth up-sampling layer, until the last-level mouth image feature is generated and converted into a mouth image.

[0121] The fusion module adopts a Unet network, takes the face image after erasing the mouth as the input of the encoder in the Unet network, fuses the output of each layer of the encoder and the multi-level mouth image features generated by the mouth generation module into the input of each layer of the decoder, and generates a complete face image after fusion.

[0122] In the technical scheme, the method for establishing the video signal library is as follows:

[0123] Step one: a small plastic ball with a reflective outer wall is attached to the tip of the nose of the experimental personnel, and a small black paper sheet is attached to the key points of the head posture of the experimental personnel;

[0124] Step two: a camera is arranged opposite the experimental personnel, and a signal transmitting end and a receiving end are arranged on both sides of the face of the experimental personnel, wherein the signal transmitting end and the receiving end are in a straight line with the small plastic ball, and the distance between the signal transmitting end and the receiving end is 1m;

[0125] Step three: a three-dimensional coordinate system is established with the center point of the camera as the center, the signal transmitting end is turned on to emit signals, the camera is turned on, and the experimental personnel reads aloud;

[0126] Step four: after the experimental personnel finish reading, the face audio and video recorded by the camera and the signal data received by the corresponding receiving end are saved to the video signal library.

[0127] Further, in the technical scheme, S10 specifically includes:

[0128] The video in the video signal library is obtained, each frame in the video contains a complete face image and contains audio of a person speaking;

[0129] It is determined whether the head posture of the experimental personnel is changed according to the signal data received by the receiving end corresponding to the video;

[0130] If the head posture of the experimental personnel is not changed, a face image set is extracted from all frames in the video, and a lip shape part in the face image is intercepted as a sample lip shape image;

[0131] If the head posture of the experimenter does not change, the image of the plastic ball is extracted from all frames of the video, the three-dimensional coordinates of the plastic ball are established in the three-dimensional coordinate system, and the corresponding lip shape in the phoneme lip driving method is used as a sample lip shape image;

[0132] The mouth erasing network is constructed, part of the face images are randomly taken from the face image set and the mouth position is marked, the mouth erasing network is trained, the mouth position of the face image without marking the mouth position is recognized and erased by using the trained mouth erasing network, and the face image is reserved;

[0133] The time-domain audio is converted into the frequency-domain mel spectrum, and the frequency-domain sampling rate is consistent with the video frame sampling rate.

[0134] The phoneme lip driving method first converts the speech or text into a phoneme sequence, and each phoneme corresponds to a specific viseme (corresponding to a specific lip shape). In order to make the mouth shape consistent with the real scene, time smoothing needs to be performed on the viseme sequence according to the designed rules. The algorithm includes two stages:

[0135] The first stage is independent of a specific speaker and includes three parallel networks for generating three groups of action parameters of lip shape, eyebrow and eye expression, and head movement;

[0136] The second stage synthesizes the video of a specific speaker, and an adaptive attention network based on three-dimensional face information supervision is used to generate different specific speaker videos.

[0137] Further, in the above technical solution, the step of "judging whether the head posture of the experimenter changes according to the signal data received by the video corresponding receiving end" specifically includes:

[0138] Step 1: data processing is performed on the signal received by the receiving end;

[0139] Step 2: the extended Kalman filtering method is used to realize ball detection;

[0140] Step 3: the likelihood ratio value is calculated by using the obtained multipath time delay joint estimation value, the obtained likelihood ratio value is compared with the detection threshold, and the detection result of whether the ball position changes is obtained;

[0141] Step 4: if the ball position changes, it is determined that the head posture of the experimenter changes; if the ball position does not change, it is determined that the head posture of the experimenter does not change.

[0142] Further, in the above technical solution, step 1 specifically includes:

[0143] Step 1: Represent the transmitted signal in the frequency domain form, where the frequency domain form of the transmitted signal is S = [S(0), S(1), K, S(K-1)], and the frequency domain form of the signal received by the receiver after underwater propagation is matrix X;

[0144] Step 2: The frequency domain form of the received signal is estimated using a binary hypothesis testing method for a specified number of receptions. Specifically:

[0145] Based on the testing methods for different hypotheses H0 and H1 in binary hypothesis testing, the frequency domain form X of the k-th received signal is analyzed. k Parameter estimation is performed for (k = 1, 2, 3, L);

[0146] Step 3: Calculate the multipath delay of the direct-transmission wave and the multipath delay of the spherical scattered wave using the EM time delay estimation algorithm. Specifically:

[0147] The multipath delay of the direct-transmission wave is obtained using the EM time delay estimation algorithm. The multipath delay of the scattering wave from the small ball is The number of acoustic lines for the direct-transmitting wave and the scattering wave from the small sphere are set to M and N, respectively. This represents the estimated time delay for each sound ray, which can be abbreviated as: and

[0148] Furthermore, in the above technical solution, step 2 specifically includes:

[0149] Step 1: Based on the Extended Kalman Filter (EKF) method, establish the state equation and observation equation for the EKF, specifically as follows:

[0150] Based on the extended Kalman filter method, the state variables of the ball's motion are set as x = [x, v]. x ,y,v y ] T and observation Establish the state equation and observation equation for the extended Kalman filter:

[0151] x k =Fx k-1 +w k

[0152] z k =h(x k )+v k

[0153] Where: F is the state transition matrix, determined by the motion of the ball, h(·) is the observation function, and w k The state noise matrix is ​​represented by w. k ~N(0,Q) and v kFor the observation noise matrix, v k ~ N(0, R)

[0154] Second step: according to the known information at the specified time, the state prediction equation and the predicted covariance matrix at the next time are obtained by using the extended Kalman filtering method, specifically:

[0155] According to the known information at k-1 time, the state prediction equation at k time is obtained by using the extended Kalman filtering method and the predicted covariance matrix P k|k-1 :

[0156]

[0157] P k|k-1 = FP k-1|k-1 F T + Q k-1|k-1

[0158] Third step: calculate the functional relationship between the motion state of the small ball and the multipath time delay, specifically:

[0159] Since the functional relationship h(x k ) between the time delay parameter and the small ball motion parameter is nonlinear, according to the processing method of the extended Kalman filter, a first-order Taylor formula is used for linearization approximation, and the expression of the functional relationship h(x k ) needs to be obtained. The virtual source mirror method is used to represent the functional relationship, and the functional relationship between the motion state x k of the small ball and the multipath time delay is obtained.

[0160]

[0161] The virtual source mirror method is used for linearization approximation of the functional relationship h(x k ), and the propagation process of the small ball scattering wave is divided into two sections: the transmitting end-small ball and the small ball-receiving end. The two sections are described using the virtual source mirror method.

[0162] For the st section of the transmitting end-small ball, let the number of sound rays be N st , then the relationship between the sound ray travel and the small ball position is:

[0163]

[0164]

[0165]

[0166] L

[0167] For the small ball-receiving end (tr) segment, let the number of sound rays also be N. tr The relationship between the travel distance of the sound ray and the position of the ball is as follows:

[0168]

[0169]

[0170]

[0171] L

[0172] (x s ,y s ,z s ), (x t ,y t ,z t ) and (x r ,y r ,z r ( ) represent the coordinates of the transmitter, the sphere, and the receiver, respectively. The selection of the number of these two sound ray segments follows N... st ×N tr The principle of =N ensures that the dimensions of the matrix are consistent. In shallow seas, the sound speed gradient does not change significantly. When calculating the time delay, the sound speed can be set to a constant value c. Thus, the multipath time delay can be expressed as:

[0173]

[0174] Therefore, the relationship between the ball's motion state and the multipath delay is as follows:

[0175]

[0176] Find the observation function h(x) k The Jacobian matrix of H is the observation matrix H. k .

[0177]

[0178] Step 4: Calculate the prediction and Kalman gain of the observations, specifically by calculating the prediction of the observations. and Kalman gain K k The calculation formula is:

[0179]

[0180]

[0181] Step 5: Update the observations, using the updated observations to represent the joint estimate of multipath time delay obtained by combining the ball's motion information. Specifically:

[0182] At the time k, the observation z k After the update process, the state update value x k|k and the error covariance update matrix P k|k .

[0183]

[0184] P k|k = P k|k-1 -K k H k P k|k-1

[0185] z k|k = h(x k|k )

[0186] wherein the observation update value represents a joint estimation of multipath delay obtained in combination with the information of the ball movement.

[0187] Further, in the above technical solution, step 3 specifically comprises:

[0188] First step: taking the joint estimation of multipath delay as the parameter estimation value in the generalized likelihood ratio test, wherein the joint estimation of multipath delay is expressed as

[0189] Second step: using the likelihood function to calculate the likelihood ratio value based on the obtained parameter estimation value, specifically: using the likelihood functions under the hypotheses H0 and H1: to calculate the likelihood ratio value L GLRT , wherein

[0190]

[0191] Third step: comparing the obtained likelihood ratio value with a detection threshold to obtain the detection result of whether the ball has coordinate change exceeding the threshold, specifically:

[0192] After simplifying equation (1), the test statistic T(X k ) is obtained, which is compared with the corresponding detection threshold η * to determine whether the ball has coordinate change exceeding the threshold, wherein the detection threshold is 0.2-0.5 cm.

[0193]

[0194] wherein the matrices are only related to the multipath delay of the straight-through wave and the ball scattering wave, respectively, and

[0195]

[0196] is a projection matrix in space,

[0197] In the above technical solution, step S30 specifically comprises:

[0198] For the Mel spectrum of a given audio, according to the method of step 1, a plurality of frames of face images after erasing the mouth of the small ball character and corresponding head posture parameters are obtained, and the Mel spectrum in the frequency domain and the plurality of frames of face images are aligned in time;

[0199] Using the trained face forgery generation model, first, the audio feature extraction module extracts features from the Mel spectrum of the given audio to generate final audio features, then the lip synchronization module generates multi-level lip image features according to the final audio features, and then the mouth generation module is used to generate multi-level mouth image features according to the multi-level lip image features and the head posture parameters, and finally the multi-level mouth image features are fused into the plurality of frames of face images after erasing the mouth of the small ball character to generate a forged face image for the mouth action under a specific audio.

[0200] Further, in the above technical solution, the specific steps of the head posture module generating the head posture feature include:

[0201] Taking a plastic ball as the center point, according to the coordinate variation of the plastic ball and the variation of the black paper sheet, the head posture variation of the experimental personnel is determined.

[0202] Further, in the above technical solution, the head posture key points at least include points of the left corner of the eye, the right corner of the eye, the left corner of the mouth, the right corner of the mouth, the center of the top of the head, the top of the left ear, the lower tip of the left ear, the top of the right ear, and the lower tip of the right ear.

[0203] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method of simulating head movement for a three-dimensional character articulatory process, characterized by, The method comprises the following steps: S10: obtaining a face video and corresponding audio from a video signal library, aligning video frames and audio frames, and extracting multiple frames of face images, head poses, and mel spectra as training samples; preprocessing the face images to generate face images with the mouth erased; S20: establishing a three-dimensional image head model and training the three-dimensional image head model using the training samples, wherein the three-dimensional image head model comprises an audio feature extraction module, a lip shape synchronization module, a mouth generation module, a head pose control module, and a fusion module, and wherein: the audio feature extraction module is configured to perform feature extraction on the mel spectrum obtained in S10 to generate final audio features; the lip shape synchronization module is configured to generate multiple levels of lip shape image features based on the final audio features, generate a lip shape image based on the last level of lip shape image features, and calculate a lip shape loss between the generated lip shape image and a lip shape image in the face image sample, wherein the lip shape loss comprises a mean square error loss and a contrast loss; the mouth generation module is configured to generate multiple levels of mouth image features based on the multiple levels of lip shape image features, generate a mouth image based on the last level of mouth image features, and calculate a mouth loss between the generated mouth image and a mouth image in the face image sample, wherein the mouth loss uses a mean square error loss; the head pose control module is configured to generate a head image feature based on a center point; the fusion module is configured to fuse the head image feature and the multiple levels of mouth image features into the face image with the mouth erased in S10, calculate a fusion loss using a fusion loss corresponding to a PCONV network, and update parameters of the three-dimensional image head model based on a weighted sum of the lip shape loss, the mouth loss, and the fusion loss; S30: generating a three-dimensional image head model for a specific audio using the trained three-dimensional image head model; The method for establishing the video signal library comprises the following steps: Step one: affix a small plastic ball with a reflective outer wall to the tip of the nose of an experimental subject, and affix black paper pieces to key points of the head pose of the experimental subject; Step two: set up a camera opposite the experimental subject, and set up a signal transmitting end and a receiving end on both sides of the face of the experimental subject, wherein the signal transmitting end and the receiving end are in a straight line with the small plastic ball, and the distance between the signal transmitting end and the receiving end is 1 m; Step three: establish a three-dimensional coordinate system with the center point of the camera as the center, and turn on the signal transmitting end to emit a signal and turn on the camera, and the experimental subject reads aloud; Step four: after the experimental subject finishes reading, save the face audio and video recorded by the camera and the signal data received by the corresponding receiving end to the video signal library; The S10 specifically comprises: obtaining a video in the video signal library, wherein each frame of the video contains a complete face image and audio of the person speaking; determining whether the head pose of the experimental subject has changed based on the signal data received by the corresponding receiving end of the video; if the head pose of the experimental subject has not changed, extracting a set of face images from all frames of the video, and extracting a lip shape part in the face image as a sample lip shape image; If the head posture of the experimenter does not change, the image of the plastic ball is extracted from all frames in the video, the three-dimensional coordinates of the plastic ball are established in the three-dimensional coordinate system, and the corresponding lip shape in the phoneme lip driving method is used as a sample lip shape image; The mouth erasing network is constructed, part of the face images are randomly taken from the face image set and the mouth position is marked, the mouth erasing network is trained, the mouth position of the face image without marking the mouth position is recognized and erased by using the trained mouth erasing network, and the face image is reserved; The time-domain audio is converted into the frequency-domain mel spectrum, and the frequency-domain sampling rate is consistent with the video frame sampling rate; The step of judging whether the head posture of the experimenter changes according to the signal data received by the receiving end corresponding to the video specifically includes: Step 1: data processing is performed on the signal received by the receiving end; Step 2: the extended Kalman filtering method is used to realize ball detection; Step 3: the likelihood ratio value is calculated by using the obtained multi-path time delay joint estimation value, the obtained likelihood ratio value is compared with the detection threshold, and the detection result of whether the ball position changes is obtained; Step 4: if the ball position changes, it is determined that the head posture of the experimenter changes; if the ball position does not change, it is determined that the head posture of the experimenter does not change.

2. The method of claim 1, wherein, The step 1 specifically includes: The first step is to express the transmitting signal of the transmitting end in the frequency domain, wherein the frequency domain form of the transmitting signal of the transmitting end is After propagation, the frequency domain form of the signal received by the receiving end is a matrix X. Second step: parameter estimation is performed on the frequency-domain form of the specified number of received signals by using the binary hypothesis testing method, specifically: Based on different hypotheses in the binary hypothesis test and The test method for the first Frequency domain form of the received signal Perform parameter estimation; Third step: the straight-through wave multi-path time delay and the ball scattering wave multi-path time delay are calculated by using the EM time delay estimation algorithm, specifically: The direct penetration wave multipath time delay is obtained by using the EM time delay estimation algorithm and the small ball scattering wave multipath time delay is The number of sound ray lines of the direct penetration wave and the small ball scattering wave is respectively set to and , The time delay estimation value represented by each sound ray line is indicated as and .

3. The method of claim 2, wherein the head movement is simulated by a plurality of actuators. The step 3 specifically includes: Step 1: use the joint multipath delay estimation value as the parameter estimation value in the generalized likelihood ratio test, wherein the joint multipath delay estimation value is expressed as ; Second step: using the likelihood function with the estimated parameter values, calculate the likelihood ratio value, specifically: using the likelihood function under the assumption and : , , calculate the likelihood ratio value , where: ; Third step: the obtained likelihood ratio value is compared with the detection threshold, and the detection result of whether the ball has a coordinate change exceeding the threshold is obtained, specifically: After simplifying formula (1), a test statistic is obtained is compared with a corresponding detection threshold to determine whether the coordinates of the small ball have changed beyond the threshold. ; where the matrix , is only related to the multipath delay of the direct and small sphere scattered waves, respectively, and: ； is a projection matrix in space, .

4. The method of claim 1, wherein, The step S30 specifically includes: For the mel spectrum of a given audio, the multi-frame face image after the ball character erases the mouth and the corresponding head posture parameter are obtained according to the method of step 1, the mel spectrum in the frequency domain and the multi-frame face image are aligned in time; The trained face forgery generation model is used, first, the audio feature extraction module extracts features from the mel spectrum of the given audio to generate final audio features, then the lip synchronization module generates multi-level lip image features according to the final audio features, then the mouth generation module is used to generate multi-level mouth image features according to the multi-level lip image features and the head posture parameter, and finally the multi-level mouth image features are fused into the multi-frame face image after the ball character erases the mouth to generate a forged face image for the mouth action under a specific audio.

5. The method of claim 4, wherein the head movement of the three-dimensional character is simulated by a plurality of head movement simulators, each of which is associated with a corresponding one of the plurality of head movement parameters. The specific steps of the head posture control module generating the head posture feature include: The head posture of the experimenter is determined according to the coordinate change of the plastic ball and the change of the black paper.

6. The method of claim 5, wherein the head movement of the three-dimensional character is simulated by a plurality of head movement simulators. The head posture key points at least include the points of the left corner of the face, the right corner of the face, the left corner of the mouth, the right corner of the mouth, the center of the head, the left ear top, the left ear lower tip, the right ear top and the right ear lower tip.

Images