A real-time video generation method, device, equipment and storage medium based on AIGC

Abstract

The present invention provides a real-time video generation method, device, equipment and storage medium based on AIGC, including: determining an initial image frame from a target image frame corresponding to a source video image frame, implicitly calibrating the initial image frame, predicting the predicted implicit key point coordinates of facial organ key points in the target image frame, and determining the scaling factor corresponding to each face sub-image; determining a migration estimation matrix corresponding to each face sub-image, and determining new implicit key point coordinates to generate a target face expression; determining the migration estimation matrix corresponding to each face sub-image through the scaling factor, the real implicit key point coordinates of the facial organ key points in each face sub-image and the predicted implicit key point coordinates, so that the scaling factor can be corrected and transformed for deformation problems caused by different angles, thereby stabilizing the problem of low quality of migration expression when the angle offset is too large during the processing of the target video image frame.

Description

Technical Field

[0001] The present invention relates to the technical field of facial expression migration, and in particular to a real-time video generation method, device, equipment and storage medium based on AIGC. Background Art

[0002] In an era of rapid technological innovation, artificial intelligence has become a topic that people cannot avoid. Especially in the field of human face, research has reached its peak, and facial expressions are one of the important topics. Facial expression transfer technology has gradually become the focus of the expression field. This technology mainly transfers the expression of a face in one picture to the face of another picture. In this process, it is necessary to ensure that the facial identity information remains unchanged, and the facial features are processed in detail to achieve the purpose of expression replacement.

[0003] In the existing technology, the source expression face and the target expression face are generally reconstructed in three dimensions, and then the face mesh parameters are constructed. The generated texture map and the face mesh are processed through an encoder, decoder and renderer to finally generate a face expression picture. The second is 2D face migration technology, which is to train a face expression generator, encode and decode the source face image and the target face image through a deep neural network, and generate the final face expression image. However, the facial expression images generated by the above two methods have a very serious loss of similarity. Not only are the facial expressions stiff, but the details of the facial features are also poorly processed, and the effect is not very good. Summary of the invention

[0004] In view of this, the present invention provides a real-time video generation method, device, equipment and storage medium based on AIGC, which is used to solve the problem that the similarity of facial expressions generated by traditional face migration technology is seriously lost, not only the facial expression is stiff, but also the details of the facial features are poorly processed, and a good effect cannot be achieved.

[0005] The technical solution of the embodiment of the present invention is achieved as follows:

[0006] In a first aspect, the present invention provides a real-time video generation method based on AIGC, comprising:

[0007] Performing face detection and tracking on the acquired source video image frame and target video image frame respectively, and acquiring a target image frame having a face from the source video image frame and the target video image frame based on the detection and tracking results;

[0008] The face in the target image frame corresponding to the source video image frame is divided into a preset number to obtain a number of face sub-images corresponding to the preset number, and the real implicit key point coordinates of the key points of the facial organs in each of the face sub-images are extracted;

[0009] Determine an initial image frame from a target image frame corresponding to the source video image frame, perform implicit calibration on the initial image frame, predict the predicted implicit key point coordinates of the facial organ key points in the target image frame based on the implicit calibration of the initial image frame, and determine the relative angle of the facial organ key points in each of the face sub-images relative to the initial image frame;

[0010] Determining a scaling factor corresponding to each of the face sub-images according to a relative angle of the key points of the facial organs of each of the face sub-images relative to the initial image frame;

[0011] Determine a migration estimation matrix corresponding to each of the face sub-images according to the scaling factor, the real implicit key point coordinates of the facial organ key points in each of the face sub-images, and the predicted implicit key point coordinates;

[0012] According to the migration estimation matrix, feature encoding is performed on the target image frame having a human face in the source video image frame and the target video image frame, new implicit key point coordinates are determined, and a target facial expression is generated according to the new implicit key point coordinates;

[0013] A facial expression replacement area is determined from each target image frame in which a human face exists in the target video image frame according to the target facial expression, and pixels of the facial expression replacement area are replaced by the facial expression to obtain a replaced target video image frame.

[0014] Further preferably, the step of determining the migration estimation matrix corresponding to each of the face sub-images according to the scaling factor, the real implicit key point coordinates of the face organ key points in each of the face sub-images, and the predicted implicit key point coordinates comprises:

[0015] Determine a rotation matrix of the face in each of the face sub-images relative to the initial image frame according to the relative angle of the facial organ key points in each of the face sub-images relative to the initial image frame, and determine a translation matrix for fine-tuning the facial organ key points in each of the face sub-images;

[0016] Using the rotation matrix and the translation matrix as conditional controls for locating the facial organs in each of the facial sub-images, and determining the posture deformation parameters corresponding to each of the facial sub-images according to the predicted implicit key point coordinates of the facial organ key points in each of the facial sub-images and the corresponding scaling factors;

[0017] According to the real implicit key point coordinates of the facial organ key points in each of the facial sub-images and the corresponding posture deformation parameters, a migration estimation matrix corresponding to each of the facial sub-images is determined.

[0018] Further preferably, the migration estimation matrix includes: a head posture estimation matrix and an expression estimation matrix; the migration estimation matrix corresponding to each face sub-image is determined according to the real implicit key point coordinates of the face organ key points in each face sub-image and the corresponding posture deformation parameters, including:

[0019] According to the real implicit key point coordinates of the facial organ key points in each of the face sub-images and the corresponding posture deformation parameters, constructing a unit matrix in a homogeneous coordinate system as an initial matrix;

[0020] Copy the first three rows and first three columns of the rotation matrix to the corresponding positions of the initial matrix and copy the first three elements of the translation matrix to the first three columns of the last row of the initial matrix;

[0021] Integrate the scaling factor and the representation transformation matrix into the rotation matrix, and then copy the integrated result to the corresponding position of the initial matrix;

[0022] The last element of the last row of the initial matrix is ​​set to 1 to maintain the correctness of the homogeneous coordinate system, thereby obtaining the head posture estimation matrix and the expression estimation matrix.

[0023] Further preferably, the expression situation-dependent matrix is ​​determined by the following method:

[0024] According to the position changes of the key points of the facial organs in the target image frame corresponding to the source video image frame, estimating the deformation of the face under different expressions;

[0025] According to the difference in face size between the source video image frame and the target video image frame and the deformation, taking the source video image frame as the position of the key points of the facial organs under the standard expression, for each of the target image frames in the target video image frame, a deformation vector between the position of the key points of the facial organs and the position of the key points of the facial organs in the standard expression is calculated;

[0026] Determining the dimension of the representation state-deformation matrix according to the dimension of the target video image frame and the number of preset facial organ key points;

[0027] According to the dimension of the representation deformation matrix, the calculated deformation vectors are arranged in the order of key points to obtain the representation deformation matrix.

[0028] Further preferably: the head posture estimation matrix and the expression estimation matrix are determined by the following formula:

[0029] X=W×(Xc+E)×R+(Ts+Td)

[0030] Among them, X is the predicted implicit key point coordinates, W is the scaling factor, Xc is the true implicit key point coordinates, E is the expression estimation matrix, R is the head pose estimation matrix, Ts and Td are the rotation matrix and translation matrix respectively.

[0031] Further preferably, the step of encoding the target image frame having a human face in the source video image frame and the target video image frame according to the migration estimation matrix, determining new implicit key point coordinates, and generating a target facial expression according to the new implicit key point coordinates comprises:

[0032] Compare the real implicit key point coordinates of the target image frame in which a human face exists in the source video image frame and the target video image frame to generate a first transformation relationship matrix, wherein the first transformation relationship matrix is ​​used to describe the transformation relationship of facial organs from the source video image frame to the target video image frame;

[0033] Applying the migration estimation matrix to the face region in the source video image frame to transform it into the coordinate system of the target video image frame;

[0034] After transforming the face region of the source video image frame to the coordinate system of the target video image frame, feature encoding is performed on the source video image frame and the target video image frame based on the texture, shape, and expression of the face region of the source video image frame in the coordinate system of the target video image frame to generate a feature vector;

[0035] Using the feature-encoded information in combination with a facial organ key point detection algorithm, new implicit key point coordinates are determined in the target video image frame, wherein the new key point coordinates are used to represent the precise position of the transformed face in the target video image frame;

[0036] Aligning the new implicit key point coordinates with the real implicit key point coordinates of the facial organ key points in the target video image frame, analyzing the stretching, compression, and rotation of facial muscles, and generating a second transformation relationship matrix, wherein the second transformation relationship matrix is ​​used to describe the transformation relationship from the facial expression in the source video image frame to the expected facial expression in the target video image frame;

[0037] Based on matrix multiplication, mapping the second transformation relationship matrix to the face area in the source video image frame to generate a new face image with a target expression;

[0038] The generated new face image is rendered to match the background and lighting conditions of the target video image frame, thereby generating the target face expression.

[0039] Further preferably, the dividing the face in the target image frame corresponding to the source video image frame into a preset number to obtain face sub-images corresponding to the preset number includes:

[0040] The face in the target image frame corresponding to the source video image frame is divided into a preset number, and a plurality of facial organ key points in each divided image are obtained;

[0041] Determine the face tilt angle of each image according to the connecting lines between the multiple facial organ key points in each image to obtain a corrected tilt angle;

[0042] According to the correction tilt angle, the corresponding images are respectively subjected to face correction to obtain face sub-images of a number corresponding to the preset number.

[0043] A second aspect: A real-time video generation device based on AIGC is provided, comprising:

[0044] An acquisition module is configured to perform face detection and tracking on the acquired source video image frame and target video image frame respectively, and based on the detection and tracking results, acquire a target image frame with a face from the source video image frame and the target video image frame;

[0045] An extraction module is configured to divide the face in the target image frame corresponding to the source video image frame into a preset number, obtain a number of face sub-images corresponding to the preset number, and extract the real implicit key point coordinates of the key points of the facial organs in each of the face sub-images;

[0046] A calibration module is configured to determine an initial image frame from a target image frame corresponding to the source video image frame, perform implicit calibration on the initial image frame, predict the predicted implicit key point coordinates of the facial organ key points in the target image frame based on the implicit calibration of the initial image frame, and determine the relative angle of the facial organ key points in each of the face sub-images relative to the initial image frame;

[0047] A first determination module is configured to determine a scaling factor corresponding to each of the face sub-images according to a relative angle of the facial organ key points of each of the face sub-images relative to the initial image frame;

[0048] A generating module configured to determine a migration estimation matrix corresponding to each of the face sub-images according to the scaling factor, the real implicit key point coordinates of the face organ key points in each of the face sub-images, and the predicted implicit key point coordinates;

[0049] A second determination module is configured to perform feature encoding on the source video image frame and the target video image frame having a human face according to the migration estimation matrix, determine new implicit key point coordinates, and generate a target facial expression according to the new implicit key point coordinates;

[0050] The replacement module is configured to determine a facial expression replacement area from each target image frame in which a human face exists in the target video image frame according to the target facial expression, and perform pixel replacement on the facial expression replacement area according to the facial expression to obtain a replaced target video image frame.

[0051] A third aspect: An electronic device is provided, comprising:

[0052] processor;

[0053] a memory for storing processor-executable instructions;

[0054] The processor is configured to execute the executable instructions stored in the memory, and execute the executable instructions to implement a real-time video generation method based on AIGC as provided in the first aspect.

[0055] A fourth aspect: A computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the processor implements the real-time video generation method based on AIGC as described in the first aspect.

[0056] The embodiment of the present invention has the following advantages due to the adoption of the above technical solution:

[0057] First: by determining the initial image frame from the target image frame corresponding to the source video image frame, implicitly calibrating the initial image frame, predicting the predicted implicit key point coordinates of the facial organ key points in the target image frame, and determining the relative angle of the facial organ key points in each face sub-image relative to the initial image frame to determine the scaling factor corresponding to each face sub-image; according to the scaling factor, the real implicit key point coordinates of the facial organ key points in each face sub-image and the predicted implicit key point coordinates, determine the migration estimation matrix corresponding to each face sub-image, and determine the new implicit key point coordinates, and generate according to the new implicit key point coordinates. Target facial expression; determine the facial expression replacement area in each target image frame where there is a face in the target video image frame according to the target facial expression, and replace pixels in the facial expression replacement area through the facial expression to obtain the replaced target video image frame, and then determine the migration estimation matrix corresponding to each face sub-image through the scaling factor, the real implicit key point coordinates of the key points of the facial organs in each face sub-image, and the predicted implicit key point coordinates, so that the scaling factor can be corrected and transformed for the deformation problem caused by different angles, so as to stabilize the problem of low quality generated when the angle offset is too large during the processing of the target video image frame.

[0058] Second, by taking the rotation matrix and the translation matrix as the conditional control for locating the facial organs in each face sub-image, the posture deformation parameters corresponding to each face sub-image are determined according to the predicted implicit key point coordinates of the facial organ key points in each face sub-image and the corresponding scaling factors, and then the migration estimation matrix corresponding to each face sub-image is determined. By controlling the predicted implicit key point coordinates of the facial organ key points in each face sub-image and the corresponding scaling factors, the migration estimation matrix corresponding to a face sub-image can be changed, and then the control vectors of the eyes, nose and mouth can be increased by changing the migration estimation matrix.

[0059] The above summary is for illustrative purposes only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments and features described above, further aspects, embodiments and features of the present invention will be readily apparent by reference to the accompanying drawings and the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0061] Figure 1 A flowchart of a real-time video generation method based on AIGC of the present invention;

[0062] Figure 2 A block diagram of a real-time video generation device based on AIGC of the present invention;

[0063] Figure 3 This is a schematic diagram of the structure of an electronic device according to Embodiment 3 of the present invention. DETAILED DESCRIPTION

[0064] In the following, only some exemplary embodiments are briefly described. As those skilled in the art will appreciate, the described embodiments may be modified in various ways without departing from the spirit or scope of the present invention. Therefore, the drawings and descriptions are considered to be exemplary and non-restrictive in nature.

[0065] The present invention provides a real-time video generation method based on AIGC, comprising:

[0066] In step S10, face detection and tracking are performed on the acquired source video image frame and target video image frame respectively, and based on the detection and tracking results, a target image frame with a face is acquired from the source video image frame and the target video image frame;

[0067] Among them, face detection and tracking is used to automatically detect and track the location of human faces in video image frames. This usually involves using machine learning or deep learning algorithms to identify facial features in images and track changes in these features over time.

[0068] In the disclosed embodiment, an advanced deep neural network model is used to extract facial features in an image using multi-scale feature fusion. The detection layer decodes features to identify and locate the facial area in the image and the corresponding five facial key points. The five key points here are only used for facial posture correction and tracking. The five facial key points may include the left and right eye corners, the nose tip, and the left and right mouth corners. Based on the detection and tracking results, the target image frame containing the face is selected to provide basic data for subsequent processing.

[0069] In step S20, the face in the target image frame corresponding to the source video image frame is cut into a preset number of sub-images to obtain a number of sub-images corresponding to the preset number, and the real implicit key point coordinates of the key points of the facial organs in each sub-image are extracted;

[0070] The face sub-image is the image portion obtained by cutting the face in the target image frame corresponding to the source video image frame according to a preset number. Each cut face portion is called a face sub-image. The real implicit key point coordinates of the key points of the facial organs in each face sub-image are extracted using a convolutional neural network extractor. The convolutional neural network processes the continuous frames in the video to facilitate the accurate extraction of the real implicit key point coordinates of the key points of the facial organs in each face sub-image.

[0071] Among them, the real implicit key point coordinates are extracted by some algorithm (such as deep learning model) and represent the real position of facial organs (such as eyes, nose, mouth, etc.). These coordinates are implicit, that is, they are not directly calculated from the image pixel position, but are learned by the model.

[0072] In the disclosed embodiment, the face area in the target image frame corresponding to the source video image frame is cut according to a preset number (such as the number of key parts such as eyes, nose, mouth, etc.) to obtain multiple face sub-images containing different facial organs. Then, a special standard implicit key point detector is used to extract the real implicit key point coordinates, which are used to describe the shape and position of the facial organs. The implicit key point can be a set of multi-dimensional key points, such as the coordinate points of the face key points represented by xy values, and also includes the feature vectors corresponding to the face key points, which are used to express the facial features information.

[0073] In the face recognition system, the system needs to perform a detailed analysis of the face. First, the face area is divided into sub-areas such as eyes, nose, and mouth. Then, the key point detection algorithm is used to extract the coordinates of key points in each sub-area, such as the corners of the eyes and mouth, to provide accurate data support for subsequent face comparison and expression migration.

[0074] In step S30, an initial image frame is determined from a target image frame corresponding to a source video image frame, and the initial image frame is implicitly calibrated. Based on the implicit calibration of the initial image frame, the predicted implicit key point coordinates of the facial organ key points in the target image frame are predicted, and the relative angle of the facial organ key points in each face sub-image relative to the initial image frame is determined;

[0075] Among them, implicit calibration is used to calibrate the key points in the image, but the calibration process is not directly obtained by measurement or calculation, but learned by some algorithm (such as deep learning model). In this technical content, implicit calibration is used to calibrate the key points of facial organs in the initial image frame.

[0076] The key points of facial organs in each face sub-image include key points of facial contour, eyes, eyebrows, lips, nose and other organs.

[0077] The predicted implicit key point coordinates are based on the implicit calibration results, for example, the predicted implicit key point coordinates can be determined based on the key point detector of the deep learning model, wherein the predicted implicit key point coordinates can be used to characterize the initial calibration position of the key point, and used to assist in predicting new key point coordinates in each frame. In other words, these coordinates are also implicit, that is, they are predicted values ​​obtained through model learning.

[0078] In the disclosed embodiment, an initial image frame is determined from the target image frame corresponding to the source video image frame as a reference. The initial image frame is implicitly calibrated, that is, a spatial relationship model of facial organs is established using known key point coordinate information. Then, the predicted implicit key point coordinates of the facial organ key points in the target image frame are predicted based on the model. At the same time, the relative angle of the facial organ key points in each face sub-image relative to the initial image frame is calculated to reflect the change of facial posture.

[0079] In the application of facial expression transfer, the system first selects an initial image frame containing a standard expression (such as a neutral expression) for implicit calibration. Then, the key point positions of facial organs in other target image frames are predicted based on the calibration results. At the same time, the relative angles of facial organs in each target image frame relative to the initial image frame are calculated to facilitate posture adjustment during subsequent expression transfer.

[0080] In step S40, a scaling factor corresponding to each face sub-image is determined according to the relative angle of the facial organ key points of each face sub-image relative to the initial image frame;

[0081] The relative angle refers to the angle change of the facial organ key points in each face sub-image relative to the corresponding key points in the initial image frame. This angle change can be used for subsequent image processing and transformation. The scaling factor is determined based on the relative angle of the facial organ key points in each face sub-image relative to the initial image frame, and is used to adjust the size of the face sub-image to match the target facial expression.

[0082] In the disclosed embodiment, the scaling factor corresponding to each face sub-image is determined based on the relative angle of the key points of the facial organs of each face sub-image relative to the initial image frame and a preset scaling rule (such as adjusting the scaling ratio according to the angle change). These scaling factors are used to adjust the size and shape of the face sub-image in the subsequent expression migration process to match the facial posture in the target video image frame.

[0083] During the facial expression transfer process, the system calculates the corresponding scaling factors based on the relative angles of the facial organs in each target image frame relative to the initial image frame. These scaling factors are then used to adjust the face sub-image to keep it consistent with the facial posture in the target video image frame.

[0084] In step S50, a migration estimation matrix corresponding to each face sub-image is determined according to the scaling factor, the real implicit key point coordinates of the facial organ key points in each face sub-image, and the predicted implicit key point coordinates;

[0085] The shift estimation matrix is ​​determined according to the scaling factor, the real implicit key point coordinates and the predicted implicit key point coordinates, and is used to migrate the facial features in the source video image frame to the target video image frame.

[0086] In the disclosed embodiment, based on the scaling factor, the real implicit key point coordinates and the predicted implicit key point coordinates of the key points of the facial organs in each face sub-image, an affine transformation, a perspective transformation or a deep learning model algorithm is used to determine the migration estimation matrix corresponding to each face sub-image. This matrix is ​​used to migrate the facial expression in the source video image frame to the target video image frame while maintaining the consistency of the shape and position of the facial organs.

[0087] In the facial expression transfer application, the system uses the transfer estimation matrix to transfer the expression in the source video image frame to the target video image frame. First, the transfer estimation matrix is ​​calculated based on the scaling factor and key point coordinate information. Then, the matrix is ​​used to transform the face in the source video image frame to match the facial posture and expression in the target video image frame.

[0088] In step S60, according to the migration estimation matrix, feature encoding is performed on the target image frame with a human face in the source video image frame and the target video image frame, new implicit key point coordinates are determined, and the target facial expression is generated according to the new implicit key point coordinates;

[0089] Feature encoding is the process of converting features in an image (such as color, texture, shape, etc.) into a computable representation. For example, feature encoding is used to convert a target image frame with a face in the source video image frame and the target video image frame into a new implicit key point coordinate representation. The target facial expression is generated based on the new implicit key point coordinates and is used to replace the original facial expression in the target video image frame.

[0090] In the disclosed embodiment, according to the migration estimation matrix, feature encoding is performed on the target image frame with a face in the source video image frame and the target video image frame to extract the feature information of the facial expression. Then, new implicit key point coordinates are generated using this information, and these coordinates describe the shape and position of the target facial expression. Finally, the target facial expression image is generated according to the new implicit key point coordinates.

[0091] In the process of facial expression migration, the system first performs feature encoding on the source video image frame and the target video image frame to extract the feature information of the facial expression. Then, it uses this information to generate new implicit key point coordinates, and then generates the target facial expression image based on this information. In this way, the expression in the source video image frame is successfully migrated to the target video image frame.

[0092] In step S70, a facial expression replacement area is determined from each target image frame in which a face exists in the target video image frame according to the target facial expression, and pixels of the facial expression replacement area are replaced by the facial expression to obtain a replaced target video image frame.

[0093] The facial expression replacement area is the area in the target video image frame where the facial expression needs to be replaced. This area is usually determined based on the shape and size of the target facial expression to ensure that the replaced expression blends seamlessly with the background and other elements in the original video frame. Pixel replacement is the process of replacing the pixel values ​​of a certain area in an image with other pixel values, which is then used to replace the pixel values ​​of the target facial expression into the facial expression replacement area, thereby generating a replaced target video image frame.

[0094] In the disclosed embodiment, a facial expression replacement region is determined from each target image frame in which a face exists in the target video image frame according to the target facial expression. Then, the facial expression replacement region is pixel-replaced using the facial expression, that is, the pixel values ​​in the target facial expression image are replaced with the pixel values ​​of the corresponding region in the target video image frame. In this way, a replaced target video image frame is obtained, which contains the migrated expression.

[0095] In the facial expression transfer application, the system first determines the area in the target video image frame where the expression needs to be replaced. Then, the generated target facial expression image is used to replace the pixels in the area. After the replacement is completed, the target video image frame containing the transferred expression is obtained. In this way, the user can see that the expression in the source video image frame has been successfully transferred to the face of the person in the target video image frame.

[0096] The above technical solution determines the initial image frame from the target image frame corresponding to the source video image frame, implicitly calibrates the initial image frame, predicts the predicted implicit key point coordinates of the facial organ key points in the target image frame, and determines the relative angle of the facial organ key points in each face sub-image relative to the initial image frame to determine the scaling factor corresponding to each face sub-image; determines the migration estimation matrix corresponding to each face sub-image according to the scaling factor, the real implicit key point coordinates of the facial organ key points in each face sub-image, and the predicted implicit key point coordinates, and determines the new implicit key point coordinates, and generates a facial sub-image according to the new implicit key point coordinates. The target facial expression is formed; according to the target facial expression, the facial expression replacement area is determined from each target image frame where there is a face in the target video image frame, and the facial expression replacement area is pixel-replaced by the facial expression to obtain the replaced target video image frame, and then the migration estimation matrix corresponding to each face sub-image is determined by the scaling factor, the real implicit key point coordinates of the facial organ key points in each face sub-image, and the predicted implicit key point coordinates, so that the scaling factor can be corrected and transformed for the deformation problem caused by different angles, so as to stabilize the problem of low quality generated when the angle offset is too large during the processing of the target video image frame.

[0097] Then, by taking the rotation matrix and the translation matrix as the conditional control for locating the facial organs in each face sub-image, the posture deformation parameters corresponding to each face sub-image are determined according to the predicted implicit key point coordinates of the facial organ key points in each face sub-image and the corresponding scaling factors, and then the migration estimation matrix corresponding to each face sub-image is determined. By controlling the predicted implicit key point coordinates of the facial organ key points in each face sub-image and the corresponding scaling factors, the migration estimation matrix corresponding to a face sub-image can be changed, and then the control vectors of the eyes, nose and mouth can be increased by changing the migration estimation matrix.

[0098] Specifically: in step S50, the migration estimation matrix corresponding to each face sub-image is determined according to the scaling factor, the real implicit key point coordinates and the predicted implicit key point coordinates of the facial organ key points in each face sub-image, including:

[0099] In step S501, according to the relative angle of the facial organ key points of each face sub-image relative to the initial image frame, a rotation matrix of the face in each face sub-image relative to the initial image frame is determined, and a translation matrix for fine-tuning the facial organ key points in each face sub-image is determined;

[0100] In the disclosed embodiment, firstly, the corresponding rotation matrix is ​​calculated according to the relative angle of the key points of the facial organs in each face sub-image relative to the initial image frame. The rotation matrix is ​​used to describe the rotation of the face in three-dimensional space, including rotation around the X-axis, Y-axis and Z-axis. At the same time, a translation matrix is ​​also determined to fine-tune the key points of the facial organs in each face sub-image to more accurately locate the position of the facial organs in the image.

[0101] Assume that the face in the initial image frame is facing forward, but in the target image frame, the face is rotated and offset to a certain extent. The rotation matrix can be calculated based on the relative angles of the key points of the facial organs (such as the corners of the eyes and the corners of the mouth) relative to the initial image frame. At the same time, in order to locate the facial organs more accurately, it may be necessary to fine-tune the key points, and then the translation matrix is ​​needed.

[0102] In step S502, the rotation matrix and the translation matrix are used as conditional controls for locating the facial organs in each face sub-image, and the posture deformation parameters corresponding to each face sub-image are determined according to the predicted implicit key point coordinates of the facial organ key points in each face sub-image and the corresponding scaling factors;

[0103] In the disclosed embodiment, after the rotation matrix and the translation matrix are obtained, they are used as the condition control for locating the facial organs in each face sub-image. Then, according to the predicted implicit key point coordinates of the key points of the facial organs in each face sub-image and the corresponding scaling factor, the posture deformation parameters corresponding to each face sub-image are calculated. These parameters describe the shape and position changes of the face after rotation, scaling and other deformations.

[0104] In the process of facial expression transfer, the facial expression in the source video image frame is transferred to the target video image frame. Before the transfer, the posture deformation parameters of the face in the target image frame are calculated. These parameters can be calculated by comparing the predicted implicit key point coordinates of the facial organ key points in the source image frame and the target image frame, and combining the scaling factor.

[0105] Specifically: the posture deformation parameters are three-dimensional deformation models that can be used to model and deform human faces. They refer to the parameters that control the posture transformation of the model in three-dimensional space.

[0106] In step S503, the migration estimation matrix corresponding to each face sub-image is determined according to the real implicit key point coordinates of the facial organ key points in each face sub-image and the corresponding posture deformation parameters.

[0107] Wherein, according to the need to determine the relative angle of the key points of the facial organs in each face sub-image relative to the initial image frame, the number of extracted key points of the facial organs may also be increased.

[0108] In the disclosed embodiment, the migration estimation matrix corresponding to each face sub-image is calculated based on the real implicit key point coordinates of the key points of the facial organs in each face sub-image and the corresponding posture deformation parameters. The migration estimation matrix is ​​a two-dimensional or three-dimensional transformation matrix used to accurately migrate the facial expression in the source image frame to the target image frame while maintaining the consistency of the shape and position of the facial organs.

[0109] In the actual application of facial expression migration, the source image frame and the target image frame are first preprocessed to extract the key point coordinates of the facial organs. Then, the migration estimation matrix is ​​calculated based on these key point coordinates and posture deformation parameters. Finally, this matrix is ​​used to transform the face in the source image frame to match the facial posture and expression in the target image frame. In this way, the migrated facial expression image can be obtained and applied to the target video image frame.

[0110] Through the above steps, the migration estimation matrix corresponding to each face sub-image can be accurately calculated and applied to the process of facial expression migration. This method can not only maintain the consistency of the shape and position of facial organs, but also achieve a more natural and realistic expression migration effect.

[0111] In a possible implementation, the migration estimation matrix includes: a head posture estimation matrix and an expression estimation matrix; in step S503, the migration estimation matrix corresponding to each face sub-image is determined according to the real implicit key point coordinates of the facial organ key points in each face sub-image and the corresponding posture deformation parameters, including:

[0112] In step S5031, a unit matrix in a homogeneous coordinate system is constructed as an initial matrix according to the real implicit key point coordinates of the key points of the facial organs in each face sub-image and the corresponding posture deformation parameters;

[0113] In the disclosed embodiment, first, a unit matrix in a homogeneous coordinate system is constructed as an initial matrix based on the real implicit key point coordinates of the key points of the facial organs in each face sub-image and the corresponding posture deformation parameters. A homogeneous coordinate system is a coordinate system used to represent points in a two-dimensional or three-dimensional space. By adding an extra dimension (usually 1), it is convenient to represent transformations such as translation and scaling. The unit matrix is ​​a matrix in which all diagonal elements are 1 and the remaining elements are 0, which represents the initial state without any transformation.

[0114] Suppose we are processing a video frame containing a face and want to transfer the facial expression to another video frame. To do this, we first extract the coordinates of the key points of the facial organs in the face sub-image and calculate the posture deformation parameters based on these coordinates. Then, we construct a unit matrix in a homogeneous coordinate system as the starting point for subsequent transformations.

[0115] In step S5032, the first three rows and first three columns of the rotation matrix are copied to the corresponding positions of the initial matrix, and the first three elements of the translation matrix are copied to the first three columns of the last row of the initial matrix;

[0116] In the disclosed embodiment, the first three rows and first three columns of the rotation matrix are copied to the corresponding positions of the initial matrix to represent the rotation of the face in three-dimensional space. At the same time, the first three elements of the translation matrix are copied to the first three columns of the last row of the initial matrix to represent the translation of the face in a two-dimensional plane. In this way, the rotation and translation transformations are integrated into the initial matrix.

[0117] During the facial expression transfer process, the rotation and translation of the face are considered. For example, if the face in the target video frame has a certain rotation and offset relative to the face in the source video frame, corresponding adjustments need to be made during the transfer process. By copying the rotation matrix and the translation matrix to the initial matrix, the posture consistency of the face can be ensured during the transfer process.

[0118] In step S5033, the scaling factor and the representation transformation matrix are integrated into the rotation matrix, and the integrated result is copied to the corresponding position of the initial matrix;

[0119] In the disclosed embodiment, after obtaining the initial matrix including rotation and translation transformation, the scaling factor and the expression deformation matrix are considered. The scaling factor is used to adjust the size of the face, while the expression deformation matrix is ​​used to describe the shape change of the face when the expression changes. The scaling factor and the expression deformation matrix are integrated into the rotation matrix (usually through matrix multiplication), and then the integrated result is copied to the corresponding position of the initial matrix. In this way, a complete transformation matrix including rotation, translation, scaling and expression deformation is obtained.

[0120] In the practical application of facial expression transfer, the face needs to be resized to match the size in the target video frame. At the same time, the impact of expression changes on the face shape needs to be considered. By integrating the scaling factor and the expression deformation matrix into the rotation matrix, a transformation matrix that can handle both resizing and expression deformation can be obtained.

[0121] In step S5034, the last element of the last row of the initial matrix is ​​set to 1 to maintain the correctness of the homogeneous coordinate system, thereby obtaining a head posture estimation matrix and an expression estimation matrix.

[0122] Among them, by integrating the scaling factor and the representation deformation matrix into the rotation matrix, the quality of the facial organs in a face sub-image after enlargement or reduction can be effectively guaranteed.

[0123] In the disclosed embodiment, the last element of the last row of the initial matrix is ​​set to 1 to maintain the correctness of the homogeneous coordinate system. In the homogeneous coordinate system, the last element is usually used to represent the depth information of the point (usually 1 in a two-dimensional transformation) to ensure that the transformed coordinates are still located on the correct plane.

[0124] In the process of facial expression migration, it is necessary to ensure that the transformed coordinates are still located on the two-dimensional plane. By setting the last element of the last row of the initial matrix to 1, it can be ensured that no additional depth information is introduced during the transformation process, thereby maintaining the flatness of the face.

[0125] Through the above steps, a complete transformation matrix including rotation, translation, scaling and expression deformation is obtained, namely the migration estimation matrix. This matrix can be used to accurately migrate the facial expression in the source video image frame to the target video image frame while maintaining the consistency of the shape and position of the facial organs.

[0126] Specifically: the expression situational matrix is ​​determined by the following method:

[0127] In step S21, the deformation of the face under different expressions is estimated according to the position changes of the key points of the facial organs in the target image frame corresponding to the source video image frame;

[0128] In the disclosed embodiment, the deformation of the face under different expressions is estimated by analyzing the position changes of key points of facial organs in the source video image frame (usually representing a standard expression or a neutral expression) and the target video image frame (containing different expressions). This involves accurate tracking and comparison of the key point positions, as well as deformation modeling based on these position changes.

[0129] Assume that there is a source video image frame in which the face is in a neutral expression state, and the target video image frame contains images of the face with different expressions (such as smile, anger, etc.). Facial feature recognition technology can be used to track the key points of facial organs (such as eyes, mouth, nose, etc.) in the source video and the target video. By comparing the changes in these positions in the source video and the target video, the deformation of the face under different expressions can be estimated, such as the size of the mouth opening, the degree of squinting of the eyes, etc.

[0130] In step S22, according to the difference in face size and deformation between the source video image frame and the target video image frame, the source video image frame is used as the position of the key points of the facial organs under the standard expression, and the deformation vector between the position of the key points of the facial organs and the position of the key points of the facial organs in the standard expression is calculated for each target image frame in the target video image frame;

[0131] The deformation vector is the deformation value of the key points of the facial organs in the standard expression to the key points of the facial organs in the target image frame.

[0132] In the disclosed embodiment, according to the difference in face size and deformation between the source video image frame and the target video image frame, the source video image frame is used as the position of the key points of the facial organs under the standard expression, and the deformation vectors between the key points of the facial organs in each target image frame in the target video image frame and the key points of the facial organs in the standard expression are calculated. These deformation vectors describe the deformation direction and size from the standard expression to the target expression.

[0133] Continuing with the above example, the key point positions of the facial organs in the source video image frame (neutral expression) and the target video image frame (different expressions) are obtained, and the deformation vector of the key point position in each target image frame relative to the key point position in the source image frame is calculated. For example, if the mouth key point position in the target image frame moves downward by a certain distance relative to the position in the source image frame, then this downward deformation vector can be calculated.

[0134] In step S23, the dimension of the representation state-deformation matrix is ​​determined according to the dimension of the target video image frame and the number of preset facial organ key points;

[0135] In the disclosed embodiment, the dimension of the deformation matrix depends on the dimension of the target video image frame (usually two-dimensional, i.e., width and height) and the number of preset facial organ key points. Each key point corresponds to a deformation vector, so the number of columns of the deformation matrix will be equal to the number of key points, and the number of rows will usually be equal to the dimension of the target image frame (if three-dimensional deformation is considered, the number of rows will be more).

[0136] Assuming that 20 facial organ key points are preset and the target video image frame is two-dimensional (i.e., has width and height), then the representation deformation matrix will be a 2x20 matrix, where each column represents the deformation vector of a key point (in two-dimensional space).

[0137] In step S24, according to the dimension of the representation deformation matrix, the calculated deformation vectors are arranged in the order of key points to obtain the representation deformation matrix.

[0138] In the disclosed embodiment, according to the dimension of the expression deformation matrix, the calculated deformation vectors are arranged in the order of key points to obtain the expression deformation matrix. This matrix describes the entire deformation process from the standard expression to the target expression, and can be used for subsequent tasks such as facial expression migration or animation production.

[0139] The deformation vectors of the key points in each target image frame relative to the source image frame have been calculated, and the dimension of the expression deformation matrix (256x256) has been determined. These deformation vectors can be arranged into a 256x256 matrix according to the order of the key points, namely the expression deformation matrix. This matrix will be used in the subsequent facial expression migration process to migrate the neutral expression in the source video image frame to the different expressions in the target video image frame.

[0140] Through the above steps, the deformation of the face under different expressions is estimated, and the expression deformation matrix is ​​calculated, which provides important data support for subsequent tasks such as facial expression migration or animation production.

[0141] Specifically: The head posture estimation matrix and expression estimation matrix are determined by the following formula:

[0142] X=W×(Xc+E)×R+(Ts+Td)

[0143] Among them, X is the predicted implicit key point coordinates, W is the scaling factor, Xc is the true implicit key point coordinates, E is the expression estimation matrix, R is the head pose estimation matrix, Ts and Td are the rotation matrix and translation matrix respectively.

[0144] Specifically: in step S60, according to the migration estimation matrix, feature encoding is performed on the target image frame with a human face in the source video image frame and the target video image frame, new implicit key point coordinates are determined, and the target facial expression is generated according to the new implicit key point coordinates, including:

[0145] In step S601, the real implicit key point coordinates of the target image frame in which a human face exists in the source video image frame and the target video image frame are compared to generate a first transformation relationship matrix, wherein the first transformation relationship matrix is ​​used to describe the transformation relationship of facial organs from the source video image frame to the target video image frame;

[0146] In the disclosed embodiment, the real implicit key point coordinates of the target image frame where the face exists in the source video image frame and the target video image frame are compared. These key point coordinates represent the precise position of the facial organs in the image. By comparing these coordinates, a first transformation relationship matrix can be generated, which describes the transformation relationship of the facial organs from the source video image frame to the target video image frame. This transformation relationship includes translation, rotation, scaling, etc., and is the basis for subsequent facial expression migration.

[0147] Assume that there is a source video image frame in which the face is in a neutral expression, and the face in the target video image frame is in a smiling expression. By comparing the key point coordinates of the facial organs in the two image frames, a transformation relationship matrix describing the transformation from neutral expression to smiling expression can be generated. This matrix will be used to guide the subsequent facial expression migration process.

[0148] In step S602, the migration estimation matrix is ​​applied to the face region in the source video image frame to transform it into the coordinate system of the target video image frame;

[0149] In the disclosed embodiment, the migration estimation matrix is ​​applied to the face region in the source video image frame. The migration estimation matrix is ​​a matrix containing transformation information such as rotation, translation, and scaling, which can transform the face region in the source video image frame to the coordinate system of the target video image frame. This process is one of the key steps in facial expression migration.

[0150] Continuing with the above example, a migration estimation matrix from the source video image frame to the target video image frame is generated, and this matrix is ​​applied to the face region in the source video image frame to transform it into the coordinate system of the target video image frame. In this way, a face image with a neutral expression can be obtained in the coordinate system of the target video image frame.

[0151] In step S603, after the face region of the source video image frame is transformed into the coordinate system of the target video image frame, feature encoding is performed on the source video image frame and the target video image frame based on the texture, shape, and expression of the face region of the source video image frame in the coordinate system of the target video image frame to generate a feature vector;

[0152] In the disclosed embodiment, feature encoding is performed on the face region of the transformed source video image frame. This process includes extracting features such as texture, shape, expression, etc. of the face region and generating feature vectors. These feature vectors will be used in the subsequent facial expression migration and generation process.

[0153] The face region in the source video image frame is transformed to the coordinate system of the target video image frame, the texture, shape and expression features of this region are extracted, and a feature vector is generated. These feature vectors will be used to guide the subsequent facial expression migration process to ensure that the generated facial expression not only conforms to the expression features of the target video image frame, but also maintains the facial identity features of the source video image frame.

[0154] In step S604, the feature-encoded information is used in combination with a facial organ key point detection algorithm to determine new implicit key point coordinates in the target video image frame, and the new key point coordinates are used to represent the precise position of the transformed face in the target video image frame;

[0155] In the disclosed embodiment, the feature-encoded information and the facial organ key point detection algorithm are used to determine new implicit key point coordinates in the target video image frame. These new key point coordinates represent the precise position of the transformed face in the target video image frame, which is the basis for subsequent facial expression generation.

[0156] The feature vector of the transformed face region is extracted, and the new implicit key point coordinates are determined in the target video image frame using the facial organ key point detection algorithm. These coordinates represent the precise position of the transformed face in the target video image frame and can be used to guide the subsequent facial expression generation process.

[0157] In step S605, the new implicit key point coordinates are aligned with the real implicit key point coordinates of the facial organ key points in the target video image frame, the facial muscle stretching, compression, and rotation are analyzed, and a second transformation relationship matrix is ​​generated, where the second transformation relationship matrix is ​​used to describe the transformation relationship from the facial expression in the source video image frame to the expected facial expression in the target video image frame;

[0158] In the disclosed embodiment, the new implicit key point coordinates are aligned with the real implicit key point coordinates of the facial organ key points in the target video image frame. By comparing the two coordinate sets, changes such as facial muscle stretching, compression, and rotation can be analyzed, and a second transformation relationship matrix is ​​generated. This matrix describes the transformation relationship from the facial expression in the source video image frame to the expected facial expression in the target video image frame.

[0159] The new implicit keypoint coordinates are determined and aligned with the real implicit keypoint coordinates in the target video image frame. By comparing these two coordinate sets, changes such as facial muscle stretching, compression, and rotation can be analyzed, and a second transformation relationship matrix describing these changes can be generated. This matrix will be used to guide the subsequent facial expression generation process.

[0160] In step S606, based on matrix multiplication, the second transformation relationship matrix is ​​mapped to the face area in the source video image frame to generate a new face image with the target expression;

[0161] In the disclosed embodiment, the second transformation relationship matrix is ​​mapped to the face region in the source video image frame based on matrix multiplication. This process transforms the facial expression in the source video image frame into the desired facial expression in the target video image frame, thereby generating a new face image with the target expression.

[0162] A second transformation relationship matrix describing the transformation relationship from the facial expression in the source video image frame to the desired facial expression in the target video image frame is generated, and this matrix is ​​mapped to the face area in the source video image frame to generate a new face image with the target expression (such as a smile).

[0163] In step S607, the generated new face image is rendered to match the background and lighting conditions of the target video image frame, thereby generating a target facial expression.

[0164] In the disclosed embodiment, the generated new face image is rendered. This process includes adjusting the image's color, brightness, contrast and other parameters to match the background and lighting conditions of the target video image frame. Through this process, a more natural and realistic target face expression image can be generated.

[0165] A new face image with the target expression is generated, and this image is rendered and its color, brightness, and contrast are adjusted to match the background and lighting conditions of the target video image frame. Through this process, a more natural and realistic target facial expression image can be obtained, which can be seamlessly integrated into the target video image frame.

[0166] Specifically: in step S20, the face in the target image frame corresponding to the source video image frame is divided into a preset number of sub-images of the face corresponding to the preset number, including:

[0167] In step S201, the face in the target image frame corresponding to the source video image frame is segmented according to a preset number, and a plurality of facial organ key points in each segmented image are obtained;

[0168] In the disclosed embodiment, the face in the target image frame corresponding to the source video image frame is cropped. This usually means dividing the entire face area into a plurality of smaller sub-images according to a preset number (such as cropping the face horizontally or vertically into several parts). Each sub-image will contain a portion of the face, and these sub-images will be used for subsequent analysis and processing. While cropping, key points of multiple facial organs in each sub-image are obtained, and these key points are used to describe the shape and features of the face.

[0169] Assume that there is a source video image frame, and its corresponding target image frame contains a complete human face. The goal is to split the face horizontally into three sub-images: left face, middle face, and right face. During the splitting process, the facial organ key point detection algorithm is used to obtain the positions of key points such as eyes, nose, and mouth in each sub-image. These key points will be used for subsequent face tilt angle determination and correction.

[0170] In step S202, the face tilt angle of each image is determined based on the connection lines between the multiple facial organ key points in each image to obtain a corrected tilt angle;

[0171] In the disclosed embodiment, the tilt angle of the face is determined based on the connection lines between multiple facial organ key points in each sub-image. This usually involves calculating the relative position and direction between the key points, and estimating the tilt degree of the face based on this information. The determination of the tilt angle is crucial for subsequent face correction because it can help restore the normal posture of the face.

[0172] Continuing with the above example, the key point positions of the facial organs in the three sub-images of the left face, middle face, and right face are obtained, and the lines between these key points (such as the line from the left eye to the right eye, the line from the nose to the mouth, etc.) are used to calculate the tilt angle of the face. If the face is tilted to the left, then the line from the left eye to the right eye may show an angle tilted to the left; similarly, if the face is tilted upward, then the line from the nose to the mouth may show an angle tilted upward. By calculating the tilt angles of these lines, the overall tilt of the face can be obtained.

[0173] In step S203, face correction is performed on the corresponding images according to the correction tilt angle to obtain a preset number of corresponding face sub-images.

[0174] In the disclosed embodiment, face correction is performed on each sub-image according to the correction tilt angle determined in step S202. The correction process usually involves operations such as rotating, scaling or translating the image to restore the normal posture of the face. The corrected sub-image will have a more standard face shape and posture, which will help subsequent tasks such as facial expression migration or recognition.

[0175] The tilt angles of the three sub-images of the left face, the middle face, and the right face are determined, and these angles will be used to correct each sub-image. For example, if the left face sub-image is tilted 10 degrees to the left, then this sub-image will be rotated 10 degrees counterclockwise to correct it. Similarly, if the middle face sub-image is tilted 5 degrees upward, then this sub-image will be rotated 5 degrees clockwise (or translated downward) to correct it. Through this process, three corrected sub-images can be obtained, which will have more standard facial shapes and postures.

[0176] Through the cropping and correction process, a preset number of face sub-images can be obtained, and these sub-images will have more standard face shapes and postures.

[0177] Specifically, the number of faces in the target image frame corresponding to the source video image frame may be 5 according to a preset number, and the cropping may be performed according to a size of 224.

[0178] See also Figure 2 As shown, the present application provides a real-time video generation device based on AIGC, comprising:

[0179] The acquisition module 210 is configured to perform face detection and tracking on the acquired source video image frame and target video image frame respectively, and acquire a target image frame having a face from the source video image frame and the target video image frame based on the detection and tracking results;

[0180] The extraction module 220 is configured to divide the face in the target image frame corresponding to the source video image frame into a preset number, obtain a number of face sub-images corresponding to the preset number, and extract the real implicit key point coordinates of the key points of the facial organs in each of the face sub-images;

[0181] The calibration module 230 is configured to determine an initial image frame from a target image frame corresponding to the source video image frame, perform implicit calibration on the initial image frame, predict the predicted implicit key point coordinates of the facial organ key points in the target image frame according to the implicit calibration of the initial image frame, and determine the relative angle of the facial organ key points in each of the face sub-images relative to the initial image frame;

[0182] A first determination module 240 is configured to determine a scaling factor corresponding to each of the face sub-images according to a relative angle of the facial organ key points of each of the face sub-images relative to the initial image frame;

[0183] A generating module 250 is configured to determine a migration estimation matrix corresponding to each of the face sub-images according to the scaling factor, the real implicit key point coordinates of the face organ key points in each of the face sub-images, and the predicted implicit key point coordinates;

[0184] The second determination module 260 is configured to perform feature encoding on the target image frame having a human face in the source video image frame and the target video image frame according to the migration estimation matrix, determine new implicit key point coordinates, and generate a target facial expression according to the new implicit key point coordinates;

[0185] The replacement module 270 is configured to determine a facial expression replacement area from each target image frame in which a face exists in the target video image frame according to the target facial expression, and perform pixel replacement on the facial expression replacement area according to the facial expression to obtain a replaced target video image frame.

[0186] In a possible implementation, the generating module 250 is configured to:

[0187] Determine a rotation matrix of the face in each of the face sub-images relative to the initial image frame according to the relative angle of the facial organ key points in each of the face sub-images relative to the initial image frame, and determine a translation matrix for fine-tuning the facial organ key points in each of the face sub-images;

[0188] Using the rotation matrix and the translation matrix as conditional controls for locating the facial organs in each of the facial sub-images, and determining the posture deformation parameters corresponding to each of the facial sub-images according to the predicted implicit key point coordinates of the facial organ key points in each of the facial sub-images and the corresponding scaling factors;

[0189] According to the real implicit key point coordinates of the facial organ key points in each of the facial sub-images and the corresponding posture deformation parameters, a migration estimation matrix corresponding to each of the facial sub-images is determined.

[0190] In a possible implementation, the generating module 250 is configured to:

[0191] According to the real implicit key point coordinates of the facial organ key points in each of the face sub-images and the corresponding posture deformation parameters, constructing a unit matrix in a homogeneous coordinate system as an initial matrix;

[0192] Copy the first three rows and first three columns of the rotation matrix to the corresponding positions of the initial matrix and copy the first three elements of the translation matrix to the first three columns of the last row of the initial matrix;

[0193] Integrate the scaling factor and the representation transformation matrix into the rotation matrix, and then copy the integrated result to the corresponding position of the initial matrix;

[0194] The last element of the last row of the initial matrix is ​​set to 1 to maintain the correctness of the homogeneous coordinate system, thereby obtaining the head posture estimation matrix and the expression estimation matrix.

[0195] In a possible implementation, the generating module 250 is configured to determine the representation situation transformation matrix in the following manner:

[0196] According to the position changes of the key points of the facial organs in the target image frame corresponding to the source video image frame, estimating the deformation of the face under different expressions;

[0197] According to the difference in face size between the source video image frame and the target video image frame and the deformation, taking the source video image frame as the position of the key points of the facial organs under the standard expression, for each of the target image frames in the target video image frame, a deformation vector between the position of the key points of the facial organs and the position of the key points of the facial organs in the standard expression is calculated;

[0198] Determining the dimension of the representation state-deformation matrix according to the dimension of the target video image frame and the number of preset facial organ key points;

[0199] According to the dimension of the representation deformation matrix, the calculated deformation vectors are arranged in the order of key points to obtain the representation deformation matrix.

[0200] In a possible implementation, the generating module 250 is configured to determine the head posture estimation matrix and the expression estimation matrix by the following formula:

[0201] X=W×(Xc+E)×R+(Ts+Td)

[0202] Among them, X is the predicted implicit key point coordinates, W is the scaling factor, Xc is the true implicit key point coordinates, E is the expression estimation matrix, R is the head pose estimation matrix, Ts and Td are the rotation matrix and translation matrix respectively.

[0203] In a possible implementation, the second determining module 260 is configured to:

[0204] Compare the real implicit key point coordinates of the target image frame in which a human face exists in the source video image frame and the target video image frame to generate a first transformation relationship matrix, wherein the first transformation relationship matrix is ​​used to describe the transformation relationship of facial organs from the source video image frame to the target video image frame;

[0205] Applying the migration estimation matrix to the face region in the source video image frame to transform it into the coordinate system of the target video image frame;

[0206] After transforming the face region of the source video image frame to the coordinate system of the target video image frame, feature encoding is performed on the source video image frame and the target video image frame based on the texture, shape, and expression of the face region of the source video image frame in the coordinate system of the target video image frame to generate a feature vector;

[0207] Using the feature-encoded information in combination with a facial organ key point detection algorithm, new implicit key point coordinates are determined in the target video image frame, wherein the new key point coordinates are used to represent the precise position of the transformed face in the target video image frame;

[0208] Aligning the new implicit key point coordinates with the real implicit key point coordinates of the facial organ key points in the target video image frame, analyzing the stretching, compression, and rotation of facial muscles, and generating a second transformation relationship matrix, wherein the second transformation relationship matrix is ​​used to describe the transformation relationship from the facial expression in the source video image frame to the expected facial expression in the target video image frame;

[0209] Mapping the second transformation relationship matrix to the face region in the source video image frame to generate a new face image with a target expression;

[0210] The generated new face image is rendered to match the background and lighting conditions of the target video image frame, thereby generating the target face expression.

[0211] In a possible implementation, the extraction module 220 is configured to:

[0212] The face in the target image frame corresponding to the source video image frame is divided into a preset number, and a plurality of facial organ key points in each divided image are obtained;

[0213] Determine the face tilt angle of each image according to the connecting lines between the multiple facial organ key points in each image to obtain a corrected tilt angle;

[0214] According to the correction tilt angle, the corresponding images are respectively subjected to face correction to obtain face sub-images of a number corresponding to the preset number.

[0215] Based on the same concept, an embodiment of the present disclosure further provides an electronic device, including:

[0216] processor;

[0217] a memory for storing processor-executable instructions;

[0218] The processor is configured to execute executable instructions stored in the memory, and execute the executable instructions to implement a real-time video generation method based on AIGC as provided in the above method embodiment.

[0219] Based on the same concept, an embodiment of the present disclosure also provides a computer-readable storage medium on which a computer program is stored, characterized in that when the computer program is executed by a processor, the steps of a real-time video generation method based on AIGC as provided in the above method embodiment are implemented.

[0220] Figure 3 A schematic diagram of the hardware structure of a device for implementing the AIGC-based real-time video generation method provided in an embodiment of the present application is shown. The device may participate in or include the apparatus or system provided in an embodiment of the present application. The computer device 10 may include one or more processors 101 (the processor may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 102 for storing data, and a transmission device 103 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 3The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 3 More or fewer components as shown, or with Figure 3 Different configurations are shown.

[0221] The memory 102 can be used to store software programs and modules of application software, such as a program instruction / data storage device corresponding to a real-time video generation method based on AIGC in an embodiment of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 102, that is, implementing the above-mentioned method. The memory 102 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 102 may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the computer device 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0222] The transmission device 103 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer device 10. In one example, the transmission device 103 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 103 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0223] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer device 10 (or mobile device).

[0224] It should be noted that the one or more processors and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer device 10 (or mobile device). As involved in the embodiments of the present application, the data processing circuit is controlled as a processor.

[0225] Embodiment 4: A computer-readable storage medium stores a computer program, which, when executed by a processor, enables the processor to implement a real-time video generation method based on AIGC.

[0226] In this embodiment, the storage medium may be located in at least one of the multiple network servers of the computer network. Optionally, in this embodiment, the storage medium may include but is not limited to: a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and other media that can store program codes.

[0227] The embodiment of the present invention has the following advantages due to the adoption of the above technical solution:

[0228] First: by determining the initial image frame from the target image frame corresponding to the source video image frame, implicitly calibrating the initial image frame, predicting the predicted implicit key point coordinates of the facial organ key points in the target image frame, and determining the relative angle of the facial organ key points in each face sub-image relative to the initial image frame to determine the scaling factor corresponding to each face sub-image; according to the scaling factor, the real implicit key point coordinates of the facial organ key points in each face sub-image and the predicted implicit key point coordinates, determine the migration estimation matrix corresponding to each face sub-image, determine the new implicit key point coordinates, and generate the target facial expression according to the new implicit key point coordinates; according to the target facial expression, determine the facial expression replacement area from each target image frame where there is a face in the target video image frame, and replace the pixels of the facial expression replacement area by the facial expression to obtain the replaced target video image frame. The migration estimation matrix corresponding to each face sub-image is determined by the scaling factor, the real implicit key point coordinates and the predicted implicit key point coordinates of the key points of the facial organs in each face sub-image. The scaling factor can be used to correct the deformation caused by different angles, thereby stabilizing the problem of low quality generated when the angle offset is too large during the processing of the target video image frame.

[0229] Second, by taking the rotation matrix and the translation matrix as the conditional control for locating the facial organs in each face sub-image, the posture deformation parameters corresponding to each face sub-image are determined according to the predicted implicit key point coordinates of the facial organ key points in each face sub-image and the corresponding scaling factors, and then the migration estimation matrix corresponding to each face sub-image is determined. By controlling the predicted implicit key point coordinates of the facial organ key points in each face sub-image and the corresponding scaling factors, the migration estimation matrix corresponding to a face sub-image can be changed, and then the control vectors of the eyes, nose and mouth can be increased by changing the migration estimation matrix.

[0230] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different systems to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0231] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and systems can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of each unit is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0232] The steps in the system of the embodiment of the present invention can be adjusted in order, combined and deleted according to actual needs. The units in the device of the embodiment of the present invention can be combined, divided and deleted according to actual needs. In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0233] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, terminal, or network device, etc.) to execute all or part of the steps of the system described in each embodiment of the present invention.

[0234] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of various changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A real-time video generation method based on AIGC, characterized in that: include: Performing face detection and tracking on the acquired source video image frame and target video image frame respectively, and acquiring a target image frame having a face from the source video image frame and the target video image frame based on the detection and tracking results; The face in the target image frame corresponding to the source video image frame is divided into a preset number to obtain a number of face sub-images corresponding to the preset number, and the real implicit key point coordinates of the key points of the facial organs in each of the face sub-images are extracted; Determine an initial image frame from a target image frame corresponding to the source video image frame, perform implicit calibration on the initial image frame, predict the predicted implicit key point coordinates of the facial organ key points in the target image frame based on the implicit calibration of the initial image frame, and determine the relative angle of the facial organ key points in each of the face sub-images relative to the initial image frame; Determining a scaling factor corresponding to each of the face sub-images according to a relative angle of the key points of the facial organs of each of the face sub-images relative to the initial image frame; Determine a migration estimation matrix corresponding to each of the face sub-images according to the scaling factor, the real implicit key point coordinates of the facial organ key points in each of the face sub-images, and the predicted implicit key point coordinates; According to the migration estimation matrix, feature encoding is performed on the target image frame having a human face in the source video image frame and the target video image frame, new implicit key point coordinates are determined, and a target facial expression is generated according to the new implicit key point coordinates; Determine a facial expression replacement region from each of the target image frames in which a human face exists in the target video image frame according to the target facial expression, and replace pixels of the facial expression replacement region according to the facial expression to obtain a replaced target video image frame, Determining a migration estimation matrix corresponding to each of the face sub-images according to the scaling factor, the real implicit key point coordinates of the face organ key points in each of the face sub-images, and the predicted implicit key point coordinates, including: Determine a rotation matrix of the face in each of the face sub-images relative to the initial image frame according to the relative angle of the facial organ key points in each of the face sub-images relative to the initial image frame, and determine a translation matrix for fine-tuning the facial organ key points in each of the face sub-images; Using the rotation matrix and the translation matrix as conditional controls for locating the facial organs in each of the facial sub-images, and determining the posture deformation parameters corresponding to each of the facial sub-images according to the predicted implicit key point coordinates of the facial organ key points in each of the facial sub-images and the corresponding scaling factors; According to the real implicit key point coordinates of the facial organ key points in each of the facial sub-images and the corresponding posture deformation parameters, a migration estimation matrix corresponding to each of the facial sub-images is determined.

2. The real-time video generation method based on AIGC according to claim 1, characterized in that: The migration estimation matrix includes: a head posture estimation matrix and an expression estimation matrix; the migration estimation matrix corresponding to each face sub-image is determined according to the real implicit key point coordinates of the face organ key points in each face sub-image and the corresponding posture deformation parameters, including: According to the real implicit key point coordinates of the facial organ key points in each of the face sub-images and the corresponding posture deformation parameters, constructing a unit matrix in a homogeneous coordinate system as an initial matrix; Copy the first three rows and first three columns of the rotation matrix to the corresponding positions of the initial matrix and copy the first three elements of the translation matrix to the first three columns of the last row of the initial matrix; Integrate the scaling factor and the representation transformation matrix into the rotation matrix, and then copy the integrated result to the corresponding position of the initial matrix; The last element of the last row of the initial matrix is ​​set to 1 to maintain the correctness of the homogeneous coordinate system, thereby obtaining the head posture estimation matrix and the expression estimation matrix.

3. The real-time video generation method based on AIGC according to claim 2, characterized in that: The representation situational matrix is ​​determined in the following manner: According to the position changes of the key points of the facial organs in the target image frame corresponding to the source video image frame, estimating the deformation of the face under different expressions; According to the difference in face size between the source video image frame and the target video image frame and the deformation, taking the source video image frame as the position of the key points of the facial organs under the standard expression, for each of the target image frames in the target video image frame, a deformation vector between the position of the key points of the facial organs and the position of the key points of the facial organs in the standard expression is calculated; Determining the dimension of the representation state-deformation matrix according to the dimension of the target video image frame and the number of preset facial organ key points; According to the dimension of the representation deformation matrix, the calculated deformation vectors are arranged in the order of key points to obtain the representation deformation matrix.

4. The real-time video generation method based on AIGC according to claim 2, characterized in that: The head posture estimation matrix and the expression estimation matrix are determined by the following formula: X=W×(Xc+E)×R+(Ts+Td) Among them, X is the predicted implicit key point coordinates, W is the scaling factor, Xc is the true implicit key point coordinates, E is the expression estimation matrix, R is the head pose estimation matrix, Ts and Td are the rotation matrix and translation matrix respectively.

5. The real-time video generation method based on AIGC according to claim 1, characterized in that: The step of encoding the target image frame having a human face in the source video image frame and the target video image frame according to the migration estimation matrix, determining new implicit key point coordinates, and generating a target facial expression according to the new implicit key point coordinates includes: Compare the real implicit key point coordinates of the target image frame in which a human face exists in the source video image frame and the target video image frame to generate a first transformation relationship matrix, wherein the first transformation relationship matrix is ​​used to describe the transformation relationship of facial organs from the source video image frame to the target video image frame; Applying the migration estimation matrix to the face region in the source video image frame to transform it into the coordinate system of the target video image frame; After transforming the face region of the source video image frame to the coordinate system of the target video image frame, feature encoding is performed on the source video image frame and the target video image frame based on the texture, shape, and expression of the face region of the source video image frame in the coordinate system of the target video image frame to generate a feature vector; Using the feature-encoded information in combination with a facial organ key point detection algorithm, new implicit key point coordinates are determined in the target video image frame, wherein the new implicit key point coordinates are used to represent the precise position of the transformed face in the target video image frame; Aligning the new implicit key point coordinates with the real implicit key point coordinates of the facial organ key points in the target video image frame, analyzing the stretching, compression, and rotation of facial muscles, and generating a second transformation relationship matrix, wherein the second transformation relationship matrix is ​​used to describe the transformation relationship from the facial expression in the source video image frame to the expected facial expression in the target video image frame; Based on matrix multiplication, mapping the second transformation relationship matrix to the face area in the source video image frame to generate a new face image with a target expression; The generated new face image is rendered to match the background and lighting conditions of the target video image frame, thereby generating the target face expression.

6. The AIGC-based real-time video generation method according to any one of claims 1 to 5, characterized in that: The face in the target image frame corresponding to the source video image frame is divided into a preset number of sub-images to obtain a number of face sub-images corresponding to the preset number, including: The face in the target image frame corresponding to the source video image frame is divided into a preset number, and a plurality of facial organ key points in each divided image are obtained; Determine the face tilt angle of each image according to the connecting lines between the multiple facial organ key points in each image to obtain a corrected tilt angle; According to the correction tilt angle, the corresponding images are respectively subjected to face correction to obtain face sub-images of a number corresponding to the preset number.

7. A real-time video generation device based on AIGC, characterized in that: The AIGC-based real-time video generation method applied to any one of claims 1 to 6 comprises: An acquisition module is configured to perform face detection and tracking on the acquired source video image frame and target video image frame respectively, and based on the detection and tracking results, acquire a target image frame with a face from the source video image frame and the target video image frame; An extraction module is configured to divide the face in the target image frame corresponding to the source video image frame into a preset number, obtain a number of face sub-images corresponding to the preset number, and extract the real implicit key point coordinates of the key points of the facial organs in each of the face sub-images; A calibration module is configured to determine an initial image frame from a target image frame corresponding to the source video image frame, perform implicit calibration on the initial image frame, predict the predicted implicit key point coordinates of the facial organ key points in the target image frame based on the implicit calibration of the initial image frame, and determine the relative angle of the facial organ key points in each of the face sub-images relative to the initial image frame; A first determination module is configured to determine a scaling factor corresponding to each of the face sub-images according to a relative angle of the facial organ key points of each of the face sub-images relative to the initial image frame; A generating module configured to determine a migration estimation matrix corresponding to each of the face sub-images according to the scaling factor, the real implicit key point coordinates of the face organ key points in each of the face sub-images, and the predicted implicit key point coordinates; A second determination module is configured to perform feature encoding on the source video image frame and the target video image frame having a human face according to the migration estimation matrix, determine new implicit key point coordinates, and generate a target facial expression according to the new implicit key point coordinates; The replacement module is configured to determine a facial expression replacement area from each target image frame in which a human face exists in the target video image frame according to the target facial expression, and perform pixel replacement on the facial expression replacement area according to the facial expression to obtain a replaced target video image frame.

8. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to execute the executable instructions stored in the memory to implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images