Image processing method and device, computer equipment and storage medium

By obtaining face videos and three-dimensional models, determining the degree of deformation information and performing three-dimensional reconstruction, the problem of unnatural deformation of face images is solved, and a more natural and delicate deformation effect is achieved.

CN119027577APending Publication Date: 2024-11-26BEIJING SWEET SUGARSOFT TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410896669.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-04
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

The prior art has problems in the deformation processing of face image deformation, which leads to poor deformation effect.

Method used

By obtaining face video, standard face three-dimensional model and target face three-dimensional model, we determine the degree of deformation information of each frame of face image, and perform three-dimensional face reconstruction and deformation processing based on this information to generate target deformation images to realize face The smooth and natural changes of special effects videos.

Benefits of technology

It improves the naturalness and fusion of facial image deformation processing, improves the deformation effect, and makes the facial special effects video more delicate and natural.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119027577A_ABST
    Figure CN119027577A_ABST
Patent Text Reader

Abstract

The invention provides an image processing method and device, computer equipment and a storage medium, and the method comprises the steps: obtaining a face video containing a first face, a standard face three-dimensional model, and a target face three-dimensional model; the target face three-dimensional model is a three-dimensional model obtained by deforming the standard face three-dimensional model based on a target deformation effect; aiming at each frame of face image in the face video, based on the position of each frame of face image in the face video, the number of face images included in the face video, the standard face three-dimensional model and the target face three-dimensional model, obtaining the target face three-dimensional model; determining deformation degree information of deformation of a first face in each frame of face image; and based on the deformation degree information, performing deformation processing on the first face in each frame of face image to obtain a target deformation image corresponding to each frame of face image.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of computer vision technology, and in particular to an image processing method, apparatus, computer device and storage medium. Background Art

[0002] With the development of image processing technology, in some games, social networking, and short video platforms, users can select their favorite target effects to deform real face images. For example, in a short video platform, users can select a target effect to deform the face image in the video to obtain a target deformation effect. The target deformation effect can be, for example, to enlarge, reduce, rotate, or displace the entire or partial face in the face image. The current face deformation has the problem of poor deformation effect. Summary of the invention

[0003] The embodiments of the present disclosure at least provide an image processing method, apparatus, computer equipment, and storage medium.

[0004] In a first aspect, an embodiment of the present disclosure provides an image processing method, including:

[0005] Acquire a face video containing a first face, a standard face 3D model, and a target face 3D model; the target face 3D model is a 3D model obtained by deforming the standard face 3D model based on a target deformation effect;

[0006] For each face image frame in the face video, based on the position of each face image frame in the face video, the number of face images included in the face video, the standard face 3D model, and the target face 3D model, determine deformation degree information for deforming a first face in each face image frame;

[0007] Based on the deformation degree information, a deformation process is performed on the first face in each frame of the face image to obtain a target deformed image corresponding to each frame of the face image.

[0008] In this way, the position of each frame of facial image in the facial video is used to determine the deformation degree information corresponding to each frame of facial image in the dynamic change process of transforming the target face in the facial video to have the same deformation effect as the target face three-dimensional model. Then, the deformation degree information is used to deform each frame of facial image in the facial video to obtain a facial special effects video composed of multiple frames of target deformed images. The special effects video can finely reflect the changing process of deforming the facial image based on the target deformation effect. The deformation process is smooth and natural, and has a better deformation effect.

[0009] In an optional implementation manner, obtaining a face video containing a first face includes:

[0010] In response to triggering, deforming the first face into a second face based on the target deformation effect, acquiring an original video including the first face;

[0011] Based on preset image quantity information, the face video is captured from the original video.

[0012] In this way, by using the preset image quantity information, the number of images corresponding to the preset image quantity information can be cut out from the multiple frames of images included in the original video, and these images can be used as facial videos containing the first face. This realizes the use of this method to process facial videos from any original video that meets the conditions, thereby increasing the applicability of the application of this method.

[0013] In an optional implementation, the determining, based on the position of each frame of the face image in the face video, the number of face images included in the face video, the standard face 3D model, and the target face 3D model, the deformation degree information of the first face in each frame of the face image includes:

[0014] Performing three-dimensional face reconstruction on each frame of the face image to obtain a reconstructed three-dimensional face model;

[0015] Determining first deformation degree information of the reconstructed human face 3D model relative to the standard human face 3D model; and determining second deformation degree information of the target human face 3D model relative to the standard human face 3D model;

[0016] Based on the first deformation degree information, the second deformation degree information, the position of each frame of facial image in the facial image sequence corresponding to the facial video, and the number of facial images included in the facial image sequence, the deformation degree information of the first face in each frame of facial image is determined.

[0017] In this way, the difference between the reconstructed three-dimensional face model and the standard three-dimensional face model is determined through the first deformation degree information, and the target deformation effect is determined through the second deformation degree information. Then, according to the position of each face frame in the face video in the face image sequence, the deformation degree information of each face frame in the face video is determined, thereby achieving fine adjustment of each face frame in the face video, making the deformation effect smoother and more natural, and improving the deformation effect.

[0018] In an optional implementation, the reconstructed three-dimensional face model includes: a plurality of first vertices, and first position information of the plurality of first vertices in a camera coordinate system corresponding to each frame of the face image; the standard three-dimensional face model includes: second vertices corresponding to the plurality of first vertices, and second position information of the second vertices in a model coordinate system;

[0019] The determining of first deformation degree information of the reconstructed three-dimensional face model relative to the standard three-dimensional face model includes:

[0020] Converting first position information of the first vertices in the camera coordinate system into third position information in the model coordinate system;

[0021] Determine the displacements corresponding to the plurality of first vertices based on the third position information corresponding to the plurality of first vertices and the second position information corresponding to the plurality of second vertices;

[0022] The first deformation degree information is constructed based on the displacement amounts respectively corresponding to the plurality of first vertices.

[0023] In this way, through the conversion of the coordinate system, the two models are calculated in the same coordinate system, and then the first deformation degree information is obtained according to the displacement of the first vertex relative to the second vertex.

[0024] In an optional implementation, the deforming process is performed on the first face in each frame of the face image based on the deformation degree information to obtain a target deformed image corresponding to each frame of the face image, including:

[0025] Based on the deformation degree information and the standard human face three-dimensional model, generating a deformed human face three-dimensional model corresponding to each frame of the human face image;

[0026] The first face in each frame of the face image is deformed based on the deformed three-dimensional face model to obtain a target deformed image corresponding to each frame of the face image.

[0027] In this way, by performing the first face deformation processing on the deformed face three-dimensional model corresponding to each frame of face image, the changing process of the face special effect can be finely reflected, making the deformation effect more delicate and natural.

[0028] In an optional implementation, the deforming the first face in each frame of the face image based on the deformed three-dimensional face model to obtain a target deformed image corresponding to each frame of the face image includes:

[0029] Converting the deformed 3D face model from a model coordinate system to a camera coordinate system corresponding to the reconstructed 3D face model to obtain fourth position information of a plurality of third vertices in the deformed 3D face model in the camera coordinate system;

[0030] Based on the fourth position information, projecting the plurality of third vertices respectively into each frame of the face image, and obtaining positions of projection points respectively corresponding to the plurality of third vertices in each frame of the face image;

[0031] Determine the displacement information of each pixel in each frame of the face image based on the positions of the projection points corresponding to the plurality of third vertices in each frame of the face image and the positions of the pixel points corresponding to the first vertices corresponding to the plurality of third vertices in each frame of the face image;

[0032] The position of each pixel in each frame of the face image is adjusted based on the displacement information to obtain a target deformed image corresponding to each frame of the face image.

[0033] In this way, the displacement information can be used to adjust the positions of the pixels of the original face image to obtain a target deformed image that corresponds to the face image of the current frame.

[0034] In an optional implementation, deforming the first face in each frame of the face image based on the deformed three-dimensional face model to obtain a target deformed image corresponding to each frame of the face image includes:

[0035] Converting the deformed 3D face model from a model coordinate system to a camera coordinate system corresponding to the reconstructed 3D face model to obtain a converted 3D face model;

[0036] The transformed face three-dimensional model is rendered using each frame of the face image to obtain the target deformed image.

[0037] In this way, through the conversion of the coordinate system, the pixel values ​​of the face image can be used to render the transformed face 3D model, making the target deformed image more realistic and natural.

[0038] In an optional implementation, using each frame of the face image to render the transformed face 3D model to obtain the target deformed image includes:

[0039] Determine pixel values ​​corresponding to each fourth vertex in the converted three-dimensional face model based on a mapping relationship between pixel points in each frame of the face image and the first vertex in the reconstructed three-dimensional face model;

[0040] Model rendering is performed based on the pixel values ​​corresponding to each fourth vertex in the converted three-dimensional face model to obtain the target deformed image.

[0041] In an optional implementation, the method further includes: acquiring a special effect map corresponding to the target face three-dimensional model;

[0042] The step of rendering the transformed face 3D model using each frame of the face image to obtain the target deformed image includes:

[0043] The transformed face three-dimensional model is rendered using each frame of the face image and the special effect map to obtain the target deformed image.

[0044] In this way, by using the special effect map corresponding to the target face 3D model, the deformation effect of the target deformed image can be made more diverse.

[0045] In an optional implementation, the transformed face 3D model is rendered using each frame of the face image and the special effect map to obtain the target deformed image.

[0046] Based on the position of each frame of the face image in the face video and the number of face images included in the face video, fusing each frame of the face image with the special effect map to obtain a target effect map;

[0047] The target effect map is used to render the transformed three-dimensional face model to obtain the target deformed image.

[0048] In an optional implementation, based on the position of each frame of the face image in the face video and the number of face images included in the face video, each frame of the face image and the special effect map are fused to obtain a target effect map, including:

[0049] Determine, based on the position of each frame of the face image in the face video and the number of face images included in the face video, a fusion ratio corresponding to the special effect map when fusion processing is performed on each frame of the face image and the special effect map;

[0050] Determine a pixel correspondence relationship between each frame of the face image and the special effect map based on a mapping relationship between a pixel in each frame of the face image and a first vertex of the reconstructed three-dimensional face model, a mapping relationship between a pixel in the special effect map and a fifth vertex of the target three-dimensional face model, and a vertex correspondence relationship between the reconstructed three-dimensional face model and the target three-dimensional face model;

[0051] Based on the pixel correspondence and the fusion ratio, the pixel values ​​of the pixels in each frame of the face map and the pixel values ​​of each pixel in the special effect map are fused to obtain the target effect map.

[0052] In this way, by utilizing the fusion ratio of the special effect map, the target effect map corresponding to each frame of the face image can be obtained, thereby generating the deformation process of the face video, making the deformation process smoother and more natural.

[0053] In a second aspect, an embodiment of the present disclosure further provides an image processing device, the device comprising:

[0054] An acquisition module, used to acquire a face video containing a first face, a standard face 3D model, and a target face 3D model; the target face 3D model is a 3D model obtained by deforming the standard face 3D model based on a target deformation effect;

[0055] a processing module, configured to determine, for each face image frame in the face video, deformation degree information for deforming a first face in each face image frame based on a position of each face image frame in the face video, a number of face images included in the face video, the standard three-dimensional face model, and the target three-dimensional face model;

[0056] A generating module is used to perform deformation processing on the first face in each frame of face image based on the deformation degree information to obtain a target deformed image corresponding to each frame of face image.

[0057] In an optional implementation, the device further includes an interception module, wherein the interception module is configured to:

[0058] In response to triggering, deforming the first face into a second face based on the target deformation effect, acquiring an original video including the first face;

[0059] Based on preset image quantity information, the face video is captured from the original video.

[0060] In an optional implementation manner, the processing module is used to:

[0061] Performing three-dimensional face reconstruction on each frame of the face image to obtain a reconstructed three-dimensional face model;

[0062] Determining first deformation degree information of the reconstructed human face 3D model relative to the standard human face 3D model; and determining second deformation degree information of the target human face 3D model relative to the standard human face 3D model;

[0063] Based on the first deformation degree information, the second deformation degree information, the position of each frame of facial image in the facial image sequence, and the number of facial images included in the facial image sequence, the deformation degree information of the first face in each frame of facial image is determined.

[0064] In an optional implementation, the device further includes a vertex displacement calculation module, and the vertex displacement calculation module is used to:

[0065] Converting first position information of the first vertices in the camera coordinate system into third position information in the model coordinate system;

[0066] Determine the displacements corresponding to the plurality of first vertices based on the third position information corresponding to the plurality of first vertices and the second position information corresponding to the plurality of second vertices;

[0067] The first deformation degree information is constructed based on the displacement amounts respectively corresponding to the plurality of first vertices.

[0068] In an optional implementation manner, the generating module is further used for:

[0069] The first face in each frame of the face image is deformed based on the deformed three-dimensional face model to obtain a target deformed image corresponding to each frame of the face image.

[0070] In an optional implementation manner, the vertex displacement calculation module is further used for:

[0071] Converting the deformed 3D face model from a model coordinate system to a camera coordinate system corresponding to the reconstructed 3D face model to obtain fourth position information of a plurality of third vertices in the deformed 3D face model in the camera coordinate system;

[0072] Based on the fourth position information, projecting the plurality of third vertices respectively into each frame of the face image, and obtaining positions of projection points respectively corresponding to the plurality of third vertices in each frame of the face image;

[0073] Determine the displacement information of each pixel in each frame of the face image based on the positions of the projection points corresponding to the plurality of third vertices in each frame of the face image and the positions of the pixel points corresponding to the first vertices corresponding to the plurality of third vertices in each frame of the face image;

[0074] The position of each pixel in each frame of the face image is adjusted based on the displacement information to obtain a target deformed image corresponding to each frame of the face image.

[0075] In an optional implementation manner, the device further includes a rendering module, and the rendering module is used to:

[0076] Converting the deformed 3D face model from a model coordinate system to a camera coordinate system corresponding to the reconstructed 3D face model to obtain a converted 3D face model;

[0077] The transformed face three-dimensional model is rendered using each frame of the face image to obtain the target deformed image.

[0078] In an optional implementation manner, the rendering module is further used for:

[0079] The step of rendering the transformed face 3D model using each frame of the face image to obtain the target deformed image includes:

[0080] The transformed face three-dimensional model is rendered using each frame of the face image and the special effect map to obtain the target deformed image.

[0081] In an optional implementation manner, the rendering module is further used for:

[0082] Based on the position of each frame of the face image in the face video and the number of face images included in the face video, fusing each frame of the face image with the special effect map to obtain a target effect map;

[0083] The target effect map is used to render the transformed three-dimensional face model to obtain the target deformed image.

[0084] In an optional implementation, the device further includes a fusion module, and the fusion module is used to:

[0085] Determine a pixel correspondence relationship between each frame of the face image and the special effect map based on a mapping relationship between a pixel in each frame of the face image and a first vertex of the reconstructed three-dimensional face model, a mapping relationship between a pixel in the special effect map and a fifth vertex of the target three-dimensional face model, and a vertex correspondence relationship between the reconstructed three-dimensional face model and the target three-dimensional face model;

[0086] Based on the pixel correspondence and the fusion ratio, the pixel values ​​of the pixels in each frame of the face map and the pixel values ​​of each pixel in the special effect map are fused to obtain the target effect map.

[0087] In a third aspect, an optional implementation of the present disclosure further provides a computer device, a processor, and a memory, wherein the memory stores machine-readable instructions executable by the processor, and the processor is used to execute the machine-readable instructions stored in the memory, and when the machine-readable instructions are executed by the processor, the machine-readable instructions perform the steps of the above-mentioned first aspect, or any possible implementation of the first aspect.

[0088] In a fourth aspect, an optional implementation of the present disclosure further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed, the steps of the above-mentioned first aspect, or any possible implementation of the first aspect are executed.

[0089] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0090] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following is a brief introduction to the drawings required for use in the embodiments. The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used together with the specification to illustrate the technical solutions of the present disclosure. It should be understood that the following drawings only illustrate certain embodiments of the present disclosure and should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can also be obtained based on these drawings without creative work.

[0091] Figure 1 A flowchart of an image processing method provided by an embodiment of the present disclosure is shown;

[0092] Figure 2 A flow chart of a method for acquiring a target deformed image in the image processing method provided by an embodiment of the present disclosure is shown;

[0093] Figure 3 A flowchart of another method for acquiring a target deformed image in the image processing method provided by an embodiment of the present disclosure is shown;

[0094] Figure 4 An example diagram of a special effect map in the image processing method provided by an embodiment of the present disclosure is shown;

[0095] Figure 5a An example diagram showing the fusion processing of a face image and a special effect map at the beginning of deformation in the image processing method provided by the embodiment of the present disclosure;

[0096] Figure 5bOne of the example diagrams of the process of fusing a face image with a special effect map in the image processing method provided by the embodiment of the present disclosure is shown;

[0097] Figure 5c The second example diagram shows the process of fusing a face image with a special effect map in the image processing method provided by the embodiment of the present disclosure;

[0098] Figure 5d An example diagram showing the fusion processing of the face image and the special effect map at the end of deformation in the image processing method provided by the embodiment of the present disclosure;

[0099] Figure 6 A schematic diagram of an image processing device provided by an embodiment of the present disclosure is shown;

[0100] Figure 7 A schematic diagram of a computer device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0101] In order to make the purpose, technical scheme and advantages of the embodiments of the present disclosure clearer, the technical scheme in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. The components of the embodiments of the present disclosure generally described and shown here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure is not intended to limit the scope of the present disclosure claimed for protection, but merely represents the selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present disclosure.

[0102] Research has found that in some games, social networking, and short video platforms, users can select their favorite target effects to deform real face images. For example, in short video platforms, users can select target effects to deform the face images in the video to obtain target deformation effects. The target deformation effects can, for example, enlarge, reduce, rotate, or shift the entire or part of the face in the face image.

[0103] However, when processing facial images in videos based on target deformation effects, most of the time, the facial images are deformed directly, so that the facial images are directly transformed into images with target deformation effects; as a result, the current method of deforming faces in images usually lacks an intermediate change process, and there are problems of stiff and unnatural deformation, and the deformation effect is poor.

[0104] In addition, when deforming a face, since it is achieved by directly adjusting the image based on the image, and the facial postures in the image are different, this leads to poor integration between the deformed image and the original face, which also causes the problem of poor deformation effect.

[0105] Based on the above research, the present disclosure provides an image processing method, which uses the position of each frame of facial images in a facial video to determine the deformation degree information corresponding to each frame of facial image in the dynamic change process of transforming a target face in the facial video to have the same deformation effect as the target face three-dimensional model, and then uses the deformation degree information to deform each frame of facial image in the facial video to obtain a facial special effects video composed of multiple frames of target deformed images. The special effects video can finely reflect the changing process of deforming the facial image based on the target deformation effect. The deformation process is smooth and natural, and has a better deformation effect.

[0106] In addition, in the embodiments of the present disclosure, when the first face in each frame of the facial image is deformed based on the deformation degree information, the first face in the facial image is three-dimensionally reconstructed to obtain a reconstructed three-dimensional facial model of the first face, and then the reconstructed three-dimensional facial model is deformed using the deformation degree information, thereby deforming the face based on the three-dimensional model, so that the deformation effect has a higher degree of integration with the first face, thereby further improving the deformation effect.

[0107] The defects existing in the relevant technologies are the results obtained by the inventors after practice and careful research. Therefore, the discovery process of the above problems and the solutions proposed by the present disclosure for the above problems below should be the contributions made by the inventors to the present disclosure during the disclosure process.

[0108] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0109] To facilitate understanding of the present embodiment, an image processing method disclosed in the embodiment of the present disclosure is first introduced in detail. The execution subject of the image processing method provided in the embodiment of the present disclosure is generally a computer device with certain computing capabilities, and the computer device includes, for example: a terminal device or a server or other processing device, and the terminal device can be a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a wearable device, etc. In some possible implementations, the image processing method can be implemented by a processor calling a computer-readable instruction stored in a memory.

[0110] The image processing method provided by the embodiment of the present disclosure is described below.

[0111] See also Figure 1 FIG. 1 is a flowchart of an image processing method provided by an embodiment of the present disclosure, wherein the method comprises steps S101 to S103, wherein:

[0112] S101: Acquire a facial video including a first face, a standard face 3D model, and a target face 3D model; the target face 3D model is a 3D model obtained by deforming the standard face 3D model based on a target deformation effect.

[0113] S102: for each face image frame in the face video, determining deformation degree information for deforming a first face in each face image frame based on a position of each face image frame in the face video, the number of face images included in the face video, the standard three-dimensional face model, and the target three-dimensional face model;

[0114] S103: Based on the deformation degree information, deform the first face in each frame of the face image to obtain a target deformed image corresponding to each frame of the face image.

[0115] The present disclosure provides an image processing method. First, through each frame of face image in a face video, according to the position of the current frame of face image in the face image sequence corresponding to the face video and the number of face image sequences corresponding to the face video, a deformation degree ratio is obtained; secondly, according to the model difference between the standard face three-dimensional model and the target face three-dimensional model, the final deformation degree information is determined; finally, according to the deformation degree information and the deformation degree ratio of the current frame of face image, the deformation degree information corresponding to the current frame of face image is determined. In this way, using the deformation degree information, each frame of face image in the face video can be deformed to different degrees to obtain a face special effects video composed of multiple frames of target deformed images. The face special effects video can finely reflect the change process of deforming the face image based on the target deformation effect, so that the deformation process is smooth and natural, and has a better deformation effect.

[0116] The above S101 to S103 are described in detail below.

[0117] With respect to the above S101, the standard human face 3D model can be generated by selecting a frontal face image with clear facial features from a face database and using a facial 3DMesh reconstruction algorithm to generate the standard human face 3D model.

[0118] The three-dimensional face model generally includes a plurality of vertices and the three-dimensional coordinates of the vertices in the model coordinate system. The vertices are connected to each other to form a plurality of meshes for constituting the surface of the three-dimensional face model.

[0119] Different meshes may also have a certain linkage relationship; the linkage relationship is used to represent the change amplitude information of the position change of other meshes adjacent to the mesh when the position of the vertex in one mesh changes, such as displacement or scale scaling.

[0120] In addition, the standard human face three-dimensional model can also be a three-dimensional model directly obtained by manual modeling using modeling software. For example, multiple vertices, facial positions corresponding to the multiple vertices, and connection relationships between multiple different vertices can be defined. The standard human face three-dimensional model can be generated based on the above-defined information.

[0121] When designing a face deformation effect, a target face 3D model with a target deformation effect can be generated by adjusting the relative positions of the vertices of the standard face 3D model on the basis of the standard face 3D model. The target deformation effect here can be a 2D or 3D deformation effect, such as a 2D animation head portrait effect or a 3D animal head portrait effect.

[0122] Among them, the target deformation effects are different. When adjusting the vertices of the standard human face three-dimensional model, the adjusted vertices can be different and the adjustment amounts of the adjusted vertices can also be different.

[0123] For example, based on the standard face model, the displacement information of the vertices is obtained by adjusting the coordinate positions of the vertices without adjusting the topological structure of the model, and the target deformation effect is obtained according to the displacement information of the vertices. In this way, the target face 3D model can be obtained according to the target deformation effect.

[0124] The three-dimensional face model in the embodiment of the present application can be generated based on the framework of the facial three-dimensional mesh (3DMesh). Under the 3D mesh framework, the number of vertices of each three-dimensional face model is the same, and the vertices of each three-dimensional face model can correspond one-to-one with the vertices of the standard three-dimensional face model. The desired three-dimensional face model can be obtained by adjusting the relative positions of the vertices of the standard three-dimensional face model.

[0125] In one embodiment provided by the present disclosure, a specific method for obtaining a face video is provided, including: in response to a trigger, deforming the first face into a second face based on the target deformation effect, obtaining an original video containing the first face; based on preset image quantity information, intercepting the face video from the original video. The preset image quantity information may be the number of image frames required to complete the face deformation effect, and may be pre-set by the face deformation effect producer.

[0126] In different application scenarios, there may be multiple ways to obtain face videos. Exemplary application scenarios for triggering the acquisition of face videos include but are not limited to the following two scenarios m1 to m2:

[0127] m1: In the application scenarios of photo editing software or short videos, in an original video including a first face uploaded by a user, the user can trigger the operation of obtaining the face video by clicking on a target special effect in the special effects module of the application. The application will capture the face video corresponding to the number of video frames required by the target special effect from the original video.

[0128] m2: In the live broadcast application scenario, when the user triggers the target special effect, based on the number of video frames required for the user to trigger the target special effect, the live broadcast video data corresponding to the number of video frames can be intercepted from the moment of triggering as the face video.

[0129] With respect to the above S102, the number of facial images included in the facial video refers to the total number of frames of facial images in the facial video, and the position of each facial image frame in the facial video refers to the temporal position of the facial image frame in the facial video. Specifically, if the facial video is composed of an image sequence comprising several facial image frames, then the sequence number of each facial image frame in the image sequence represents the position of the facial image frame in the facial video.

[0130] In some embodiments of the present application, the degree of deformation of each frame of a face image in a face video is related to the position of the frame of the face image in the face video. For example, the closer the position of a frame of an image in the video is, the smaller its degree of deformation is, and the closer the deformation effect is to the face in the original face video; conversely, the closer the position of a frame of an image in the video is, the greater its degree of deformation is, and the closer the deformation effect is to the face represented by the target face 3D model. Optionally, a relationship model between the degree of deformation information and the position of the face image in the face video, the number of face images included in the face video, the standard face 3D model, and the target face 3D model can be pre-constructed, and the degree of deformation information of each frame of the face image is determined based on the relationship model.

[0131] Optionally, for the above S102, after obtaining the face video, the standard face 3D model, and the target face 3D model, the embodiment of the present disclosure provides a specific method for obtaining deformation degree information of the first face in each frame of the face image, including the following steps A to D:

[0132] Step A: Perform three-dimensional face reconstruction on the first face in each frame of the face image to obtain a reconstructed three-dimensional face model corresponding to each frame of the face image. The reconstructed three-dimensional face model can be in the corresponding face image coordinate system, or the reconstructed face model can be converted to the camera coordinate system of the camera that captures the corresponding face image.

[0133] For example, the facial 3DMesh reconstruction algorithm can be used to analyze and process the first face corresponding to each face image in multiple face images in the face video to construct a reconstructed face 3D model, wherein each vertex in the reconstructed face 3D model and the standard face 3D model has a one-to-one correspondence.

[0134] In another example, taking video processing software equipped with a facial 3D Mesh reconstruction algorithm as an example, the user imports the facial video to be processed into the software, and can start according to a special effect selected by the user, or, starting from the user importing the facial video into the software, the software identifies the facial feature areas of each frame of the facial image in the facial video, and uses multiple vertices and line segments connecting multiple vertices to form multiple triangular faces. Through three-dimensional changes, such as affine transformation, rotation, and scaling, a reconstructed three-dimensional face model is constructed, and finally the reconstructed three-dimensional face model is displayed in the model display area of ​​the software.

[0135] Step B: Determine first deformation degree information of the reconstructed three-dimensional face model relative to the standard three-dimensional face model.

[0136] Among them, the first deformation degree information is expressed as the displacement of the vertex position coordinates of the reconstructed face 3D model relative to the vertex position coordinates of the standard face 3D model. For each vertex of the face 3D model, the displacement of the vertex between the vertex position coordinates of the reconstructed face 3D model and the corresponding vertex position coordinates of the standard face 3D model is calculated; the displacements corresponding to multiple vertices in the face 3D model represent the deformation degree of the reconstructed face 3D model relative to the standard face 3D model, that is, the first deformation degree information is obtained based on the calculation result.

[0137] For example, in order to distinguish the vertices of the reconstructed 3D face model from the vertices of the standard 3D face model, the vertices of the reconstructed 3D face model can be expressed as first vertices, and the position coordinates of the first vertex in the camera coordinate system can be expressed as first position information; the vertices of the standard 3D face model can be expressed as second vertices, and the coordinates of the second vertex in the model coordinate system can be expressed as second position information. The corresponding relationship between the first vertex and the second vertex has been expressed in the above text, and will not be repeated here.

[0138] In one embodiment provided by the present disclosure, obtaining the first deformation degree information includes but is not limited to the following steps b1 to b3:

[0139] Step b1: converting first position information of the plurality of first vertices in the camera coordinate system into third position information in the model coordinate system.

[0140] Among them, the camera coordinate system is represented by the optical center of the camera as the origin of the coordinate system, the optical axis of the camera as the Z axis, and the X axis and Y axis are parallel to the X axis and Y axis of the image coordinate system where the face image is located; the model coordinate system is represented by the absolute coordinates of the three-dimensional face model in the model space. The camera coordinate system can be obtained by rotation and translation of the model coordinate system.

[0141] Exemplarily, the conversion process includes, for example:

[0142] The coordinate values ​​corresponding to the first position information of the first vertex in the camera coordinate system are normalized according to the orthogonal projection method. The first vertex of the reconstructed face 3D model is transformed from the camera coordinate system to the model coordinate system using rigid body transformation to obtain the third position information of the multiple first vertices in the model coordinate system.

[0143] In a specific implementation, coordinate value normalization can be understood as converting coordinate values ​​in different coordinate systems into coordinate values ​​in a unified coordinate system, that is, converting the coordinate values ​​of the reconstructed three-dimensional face model in the camera coordinate system into coordinate values ​​in the model coordinate system.

[0144] Rigid body transformation can be understood as changing the spatial position (translation) or orientation (rotation) of the reconstructed 3D face model without changing the shape of the reconstructed 3D face model, that is, the relative positions of the first vertices of the reconstructed 3D face model do not change.

[0145] In this way, the first position information of the first vertex in the camera coordinate system can be converted into the third position information in the model coordinate system according to the conversion process in the above manner.

[0146] Following the above step b1, the specific method for obtaining the first deformation degree information in the embodiment of the present disclosure further includes:

[0147] b2: Determine the displacement amounts corresponding to the plurality of first vertices based on the third position information corresponding to the plurality of first vertices and the second position information corresponding to the plurality of second vertices.

[0148] The third position information may be expressed as the position coordinates of the first vertex in the model coordinate system; and the second position information may be expressed as the position coordinates of the second vertex in the model coordinate system.

[0149] Exemplarily, in the model coordinate system, the position coordinates of the second vertex are compared with the position coordinates of the first vertex corresponding to the second vertex to obtain the displacement of the first vertex relative to the second vertex, which can be expressed as calculating the coordinate difference between the two vertices. For example, in the model coordinate system, the coordinates of a first vertex are (X1, Y1, Z1), and the position coordinates of a second vertex are (X2, Y2, Z2), then the displacement of the first vertex relative to the second vertex is (X2-X1, Y2-Y1, Z2-Z1), which is expressed as taking the position coordinates of the second vertex of the standard human face 3D model in the model coordinate system as the starting point and the position coordinates of the first vertex of the reconstructed human face 3D model in the model coordinate system as the end point, and calculating the coordinate difference between the starting point and the end point.

[0150] b3: constructing the first deformation degree information based on the displacement amounts corresponding to the plurality of first vertices respectively.

[0151] Exemplarily, a calculation result is obtained according to the displacement of the first vertex and the second vertex respectively, and the calculation result represents the degree of deformation of the reconstructed three-dimensional face model relative to the standard three-dimensional face model, that is, the first deformation degree information is obtained according to the calculation result. For example, the calculation result can be stored as an array, which includes the coordinate differences between all the first vertices of the reconstructed three-dimensional face model and all the second vertices of the corresponding standard three-dimensional face model, and can also be expressed as the degree of deformation of the reconstructed three-dimensional face model relative to the standard three-dimensional face model.

[0152] Step C: Determine second deformation degree information of the target human face 3D model relative to the standard human face 3D model.

[0153] Since the target human face 3D model and the standard human face 3D model are both in the model coordinate system, there is no need to perform a coordinate system conversion process to obtain the second deformation degree information.

[0154] Exemplarily, in the model coordinate system, obtaining the second deformation degree information is similar to the above steps b2 to b3, except that the calculation result is the difference between the vertex coordinates of the target face 3D model and the vertex coordinates of the standard face 3D model. The calculation result characterizes the target deformation effect of the target face 3D model relative to the standard face 3D model, that is, the second deformation degree information is obtained according to the calculation result. For example, similar to the above first deformation degree information, the calculation result can be stored as an array, which includes the difference between all vertex coordinates of the target face 3D model and all vertex coordinates of the standard face 3D model, and can also be expressed as the deformation degree of the target face 3D model relative to the standard face 3D model.

[0155] Step D: Based on the first deformation degree information, the second deformation degree information, the position of each frame of face image in the face image sequence, and the number of face images included in the face image sequence, determine the deformation degree information of the first face in each frame of face image.

[0156] The first deformation degree information and the second deformation degree information are obtained through the above steps B to C, which will not be described in detail here.

[0157] Exemplarily, the facial image sequence can be represented as a combination of multiple facial images in a facial video in a time sequence, and each facial image frame can be numbered in time sequence to obtain an image sequence number representing each facial image frame in the facial image sequence and the total number of facial images in the facial image sequence. According to the proportional relationship between the image sequence number and the total number, as well as the first deformation degree information and the second deformation degree information, the deformation degree information of the first face in each facial image frame is obtained.

[0158] With respect to the above S103, a deformed face three-dimensional model representing the deformation process can be generated according to the deformation degree information corresponding to the first face of any frame face image in the face video, and a target deformed image in the face image of the corresponding frame number can be generated according to the three-dimensional face model.

[0159] In one embodiment provided by the present disclosure, a deformed three-dimensional face model corresponding to each frame of the face image is generated based on the deformation degree information and the standard three-dimensional face model; the first face in each frame of the face image is deformed based on the deformed three-dimensional face model to obtain a target deformed image corresponding to each frame of the face image.

[0160] For example, the vertex position coordinates of the deformed face 3D model of the current frame can be determined according to the deformation degree information of the current frame and the vertex position coordinates of the standard face 3D model. Specifically, it can be obtained by the following formula (1):

[0161] Render_Mesh=Basic_Mesh+Offset_Face+Offset_Morph×Ic / N(1)

[0162] Among them, Render_Mesh represents the vertex position coordinates of the deformed face 3D model; Basic_Mesh represents the vertex position coordinates of the standard face 3D model; Offset_Face represents the vertex displacement of the reconstructed face 3D model relative to the standard face 3D model; Offset_Morph represents the vertex displacement of the target face 3D model relative to the standard face 3D model; Ic represents the image sequence number of the current frame face image in the face image sequence that constitutes the face video, or represents the sequence number difference between the current frame face image and the first frame face image that performs face deformation in the face video; N represents the number of face images in the face image sequence.

[0163] It can be understood that Basic_Mesh+Offset_Face can obtain a reconstructed three-dimensional face model; Offset_Morph×Ic / N represents the deformation degree of the target deformation effect.

[0164] Exemplarily, when Ic / N approaches 1, it means that the vertex position coordinates of the deformed 3D face model are close to the target 3D face model, and when Ic / N approaches 0, it means that the vertex position coordinates of the deformed 3D face model are close to the vertex position coordinates of the reconstructed 3D face model.

[0165] Furthermore, the first face in each frame of the face image is deformed based on the deformable face three-dimensional model to obtain a target deformed image corresponding to each frame of the face image.

[0166] See also Figure 2 FIG. 1 is a flow chart of a method for acquiring a target deformed image in an embodiment of the image processing method provided by the present disclosure, and the flow chart includes the following S201 to S204:

[0167] S201: Convert the deformed 3D face model from the model coordinate system to the camera coordinate system corresponding to the reconstructed 3D face model, and obtain fourth position information of multiple third vertices in the deformed 3D face model in the camera coordinate system.

[0168] Here, the deformed 3D face model is converted to the camera coordinate system, and the displacement between the third vertex of the deformed 3D face model and the first vertex of the reconstructed 3D face model in the camera coordinate system can be calculated.

[0169] Exemplarily, when the deformed 3D face model does not deform, different angles are rotated around the X-axis, Y-axis, and Z-axis of the deformed 3D face model, and corresponding rotation matrices are obtained. The position information of the deformed 3D face model in the camera coordinate system is obtained according to the rotation matrix.

[0170] S202: Based on the fourth position information, project the plurality of third vertices respectively into each frame of the face image to obtain positions of projection points corresponding to the plurality of third vertices respectively in each frame of the face image.

[0171] Here, the three-dimensional coordinates of the third vertex of the deformed three-dimensional face model can be transformed into two-dimensional coordinates by projection, so that the position of the projection point of the third vertex on the two-dimensional image relative to the face image can be obtained.

[0172] S203: Determine the displacement information of each pixel point in each frame of the face image based on the positions of the projection points corresponding to the multiple third vertices in each frame of the face image and the positions of the pixel points corresponding to the first vertices corresponding to the multiple third vertices in each frame of the face image.

[0173] Exemplarily, the vertices of the deformed 3D face model and the reconstructed 3D face model can be projected onto the 2D face image, that is, the vertex position coordinates in the camera coordinate system are converted into vertex position coordinates in the pixel coordinate system, and the vertex displacement of the deformed 3D face model relative to the reconstructed 3D face model in the pixel coordinate system is calculated to determine the displacement information of the pixel point in each frame of the face image that has the same coordinate position as the first vertex of the reconstructed 3D face model.

[0174] S204: adjusting the position of each pixel in each frame of the face image based on the displacement information to obtain a target deformed image corresponding to each frame of the face image.

[0175] Exemplarily, based on the displacement information, in the pixel coordinate system, the position of each pixel in each frame of the face image is adjusted to be the same as the vertex position coordinates of the deformed face 3D model of that frame, thereby obtaining a target deformed image corresponding to each frame of the face image.

[0176] Through the above method, each frame of facial image can be transformed to obtain a target deformed image corresponding to each frame of facial image, wherein the target deformed image has a certain degree of features of the first face and part of the features of the target deformation effect, and the degree of the two features contained therein is related to the position of each frame of facial image in the facial video.

[0177] See also Figure 3 FIG. 1 is a flow chart of another method of acquiring a target deformed image in the image processing method of an embodiment of the present disclosure, and the flow chart includes but is not limited to the following steps S301 to S302:

[0178] S301: Convert the deformed 3D face model from a model coordinate system to a camera coordinate system corresponding to the reconstructed 3D face model to obtain a converted 3D face model.

[0179] Here, the vertex coordinates of the transformed three-dimensional face model can be directly projected onto the two-dimensional face image in the camera coordinate system.

[0180] S302: Rendering the transformed three-dimensional face model using each frame of the face image to obtain the target deformed image.

[0181] Specifically, based on the mapping relationship between the pixel points in each frame of the facial image and the first vertex in the reconstructed three-dimensional facial model, the pixel values ​​corresponding to each fourth vertex in the converted three-dimensional facial model are determined; model rendering is performed based on the pixel values ​​corresponding to each fourth vertex in the converted three-dimensional facial model to obtain the target deformed image.

[0182] Exemplarily, the mapping relationship can be expressed as a plurality of first vertices in the reconstructed face 3D model, which correspond to face pixels in each frame of the face image, for example, corresponding to the pixel values ​​of the pixels. In this way, since the transformed face 3D model is a deformed face 3D model generated according to the change of the first vertex of the reconstructed face 3D model, and then the coordinate system is transformed, that is, the fourth vertex of the transformed face 3D model corresponds to the first vertex of the reconstructed face 3D model, and there is a displacement between the fourth vertex and the first vertex in the same camera coordinate system, therefore, the pixel value corresponding to the face pixel in the face image corresponding to the first vertex can be assigned to the corresponding fourth vertex, so that the model is rendered according to the pixel values ​​corresponding to each fourth vertex in the transformed face 3D model, and the target deformed image is obtained.

[0183] In order to make the deformation effect richer, the corresponding special effect map can also be matched to the target face 3D model according to the target effect. The special effect map represents the texture of the face 3D model, and presents the corresponding texture when attached to the face 3D model.

[0184] In one embodiment provided by the present disclosure, the transformed face three-dimensional model is rendered using each frame of the face image and the special effect map to obtain a target deformed image.

[0185] Specifically, the specific method of rendering the converted face 3D model to obtain the target deformed image includes the following steps E to F:

[0186] Step E: Based on the position of each frame of the face image in the face video and the number of face images included in the face video, each frame of the face image and the special effect map are merged to obtain a target effect map.

[0187] For example, the user can select a special effect sticker in the sticker interface of the application loaded with the method according to his / her own preferences. The special effect sticker can be, for example, an image of a small animal such as a kitten or a puppy; or a decorative image such as a hat, glasses, or hair. Figure 4 An example of a special effect map is shown, where the special effect map is a "Shiba Inu" image. The specific map content can be selected according to the actual application, and will not be described here.

[0188] After the special effect map is selected, the current frame face image and the special effect map are fused according to the fusion ratio to obtain the target effect map.

[0189] Wherein, obtaining the target effect map includes the following steps e1 to e3:

[0190] Step e1: Based on the position of each face image frame in the face video and the number of face images included in the face video, determine the fusion ratio corresponding to the special effect map when fusing each face image frame with the special effect map.

[0191] For example, the fusion ratio can be determined by the following formula (2):

[0192] Blend_Alpha=Ic / N(2)

[0193] Wherein, Blend_Alpha represents the fusion ratio; Ic represents the image sequence number of the current frame face image in the face image sequence; and N represents the number of face images in the face image sequence.

[0194] For example, see Figure 5a to Figure 5d As shown in FIG. , it is an example of the fusion processing of the current frame face image and the special effect map, where: Figure 5a It is an example of the fusion processing of the face image and the special effect map at the beginning of deformation. It can be seen that at the beginning of deformation, since the current face image is the first face image in the face image sequence, Ic / N is equal to 0. Figure 5a In the image, the face image and the special effect map are not fused.

[0195] Figure 5d It is an example of the fusion process of the face image and the special effect map at the end of the deformation. It can be seen that at the end of the deformation, since the current face image is the last face image in the face image sequence, Ic / N is equal to 1. Figure 5d In the image, the face image is completely integrated with the special effect map.

[0196] Figure 5b to Figure 5c They represent example images of the fusion process of face image and special effect map, assuming Figure 5bThe position of the current face image in the face image sequence is Ic1, Figure 5c The position of the current face image in the face image sequence is Ic2, and Ic1 is less than Ic2. Since Ic1 is less than Ic2, Ic1 / N is less than Ic2 / N. It can be seen that Figure 5c Compared to Figure 5b , the fusion effect of its special effect maps is more saturated.

[0197] Step e2: Based on the mapping relationship between the pixel points in each frame of the facial image and the first vertex of the reconstructed three-dimensional facial model, the mapping relationship between the pixel points in the special effect map and the fifth vertex of the target three-dimensional facial model, and the vertex correspondence between the reconstructed three-dimensional facial model and the target three-dimensional facial model, determine the pixel point correspondence between each frame of the facial image and the special effect map.

[0198] Exemplarily, based on the mapping relationship between the pixel points in each frame of the facial image and the first vertex of the reconstructed three-dimensional facial model, the pixel value information of the pixel points in the facial image corresponding to the multiple first vertices in the reconstructed three-dimensional facial model can be obtained; based on the mapping relationship between the pixel points in the special effects map and the fifth vertex of the target three-dimensional facial model, the pixel value information of the pixel points in the special effects map corresponding to the multiple fifth vertices in the target three-dimensional facial model can be obtained; based on the displacement information of the vertices between the reconstructed three-dimensional facial model and the target three-dimensional facial model, the coordinate positions of the pixel points of the special effects map can be projected onto each frame of the facial image.

[0199] Step e3: Based on the pixel correspondence and the fusion ratio, the pixel values ​​of the pixels in each frame of the face map and the pixel values ​​of each pixel in the special effect map are fused to obtain the target effect map.

[0200] Exemplarily, the coordinate positions of the pixels of the special effect map are projected onto each frame of the face image, and then the color saturation of the special effect map projected onto the face image is obtained based on the fusion ratio, and finally the target effect map corresponding to each frame of the face image is obtained.

[0201] Following the above step E, the rendering process further includes:

[0202] Step F: Utilizing the target effect map, rendering the transformed face three-dimensional model to obtain the target deformed image.

[0203] Exemplarily, the target effect map is used to render the transformed face 3D model corresponding to the current frame face image to obtain a target deformed image.

[0204] Through the above process, the first face in the face video and the target effect map can be merged to obtain a target deformed image with facial features and map features, so that based on the target deformation effects corresponding to multiple face images, a face special effects video can be obtained, which can show the dynamic change process of transforming the first face into a second face with the target deformation effect.

[0205] The present disclosure also provides a specific example of an image processing method, the method comprising the following steps G to H:

[0206] Step G: Material preparation. Taking the execution subject as material production software as an example, the steps specifically include the following steps g1 to g3:

[0207] Step g1: Use the facial mesh reconstruction algorithm to select a frontal face image with appropriate facial expression from the standard face database and generate a face 3D model as the standard face 3D model BasicMesh. The standard face 3D model is the basis for the facial mesh reconstruction algorithm to perform 3D reconstruction on different faces. By rotating and scaling the standard face 3D model as a whole and adjusting the relative positions of each vertex in the model, a reconstructed face 3D model can be obtained for the position, orientation, and shape of the face in the face video uploaded by the user.

[0208] Step g2: Based on the standard 3D face model mentioned in step g1 above, the artist constructs a target 3D face model by adjusting the vertices without adjusting the topological structure of the model, and matches it with a suitable target map according to the deformation effect selected by the user to generate the target deformation material.

[0209] Step g3: Special effects material production of human face The number of frames N required for the entire process of the target deformation material mentioned in step g2 above is specified according to the specific application.

[0210] Following the above step G, after the material preparation stage is completed, a specific example of the image processing method further includes:

[0211] Step H: actual rendering. Taking the execution subject as image processing software as an example, in the image processing software, the user can upload the face video that he has taken. When the user triggers a certain target special effect of the image processing software, the software calls the target deformation material corresponding to the target special effect to render the face video. The step H can be specifically implemented by executing the following steps h1 to h6:

[0212] h1: Analyze the standard face 3D model prepared in step G, the target face 3D model, and the target map prepared according to the deformation effect selected by the user, calculate the displacement of each vertex in the target 3D model relative to each vertex in the standard face 3D model, and record the displacement of each corresponding vertex in an array named Offset_Morph, and store this array in memory. Record the initial frame face image Is that starts rendering based on the target deformation material.

[0213] h2: Utilize AI (artificial intelligence) algorithm to analyze and process each face frame in the face video, and generate a reconstructed face 3D model for the current face frame. Here, the reconstructed face 3D model is in the camera coordinate system.

[0214] h3: Normalize the coordinate values ​​of each vertex in the reconstructed 3D face model according to the orthogonal projection algorithm, and then use the displacement matrix to transform the reconstructed 3D face model from the camera coordinate system to the model coordinate system, and calculate the displacement between the vertices of the reconstructed 3D face model in the model coordinate system relative to the vertices of the standard 3D face model, and record the calculation results in an array named Offset_Face.

[0215] h4: Record the current frame face image Ic and calculate the point position of the deformed face 3D model RenderMesh currently used for rendering in the standard coordinate system based on formula (3):

[0216] Render_Mesh=Basic_Mesh+Offset_Face+Offset_Morph×(Ic-Is) / N (3)

[0217] h5: Calculate the fusion ratio Blend_Alpha of the current target texture as shown in the following formula (4):

[0218] Blend_Alpha=(Ic-Is) / N (4)

[0219] h6: Use the inverse matrix of the displacement matrix to calculate the result of the current deformed face 3D model back to the camera coordinate system, and mix the target map and the current frame face image according to the fusion ratio to render the current frame deformed face 3D model.

[0220] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0221] Based on the same inventive concept, an image processing device corresponding to the image processing method is also provided in the embodiment of the present disclosure. Since the principle of solving the problem by the device in the embodiment of the present disclosure is similar to the above-mentioned image processing method in the embodiment of the present disclosure, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0222] Reference Figure 6 FIG. 1 is a schematic diagram of an image processing device provided by an embodiment of the present disclosure, wherein the device comprises: an acquisition module 61, a processing module 62, and a generation module 63; wherein:

[0223] An acquisition module 61 is used to acquire a face video including a first face, a standard face 3D model, and a target face 3D model; the target face 3D model is a 3D model obtained by deforming the standard face 3D model based on a target deformation effect;

[0224] The processing module 62 is used to determine, for each face image frame in the face video, deformation degree information for deforming the first face in each face image frame based on the position of each face image frame in the face video, the number of face images included in the face video, the standard three-dimensional face model, and the target three-dimensional face model;

[0225] The generating module 63 is used to perform deformation processing on the first face in each frame of the face image based on the deformation degree information to obtain a target deformed image corresponding to each frame of the face image.

[0226] In an optional implementation, the device further includes an interception module, wherein the interception module is configured to:

[0227] In response to triggering, deforming the first face into a second face based on the target deformation effect, acquiring an original video including the first face;

[0228] Based on preset image quantity information, the face video is captured from the original video.

[0229] In an optional implementation, the processing module 62 is used to:

[0230] Performing three-dimensional face reconstruction on each frame of the face image to obtain a reconstructed three-dimensional face model;

[0231] Determining first deformation degree information of the reconstructed human face 3D model relative to the standard human face 3D model; and determining second deformation degree information of the target human face 3D model relative to the standard human face 3D model;

[0232] Based on the first deformation degree information, the second deformation degree information, the position of each frame of facial image in the facial image sequence corresponding to the facial video, and the number of facial images included in the facial image sequence, the deformation degree information of the first face in each frame of facial image is determined.

[0233] In an optional implementation, the device further includes a vertex displacement calculation module 64, and the vertex displacement calculation module 64 is used to:

[0234] Converting first position information of the first vertices in the camera coordinate system into third position information in the model coordinate system;

[0235] Determine the displacements corresponding to the plurality of first vertices based on the third position information corresponding to the plurality of first vertices and the second position information corresponding to the plurality of second vertices;

[0236] The first deformation degree information is constructed based on the displacement amounts respectively corresponding to the plurality of first vertices.

[0237] In an optional implementation manner, the generating module 63 is further configured to:

[0238] The first face in each frame of the face image is deformed based on the deformed three-dimensional face model to obtain a target deformed image corresponding to each frame of the face image.

[0239] In an optional implementation manner, the vertex displacement calculation module 64 is further used for:

[0240] Converting the deformed 3D face model from a model coordinate system to a camera coordinate system corresponding to the reconstructed 3D face model to obtain fourth position information of a plurality of third vertices in the deformed 3D face model in the camera coordinate system;

[0241] Based on the fourth position information, projecting the plurality of third vertices respectively into each frame of the face image, and obtaining positions of projection points respectively corresponding to the plurality of third vertices in each frame of the face image;

[0242] Determine the displacement information of each pixel in each frame of the face image based on the positions of the projection points corresponding to the plurality of third vertices in each frame of the face image and the positions of the pixel points corresponding to the first vertices corresponding to the plurality of third vertices in each frame of the face image;

[0243] The position of each pixel in each frame of the face image is adjusted based on the displacement information to obtain a target deformed image corresponding to each frame of the face image.

[0244] In an optional implementation manner, the device further includes a rendering module 66, and the rendering module 66 is used to:

[0245] Converting the deformed 3D face model from a model coordinate system to a camera coordinate system corresponding to the reconstructed 3D face model to obtain a converted 3D face model;

[0246] The transformed face three-dimensional model is rendered using each frame of the face image to obtain the target deformed image.

[0247] In an optional implementation manner, the rendering module 66 is further used for:

[0248] The step of rendering the transformed face 3D model using each frame of the face image to obtain the target deformed image includes:

[0249] The transformed face three-dimensional model is rendered using each frame of the face image and the special effect map to obtain the target deformed image.

[0250] In an optional implementation manner, the rendering module 66 is further used for:

[0251] Based on the position of each frame of the face image in the face video and the number of face images included in the face video, fusing each frame of the face image with the special effect map to obtain a target effect map;

[0252] The target effect map is used to render the transformed three-dimensional face model to obtain the target deformed image.

[0253] In an optional implementation, the device further includes a fusion module 65, and the fusion module 65 is used to:

[0254] Determine a pixel correspondence relationship between each frame of the face image and the special effect map based on a mapping relationship between a pixel in each frame of the face image and a first vertex of the reconstructed three-dimensional face model, a mapping relationship between a pixel in the special effect map and a fifth vertex of the target three-dimensional face model, and a vertex correspondence relationship between the reconstructed three-dimensional face model and the target three-dimensional face model;

[0255] Based on the pixel correspondence and the fusion ratio, the pixel values ​​of the pixels in each frame of the face map and the pixel values ​​of each pixel in the special effect map are fused to obtain the target effect map.

[0256] For descriptions of the processing flow of each module in the device and the interaction flow between each module, reference may be made to the relevant descriptions in the above method embodiment, which will not be described in detail here.

[0257] The present disclosure also provides a computer device, such as Figure 7 FIG. 1 is a schematic diagram of a computer device structure provided in an embodiment of the present disclosure, including:

[0258] A processor 71 and a memory 72; the memory 72 stores machine-readable instructions executable by the processor 71, and the processor 71 is used to execute the machine-readable instructions stored in the memory 72. When the machine-readable instructions are executed by the processor 71, the processor 71 performs the following steps:

[0259] Acquire a face video containing a first face, a standard face 3D model, and a target face 3D model; the target face 3D model is a 3D model obtained by deforming the standard face 3D model based on a target deformation effect;

[0260] For each face image frame in the face video, based on the position of each face image frame in the face video, the number of face images included in the face video, the standard face 3D model, and the target face 3D model, determine deformation degree information for deforming a first face in each face image frame;

[0261] Based on the deformation degree information, a deformation process is performed on the first face in each frame of the face image to obtain a target deformed image corresponding to each frame of the face image.

[0262] The above-mentioned memory 72 includes internal memory 721 and external memory 722; the memory 721 here is also called internal memory, which is used to temporarily store the calculation data in the processor 71, as well as the data exchanged with the external memory 722 such as the hard disk. The processor 71 exchanges data with the external memory 722 through the internal memory 721.

[0263] The specific execution process of the above instructions can refer to the steps of the image processing method described in the embodiment of the present disclosure, which will not be repeated here.

[0264] The present disclosure also provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the steps of the image processing method described in the above method embodiment are executed. The storage medium can be a volatile or non-volatile computer-readable storage medium.

[0265] The embodiments of the present disclosure also provide a computer program product, which carries a program code. The instructions included in the program code can be used to execute the steps of the image processing method described in the above method embodiment. For details, please refer to the above method embodiment, which will not be repeated here.

[0266] The computer program product may be implemented in hardware, software or a combination thereof. In one optional embodiment, the computer program product is implemented as a computer storage medium. In another optional embodiment, the computer program product is implemented as a software product, such as a software development kit (SDK).

[0267] The present disclosure relates to the field of augmented reality. By acquiring the image information of the target object in the real environment, the relevant features, states and attributes of the target object are detected or identified by various visual related algorithms, so as to obtain an AR effect that combines virtuality and reality and matches the specific application. Exemplarily, the target object may involve the face, limbs, gestures, actions, etc. related to the human body, or markers, markers related to objects, or sand tables, display areas or display items related to venues or places. Visual related algorithms may involve visual positioning, SLAM, three-dimensional reconstruction, image registration, background segmentation, key point extraction and tracking of objects, and position or depth detection of objects. Specific applications may not only involve interactive scenes such as guided tours, navigation, explanations, reconstruction, virtual effect superposition and display related to real scenes or objects, but also special effects processing related to people, such as makeup beautification, limb beautification, special effects display, virtual model display and other interactive scenes. The relevant features, states and attributes of the target object can be detected or identified by a convolutional neural network. The above-mentioned convolutional neural network is a network model obtained by model training based on a deep learning framework.

[0268] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the several embodiments provided in the present disclosure, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the device or unit can be electrical, mechanical or other forms.

[0269] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0270] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0271] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0272] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present disclosure, which are used to illustrate the technical solutions of the present disclosure, rather than to limit them. The protection scope of the present disclosure is not limited thereto. Although the present disclosure is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed in the present disclosure, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. An image processing method, characterized in that: include: Acquire a face video including a first face, a standard face 3D model, and a target face 3D model; The target human face three-dimensional model is a three-dimensional model obtained by deforming the standard human face three-dimensional model based on the target deformation effect; For each face image frame in the face video, based on the position of each face image frame in the face video, the number of face images included in the face video, the standard face 3D model, and the target face 3D model, determine deformation degree information for deforming a first face in each face image frame; Based on the deformation degree information, a deformation process is performed on the first face in each frame of the face image to obtain a target deformed image corresponding to each frame of the face image.

2. The method according to claim 1, characterized in that Get the face video containing the first face, including: In response to triggering, deforming the first face into a second face based on the target deformation effect, acquiring an original video including the first face; Based on preset image quantity information, the face video is captured from the original video.

3. The method according to claim 1, characterized in that: The step of determining deformation degree information for deforming the first face in each frame of the face image based on the position of each frame of the face image in the face video, the number of face images included in the face video, the standard face 3D model, and the target face 3D model comprises: Performing three-dimensional face reconstruction on the first face in each frame of face image to obtain a reconstructed three-dimensional face model corresponding to each frame of face image; Determining first deformation degree information of the reconstructed human face 3D model relative to the standard human face 3D model; and determining second deformation degree information of the target human face 3D model relative to the standard human face 3D model; Based on the first deformation degree information, the second deformation degree information, the position of each frame of facial image in the facial image sequence corresponding to the facial video, and the number of facial images included in the facial image sequence, the deformation degree information of the first face in each frame of facial image is determined.

4. The method according to claim 3, characterized in that The reconstructed 3D face model includes: a plurality of first vertices, and first position information of the plurality of first vertices in a camera coordinate system corresponding to each frame of the face image; the standard 3D face model includes: second vertices corresponding to the plurality of first vertices, and second position information of the second vertices in a model coordinate system; The determining of first deformation degree information of the reconstructed three-dimensional face model relative to the standard three-dimensional face model includes: Converting first position information of the first vertices in the camera coordinate system into third position information in the model coordinate system; Determine the displacements corresponding to the plurality of first vertices based on the third position information corresponding to the plurality of first vertices and the second position information corresponding to the plurality of second vertices; The first deformation degree information is constructed based on the displacement amounts respectively corresponding to the plurality of first vertices.

5. The method according to any one of claims 1 to 4, characterized in that: The deforming process is performed on the first face in each frame of the face image based on the deformation degree information to obtain a target deformed image corresponding to each frame of the face image, including: Based on the deformation degree information and the standard human face three-dimensional model, generating a deformed human face three-dimensional model corresponding to each frame of the human face image; The first face in each frame of the face image is deformed based on the deformed three-dimensional face model to obtain a target deformed image corresponding to each frame of the face image.

6. The method according to claim 5, characterized in that The deforming the first face in each frame of the face image based on the deformed face three-dimensional model to obtain a target deformed image corresponding to each frame of the face image includes: Converting the deformed 3D face model from a model coordinate system to a camera coordinate system corresponding to the reconstructed 3D face model to obtain fourth position information of a plurality of third vertices in the deformed 3D face model in the camera coordinate system; Based on the fourth position information, projecting the plurality of third vertices respectively into each frame of the face image, and obtaining positions of projection points respectively corresponding to the plurality of third vertices in each frame of the face image; Determine the displacement information of each pixel in each frame of the face image based on the positions of the projection points corresponding to the plurality of third vertices in each frame of the face image and the positions of the pixel points corresponding to the first vertices corresponding to the plurality of third vertices in each frame of the face image; The position of each pixel in each frame of the face image is adjusted based on the displacement information to obtain a target deformed image corresponding to each frame of the face image.

7. The method according to claim 5, characterized in that The deforming process is performed on the first face in each frame of the face image based on the deformed face three-dimensional model to obtain a target deformed image corresponding to each frame of the face image, comprising: Converting the deformed 3D face model from a model coordinate system to a camera coordinate system corresponding to the reconstructed 3D face model to obtain a converted 3D face model; The transformed face three-dimensional model is rendered using each frame of the face image to obtain the target deformed image.

8. The method according to claim 7, characterized in that Rendering the transformed face three-dimensional model using each frame of the face image to obtain the target deformed image includes: Determine pixel values ​​corresponding to each fourth vertex in the converted three-dimensional face model based on a mapping relationship between pixel points in each frame of the face image and the first vertex in the reconstructed three-dimensional face model; Model rendering is performed based on the pixel values ​​corresponding to each fourth vertex in the converted three-dimensional face model to obtain the target deformed image.

9. The method according to claim 7 or 8, characterized in that: Also includes: Obtaining a special effects map corresponding to the target face three-dimensional model; The step of rendering the transformed face three-dimensional model using each frame of the face image to obtain the target deformed image includes: The transformed face three-dimensional model is rendered using each frame of the face image and the special effect map to obtain the target deformed image.

10. The method according to claim 9, characterized in that The step of rendering the transformed face three-dimensional model by using each frame of the face image and the special effect map to obtain the target deformed image includes: Based on the position of each frame of the face image in the face video and the number of face images included in the face video, fusing each frame of the face image with the special effect map to obtain a target effect map; The target effect map is used to render the transformed three-dimensional face model to obtain the target deformed image.

11. The method according to claim 10, characterized in that Based on the position of each frame of the face image in the face video and the number of face images included in the face video, each frame of the face image and the special effect map are fused to obtain a target effect map, including: Determine, based on the position of each frame of the face image in the face video and the number of face images included in the face video, a fusion ratio corresponding to the special effect map when fusion processing is performed on each frame of the face image and the special effect map; Determine a pixel correspondence relationship between each frame of the face image and the special effect map based on a mapping relationship between a pixel in each frame of the face image and a first vertex of the reconstructed three-dimensional face model, a mapping relationship between a pixel in the special effect map and a fifth vertex of the target three-dimensional face model, and a vertex correspondence relationship between the reconstructed three-dimensional face model and the target three-dimensional face model; Based on the pixel correspondence and the fusion ratio, the pixel values ​​of the pixels in each frame of the face map and the pixel values ​​of each pixel in the special effect map are fused to obtain the target effect map.

12. An image processing device, characterized in that: include: A first acquisition module, used to acquire a face video including a first face, a standard face 3D model, and a target face 3D model; The target human face three-dimensional model is a three-dimensional model obtained by deforming the standard human face three-dimensional model based on the target deformation effect; a processing module, configured to determine, for each face image frame in the face video, deformation degree information for deforming a first face in each face image frame based on a position of each face image frame in the face video, a number of face images included in the face video, the standard three-dimensional face model, and the target three-dimensional face model; A generating module is used to perform deformation processing on the first face in each frame of face image based on the deformation degree information to obtain a target deformed image corresponding to each frame of face image.

13. A computer device, characterized in that: include: A processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and the processor is used to execute the machine-readable instructions stored in the memory. When the machine-readable instructions are executed by the processor, the processor performs the steps of the image processing method according to any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program. When the computer program is executed by a computer device, the computer device executes the steps of the image processing method according to any one of claims 1 to 11.