Image processing method and related device

Abstract

The embodiment of the present application discloses a kind of image processing method and related device, and the reference control point is obtained by editing face deformation effect to reference face image Reference offset, and the reference face scale in reference face image is obtained.In the face of input face image, face relative to the face of reference face image occurs side face rotation, when face deformation effect is applied to input face image, input face scale, and horizontal axis unit direction vector and vertical axis unit direction vector in two-dimensional coordinate system are determined;With reference face scale, input face scale, horizontal axis unit direction vector and vertical axis unit direction vector, corresponding adjustment reference offset is carried out according to left half face area, middle area and right half face area, to obtain the first offset of corresponding input control point;Through the original coordinate of input control point and the first offset, two-dimensional face deformation processing is carried out to input face image, to obtain the deformation face image of applying face deformation effect.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to an image processing method and related apparatus. Background Technology

[0002] Currently, in scenarios such as short videos or live streaming, users can apply facial effects or facial beautification and other facial deformation effects. Generally, traditional two-dimensional facial deformation methods are used to edit the original facial image to obtain a deformed facial image with the above-mentioned facial deformation effects.

[0003] However, in scenarios such as short videos or live streaming, users often rotate their faces from various angles. For face images with side profile rotation, the traditional two-dimensional face deformation method mentioned above cannot adapt to the situation of side profile rotation, resulting in face distortion, that is, the face deformation effect is not natural and smooth enough. Summary of the Invention

[0004] To address the aforementioned technical problems, this application provides an image processing method and related apparatus that, when applying a face deformation effect, adapts to the rotation of the face to avoid facial distortion and make the face deformation effect more natural and smooth.

[0005] The embodiments of this application disclose the following technical solutions:

[0006] On one hand, this application provides an image processing method, the method comprising:

[0007] When applying a face deformation effect to an input face image, the reference offset of the reference control point in the reference face image is obtained; the face in the input face image rotates to the side relative to the face in the reference face image, and the reference offset is obtained by editing the face deformation effect on the reference face image;

[0008] Determine the input face scale corresponding to the reference face scale, as well as the horizontal axis unit direction vector and the vertical axis unit direction vector in the two-dimensional coordinate system; the reference face scale includes the preset scale of the left half face region, the middle region and the right half face region in the reference face image, and the two-dimensional coordinate system takes the input face rotation direction vector in the input face image as the vertical axis;

[0009] Based on the reference face scale, the input face scale, the horizontal axis unit direction vector, and the vertical axis unit direction vector, the reference offset is adjusted corresponding to the left half face region, the middle region, and the right half face region to obtain the first offset of the input control point corresponding to the reference control point;

[0010] Based on the original coordinates of the input control point and the first offset of the input control point, the input face image is subjected to two-dimensional face deformation processing to obtain a deformed face image with the face deformation effect applied.

[0011] On the other hand, this application provides an image processing apparatus, the apparatus comprising: an acquisition unit, a determination unit, an adjustment unit, and a deformation unit;

[0012] The acquisition unit is used to acquire the reference offset of the reference control point in the reference face image when applying a face deformation effect to the input face image; the face in the input face image undergoes a side-view rotation relative to the face in the reference face image, and the reference offset is obtained by editing the face deformation effect on the reference face image;

[0013] The determining unit is used to determine the input face scale corresponding to the reference face scale, as well as the horizontal axis unit direction vector and the vertical axis unit direction vector in the two-dimensional coordinate system; the reference face scale includes the preset scale of the left half face region, the middle region and the right half face region in the reference face image, and the two-dimensional coordinate system takes the input face rotation direction vector in the input face image as the vertical axis;

[0014] The adjustment unit is used to adjust the reference offset according to the reference face scale, the input face scale, the horizontal axis unit direction vector and the vertical axis unit direction vector, corresponding to the left half face region, the middle region and the right half face region, to obtain the first offset of the input control point corresponding to the reference control point;

[0015] The deformation unit is used to perform two-dimensional face deformation processing on the input face image based on the original coordinates of the input control point and the first offset of the input control point, so as to obtain a deformed face image with the face deformation effect applied.

[0016] On the other hand, this application provides an apparatus for image processing, the apparatus including a processor and a memory:

[0017] The memory is used to store program code and transmit the program code to the processor;

[0018] The processor is configured to execute the methods described above according to instructions in the program code.

[0019] On the other hand, embodiments of this application provide a computer-readable storage medium for storing a computer program for performing the methods described above.

[0020] On the other hand, embodiments of this application provide a computer program product, characterized in that it includes a computer program or instructions; when the computer program or instructions are executed by a processor, the methods described above are performed.

[0021] As can be seen from the above technical solution, the baseline offset of the baseline control point is obtained by pre-editing the face deformation effect on the baseline face image, and the preset scales of the left half, middle, and right half of the face region in the baseline face image are used as the baseline face scale. When the face in the input face image rotates relative to the face in the baseline face image, and the face deformation effect is applied to the input face image, the input face scale corresponding to the baseline face scale, the horizontal axis unit direction vector, and the vertical axis unit direction vector in the two-dimensional coordinate system are determined, with the vertical axis of the two-dimensional coordinate system being the input face rotation direction vector in the input face image. Using the baseline face scale, input face scale, horizontal axis unit direction vector, and vertical axis unit direction vector, the baseline offset is adjusted according to the left half, middle, and right half of the face region to obtain the first offset of the input control point corresponding to the baseline control point. Using the original coordinates of the input control point and the first offset of the input control point, the input face image is subjected to two-dimensional face deformation processing to obtain the deformed face image with the face deformation effect applied.

[0022] Based on this, the system first adapts the baseline offset of the reference control points obtained from editing the face deformation effect on the reference face image by adjusting the preset scales of the left, middle, and right halves of the face image relative to the reference face image along the horizontal and vertical axes. This first offset of the input control points is used to accommodate the rotation of the face. Then, by combining the original coordinates of the input control points and the first offset of the input control points with a two-dimensional face deformation method, a deformed face image with the face deformation effect is obtained. In other words, this face deformation method can adapt to the rotation of the face when applying the face deformation effect, thus avoiding face distortion and making the face deformation effect more natural and smooth. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a schematic diagram illustrating an application scenario of the image processing method provided in the embodiments of this application;

[0025] Figure 2 A schematic flowchart of an image processing method provided in an embodiment of this application;

[0026] Figure 3 A schematic diagram of reference key points and reference face scale in a reference face image provided in an embodiment of this application;

[0027] Figure 4 This is a schematic diagram illustrating the input of key points and the input of face scale in an input face image, as provided in an embodiment of this application.

[0028] Figure 5 This is a schematic diagram of the overall process framework of an image processing method provided in an embodiment of this application;

[0029] Figure 6 A schematic diagram of an image processing apparatus provided in an embodiment of this application;

[0030] Figure 7 This is a schematic diagram of the structure of the terminal device provided in the embodiments of this application;

[0031] Figure 8 This is a schematic diagram of the server structure provided in an embodiment of this application. Detailed Implementation

[0032] The embodiments of this application will now be described with reference to the accompanying drawings.

[0033] In related technologies, when users apply face deformation effects in scenarios such as short videos or live streaming, they usually use traditional two-dimensional face deformation methods to edit the original face image to create the face deformation effect. For example, interpolation-based two-dimensional face deformation algorithms or triangular mesh partitioning-based two-dimensional face deformation algorithms are used to obtain deformed face images with the applied face deformation effect.

[0034] However, in scenarios such as short videos or live streaming, when applying face deformation effects to face images obtained by rotating the face to the side, if the traditional two-dimensional face deformation method is still used, it cannot adapt to the situation of the face rotating to the side, resulting in face deformation distortion, that is, the face deformation effect is not natural and smooth enough.

[0035] Based on this, this application provides an image processing method and related apparatus. The method involves inputting a face image in which the face rotates relative to a reference face image. First, by varying the preset scales of the left, middle, and right halves of the face image relative to the reference face image along the horizontal and vertical axes, the method adaptively adjusts the reference offset of the reference control points obtained from editing the face deformation effect on the reference face image, thus obtaining a first offset of the input control points to accommodate the face rotation. Then, by combining the original coordinates of the input control points and the first offset with a two-dimensional face deformation method, a deformed face image with the applied face deformation effect is obtained. In other words, this face deformation method can adaptively handle the face rotation when applying the face deformation effect, avoiding facial distortion and resulting in a more natural and smooth face deformation effect.

[0036] The image processing method provided in this application can be applied to image processing devices with data processing capabilities, such as servers and terminal devices. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal device can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. Furthermore, the terminal device and the server can be directly or indirectly connected via wired or wireless communication, which is not limited herein.

[0037] To facilitate understanding of the technical solution of this application, the image processing method provided in the embodiments of this application will be described below in conjunction with a practical application scenario, using a server as the image processing device.

[0038] See Figure 1 , Figure 1 This is a schematic diagram illustrating an application scenario of the image processing method provided in an embodiment of this application. Figure 1 In the application scenario shown, the application scenario includes a terminal device 100, which is equipped with a camera. Taking the user using the terminal device 100 to shoot a short video or live stream as an example, the user applies a face deformation effect. The terminal device 100 serves as the aforementioned image processing device.

[0039] Terminal device 100 obtains a reference face image of the user through a camera. The reference face image refers to a standard frontal face image of the user.

[0040] The terminal device 100 pre-edits the face deformation effect on the reference face image to obtain the reference offset of the reference control point, and obtains the preset scale of the left half face region, the middle region and the right half face region in the reference face image as the reference face scale; the terminal device 100 records the reference offset and the reference face scale.

[0041] During the process of a user shooting a short video or live stream using terminal device 100, the user rotates their face to the side, and terminal device 100 obtains an input face image. That is, the face in the input face image undergoes a side rotation relative to the face in the reference face image, meaning there is a yaw angle between the face in the input face image and the face in the reference face image. When applying face deformation effects to the input face image, terminal device 100 obtains the reference offset.

[0042] Terminal device 100 determines the input face scale corresponding to the reference face scale, as well as the horizontal axis unit direction vector and the vertical axis unit direction vector in the two-dimensional coordinate system. The two-dimensional coordinate system takes the input face rotation direction vector in the input face image as the vertical axis.

[0043] The terminal device 100 adjusts the reference offset according to the reference face scale, the input face scale, the horizontal axis unit direction vector, and the vertical axis unit direction vector, corresponding to the left half face region, the middle region, and the right half face region, to obtain the first offset of the input control point corresponding to the reference control point.

[0044] The terminal device 100 performs two-dimensional face deformation processing on the input face image based on the original coordinates of the input control point and the first offset of the input control point to obtain a deformed face image with the applied face deformation effect, so as to display the deformed face image in short videos or live broadcasts.

[0045] Based on this, the terminal device 100 first adaptively adjusts the reference offset of the reference control points obtained by editing the face deformation effect on the reference face image in the horizontal and vertical directions by varying the preset scales of the left, middle, and right half of the face region relative to the reference face image in the horizontal and vertical directions. This results in a first offset of the input control points to accommodate the rotation of the face. The terminal device 100 then uses the original coordinates of the input control points and the first offset of the input control points, combined with a two-dimensional face deformation method, to obtain a deformed face image with the applied face deformation effect. In other words, this face deformation method can adaptively handle the rotation of the face when applying the face deformation effect, thus avoiding facial distortion and making the face deformation effect more natural and smooth.

[0046] The following description, with reference to the accompanying drawings and using a terminal device as the image processing device, introduces an image processing method provided in an embodiment of this application.

[0047] See Figure 2 , Figure 2 This is a schematic flowchart illustrating an image processing method provided in an embodiment of this application. Figure 2 As shown, the image processing method includes the following steps:

[0048] S201: When applying face deformation effects to the input face image, obtain the reference offset of the reference control point in the reference face image; the face in the input face image rotates to the side relative to the face in the reference face image, and the reference offset is obtained by editing the face deformation effect on the reference face image.

[0049] In this embodiment of the application, in scenarios such as short videos or live streaming, the face image obtained by the user's face side-view rotation is used as the input face image, and the user's frontal standard face image is used as the reference face image. Then, the face in the input face image undergoes side-view rotation relative to the face in the reference face image, that is, the face in the input face image and the face in the reference face image have a yaw angle.

[0050] Specifically, based on the face in the reference face image, the rotation direction vector of the reference face in the reference face image is used as the reference y-axis, and the direction vector perpendicular to the rotation direction vector of the reference face in the reference face image is used as the reference x-axis. In this way, a reference three-dimensional coordinate system is constructed, and the face in the input face image rotates around the reference y-axis relative to the face in the reference face image.

[0051] The relevant technology directly uses traditional 2D face deformation methods to edit the face deformation effect of the input face image. These traditional 2D face deformation methods may include interpolation-based 2D face deformation algorithms or triangular mesh partitioning-based 2D face deformation algorithms. Face deformation effects may include face effects or face beautification, such as baby face or frog mouth, or face beautification such as face slimming or eye enlargement. This method does not consider the rotation of the face from the side, resulting in face distortion and an unnatural, unsmooth face deformation effect.

[0052] In this embodiment, when applying a face deformation effect to an input face image, the face deformation effect is not directly edited using traditional 2D face deformation methods. Instead, a reference face image is pre-edited using traditional 2D face deformation methods to obtain the reference offset of the reference control points in the reference face image. Based on this reference offset, the reference offset is adaptively adjusted for the input face image. The adjusted reference offset is then used to complete the application of the face deformation effect to the input face image. Therefore, in this embodiment, when applying a face deformation effect to an input face image, it is first necessary to obtain the reference offset of the reference control points in the reference face image.

[0053] The process involves first editing a reference face image using a traditional 2D face deformation method to obtain the reference offset of the reference control points in the reference face image. This essentially means editing the reference face image using a traditional 2D face deformation method. First, the coordinates of the reference control points before and after movement can be obtained, i.e., the original coordinates and offset coordinates of the reference control points. Then, the coordinate offset is calculated using the original and offset coordinates of the reference control points, and normalized by combining this with the reference image size of the reference face image to obtain the reference offset of the reference control points. Therefore, this application provides a possible implementation method, and the step of obtaining the reference offset may include, for example, the steps S1-S2 in the following steps:

[0054] S1: Edit the face deformation effect on the reference face image to obtain the original coordinates and offset coordinates of the reference control points.

[0055] In the specific implementation of S1, firstly, facial landmark detection is performed on the reference face image to obtain multiple reference landmarks; then, editing the face deformation effect on the reference face image is actually editing the face deformation effect on multiple reference landmarks in the reference face image, that is, moving some or all of the multiple reference landmarks, and the moved reference landmarks can be determined as reference control points; finally, the original coordinates of the reference landmarks and the offset coordinates of the reference control points are obtained. Therefore, this application provides a possible implementation method, and S1 may include, for example, the steps S11-S13 in the following steps:

[0056] S11: Perform facial landmark detection on the baseline face image to obtain the baseline landmarks.

[0057] The process of obtaining benchmark key points by detecting facial landmarks in a benchmark face image can be achieved by first obtaining n feature key points in the benchmark face image through a facial landmark recognition algorithm, and then obtaining m benchmark key points in the benchmark face image through an interpolation algorithm, where m is greater than n, in order to cover key points in more face regions.

[0058] S12: Edit the face deformation effect on the reference key points in the reference face image, and determine the moved reference key points as reference control points.

[0059] Among them, editing the face deformation effect of the reference key points in the reference face image can be done by moving k reference key points out of m reference key points, where k is less than or equal to m, and then the k reference key points are determined as k reference control points.

[0060] S13: Obtain the original coordinates and offset coordinates of the reference control point.

[0061] The original coordinates and offset coordinates of the k reference control points are shown below:

[0062]

[0063]

[0064] S2: Obtain the baseline offset based on the original coordinates of the baseline control points, the offset coordinates of the baseline control points, and the baseline image size of the baseline face image.

[0065] Wherein, corresponding to the original coordinates of k reference control points and the offset coordinates of k reference control points, the width of the reference image size of the reference face image is w. standard The height is h standard When the reference offsets of the k reference control points are as follows:

[0066]

[0067]

[0068]

[0069] In addition, according to k reference control points In the region located in the baseline face image, the baseline control points are divided into k1 left half-face regions. ,k2 intermediate area reference control points and k3 baseline control points for the right half of the face The sum of k1, k2, and k3 equals k. Therefore, when recording the reference offset of k reference control points, the reference control points are also divided into k1 reference control points for the left half of the face region. The reference offset is k2 intermediate region reference control points The reference offset is and k3 baseline control points for the right half of the face The reference offset is .

[0070] S202: Determine the input face scale corresponding to the reference face scale, as well as the horizontal axis unit direction vector and vertical axis unit direction vector in the two-dimensional coordinate system; the reference face scale includes the preset scale of the left half face region, the middle region and the right half face region in the reference face image, and the two-dimensional coordinate system takes the input face rotation direction vector in the input face image as the vertical axis.

[0071] In this embodiment, the face in the input face image rotates relative to the face in the reference face image. In fact, the preset scale of the input face image changes relative to the reference face image in the left half face region, the middle region, and the right half face region. In particular, the preset scale of the left half face region, the middle region, and the right half face region changes in the horizontal axis direction and the vertical axis direction.

[0072] Based on this, in order to adaptively adjust the reference offset based on the reference offset for the input face image, it is necessary to obtain the preset scales of the left half, middle, and right half of the face region in the reference face image in advance as the reference face scale; then, for the input face image, it is necessary to obtain the preset scales of the left half, middle, and right half of the face region in the input face image corresponding to the reference face scale as the input face scale; and form a two-dimensional coordinate system by taking the input face rotation direction vector in the input face image as the vertical axis and the direction vector in the input face image perpendicular to the input face rotation direction vector as the horizontal axis, and obtain the unit direction vector of the horizontal axis and the unit direction vector of the vertical axis in this two-dimensional coordinate system.

[0073] The baseline face scale can be calculated from baseline key points in the baseline face image. For example, such as... Figure 3 The diagram illustrates reference key points and reference face scales in a reference face image. The reference face scales may include, for example, the reference left cheek width, reference nose bridge height, reference right cheek width, and reference face cheek width, as shown below:

[0074]

[0075] In the specific implementation of S202, the input face scale corresponding to the reference face scale can also be calculated from the input key points corresponding to the reference key points in the input face image. Therefore, it is necessary to first perform face key point detection on the input face image to obtain the input key points, and then obtain the input face scale by using the reference face scale corresponding to the input key points. Furthermore, it is necessary to establish a two-dimensional coordinate system using the input key points, with the input face rotation direction vector as the vertical axis and the direction vector perpendicular to the input face rotation direction vector as the horizontal axis, and determine the unit direction vectors of the horizontal and vertical axes in the two-dimensional coordinate system. Therefore, this application provides a possible implementation method, and S204 may include, for example, the steps S2021-S2023 in the following steps:

[0076] S2021: Perform facial landmark detection on the input face image to obtain the input landmarks corresponding to the baseline landmarks.

[0077] In this case, facial landmark detection of the input face image can also be performed by first obtaining n feature landmarks in the input face image through a facial landmark recognition algorithm, and then obtaining m input landmarks in the input face image through an interpolation algorithm. The m input landmarks correspond to m reference landmarks.

[0078] S2022: Corresponding to the baseline face scale, the input face scale is determined based on the input key points.

[0079] Among them, the reference facial dimensions include the reference left cheek width, reference nose bridge height, reference right cheek width, and reference facial cheek width; correspondingly, for example, such as Figure 4 The diagram illustrates the input of key points and facial scales in an input face image. The input facial scales may include, for example, the width of the left cheek, the height of the bridge of the nose, the width of the right cheek, and the width of the entire face, as shown below:

[0080]

[0081] S2023: Establish a two-dimensional coordinate system based on the input key points, and determine the unit direction vector of the horizontal axis and the unit direction vector of the vertical axis in the two-dimensional coordinate system.

[0082] Among them, Figure 4 Based on this, the input face rotation direction vector can be The unit direction vector of the horizontal axis can be The unit direction vector of the vertical axis can be

[0083] S203: Based on the reference face scale, the input face scale, the horizontal axis unit direction vector, and the vertical axis unit direction vector, adjust the reference offset for the left half face region, the middle region, and the right half face region to obtain the first offset of the input control point corresponding to the reference control point.

[0084] In this embodiment, by using the reference face scale, the input face scale, the horizontal axis unit direction vector, and the vertical axis unit direction vector, the preset scale changes of the input face image relative to the reference face image in the left half face region, the middle region, and the right half face region can be analyzed. In particular, the preset scale changes of the left half face region, the middle region, and the right half face region in the horizontal axis direction and the vertical axis direction are analyzed. The reference offset is adaptively adjusted accordingly for the left half face region, the middle region, and the right half face region. The adjusted reference offset is adapted to the input control point corresponding to the reference control point and is used as the first offset of the input control point.

[0085] Among them, at k reference control points The baseline control points are divided into k1 left half-face regions. ,k2 intermediate area reference control points and k3 baseline control points for the right half of the face Based on this, there are k input control points in the input face image. The baseline control points are divided into k1 left half-face regions. k2 intermediate region input control points and k3 input control points for the right half of the face region .

[0086] In the specific implementation of S203, considering a more intuitive representation of the preset scale changes of the input face image relative to the reference face image in the left, middle, and right half-face regions, it is necessary to first decompose the reference offset into the horizontal and vertical axes in a two-dimensional coordinate system using the horizontal and vertical axis unit direction vectors to obtain the horizontal and vertical components of the reference offset; then, using the reference face scale and the input face scale, the horizontal and vertical components are adjusted accordingly for the left, middle, and right half-face regions to obtain the first offset. Therefore, this application provides a possible implementation, and S203 may include, for example, the steps S2031-S2032 in the following steps:

[0087] S2031: Decompose the reference offset based on the horizontal axis unit direction vector and the vertical axis unit direction vector to obtain the horizontal axis component and the vertical axis component of the reference offset.

[0088] in, ; Indicates the reference offset Horizontal axis component, Indicates the reference offset The vertical axis component.

[0089] S2032: Based on the reference face scale and the input face scale, adjust the horizontal axis components and vertical axis components for the left half face region, the middle region and the right half face region to obtain the first offset.

[0090] In a specific implementation of S2032, the reference face scale and the input face scale are actually used to correspond to the left half-face region, the middle region, and the right half-face region, respectively, to determine the first adjustment coefficient for the horizontal axis component and the second adjustment coefficient for the vertical axis component. Based on this, the horizontal axis component is adjusted with the first adjustment coefficient, and the vertical axis component is adjusted with the second adjustment coefficient, and the first offset is obtained by combining the results. Therefore, this application provides a possible implementation method, and S2032 may include, for example, steps S3-S5:

[0091] S3: Based on the baseline face scale and the input face scale, determine the first adjustment coefficient of the horizontal axis component and the second adjustment coefficient of the vertical axis component for the corresponding left half face region, middle region and right half face region.

[0092] S4: Adjust the horizontal axis component according to the first adjustment coefficient, and adjust the vertical axis component according to the second adjustment coefficient to obtain the first offset.

[0093] Among them, the baseline control points for k1 left half of the face region reference offset In other words, input the width of the left cheek. Compared to the baseline width of the left cheek Changes occur; input the width of the human face cheek. Compared to the standard human face cheek width The basic structure remains unchanged; then The horizontal axis component needs to change. The vertical axis component needs to remain essentially unchanged in order to obtain k1 input control points for the left half of the face region. The first offset is For example, if the input face image is obtained by rotating a face in a reference face image to the left, then... Compared to Decrease Compared to Basically unchanged, The horizontal axis component needs to be reduced. The vertical axis component remains essentially unchanged to obtain .

[0094] In practical implementation, The first adjustment factor for the horizontal axis component can be and The ratio, The second adjustment factor for the vertical axis component can be and The ratio is obtained. The details are as follows:

[0095]

[0096] Similarly, for the reference control points of k2 intermediate regions reference offset In terms of input nose bridge height Compared to the baseline nose bridge height The basic structure remains unchanged; then The horizontal axis component needs to remain essentially unchanged. The vertical axis component also needs to remain essentially unchanged in order to obtain the input control points of k2 intermediate regions. The first offset is .but The first adjustment factor for the horizontal axis component can be and The ratio, The second adjustment factor for the vertical axis component can also be and The ratio is obtained. The details are as follows:

[0097]

[0098] Similarly, for k3 baseline control points in the right half of the face region reference offset In other words, input the width of the right cheek. Compared to the baseline right cheek width Changes occur; input the width of the human face cheek. Compared to the standard human face cheek width The basic structure remains unchanged; then The horizontal axis component needs to change. The vertical axis component needs to remain essentially unchanged in order to obtain k3 input control points for the right half of the face region. The first offset is .but The first adjustment factor for the horizontal axis component can be and The ratio, The second adjustment factor for the vertical axis component can be and The ratio is obtained. The details are as follows:

[0099]

[0100] Furthermore, in this embodiment, there are cases where the input face image is obtained by the user rotating and sideways. In this case, the face in the input face image not only undergoes sideways rotation but also rotation relative to the face in the reference face image; that is, the face in the input face image and the face in the reference face image have not only a yaw angle but also a rotation angle. Based on the aforementioned reference three-dimensional coordinate system, a reference z-axis can be determined, and the face in the input face image rotates around the reference z-axis relative to the face in the reference face image. In this case, the reference offset can be adjusted first to accommodate the face rotation, and then further adjusted using the method described in S203 above to accommodate the sideways rotation.

[0101] In specific implementation, adjusting the reference offset to adapt to face rotation is actually taking the angle between the input face rotation direction vector in the input face image and the reference face rotation direction vector in the reference face image as the rotation angle. This rotation angle is used to adjust the reference offset to adapt to face rotation, resulting in a second offset of the input control point. Based on this second offset, the reference offset is replaced with the second offset, and the above S203 is executed to adapt to the side-face rotation. Therefore, this application provides a possible implementation where the face in the input face image and the face in the reference face image have rotation and yaw angles. The method may also include steps S5-S6:

[0102] S5: Determine the rotation angle based on the input face rotation direction vector and the reference face rotation direction vector in the reference face image.

[0103] S6: Obtain the second offset of the input control point based on the reference offset and the rotation angle.

[0104] Wherein, the input face rotation direction vector can be The reference face rotation direction vector can be ,calculate to The rotation angle is θ, with counterclockwise rotation being positive and clockwise rotation being negative. The reference control point is adjusted using the rotation angle θ. reference offset , obtain input control points second offset The details are as follows:

[0105]

[0106] Correspondingly, S203 may include, for example, adjusting the second offset to obtain the first offset based on the reference face scale, the input face scale, the horizontal axis unit direction vector and the vertical axis unit direction vector, corresponding to the left half face region, the middle region and the right half face region.

[0107] Furthermore, in this embodiment, the face in the input face image may have a pitch angle in addition to the rotation and yaw angles compared to the face in the reference face image. That is, the input face image is obtained by the user performing face pitch rotation, face rotation, and face side rotation. In this case, considering that the face pitch rotation is similar to the case of a frontal standard face, it can be used directly without adjusting the reference offset. This, combined with the above method, adapts to the cases of face rotation and face side rotation, thereby accommodating face rotation at various angles.

[0108] S204: Based on the original coordinates of the input control points and the first offset of the input control points, perform two-dimensional face deformation processing on the input face image to obtain a deformed face image with the applied face deformation effect.

[0109] In this embodiment of the application, after obtaining the first offset of the input control point in S203, and combining it with the original coordinates of the input control point, the traditional two-dimensional face deformation method can be used to perform two-dimensional face deformation processing on the input face image to obtain a deformed face image with the face deformation effect applied.

[0110] In the specific implementation of S204, firstly, the offset coordinates of the input control point after movement can be calculated by combining the first offset of the input control point with the input image size of the input face image and the original coordinates of the input control point; then, based on the original coordinates and offset coordinates of the input control point, a traditional two-dimensional face deformation method is used to perform two-dimensional face deformation processing on the input face image to obtain a deformed face image with the applied face deformation effect. Therefore, this application provides a possible implementation method, and S204 may include, for example, the steps S2041-S2042 in the following steps:

[0111] S2041: Obtain the offset coordinates of the input control point based on the original coordinates of the input control point, the first offset of the input control point, and the input image size of the input face image.

[0112] Among them, k1 are the baseline control points for the left half of the face. The original coordinates and the input control points of k2 intermediate regions The original coordinates and k3 input control points for the right half of the face region The original coordinates are shown below:

[0113]

[0114]

[0115]

[0116] The width of the input image is w. input The height is h input Then, k1 reference control points for the left half of the face region. The offset coordinates are as follows:

[0117]

[0118]

[0119] k2 input control points in the intermediate regions The offset coordinates are as follows:

[0120]

[0121]

[0122] k3 input control points for the right half of the face region The offset coordinates are as follows:

[0123]

[0124]

[0125] S2042: Based on the original coordinates and offset coordinates of the input control points, perform two-dimensional face deformation processing on the input face image to obtain a deformed face image.

[0126] In a specific implementation of S2042, firstly, by combining the original coordinates and offset coordinates of the input control points with the radius of influence of the input control points and the original coordinates of the input pixels, the offset coordinates of the input pixels can be calculated using the pixel offset function. Then, the input face image is processed using the offset coordinates of the input pixels to obtain the deformed face image. Therefore, this application provides a possible implementation method, and S2042 may include, for example, steps S7-S8:

[0127] S7: Determine the offset coordinates of the input pixel based on the original coordinates of the input control point, the offset coordinates of the input control point, the radius of influence of the input control point, the original coordinates of the input pixel, and the pixel offset function.

[0128] The original coordinates of any input pixel can be: If the radius of influence of the input control point is R, then the offset coordinates of the input pixel are obtained as follows:

[0129]

[0130]

[0131]

[0132]

[0133] The above To represent a linear change, the simplest form is... , For distance weighting function, Indicates distance, This represents the normalized distance weight function.

[0134] In addition, in this embodiment of the application, besides obtaining the offset coordinates of all input pixels in the input face image through the above-mentioned S7 implementation, the input face image can also be triangulated by some input pixels in the input face image, and the offset coordinates of some input pixels can be obtained by the above-mentioned S7 implementation. By solving the triangle vertices and combining centroid interpolation, the offset coordinates of other pixels in the input face image other than some input pixels can be obtained, further improving the calculation efficiency of obtaining the offset coordinates of all input pixels in the input face image.

[0135] S8: Obtain the deformed face image based on the input face image and the offset coordinates of the input pixels.

[0136] Specifically, based on the offset coordinates of the input pixels, bilinear interpolation is performed on the input face image to obtain a deformed face image. Of course, the implementation methods of S7-S8 described above are only one specific implementation of a traditional two-dimensional face deformation method, namely, the inverse distance weighted replacement method. In this embodiment, other specific implementation methods can also be used, such as the moving least squares method, etc.

[0137] The image processing method provided in the above embodiments pre-edits a face deformation effect on a reference face image to obtain the reference offset of the reference control point, and obtains the preset scales of the left half, middle, and right half of the face in the reference face image as the reference face scale. When the face in the input face image rotates relative to the face in the reference face image, and the face deformation effect is applied to the input face image, the input face scale corresponding to the reference face scale, as well as the horizontal axis unit direction vector and vertical axis unit direction vector in the two-dimensional coordinate system are determined, with the vertical axis of the two-dimensional coordinate system being the input face rotation direction vector in the input face image; using the reference face scale, input face scale, horizontal axis unit direction vector, and vertical axis unit direction vector, the reference offset is adjusted according to the left half, middle, and right half of the face to obtain the first offset of the input control point corresponding to the reference control point; using the original coordinates of the input control point and the first offset of the input control point, two-dimensional face deformation processing is performed on the input face image to obtain the deformed face image with the face deformation effect applied.

[0138] Based on this, the system first adapts the baseline offset of the reference control points obtained from editing the face deformation effect on the reference face image by adjusting the preset scales of the left, middle, and right halves of the face image relative to the reference face image along the horizontal and vertical axes. This first offset of the input control points is used to accommodate the rotation of the face. Then, by combining the original coordinates of the input control points and the first offset of the input control points with a two-dimensional face deformation method, a deformed face image with the face deformation effect is obtained. In other words, this face deformation method can adapt to the rotation of the face when applying the face deformation effect, avoiding facial distortion and making the face deformation effect more natural and smooth. This provides a key foundational technology for subsequent applications such as face animation, face fusion, or face swapping.

[0139] Based on the above explanation, see [link / reference]. Figure 5 , Figure 5 This is a schematic diagram of the overall workflow framework of an image processing method provided in an embodiment of this application. The schematic diagram of the overall workflow framework of the image processing method is divided into an offline workflow framework and an online workflow framework.

[0140] The offline workflow framework includes obtaining baseline key points, obtaining baseline offsets, and obtaining baseline face scales. Obtaining baseline key points refers to performing facial key point detection on the baseline face image to obtain the baseline key points; obtaining baseline offsets refers to editing the face deformation effects on the baseline key points in the baseline face image to obtain the baseline offset of the baseline control points in the baseline face image; obtaining baseline face scales refers to obtaining the preset scales of the left half, middle, and right half of the face.

[0141] The online workflow framework includes obtaining the reference offset, determining the input face scale, adjusting the reference offset, and obtaining the deformed face image. Specifically, obtaining the reference offset refers to acquiring the reference offset of the reference control points in the reference face image when applying the face deformation effect to the input face image; determining the input face scale refers to determining the input face scale corresponding to the reference face scale, as well as the unit direction vectors of the horizontal and vertical axes in the two-dimensional coordinate system; adjusting the reference offset refers to adjusting the reference offset according to the reference face scale, the input face scale, the unit direction vectors of the horizontal and vertical axes, corresponding to the left, middle, and right half of the face region, to obtain the first offset of the input control points corresponding to the reference control points; obtaining the deformed face image refers to performing two-dimensional face deformation processing on the input face image based on the original coordinates of the input control points and the first offset of the input control points to obtain the deformed face image with the applied face deformation effect.

[0142] In addition to the image processing method provided in the above embodiments, this application also provides an image processing apparatus.

[0143] See Figure 6 This figure is a schematic diagram of an image processing apparatus provided in an embodiment of this application. Figure 6 As shown, the image processing apparatus 600 includes: an acquisition unit 601, a determination unit 602, an adjustment unit 603, and a deformation unit 604;

[0144] The acquisition unit 601 is used to acquire the reference offset of the reference control point in the reference face image when applying face deformation effect to the input face image; the face in the input face image rotates to the side relative to the face in the reference face image, and the reference offset is obtained by editing the face deformation effect on the reference face image;

[0145] The determining unit 602 is used to determine the input face scale corresponding to the reference face scale, as well as the horizontal axis unit direction vector and the vertical axis unit direction vector in the two-dimensional coordinate system; the reference face scale includes the preset scale of the left half face region, the middle region and the right half face region in the reference face image, and the two-dimensional coordinate system takes the input face rotation direction vector in the input face image as the vertical axis;

[0146] The adjustment unit 603 is used to adjust the reference offset according to the reference face scale, the input face scale, the horizontal axis unit direction vector and the vertical axis unit direction vector, corresponding to the left half face region, the middle region and the right half face region, to obtain the first offset of the input control point corresponding to the reference control point;

[0147] The deformation unit 604 is used to perform two-dimensional face deformation processing on the input face image based on the original coordinates of the input control point and the first offset of the input control point, so as to obtain a deformed face image with the face deformation effect applied.

[0148] As one possible implementation, adjustment unit 603 is used for:

[0149] The reference offset is decomposed based on the horizontal axis unit direction vector and the vertical axis unit direction vector to obtain the horizontal axis component and the vertical axis component of the reference offset.

[0150] Based on the baseline face scale and the input face scale, the horizontal and vertical axis components are adjusted for the left, middle, and right half of the face to obtain the first offset.

[0151] As one possible implementation, adjustment unit 603 is used for:

[0152] Based on the baseline face scale and the input face scale, the first adjustment coefficient of the horizontal axis component and the second adjustment coefficient of the vertical axis component are determined for the left half face region, the middle region and the right half face region.

[0153] The horizontal axis component is adjusted according to the first adjustment factor, and the vertical axis component is adjusted according to the second adjustment factor to obtain the first offset.

[0154] As one possible implementation, the apparatus also includes an obtaining unit for:

[0155] Edit the face deformation effect on the reference face image to obtain the original coordinates and offset coordinates of the reference control points;

[0156] The baseline offset is obtained based on the original coordinates of the baseline control points, the offset coordinates of the baseline control points, and the baseline image size of the baseline face image.

[0157] As one possible implementation, the unit is obtained for:

[0158] Benchmark key points are obtained by performing facial landmark detection on a benchmark face image;

[0159] Edit the face deformation effect on the reference key points in the reference face image, and determine the moved reference key points as reference control points;

[0160] Obtain the original coordinates and offset coordinates of the reference control points.

[0161] As one possible implementation, deformable element 604 is used for:

[0162] The offset coordinates of the input control points are obtained based on the original coordinates of the input control points, the first offset of the input control points, and the input image size of the input face image.

[0163] Based on the original coordinates and offset coordinates of the input control points, a two-dimensional face deformation process is performed on the input face image to obtain a deformed face image.

[0164] As one possible implementation, deformable element 604 is used for:

[0165] The offset coordinates of the input pixel are determined based on the original coordinates of the input control point, the offset coordinates of the input control point, the radius of the influence range of the input control point, the original coordinates of the input pixel, and the pixel offset function.

[0166] A deformed face image is obtained based on the input face image and the offset coordinates of the input pixels.

[0167] As one possible implementation, unit 602 is used for:

[0168] Facial landmark detection is performed on the input face image to obtain the input landmark corresponding to the reference landmark; the reference landmark is obtained by performing facial landmark detection on the reference face image.

[0169] Based on the baseline face scale, the input face scale is determined according to the input key points;

[0170] Establish a two-dimensional coordinate system based on the input key points, and determine the unit direction vector of the horizontal axis and the unit direction vector of the vertical axis in the two-dimensional coordinate system.

[0171] As one possible implementation, if the face in the input face image rotates and lateralizes relative to the face in the reference face image, the determining unit 602 is further configured to:

[0172] The rotation angle is determined based on the input face rotation direction vector and the reference face rotation direction vector in the reference face image;

[0173] The second offset of the input control point is obtained based on the reference offset and the rotation angle.

[0174] Adjustment unit 603 is used for:

[0175] Based on the baseline face scale, the input face scale, the horizontal axis unit direction vector, and the vertical axis unit direction vector, the second offset is adjusted for the left half face region, the middle region, and the right half face region to obtain the first offset.

[0176] The image processing apparatus provided in the above embodiments pre-edits a face deformation effect on a reference face image to obtain the reference offset of the reference control point, and obtains the preset scales of the left half, middle, and right half of the face region in the reference face image as the reference face scale. When the face in the input face image rotates relative to the face in the reference face image, and the face deformation effect is applied to the input face image, the input face scale corresponding to the reference face scale, as well as the horizontal axis unit direction vector and the vertical axis unit direction vector in the two-dimensional coordinate system are determined, with the vertical axis of the two-dimensional coordinate system being the input face rotation direction vector in the input face image; by using the reference face scale, the input face scale, the horizontal axis unit direction vector, and the vertical axis unit direction vector, the reference offset is adjusted according to the left half, middle, and right half of the face region to obtain the first offset of the input control point corresponding to the reference control point; by using the original coordinates of the input control point and the first offset of the input control point, two-dimensional face deformation processing is performed on the input face image to obtain a deformed face image with the face deformation effect applied.

[0177] Based on this, the system first adapts the baseline offset of the reference control points obtained from editing the face deformation effect on the reference face image by adjusting the preset scales of the left, middle, and right halves of the face image relative to the reference face image along the horizontal and vertical axes. This first offset of the input control points is used to accommodate the rotation of the face. Then, by combining the original coordinates of the input control points and the first offset of the input control points with a two-dimensional face deformation method, a deformed face image with the face deformation effect is obtained. In other words, this face deformation method can adapt to the rotation of the face when applying the face deformation effect, thus avoiding face distortion and making the face deformation effect more natural and smooth.

[0178] This application also provides an image processing device, which will be described below from the perspective of hardware implementation.

[0179] See Figure 7 , Figure 7 This is a schematic diagram of a terminal device provided in an embodiment of this application. For ease of explanation, only the parts related to the embodiments of this application are shown; for specific technical details not disclosed, please refer to the method section of the embodiments of this application. The terminal device can be any terminal device, including mobile phones, tablet computers, personal digital assistants (PDAs), etc. Taking a mobile phone as an example:

[0180] Figure 7 This diagram illustrates a partial structural representation of a mobile phone related to the terminal device provided in this embodiment. (Reference) Figure 7 The mobile phone includes components such as a radio frequency (RF) circuit 710, a memory 720, an input unit 730, a display unit 740, a sensor 750, an audio circuit 760, a wireless Fidelity (WiFi) module 770, a processor 780, and a power supply 790. Those skilled in the art will understand that... Figure 7 The mobile phone structure shown is not a limitation on the construction of a mobile phone and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0181] The following is combined with Figure 7 A detailed introduction to each component of the mobile phone:

[0182] RF circuit 710 can be used for receiving and transmitting signals during information transmission or calls. Specifically, it receives downlink information from the base station and processes it with processor 780; additionally, it transmits uplink data to the base station. Typically, RF circuit 710 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low-noise amplifier (LNA), and a duplexer. Furthermore, RF circuit 710 can also communicate wirelessly with networks and other devices. The aforementioned wireless communication can use any communication standard or protocol, including but not limited to Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, and Short Messaging Service (SMS).

[0183] The memory 720 can be used to store software programs and modules. The processor 780 runs the software programs and modules stored in the memory 720 to realize various functions and data processing of the mobile phone. The memory 720 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory 720 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0184] The input unit 730 can be used to receive input numerical or character information, and to generate key signal inputs related to user settings and function control of the mobile phone. Specifically, the input unit 730 may include a touch panel 731 and other input devices 732. The touch panel 731, also known as a touch screen, can collect touch operations performed by the user on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch panel 731), and drive the corresponding connected devices according to a pre-set program. Optionally, the touch panel 731 may include two parts: a touch detection device and a touch controller. The touch detection device detects the user's touch position and the signal generated by the touch operation, and transmits the signal to the touch controller; the touch controller receives touch information from the touch detection device, converts it into touch point coordinates, and sends it to the processor 780, and can also receive and execute commands sent by the processor 780. In addition, the touch panel 731 can be implemented using various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch panel 731, the input unit 730 may also include other input devices 732. Specifically, other input devices 732 may include, but are not limited to, one or more of the following: physical keyboard, function keys (such as volume control buttons, power buttons, etc.), trackball, mouse, joystick, etc.

[0185] The display unit 740 can be used to display information input by the user or information provided to the user, as well as various menus of the mobile phone. The display unit 740 may include a display panel 741, which may optionally be configured as a Liquid Crystal Display (LCD), Organic Light-Emitting Diode (OLED), or similar display panel. Further, a touch panel 731 may cover the display panel 741. When the touch panel 731 detects a touch operation on or near it, it transmits the information to the processor 780 to determine the type of touch event. Subsequently, the processor 780 provides corresponding visual output on the display panel 741 based on the type of touch event. Although in Figure 7 In this embodiment, the touch panel 731 and the display panel 741 are two separate components to realize the input and output functions of the mobile phone. However, in some embodiments, the touch panel 731 and the display panel 741 can be integrated to realize the input and output functions of the mobile phone.

[0186] The mobile phone may also include at least one sensor 750, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor. The ambient light sensor can adjust the brightness of the display panel 741 according to the ambient light level, and the proximity sensor can turn off the display panel 741 and / or backlight when the phone is moved to the ear. As a type of motion sensor, an accelerometer sensor can detect the magnitude of acceleration in various directions (generally three axes). When stationary, it can detect the magnitude and direction of gravity and can be used for applications that recognize the phone's posture (such as landscape / portrait switching, related games, magnetometer posture calibration), vibration recognition functions (such as pedometer, taps), etc. Other sensors that may be configured in the mobile phone, such as gyroscopes, barometers, hygrometers, thermometers, and infrared sensors, will not be described in detail here.

[0187] Audio circuit 760, speaker 761, and microphone 762 provide an audio interface between the user and the mobile phone. Audio circuit 760 converts received audio data into electrical signals and transmits them to speaker 761, where speaker 761 converts them into sound signals for output. On the other hand, microphone 762 converts collected sound signals into electrical signals, which are received by audio circuit 760, converted into audio data, and then processed by processor 780 before being transmitted via RF circuit 710 to, for example, another mobile phone, or the audio data can be output to memory 720 for further processing.

[0188] WiFi is a short-range wireless transmission technology. Through the WiFi module 770, mobile phones can help users send and receive emails, browse web pages, and access streaming media, providing users with wireless broadband internet access. Although Figure 7 The WiFi module 770 is shown, but it is understood that it is not an essential component of the mobile phone and can be omitted as needed without changing the essence of the invention.

[0189] The processor 780 is the control center of the mobile phone, connecting various parts of the phone through various interfaces and lines. It performs various functions and processes data by running or executing software programs and / or modules stored in the memory 720, and by calling data stored in the memory 720. Optionally, the processor 780 may include one or more processing units; preferably, the processor 780 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may also not be integrated into the processor 780.

[0190] The mobile phone also includes a power supply 790 (such as a battery) that supplies power to various components. Preferably, the power supply can be logically connected to the processor 780 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system.

[0191] Although not shown, mobile phones may also include a camera, Bluetooth module, etc., which will not be described in detail here.

[0192] In this embodiment of the application, the memory 720 included in the mobile phone can store program code and transmit the program code to the processor.

[0193] The processor 780 included in the mobile phone can execute the image processing method provided in the above embodiments according to the instructions in the program code.

[0194] See Figure 8 , Figure 8 This is a schematic diagram of a server structure provided in an embodiment of this application. The server 800 can vary significantly due to differences in configuration or performance. It may include one or more central processing units (CPUs) 822 (e.g., one or more processors) and memory 832, and one or more storage media 830 (e.g., one or more mass storage devices) for storing application programs 842 or data 844. The memory 832 and storage media 830 can be temporary or persistent storage. The program stored in the storage media 830 may include one or more modules (not shown in the diagram), each module may include a series of instruction operations on the server. Furthermore, the CPU 822 may be configured to communicate with the storage media 830 and execute the series of instruction operations in the storage media 830 on the server 800.

[0195] Server 800 may also include one or more power supplies 826, one or more wired or wireless network interfaces 850, one or more input / output interfaces 858, and / or one or more operating systems 841, such as Windows Server. TM Mac OS X TM Unix TM Linux TM FreeBSD TM etc.

[0196] The steps performed by the server in the above embodiments can be based on this Figure 8 The server structure shown.

[0197] The CPU 822 is used to perform the following steps:

[0198] When applying face deformation effects to an input face image, the reference offset of the reference control point in the reference face image is obtained; the face in the input face image rotates to the side relative to the face in the reference face image, and the reference offset is obtained by editing the face deformation effect on the reference face image;

[0199] Determine the input face scale corresponding to the baseline face scale, as well as the horizontal axis unit direction vector and vertical axis unit direction vector in the two-dimensional coordinate system; the baseline face scale includes the preset scales of the left half face region, the middle region and the right half face region in the baseline face image, and the two-dimensional coordinate system takes the input face rotation direction vector in the input face image as the vertical axis;

[0200] Based on the reference face scale, the input face scale, the horizontal axis unit direction vector, and the vertical axis unit direction vector, the reference offset is adjusted for the left half face region, the middle region, and the right half face region to obtain the first offset of the input control point corresponding to the reference control point;

[0201] Based on the original coordinates and the first offset of the input control points, the input face image is subjected to two-dimensional face deformation processing to obtain a deformed face image with the applied face deformation effect.

[0202] In addition to the image processing method described above, this application also provides a face deformation terminal device to enable the above image processing method to be implemented and applied in practice.

[0203] This application also provides a computer-readable storage medium for storing a computer program that performs the image processing method provided in the above embodiments.

[0204] This application also provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of an image processing apparatus reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the image processing apparatus to perform the image processing methods provided in the various optional implementations of the above aspects.

[0205] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium can be at least one of the following media: read-only memory (ROM), RAM, magnetic disk or optical disk, and other media that can store program code.

[0206] It should be noted that the various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for the device and system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiments. The device and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the solution in this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0207] The above description is merely one specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image processing method, characterized in that, The method for adapting to the rotation of the face when applying a face deformation effect includes: When applying a face deformation effect to an input face image, the reference offset of the reference control point in the reference face image is obtained; the face in the input face image rotates to the side relative to the face in the reference face image, and the reference offset is obtained by editing the face deformation effect on the reference face image; The input face scale corresponding to the reference face scale is determined, as well as the horizontal axis unit direction vector and the vertical axis unit direction vector in the two-dimensional coordinate system; the reference face scale includes the preset scale of the left half face region, the middle region and the right half face region in the reference face image, the two-dimensional coordinate system takes the input face rotation direction vector in the input face image as the vertical axis, and the reference face scale includes the reference left cheek width, the reference bridge of the nose height, the reference right cheek width and the reference face cheek width; Based on the reference face scale, the input face scale, the horizontal axis unit direction vector, and the vertical axis unit direction vector, the reference offset is adjusted corresponding to the left half face region, the middle region, and the right half face region to obtain the first offset of the input control point corresponding to the reference control point; Based on the original coordinates of the input control point and the first offset of the input control point, the input face image is subjected to two-dimensional face deformation processing to obtain a deformed face image with the face deformation effect applied.

2. The method according to claim 1, characterized in that, The step of adjusting the reference offset according to the reference face scale, the input face scale, the horizontal axis unit direction vector, and the vertical axis unit direction vector, corresponding to the left half-face region, the middle region, and the right half-face region, to obtain the first offset of the input control point corresponding to the reference control point includes: The reference offset is decomposed based on the horizontal axis unit direction vector and the vertical axis unit direction vector to obtain the horizontal axis component and the vertical axis component of the reference offset. Based on the reference face scale and the input face scale, the horizontal axis component and the vertical axis component are adjusted to obtain the first offset, corresponding to the left half face region, the middle region and the right half face region.

3. The method according to claim 2, characterized in that, The step of adjusting the horizontal axis component and the vertical axis component according to the reference face scale and the input face scale, corresponding to the left half-face region, the middle region, and the right half-face region, to obtain the first offset includes: Based on the reference face scale and the input face scale, corresponding to the left half-face region, the middle region and the right half-face region, the first adjustment coefficient of the horizontal axis component and the second adjustment coefficient of the vertical axis component are determined; The horizontal axis component is adjusted according to the first adjustment coefficient, and the vertical axis component is adjusted according to the second adjustment coefficient to obtain the first offset.

4. The method according to any one of claims 1-3, characterized in that, The steps for obtaining the reference offset include: Edit the face deformation effect on the reference face image to obtain the original coordinates and offset coordinates of the reference control points; The reference offset is obtained based on the original coordinates of the reference control point, the offset coordinates of the reference control point, and the reference image size of the reference face image.

5. The method according to claim 4, characterized in that, The step of editing the face deformation effect on the reference face image to obtain the original coordinates and offset coordinates of the reference control points includes: Facial landmark detection is performed on the reference face image to obtain reference landmarks; Edit the face deformation effect on the reference key points in the reference face image, and determine the moved reference key points as the reference control points; Obtain the original coordinates and offset coordinates of the reference control point.

6. The method according to claim 1, characterized in that, The step of performing two-dimensional face deformation processing on the input face image based on the original coordinates of the input control point and the first offset of the input control point to obtain a deformed face image with the applied face deformation effect includes: The offset coordinates of the input control point are obtained based on the original coordinates of the input control point, the first offset of the input control point, and the input image size of the input face image. Based on the original coordinates and offset coordinates of the input control points, the input face image is subjected to two-dimensional face deformation processing to obtain the deformed face image.

7. The method according to claim 6, characterized in that, The step of performing two-dimensional face deformation processing on the input face image based on the original coordinates and offset coordinates of the input control points to obtain the deformed face image includes: The offset coordinates of the input pixel are determined based on the original coordinates of the input control point, the offset coordinates of the input control point, the radius of influence of the input control point, the original coordinates of the input pixel, and the pixel offset function. The deformed face image is obtained based on the input face image and the offset coordinates of the input pixels.

8. The method according to claim 1, characterized in that, The determination of the input face scale corresponding to the reference face scale, and the unit direction vectors of the horizontal axis and vertical axis in the two-dimensional coordinate system, includes: Facial landmark detection is performed on the input face image to obtain the input landmark corresponding to the reference landmark; the reference landmark is obtained by performing facial landmark detection on the reference face image. The input face scale is determined based on the input key points, corresponding to the baseline face scale. The two-dimensional coordinate system is established based on the input key points, and the unit direction vectors of the horizontal axis and vertical axis in the two-dimensional coordinate system are determined.

9. The method according to claim 1, characterized in that, The method further includes rotating and side-viewing the face in the input face image relative to the face in the reference face image. The rotation angle is determined based on the input face rotation direction vector and the reference face rotation direction vector in the reference face image; The second offset of the input control point is obtained based on the reference offset and the rotation angle. The step of adjusting the reference offset according to the reference face scale, the input face scale, the horizontal axis unit direction vector, and the vertical axis unit direction vector, corresponding to the left half-face region, the middle region, and the right half-face region, to obtain the first offset of the input control point corresponding to the reference control point includes: Based on the reference face scale, the input face scale, the horizontal axis unit direction vector, and the vertical axis unit direction vector, the second offset is adjusted corresponding to the left half face region, the middle region, and the right half face region to obtain the first offset.

10. An image processing apparatus, characterized in that, The device for adapting to the rotation of the side profile when applying face deformation effects includes: an acquisition unit, a determination unit, an adjustment unit, and a deformation unit; The acquisition unit is used to acquire the reference offset of the reference control point in the reference face image when applying a face deformation effect to the input face image; the face in the input face image has undergone a side rotation relative to the face in the reference face image, and the reference offset is obtained by editing the face deformation effect on the reference face image; The determining unit is used to determine the input face scale corresponding to the reference face scale, as well as the horizontal axis unit direction vector and the vertical axis unit direction vector in the two-dimensional coordinate system; the reference face scale includes the preset scales of the left half face region, the middle region and the right half face region in the reference face image, the two-dimensional coordinate system takes the input face rotation direction vector in the input face image as the vertical axis, and the reference face scale includes the reference left cheek width, the reference bridge of the nose height, the reference right cheek width and the reference face cheek width; The adjustment unit is used to adjust the reference offset according to the reference face scale, the input face scale, the horizontal axis unit direction vector and the vertical axis unit direction vector, corresponding to the left half face region, the middle region and the right half face region, to obtain the first offset of the input control point corresponding to the reference control point; The deformation unit is used to perform two-dimensional face deformation processing on the input face image based on the original coordinates of the input control point and the first offset of the input control point, so as to obtain a deformed face image with the face deformation effect applied.

11. The apparatus according to claim 10, characterized in that, The adjustment unit is used for: The reference offset is decomposed based on the horizontal axis unit direction vector and the vertical axis unit direction vector to obtain the horizontal axis component and the vertical axis component of the reference offset. Based on the reference face scale and the input face scale, the horizontal axis component and the vertical axis component are adjusted to obtain the first offset, corresponding to the left half face region, the middle region and the right half face region.

12. The apparatus according to claim 11, characterized in that, The adjustment unit is used for: Based on the reference face scale and the input face scale, corresponding to the left half-face region, the middle region and the right half-face region, the first adjustment coefficient of the horizontal axis component and the second adjustment coefficient of the vertical axis component are determined; The horizontal axis component is adjusted according to the first adjustment coefficient, and the vertical axis component is adjusted according to the second adjustment coefficient to obtain the first offset.

13. The apparatus according to any one of claims 10-12, characterized in that, The apparatus further includes an obtaining unit, the obtaining unit being configured to: Edit the face deformation effect on the reference face image to obtain the original coordinates and offset coordinates of the reference control points; The reference offset is obtained based on the original coordinates of the reference control point, the offset coordinates of the reference control point, and the reference image size of the reference face image.

14. An apparatus for image processing, characterized in that, The device includes a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is configured to execute the method according to any one of claims 1-9 according to the instructions in the program code.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a computer program for performing the method according to any one of claims 1-9.

16. A computer program product, characterized in that, Includes a computer program or instructions; when the computer program or instructions are executed by a processor, the method described in any one of claims 1-9 is performed.

Images