An image processing method, apparatus, device, medium, product

Abstract

The application discloses an image processing method, device, equipment, medium and product in the technical field of data processing, which comprises the following steps: firstly, obtaining an original image; secondly, predicting a first image, transparency information and a deformation offset according to the affine transformation result of the original image; thirdly, performing weighted fusion on the inverse affine transformation result of the first image and the original image according to the weight indicated by the inverse affine transformation result of the transparency information, so as to obtain a fusion image; and finally, determining a second image according to the fusion image, the deformation offset and a target mask capable of indicating a deformation region determined from the fusion image. Thus, the effective range of the deformation indicated by the deformation offset in the fusion image can be as far as possible limited in the deformation region by means of the target mask, so that the demand for deformation of part of the region in the original image can be better met, and the image processing effect, such as image definition, can be better improved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to an image processing method, apparatus, device, medium, or product. Background Technology

[0002] In some scenarios, such as facial enhancement, the following requirements may exist: given an image, adjust the face in that image to achieve a certain effect (such as slimming the face, removing a double chin, enlarging the eyes, highlighting the jawline, etc.). For example, in a scenario requiring a slimmer face, given an image from the user, the goal is to reduce the size of the face to achieve this effect. Similarly, in a scenario requiring the removal of a double chin, given an image from the user, the goal is to remove the double chin from the face to achieve this effect. Summary of the Invention

[0003] To achieve the above requirements, this application provides an image processing method, apparatus, device, medium, and product.

[0004] To achieve the above objectives, the technical solution provided in this application is as follows:

[0005] This application provides an image processing method, the method comprising: acquiring an original image; and predicting a first image, transparency information, and deformation offset based on the affine transformation result of the original image, wherein the original image belongs to a first space, and the affine transformation result and the first image both belong to a second space, the second space being different from the first space; the transparency information indicating the transparency of each pixel in the first image, and the deformation offset indicating the offset of coordinates in the second space; performing weighted fusion on the inverse affine transformation result of the transparency information and the original image according to the inverse affine transformation result of the transparency information to obtain a fused image, wherein the inverse affine transformation result of the transparency information indicating the weight of each pixel in the inverse affine transformation result of the first image in the weighted fusion; and determining a second image based on the fused image, the deformation offset, and a target mask determined from the fused image, wherein the target mask indicates a deformation region.

[0006] In one possible implementation, the target mask further indicates a non-deformation region; determining the second image based on the fused image, the deformation offset, and the target mask determined from the fused image includes: transforming the deformation offset from the second space to the first space according to the transformation matrix of the inverse affine transformation to obtain a transformed offset, the transformed offset indicating the offset of coordinates in the first space; updating the transformed offset according to the target mask to obtain an updated offset, the updated offset indicating that the offset of coordinates located in the non-deformation region is 0; offsetting the coordinates in the first space according to the updated offset to obtain an offset result; and performing grid sampling on the fused image according to the offset result to obtain the second image.

[0007] In one possible implementation, updating the transformed offset based on the target mask to obtain the updated offset includes: normalizing the target mask to obtain a normalization result, wherein the normalization result indicates the offset weight corresponding to the coordinates in the first space; and weighting the transformed offset according to the normalization result to obtain the updated offset.

[0008] In one possible implementation, the method further includes: performing Gaussian blurring on the target mask to obtain a blurred result; the step of normalizing the target mask to obtain a normalized result includes: normalizing the blurred result to obtain the normalized result.

[0009] In one possible implementation, transforming the deformation offset from the second space to the first space according to the transformation matrix of the inverse affine transformation to obtain the transformed offset includes: transforming the deformation offset from the coordinate system of the second space to the coordinate system of the first space according to the transformation matrix of the inverse affine transformation to obtain the transformed offset; and performing an inverse affine transformation on the transformed offset to obtain the transformed offset.

[0010] In one possible implementation, the maximum value of the deformation offset on the first coordinate axis and the maximum value of the deformation offset on the second coordinate axis are both 1; the step of transforming the deformation offset from the coordinate system of the second space to the coordinate system of the first space according to the transformation matrix of the inverse affine transformation to obtain the transformed offset includes: adjusting the range of the deformation offset according to the width and height of the second space to obtain the adjusted offset, wherein the maximum value of the adjusted offset on the first coordinate axis is the width of the second space, and the maximum value of the adjusted offset on the second coordinate axis is 1. The maximum value is the height of the second space; according to the transformation matrix of the inverse affine transformation, the adjusted offset is processed to obtain the processed offset. The maximum value of the processed offset on the first coordinate axis is the width of the first space, and the maximum value of the processed offset on the second coordinate axis is the height of the first space; based on the width and height of the first space, the processed offset is normalized to obtain the transformed offset. The maximum value of the transformed offset on the first coordinate axis and the maximum value of the transformed offset on the second coordinate axis are both 1.

[0011] In one possible implementation, the process of determining the target mask includes: determining a face mask and a skin mask from the fused image; dilating the face mask to obtain a dilated mask, and eroding the face mask to obtain an eroded mask; and determining the target mask based on the difference between the dilated mask and the eroded mask, and the difference between the skin mask and the face mask.

[0012] In one possible implementation, the process of determining the target mask is based on the face adjustment requirements achieved by the method; and / or, the affine transformation result describes a portion of the original image, the portion of the image including at least the face in the original image.

[0013] This application provides an image processing apparatus, comprising: an acquisition unit, configured to acquire an original image, and predict a first image, transparency information, and deformation offset based on an affine transformation result of the original image, wherein the original image belongs to a first space, and both the affine transformation result and the first image belong to a second space, the second space being different from the first space, the transparency information indicating the transparency of each pixel in the first image, and the deformation offset indicating the offset of coordinates in the second space; a fusion unit, configured to perform weighted fusion of the inverse affine transformation result of the first image and the original image according to the inverse affine transformation result of the transparency information to obtain a fused image, wherein the inverse affine transformation result of the transparency information indicates the weight of each pixel in the inverse affine transformation result of the first image in the weighted fusion; and a determination unit, configured to determine a second image based on the fused image, the deformation offset, and a target mask determined from the fused image, wherein the target mask indicates a deformation region.

[0014] This application provides an electronic device, the device comprising: a processor and a memory; the memory for storing instructions or computer programs; the processor for executing the instructions or computer programs in the memory, so that the electronic device performs the image processing method provided in this application.

[0015] This application provides a computer-readable medium storing instructions or a computer program that, when executed on a device, causes the device to perform the image processing method provided in this application.

[0016] This application provides a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the image processing method provided in this application.

[0017] Compared with related technologies, this application has at least the following advantages:

[0018] The technical solution provided in this application first acquires an original image, ensuring that the original image belongs to a first space; then, based on the affine transformation result of the original image, it predicts a first image, transparency information, and deformation offset, so that both the affine transformation result and the first image belong to a second space different from the first space. Furthermore, the transparency information indicates the transparency of each pixel in the first image, and the deformation offset indicates the offset of coordinates in the second space. This allows the first image to represent the result of adjusting (e.g., beautifying) the face in the affine transformation result to a certain extent. The transparency information indicates the accuracy of the predicted pixel values ​​in the first image, and the deformation offset indicates the type of deformation applied to the face in the first image. Finally, an inverse affine transformation based on the transparency information is performed. The weights indicated by the results are used to perform a weighted fusion of the inverse affine transformation result of the first image and the original image to obtain a fused image. This fused image represents the result of adjusting the face in the original image to a certain extent. Then, based on the fused image, the deformation offset, and a target mask determined from the fused image that indicates the deformation region, a second image is determined so that the changes in the second image compared to the fused image are located as close as possible within the deformation region. In this way, the effective range of the deformation indicated by the deformation offset in the fused image can be limited as much as possible within the deformation region by using the target mask. This better meets the need for deformation of certain areas (such as parts of the face) in the original image, thereby improving the image processing effect, such as image clarity. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 A schematic diagram of an image processing flow provided in an embodiment of this application;

[0021] Figure 2 A flowchart illustrating an image processing method provided in an embodiment of this application;

[0022] Figure 3 A schematic diagram illustrating another image processing flow provided in an embodiment of this application;

[0023] Figure 4 A schematic diagram illustrating a target mask determination process provided in an embodiment of this application;

[0024] Figure 5 This is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of this application;

[0025] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0026] Research has found that in certain scenarios, machine learning models, such as Generative Adversarial Networks (GANs), can be used to achieve the requirement of "adjusting the face in an image to achieve a certain effect." GANs are a commonly used image generation model in computer vision, comprising a generator and a discriminator. During training, the generator aims to generate the most realistic data possible, while the discriminator aims to accurately distinguish between generated and real data. This adversarial training process continuously optimizes the performance of both the generator and discriminator, resulting in a well-trained GAN model that can generate high-quality, indistinguishable data. Furthermore, GANs have wide-ranging applications, including image generation, style transfer, portrait retouching, facial enhancement, and special effects creation. Furthermore, in the field of portrait beautification, this GAN model can be used to achieve facial adjustments such as smoothing the face (that is, making the face more rounded), slimming the face, enlarging the eyes, removing double chins, highlighting the jawline, beautifying, whitening, improving skin texture, removing blemishes, removing acne, and removing wrinkles; and these facial adjustment needs all require certain deformation of the face.

[0027] The study also found that the process of achieving the above-mentioned facial adjustment requirements based on GAN models (such as...) Figure 1 The image processing procedure shown may include steps one through five below.

[0028] Step 1: Obtain the original image provided by the user. After that, regarding Perform affine transformations (such as) Figure 1 The affine transformation shown is used to obtain a face cropping image that includes the face. Among them, the The image space it belongs to is the original space. The image space it belongs to is the crop space.

[0029] Step 2: [The following text appears to be a separate, unrelated section:] ...the above... Inputting into a GAN model yields the target image I predicted and output by that GAN model. cAlpha Channel (α Channel or Alpha Channel) A c and deformation offset Δ c Among them, the I c It belongs to the clipping space; and this I c Compared to this Some adjustments were made (such as removing the double chin). This A c Able to indicate the I c The transparency of each pixel. This Δ c This includes the offset of each coordinate in the clipping space, so that the Δ c Able to represent I c The deformation required for the mid-face.

[0030] Step 3: For the above I c And the above A c Perform inverse affine transformations respectively (e.g.) Figure 1 The inverse affine transformation shown is used to obtain the transformed image I. o and the transformed channel A o So that A o Able to indicate the I o The transparency of each pixel in the image, and based on this A o The data at each position in the I are used as weights to calculate the weights for the I. o and the above Perform weighted fusion (e.g.) Figure 1 The weighted fusion shown above yields the fused image I. b .

[0031] Step 4: First, adjust the above Δ c With the coordinates G in clip space c Add them together to get the result G′ c Then, the G′ c Conversion (e.g.) Figure 1 The coordinate system is transformed back to the original space, resulting in the transformation G′. o ; and then the G′ o Perform an inverse affine transformation in order to transform G′ o The resolution size is transformed into the above The resolution size is used to obtain the transformation result G″. o (like Figure 1 The coordinates in the original space shown are offset.

[0032] Step 5: Use the above G″ o In the above I b Network sampling is performed on top (e.g.) Figure 1 (Sampling the grid shown) to obtain the final result image.

[0033] It should be noted that, regarding the above G″ o In this matrix, the G″ o This matrix represents the result after the coordinates in the original space have been shifted. Where G″ o (i,j) refers to the G″ o The data recorded in the i-th row and j-th column of this matrix, and the G″ o (i,j) is a coordinate value (that is, the coordinate value used when performing grid sampling) so that G″ o (i,j) can represent the pixel in the i-th row and j-th column of the final image above. b The position of the pixel in the i-th row and j-th column of the final image. It can be seen that the pixel in the i-th row and j-th column of the final image can refer to the pixel starting from the i-th column. b The sample obtained in this I b The middle is located or is closest to the G″. o The pixel at coordinate (i,j), where i is a positive integer, i ≤ the number of rows in the final image, and j is a positive integer, j ≤ the number of columns in the final image.

[0034] The study also found that for some facial adjustment needs, such as a smooth face, double chin removal, and a more defined jawline, these only require localized facial deformation, without the need for overall facial deformation. However, in the solutions shown in steps one through five above, due to the aforementioned Δ... c The deformation of the entire face is described to make it difficult for GANs running on some devices (such as terminal devices with computational limitations and real-time requirements) to accurately predict this Δ. c At that time, it is impossible to ensure that Δ c The value is 0 in the non-deformable region of the face (i.e., the region where deformation is not desired), thus causing the value based on Δ to be 0. c The resulting final image may appear blurry in many areas of the face, affecting the facial adjustment effect and user experience.

[0035] Based on the above research, in order to overcome the above problems, this application provides an image processing method, which includes: firstly acquiring an original image, such that the original image belongs to a first space; then predicting a first image, transparency information, and deformation offset based on the affine transformation result of the original image, such that the affine transformation result and the first image both belong to a second space different from the first space, and further enabling the transparency information to indicate the transparency of each pixel in the first image, and enabling the deformation offset to indicate the offset of coordinates in the second space, thereby enabling the first image to represent the result obtained by adjusting (e.g., beautifying) the face in the affine transformation result to a certain extent, enabling the transparency information to indicate the accuracy of the prediction of each pixel in the first image, and enabling the deformation offset to indicate what kind of deformation is applied to the face in the first image; then, according to the weight indicated by the inverse affine transformation result of the transparency information, predicting the first image, transparency information, and deformation offset based on the first image, such that the first image, transparency information, and deformation offset are all in the second space. The inverse affine transformation result of an image is weighted and fused with the original image to obtain a fused image that can represent the result of adjusting the face in the original image to a certain extent. Then, based on the fused image, the deformation offset, and a target mask determined from the fused image that can indicate the deformation area, a second image is determined so that the changes in the second image compared to the fused image are located as close as possible to the deformation area (i.e., the desired deformation area). In this way, the effective range of the deformation indicated by the deformation offset in the fused image can be limited as much as possible to the deformation area by means of the target mask, so as to overcome the blurring caused by the inability to ensure that the value of the deformation offset is 0 in the non-deformation area. This can better meet the needs of deforming part of the original image (such as part of the face), and thus help to improve the image processing effect, such as image clarity.

[0036] Furthermore, this application does not limit the subject executing the image processing method. For example, the method can be applied to a terminal device or a server. Alternatively, the method can be implemented through data interaction between the terminal device and the server. The terminal device can be a smartphone, computer, personal digital assistant (PDA), tablet computer, etc. The server can be a standalone server, a cluster server, or a cloud server.

[0037] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0038] To better understand the technical solution provided in this application, the image processing method provided in this application will be explained below with reference to some accompanying drawings. For example... Figure 2 As shown, the image processing method provided in this application includes S1-S3 below.

[0039] S1: Obtain the original image, and predict the first image, transparency information and deformation offset based on the affine transformation result of the original image. The original image belongs to the first space, and the affine transformation result and the first image both belong to the second space. The second space is different from the first space. The transparency information indicates the transparency of each pixel in the first image, and the deformation offset indicates the offset of the coordinates in the second space.

[0040] Here, the original image refers to the image that requires facial adjustment, such as... Figure 3 The original image shown or the above It should be noted that this application does not limit the method of acquiring the original image. For example, the original image may be provided or selected by the user through a certain method. Furthermore, when the image processing method provided in this application is applied to an electronic device (such as a terminal device), the original image may refer to an image provided by other tasks performed by the electronic device.

[0041] The affine transformation result of the original image refers to the image obtained by performing an affine transformation on the original image, such as... Figure 3 The face cropping diagram shown or the above This is to ensure that the image space to which the affine transformation result belongs is different from the image space to which the original image belongs. It can be seen that when the original image belongs to the first space (such as the cropping space mentioned above), and the affine transformation result belongs to the second space (such as the original space or...), the transformation will result in a different image space. Figure 3 When the original space is shown, the second space is different from the first space.

[0042] It should be noted that for the above affine transformations (such as...) Figure 3As shown in the affine transformation diagram, this affine transformation is a fundamental operation in image processing, and it can achieve geometric transformations of images through linear transformations and translations. Furthermore, this affine transformation can include some or all of the operations such as rotation, scaling, translation, flipping, and shearing. In addition, this affine transformation can map an image in the first space (such as the original image) to an image in the second space (such as the affine transformation result of the original image). Moreover, this affine transformation can be implemented using a pre-defined transformation matrix, which indicates what operations are performed on the image in the first space, such as cropping, scaling, or rotation, to obtain the image in the second space. This allows the image in the second space to better describe some of the information carried by the image in the first space, such as information describing facial features. Furthermore, this application does not limit the implementation of this affine transformation. For example, the affine transformation can be determined based on the image processing requirements of the actual application scenario of the image processing method provided in this application (such as facial adjustment requirements like face slimming).

[0043] As can be seen, in one possible implementation, the affine transformation result of the original image can refer to the result obtained by cropping, rotating, scaling and other operations on the original image through affine transformation, so that the face in the affine transformation result is rotated to a certain extent compared with the face in the original image, and the background information in the affine transformation result is less than the background information in the original image, and the resolution of the affine transformation result is less than the resolution of the original image.

[0044] The first image refers to the image predicted based on the affine transformation result of the original image mentioned above, such as... Figure 3 The target image shown or the above I c This is to enable the first image to represent the image obtained after certain processing of the affine transformation result (such as facial adjustment processing like removing a double chin), thereby allowing the first image to meet the processing requirements of the original image to a certain extent, such as the facial adjustment requirement of removing a double chin, and thus enabling the first image to represent the adjustment effect (such as beautification effect) on the face in the affine transformation result. Furthermore, the image space to which the first image belongs is the same as the image space to which the affine transformation result of the original image belongs, both being the aforementioned second space.

[0045] It should be noted that this application does not limit the implementation of the first image described above. For example, the first image can be a Red Green Blue (RGB) image. The RGB image is a color image synthesized based on the three basic color channels: red, green, and blue. The RGB image consists of three independent color channels: a red channel, a green channel, and a blue channel. Each channel typically uses an 8-bit binary number to represent the intensity or brightness of the color, so the value range for each channel is 0 to 255. 0 represents the minimum intensity (i.e., black) of the color channel, while 255 represents the maximum intensity (i.e., pure color). By mixing the three basic colors—red, green, and blue—at different intensities, almost all colors within the range of human vision can be produced. Specifically, when the intensity of all three colors is 0, the synthesized color is black; and when their intensity is 255, white is obtained. By adjusting the intensity values ​​of these three channels, a rich variety of colors can be obtained.

[0046] It should also be noted that this application does not limit the method of acquiring the first image. For example, it can be implemented using any pre-built model with image generation capabilities, such as any GAN model or any diffusion model. Therefore, in one possible implementation, the process of acquiring the first image may include: after obtaining the affine transformation result of the original image, inputting the affine transformation result into a pre-built GAN model (such as...). Figure 1 or Figure 3 The GAN model shown is used to obtain at least the first image predicted and output by the GAN model.

[0047] Transparency information refers to the information generated when predicting the first image based on the affine transformation result of the original image, which describes the transparency of the first image, such as... Figure 3 The Alpha channel shown or the aforementioned A c This is to enable the transparency information to indicate the transparency of each pixel in the first image, thereby allowing the transparency information to represent, to some extent, the prediction accuracy of each pixel in the first image, and further enabling the transparency information to represent, to some extent, the performance of each pixel in the first image in subsequent processes (such as...). Figure 3 The use value presented in the weighted fusion shown.

[0048] It should be noted that this application does not limit the implementation method of the above-mentioned transparency information. For example, the transparency information can be implemented using a matrix. Therefore, when the transparency information uses A... c When this matrix is ​​represented, A cThe data in the k-th row and l-th column of the first image can represent the transparency of the pixel located at the k-th row and l-th column, where k is a positive integer and k ≤ the value of A. c The number of rows in A, where l is a positive integer, l ≤ the number of rows in A. c The number of columns in the image. Furthermore, the size of this transparency information is determined based on the size of the first image, so that the transparency information can indicate the transparency of each pixel in the first image.

[0049] Deformation offset refers to the information generated when predicting the first image based on the affine transformation result of the original image, which describes the offset of coordinates in the second space, such as... Figure 3 The deformation offset shown or the above Δ c This allows the deformation offset to describe what kind of deformation is required for some body parts (such as the face and / or other parts) in the first image, thereby enabling the deformation offset to indicate to some extent what further processing is required for some body parts in the first image, and further enabling the deformation offset to indicate to some extent how to adjust some body parts indicated by the first image in subsequent processes.

[0050] It should be noted that this application does not limit the implementation method of the above-mentioned deformation offset. For example, the deformation offset can be implemented using a matrix. It can be seen that when the deformation offset uses Δ... c When this matrix is ​​represented, the Δ c The data in the k-th row and l-th column can represent the offset of the coordinate (k, l) in the second space, such as the offset on each coordinate axis, so that the Δ c The data in the k-th row and l-th column can represent the coordinate offset of the pixel located at the k-th row and l-th column in the first image belonging to the second space, where k is a positive integer and k ≤ the Δ c The number of rows in the Δ, where l is a positive integer, l ≤ the number of rows in the Δ. c The number of columns in the image. In addition, the size of the deformation offset is determined based on the size of the second space (such as the size of the first image) so that the deformation offset can indicate the offset of each coordinate in the second space.

[0051] Based on the above content related to S1, it can be understood that in one possible implementation, for the terminal device used by the user, when the terminal device receives the original image (e.g., ...) provided by the user... Figure 3 After showing the original image, an affine transformation is first performed on the original image to obtain the affine transformation result of the original image (such as...). Figure 3 The face cropping image shown is then input into a pre-built GAN model to obtain the first image predicted and output by the GAN model (e.g., the face cropping image shown). Figure 3The target image shown), transparency information (such as...) Figure 3 The alpha channel shown) and deformation offset (such as Figure 3 (The deformation offset shown).

[0052] S2: Based on the inverse affine transformation result of the transparency information, the inverse affine transformation result of the first image and the original image are weighted and fused to obtain a fused image. The inverse affine transformation result of the transparency information indicates the weight of each pixel in the inverse affine transformation result of the first image in the weighted fusion.

[0053] The inverse affine transformation result of the first image refers to the image obtained by performing an inverse affine transformation on the first image (as described in section I above). o ), so that the result of the inverse affine transformation can be realized in the first space (e.g., Figure 3 The image space described by the first image (shown in the original space) describes the facial information carried by the first image, so that the inverse affine transformation result can represent the adjustment effect (such as beautification effect) on the face in the original image. It can be seen that the image space to which the inverse affine transformation result of the first image belongs is the same as the image space to which the original image belongs, both being the first space.

[0054] It should be noted that the aforementioned inverse affine transformation refers to the inverse process of the affine transformation described above, enabling the inverse affine transformation to map the image in the second space back to the image in the first space. For example, the original image can be obtained by performing an inverse affine transformation on the affine transformation result. Furthermore, the inverse affine transformation can be implemented using a transformation matrix, which indicates what operation is performed on the image in the second space to obtain the image in the first space. Moreover, the inverse affine transformation is determined based on the affine transformation.

[0055] The inverse affine transformation result of transparency information refers to the result obtained by performing an inverse affine transformation on the transparency information (such as A above). o This allows the inverse affine transformation result of the transparency information to represent the transparency of each pixel in the inverse affine transformation result of the first image, thereby enabling the inverse affine transformation result of the transparency information to indicate the prediction accuracy of each pixel in the inverse affine transformation result of the first image, and further enabling the inverse affine transformation result of the transparency information to indicate the transparency of each pixel in the inverse affine transformation result of the first image in the aforementioned weighted fusion (such as...). Figure 3 The weights in the weighted fusion shown are such that the inverse affine transformation result of the transparency information can indicate the likelihood of each pixel in the inverse affine transformation result of the first image being used in the weighted fusion.

[0056] A fused image is an image obtained by weighted fusion of the inverse affine transformation result of the first image and the original image, such as... Figure 3 The fused image shown or the above I b This is to ensure that some pixels in the fused image come from the inverse affine transformation result, some pixels in the fused image come from the original image, and some pixels in the fused image are obtained by fusing the corresponding pixels in the inverse affine transformation result and the corresponding pixels in the original image. This allows the fused image to better represent the adjustment effect on the face in the original image.

[0057] In addition, this application does not limit the method of obtaining the above-mentioned fused image. For example, it can be implemented using the formula (1) below.

[0058]

[0059] In the formula, I b (i,j) represents the above fused image I. b The pixel located at the i-th row and j-th column; A o (i,j) represents the inverse affine transformation result A of the above transparency information. o The data located at the i-th row and j-th column (such as the single data point of transparency); I o (i,j) represents the inverse affine transformation result I of the first image mentioned above. o The pixel located at the position of row i and column j; The above original image The pixel located at the i-th row and j-th column, where i is a positive integer and i ≤ the pixel in the fused image I. b The row number in the image, j is a positive integer, j≤ the fused image I b The number of columns in the table.

[0060] Based on the above content related to S2, in one possible implementation, for the terminal device used by the user, after the terminal device predicts the first image and transparency information, it first performs inverse affine transformation on the first image and transparency information respectively; then, according to the inverse affine transformation result of the transparency information, it performs weighted fusion of the inverse affine transformation result of the first image and the original image to obtain a fused image (e.g., ...). Figure 3 The fused image shown is used to represent the result of adjustments made to the face in the original image.

[0061] S3: Determine a second image based on the fused image, the deformation offset, and the target mask determined from the fused image, wherein the target mask indicates the deformation region.

[0062] The target mask refers to the mask determined based on the fused image that indicates the deformed region, such as... Figure 3 The target mask shown is configured to limit the effective range of the deformation indicated by the aforementioned deformation offset, thereby overcoming the blurring caused by the inability to ensure that the deformation offset has a value of 0 in the non-deformation region. It is evident that the target mask can indicate which coordinates in the first space are located in the deformation region and which coordinates are located in the non-deformation region, so that the target mask can at least indicate the deformation region in the first space, thereby ensuring that the target mask can at least indicate the deformation region within the image (such as the fused image) in the first space.

[0063] Furthermore, this application does not limit the implementation of the target mask. For example, the target mask can at least satisfy the following constraints: in the target mask, the mask value of 0 is used to represent the non-deformation area, and the mask value of 255 is used to represent the deformation area, so that the target mask can accurately represent the undesired deformation area and the desired deformation area, thereby limiting the deformation area in the process of determining the second image.

[0064] Furthermore, this application does not limit the method of obtaining the aforementioned target mask. For example, it can be manually annotated by relevant personnel based on the fused image. Alternatively, the target mask can be obtained by processing the fused image and text describing the facial adjustment requirements of the current application scenario using a pre-built machine learning model. This application does not limit the implementation method of the machine learning model.

[0065] The second image refers to an image obtained by processing the original image described above, which can represent the final adjustment effect (such as beautification effect) on the face in the original image, such as... Figure 3 The final result shown is intended to enable the second image to better meet the processing needs of the original image, such as facial adjustments like removing a double chin.

[0066] Furthermore, this application does not limit the implementation of S3 described above. For example, S3 can be implemented using any pre-trained machine learning model capable of generating images based on fused images, deformation offsets, and target masks. It should be noted that this application does not limit the implementation of this machine learning model.

[0067] For example, to better reduce computational burden, step four above can be used to implement S3. Therefore, in one possible implementation, S3 can specifically include: first adding the aforementioned deformation offset to the coordinates in the second space to obtain the sum (such as G′ above). cThis is done so that the sum can represent the offset coordinates in the second space; then the sum is transformed into a new coordinate system (e.g., ...). Figure 1 The coordinate system transformation shown is used to transform the addition result from the coordinate system of the second space to the coordinate system of the first space, so as to obtain the transformation result (such as G′ above). o Then perform an inverse affine transformation on the result (e.g., ...). Figure 1 The inverse affine transformation shown above is used to obtain the transformation result (as described above, G″). o The transformation is performed to represent the offset coordinates in the first space. Then, based on the target mask, the activation information corresponding to the transformation result is determined so that the activation information can indicate that the deformation is active within the deformation region described by the target mask as indicated by the transformation result, while the deformation is inactive in other regions. Finally, the fused image is mesh-sampled according to the activation information and the transformation result to obtain a second image, so that the changes in the second image compared to the fused image are located as close as possible within the deformation region.

[0068] It should be noted that this application does not limit the implementation method of the effective information corresponding to the above transformation result. For example, when the transformation result adopts G″ o When this matrix is ​​implemented, the effective information can be implemented using a matrix E, the size of which is the same as that of G″. o This matrix has the same size, and the data (such as 0 or 1) at the i-th row and j-th column position in this matrix E can represent G″. o Whether the data at the i-th row and j-th column position in this matrix is ​​effective, so that when effective, the pixel at the i-th row and j-th column position in the second image refers to the pixel according to G″. o The data (such as coordinates) at the position of row i and column j in this matrix are the pixels sampled from the above-mentioned fused image. However, when not in effect, the pixel at the position of row i and column j in the above-mentioned second image refers to the pixel at the position of row i and column j in the fused image.

[0069] Based on the above-mentioned content of S1 to S3, it can be seen that in the image processing scheme provided in this application, an original image is first acquired so that the original image belongs to a first space; then, based on the affine transformation result of the original image, a first image, transparency information, and deformation offset are predicted, so that the affine transformation result and the first image both belong to a second space different from the first space. Furthermore, the transparency information can indicate the transparency of each pixel in the first image, and the deformation offset can indicate the offset of coordinates in the second space. This allows the first image to represent the result of adjusting (e.g., beautifying) the face in the affine transformation result to a certain extent. The transparency information can indicate the accuracy of the prediction of each pixel in the first image, and the deformation offset can indicate what kind of deformation is applied to the face in the first image. Finally, according to the weight indicated by the inverse affine transformation result of the transparency information... The inverse affine transformation result of the first image and the original image are weighted and fused to obtain a fused image, which can represent the result of adjusting the face in the original image to a certain extent. Then, based on the fused image, the deformation offset, and the target mask determined from the fused image that can indicate the deformation area, a second image is determined so that the changes in the second image compared to the fused image are located as close as possible within the deformation area. In this way, the effective range of the deformation indicated by the deformation offset in the fused image can be limited as much as possible within the deformation area by using the target mask, so as to overcome the blurring caused by the inability to ensure that the value of the deformation offset is 0 in the non-deformation area. This can better meet the needs of deforming part of the original image (such as part of the face), and thus help to improve the image processing effect, such as image clarity.

[0070] In addition, in some scenarios, such as face adjustment scenarios, the affine transformation result of the original image can at least satisfy the following constraints: the affine transformation result describes a part of the original image (such as the face region), and the part of the image includes at least the face in the original image, so that subsequent processing of the face in the original image can be achieved by using the relevant processing of the affine transformation result. In this way, the interference caused by other content (such as the background) in the original image can be overcome as much as possible by processing the affine transformation result, thereby improving the image processing effect.

[0071] As can be seen, in one possible implementation, after obtaining the original image given by the user, affine processing is performed on the original image to obtain the affine transformation result of the original image (e.g., ...). Figure 3The face cropping diagram shown is used to make the affine transformation result better describe the face in the original image, so that subsequent adjustments (such as beautification) to the face in the original image can be achieved by processing the affine transformation result. This can effectively overcome the interference caused by factors such as the face being tilted or the presence of a large background in the original image, thereby improving the image processing effect.

[0072] Research has revealed that the implementation of S3, achieved through step four above, has the following characteristics: First, the deformation offset is added to the coordinates in the second space; then, coordinate system transformation and inverse affine transformation are performed; finally, network sampling is performed on the fused image in the first space. This means that the coordinates used for grid sampling in the first space may be obtained by interpolating the coordinates in the second space during the inverse affine transformation. Consequently, the "coordinates used for grid sampling in the first space" may not belong to the coordinates in the first space. This may lead to misalignment between the coordinate values ​​and the image space. Therefore, when sampling according to the "coordinates used for grid sampling in the first space", it may be necessary to use a certain estimation method (such as finding the nearest neighbor or averaging) to achieve sampling, which may result in sampling errors and affect the clarity of the final image.

[0073] Based on the above research, in order to overcome the above problems, this application also provides a possible implementation of the above S3. In this implementation, when the target mask also indicates a non-deformable region (that is, a region where deformation is not desired), S3 may include: transforming the deformation offset from the second space to the first space according to the transformation matrix of the inverse affine transformation to obtain the transformed offset (e.g., Figure 3 The deformation offset shown is a deformation offset belonging to the original space and having the same resolution as the original image. This transformed offset indicates the offset of the coordinates in the first space. The transformed offset is updated according to the target mask to obtain an updated offset, which indicates that the offset of the coordinates located in the non-deformation region is 0. The coordinates in the first space are offset according to the updated offset to obtain an offset result. The fused image is then mesh-sampled according to the offset result to obtain a second image. In this way, by first losslessly transforming the predicted deformation offset to the first space and then offsetting the coordinates in the first space, it can be ensured that the coordinates used for sampling all belong to the coordinates in the first space. This effectively overcomes the defect caused by the coordinates used during sampling not belonging to the coordinates in the first space, thereby improving image clarity.

[0074] It should be noted that this application does not limit the implementation of the above-mentioned step of "transforming the deformation offset from the second space to the first space according to the transformation matrix of the above-mentioned inverse affine transformation to obtain the transformed offset". For example, it can be implemented by any method that can implement this step, such as a script pre-set for the aforementioned function, or a machine learning model with the aforementioned function pre-built, or a rule pre-set for the aforementioned function.

[0075] It should also be noted that this application does not limit the implementation of the above-mentioned step of "updating the transformed offset according to the above-mentioned target mask to obtain the updated offset". For example, it can be specifically as follows: for the position of the i-th row and j-th column in the transformed offset, if the position of the i-th row and j-th column is located in the non-deformation area indicated by the target mask, it can be determined that the deformation indicated by the data (i.e., the offset) at the position of the i-th row and j-th column in the transformed offset belongs to the deformation caused by the prediction error, rather than the deformation expected in the current scene. Therefore, the data at the position of the i-th row and j-th column in the transformed offset can be directly adjusted to 0, where i is a positive integer, i≤ the number of rows in the transformed offset, and j is a positive integer, j≤ the number of columns in the transformed offset.

[0076] Based on the above three paragraphs, it can be seen that in one possible implementation, the deformation offset (such as the aforementioned Δ) is obtained through prediction. c Next, according to the transformation matrix of the inverse affine transformation described above, the deformation offset is losslessly transformed from the second space to the first space to obtain a transformed offset. This transformed offset can represent the offset of each coordinate in the first space, thus describing what kind of deformation is applied to the image (such as the fused image described above) in the first space. Then, the transformed offset is updated according to the target mask described above to obtain an updated offset, so that the updated offset can at least indicate that no deformation occurs in the non-deformation region indicated by the target mask. Then, according to the updated offset, the coordinates in the first space are offset to obtain an offset result, so that the coordinates located in the non-deformation region do not need to be offset during the determination of the offset result. Finally, according to the offset result, the fused image is mesh sampled to obtain a second image, so that the part of the second image located in the non-deformation region is not deformed compared to the part of the fused image located in the non-deformation region, thereby improving the clarity of the second image.

[0077] Research has revealed that in the aforementioned target mask, the non-deformation region is represented by the mask value 0, and the deformation region is represented by the mask value 255. This allows the normalization result of the target mask to be directly used as a weight: the offset of the coordinates in the non-deformation region in the first space is ineffective, but the offset of the coordinates in the deformation region is effective.

[0078] Based on the above research, in order to better improve the processing effect, this application also provides an implementation method for the step of "updating the transformed offset according to the target mask to obtain the updated offset". In this method, the step may include: normalizing the target mask (e.g., dividing by 255) to obtain a normalization result, the normalization result indicating the offset weight corresponding to the coordinates in the first space; and weighting the transformed offset according to the normalization result to obtain the updated offset (e.g., ...). Figure 3 The weighted deformation offset shown is used to ensure that the updated offset indicates that the coordinates located in the non-deformation region do not shift. This reduces the difficulty of obtaining the updated offset by normalization, which helps to improve processing efficiency.

[0079] As can be seen, in one possible implementation, after obtaining the target mask, the target mask is first normalized to obtain a normalized result, so that all data in the normalized result are 0 or 1, thereby enabling the normalized result to represent the possibility of each coordinate shift in the first space, and thus enabling the normalized result to represent the accuracy of each offset in the transformed offset to a certain extent. In this way, the normalized result can represent the weight of each offset in the transformed offset to a certain extent. Then, according to the normalized result, the transformed offset is weighted to obtain an updated offset, so that the updated offset can more accurately represent what kind of deformation the fused image undergoes, which is beneficial to improving the image processing effect.

[0080] Research has revealed that in the aforementioned target mask, the non-deformed region is represented by the mask value 0, while the deformed region is represented by the mask value 255. This results in a noticeable break between the edges of the non-deformed and deformed regions indicated by the target mask, thus affecting the image processing effect.

[0081] Based on the above research, in order to ensure a smooth transition between the non-deformable region and the deformable region, this application also provides an implementation method for the step of "updating the transformed offset according to the target mask to obtain the updated offset". In this method, the step may include: applying Gaussian blur to the target mask (e.g., ...). Figure 3The Gaussian blur shown is used to obtain a blurred result, which includes not only mask values ​​of 0 and 255, but also some mask values ​​between 0 and 255. This creates a smooth transition region between the non-deformed region and the deformed region indicated by the blurred result. The blurred result is then normalized to obtain a normalized result, ensuring that all data in the normalized result belong to the interval [0, 1]. This normalized result indicates that the offset weights for coordinates within the non-deformed region are 0 and those within the deformed region are 1. It also indicates that the offset weights for coordinates within the transition region between the non-deformed and deformed regions gradually increase from 0 to 1. Based on this normalized result, the transformed offset is weighted to obtain the updated offset (e.g., ...). Figure 3 The weighted deformation offset shown is used to indicate that the offset in the non-deformation region is 0, the offset in the deformation region is 1, and the offset in the transition region between the non-deformation region and the deformation region gradually increases from 0 to 1. This ensures that the offset in the deformation region in the updated offset is still obtained through prediction, thereby avoiding obvious discontinuities at the edge between the non-deformation region and the deformation region while ensuring that the final deformation effect is obvious enough. This is beneficial to improving the image processing effect.

[0082] As can be seen, in one possible implementation, after obtaining the target mask, the target mask is first subjected to Gaussian blurring to obtain a blurred result, so that the blurred result can blur the edges between the non-deformed and deformed regions while preserving as much of the non-deformed and deformed regions as possible; then the blurred result is normalized to obtain a normalized result (e.g., ...). Figure 3 The deformation offset weights shown are normalized to ensure a smooth transition from 0 to 1 between the non-deformed region and the deformed region; then, the normalized result is multiplied by the transformed offset to obtain the updated offset (e.g., ...). Figure 3 The weighted deformation offset shown can be used to avoid obvious breaks at the edge between the non-deformed area and the deformed area while ensuring that the final deformation effect is obvious enough, thus helping to improve the image processing effect.

[0083] Furthermore, to further improve efficiency, this application also provides a possible implementation of the step of "transforming the deformation offset from the second space to the first space according to the transformation matrix of the inverse affine transformation to obtain the transformed offset". In this implementation, the step may include: transforming the deformation offset from the coordinate system of the second space to the coordinate system of the first space according to the transformation matrix of the inverse affine transformation to obtain the transformed offset, so that the transformed offset can represent the offset in the first space, thereby making the transformed offset belong to the first space; performing an inverse affine transformation on the transformed offset to obtain the transformed offset, so that the transformed offset can represent the offset belonging to the first space and having the same resolution as the original image. In this way, the deformation offset can be quickly and losslessly mapped to the first space by first transforming the coordinate system and then performing an inverse affine transformation, thereby not only reducing the resource overhead of the mapping but also improving the efficiency of the mapping, which is conducive to improving the image processing effect.

[0084] It should be noted that this application does not limit the implementation of the above-mentioned step of "transforming the deformation offset from the coordinate system of the second space to the coordinate system of the first space according to the transformation matrix of the inverse affine transformation to obtain the transformed offset". For example, it can be implemented by any method that can perform coordinate system transformation, such as a script pre-set for the aforementioned function, or a machine learning model with the aforementioned function pre-built, or a rule pre-set for the aforementioned function.

[0085] Based on the above two paragraphs, it can be seen that in one possible implementation, after predicting the above deformation offset (such as...) Figure 3 After the deformation offset shown, first perform a coordinate system transformation on the deformation offset according to the transformation matrix of the inverse affine transformation mentioned above (e.g., Figure 3 The coordinate system transformation shown is used to transform the deformation offset from the coordinate system of the second space to the coordinate system of the first space, so as to obtain the transformed offset, which can represent the offset of some coordinates in the first space to a certain extent; then, according to the transformation matrix of the inverse affine transformation, the transformed offset is processed to achieve the inverse affine transformation of the transformed offset, so as to obtain the transformed offset (such as...). Figure 3The deformation offset shown belongs to the original space and has the same resolution as the original image. This is to ensure that the transformed offset not only belongs to the original space, but also that the resolution of the transformed offset is the same as the resolution of the image in the first space (such as the original image or the above-mentioned fused image). This allows the transformed offset to represent the offset of all coordinates in the first space, and thus allows the transformed offset to better represent what kind of deformation is performed on the fused image. This is beneficial to improving the efficiency of obtaining the transformed offset and reducing the difficulty of obtaining the transformed offset.

[0086] Furthermore, to better reduce the processing difficulty, this application also provides a possible implementation of the above-mentioned step of "transforming the deformation offset from the coordinate system of the second space to the coordinate system of the first space according to the transformation matrix of the inverse affine transformation to obtain the transformed offset". In this method, when the maximum value of the deformation offset on the first coordinate axis (such as the X-axis) and the maximum value of the deformation offset on the second coordinate axis (such as the Y-axis) are both 1, this step may include: adjusting the range of the deformation offset according to the width of the second space and the height of the second space to obtain an adjusted offset, wherein the maximum value of the adjusted offset on the first coordinate axis is the width of the second space, and the maximum value of the adjusted offset on the second coordinate axis is the height of the second space; processing the adjusted offset according to the transformation matrix of the inverse affine transformation to obtain a processed offset, wherein the value of the processed offset on the first coordinate axis is... The maximum value of the domain is the width of the first space, and the maximum value of the processed offset on the second coordinate axis is the height of the first space. Based on the width and height of the first space, the processed offset is normalized to obtain the transformed offset. The maximum value of the transformed offset on the first coordinate axis and the maximum value of the transformed offset on the second coordinate axis are both 1, so that the transformed offset can represent the normalized offset in the first space. In this way, the coordinate system transformation for the deformation offset can be achieved through value domain adjustment → data processing (such as rotation and scaling) → normalization. This allows the coordinate system transformation to be achieved at a lower cost (such as less computing resources and less time overhead), thereby reducing the implementation difficulty of the image processing method provided in this application and expanding the application scope of the image processing method, such as applying the image processing method to terminal devices with limited computing resources.

[0087] It can be seen that, in one possible implementation, the above deformation offset can be transformed from the second space (such as the above-mentioned clipping space) to the first space (such as the above-mentioned original space) without loss using the process shown in formulas (2)-(6) below.

[0088]

[0089] Δ o =WarpAffine(Δ t ,T′,W o H o (6)

[0090] In the formula, Δ c This indicates the aforementioned deformation offset, and Indicates will This scalar element and These scalar elements are combined into a vector Δ c , Indicates that Δ c The offset on the first coordinate axis (such as the X-axis). Indicates that Δ c Offset on the second coordinate axis (e.g., the Y-axis); W c This represents the width of the second space, which is the image within that second space (e.g., ...). Figure 3 The width of the face cropped image shown; H c T' represents the height of the second space, which is the height of the image in the second space; T' represents the transformation matrix of the inverse affine transformation described above; T' 1,1 This represents the element in the first row and first column of the transformation matrix of the inverse affine transformation; T′ 1,2 This represents the element in the first row and second column of the transformation matrix of the inverse affine transformation; T′ 2,1 This represents the element in the 2nd row and 1st column of the transformation matrix of the inverse affine transformation; T′ 2,2 W represents the element in the 2nd row and 2nd column of the transformation matrix for the inverse affine transformation. o This represents the width of the first space, which is the image within that first space (e.g., ...). Figure 3 The width of the original image shown; H o This represents the height of the first space, which is the height of the image within that first space; Δ t This indicates that the Δ c The result of the coordinate system transformation is to make the Δ t It can represent the normalized offset within the first space, and Indicates will This scalar element and These scalar elements are combined into a vector Δ o , Indicates that Δ o The offset on the first coordinate axis (such as the X-axis). Indicates that Δ oThe offset on the second coordinate axis (such as the Y-axis); WarpAffine(·) represents the transformation function, so that WarpAffine(Δ) t ,T′,W o H o It can represent the transformation matrix T′ based on the inverse affine transformation and the width W of the first space. o And the height H of the first space o To achieve this Δ t Inverse affine transformation; Δ o This indicates that the deformation offset Δ c The result obtained by lossless transformation from the second space to the first space, so that the Δ o It can represent the offset that belongs to the first space and has the same resolution as the original image.

[0091] Based on the relevant content of the above formulas (2)-(5), it can be seen that the above deformation offset Δ is obtained through prediction. c Then, the deformation offset Δ c The process of lossless transformation to the first space is as follows:

[0092] First, due to the deformation offset Δ c Its range is [0, 1], therefore this Δ c Offsets on the X and Y axes and Multiply by the width and height of the second space respectively to expand their ranges to [0, W]. c ] and [0, H c [, to obtain the two components after the value range is amplified;]

[0093] Then, based on the affine principle, since the elements of the first row and first two columns of the transformation matrix T′ of the inverse affine transformation can represent the rotation and scaling from the second space to the first space on the X-axis, we can multiply the two components after range amplification by the elements of the first row and first two columns of T′ and then sum them to obtain the Δ. c The offset along the X-axis in the first space;

[0094] Similarly, since the elements in the second row and first two columns of the transformation matrix T′ of the inverse affine transformation can represent the rotation and scaling from the second space to the first space on the Y-axis, the two components after range amplification can be multiplied by the elements in the second row and first two columns of the transformation matrix T′ of the inverse affine transformation and then summed to obtain the deformation offset Δ. c The offset along the Y-axis in the first space, where the ranges of these two offsets are [0, W], respectively. o ] and [0, H oTherefore, these two offsets are divided by the width W of the first space. o and the height H of the first space o This allows us to obtain the normalized offset Δ within the first space. t ;

[0095] Finally, based on the transformation matrix T′ of the inverse affine transformation and the width W of the first space... o And the height H of the first space o To achieve the target Δ t The inverse affine transformation yields the deformation offset Δ with the same resolution as the original image and a value range of [0, 1]. o This allows the deformation offset Δ to be achieved. c Lossless transformation to this first space to obtain Δ o .

[0096] Research has shown that, in order to improve flexibility, a target mask can be obtained by performing certain processing on the fused image (such as semantic segmentation, graphics dilation and erosion, mask elimination, mask fusion, etc.), so that the target mask can represent the deformed and non-deformed regions required in the current scene.

[0097] Research has revealed that the deformation regions specified for different facial adjustment needs are different. Therefore, to improve flexibility, this application also provides a possible implementation of the aforementioned target mask. In this implementation, the target mask can be determined based on the facial adjustment needs implemented by the image processing method provided in this application, so that the target mask can accurately represent the deformation regions and non-deformation regions specified by the facial adjustment needs. This allows the target mask to accurately represent the deformation characteristics specified by the facial adjustment needs, thereby better meeting the facial adjustment needs.

[0098] As can be seen, in one possible implementation, after determining the facial adjustment requirements specified by the user (such as removing a double chin) based on user operations, a mask determination method matching the facial adjustment requirements is first searched from a pre-built mapping relationship; then, the fused image is processed using the matching mask determination method to obtain a target mask, so that the target mask can accurately represent the deformed and non-deformed regions specified by the facial adjustment requirements, so that an image satisfying the facial adjustment requirements can be obtained by subsequently performing deformation processing on the fused image based on the target mask (such as removing a double chin). Figure 3The final effect shown (or the second image mentioned above) enhances flexibility. This mapping relationship records mask determination methods corresponding to various facial adjustment requirements, enabling the mapping relationship to indicate how the mask indicating the deformed area is determined under different facial adjustment requirements. Furthermore, this application does not limit the construction method of this mapping relationship; for example, it can be manually constructed by relevant personnel. Alternatively, the mapping relationship can be obtained by analyzing a large number of documents related to facial adjustment requirements.

[0099] To make it easier to understand, two examples will be used to illustrate the point below.

[0100] Example 1: When the user-specified facial adjustment requirement is a smooth face (e.g., making the face shape more rounded), the process of determining the target mask may include: determining a face mask and a skin mask from the fused image, such that the face mask can represent the position of the face in the fused image, and the skin mask can represent the position of the skin in the fused image; dilating the face mask to obtain a dilated mask, such that the face area indicated by the dilated mask is larger than the face area indicated by the face mask; and performing [further steps on the face mask]. Erosion yields an eroded mask, ensuring that the facial region indicated by the eroded mask is smaller than the facial region indicated by the facial mask. Based on the difference between the dilated mask and the eroded mask, and the difference between the skin mask and the facial mask, a target mask is determined. This target mask is designed to represent both the deformed regions (such as the edge regions of the face and the neck region) and the non-deformed regions (such as the middle region of the face) specified by the facial adjustment requirements. This allows the target mask to accurately represent the deformation characteristics specified by the facial adjustment requirements, thus better meeting those requirements.

[0101] It should be noted that this application does not limit the implementation of the above-mentioned step of "determining the target mask based on the difference between the dilated mask and the eroded mask, and the difference between the skin mask and the face mask". For example, it can be implemented using the formulas (7)-(9) below.

[0102] M n =M s -M f (7)

[0103]

[0104] In the formula, M f Indicates a face mask; M s Indicates skin mask; M n A mask representing skin other than facial skin (e.g.) Figure 4 The deformed region mask shown is used to make the M n It can represent areas other than the face that need to be deformed; This indicates that the face mask M f The inflated mask obtained after inflating; This indicates that the face mask M f The etched mask obtained by performing etching; A mask representing the facial contour and its adjacent areas (e.g.) Figure 4 The deformed region mask shown is used to make the face area (deformation region mask shown) so that It can represent the area of ​​the face that needs to be deformed; M represents the target mask.

[0105] It should also be noted that this application does not limit the method of obtaining the face mask. For example, it can be implemented using any method that can extract the face mask from the image, such as any face segmentation algorithm or any machine learning model with face mask determination function.

[0106] It should also be noted that this application does not limit the method of obtaining the skin mask. For example, it can be implemented using any method that can extract the skin mask from the image, such as any skin segmentation algorithm or a machine learning model with skin mask determination function.

[0107] It's also important to note that image dilation and erosion are two fundamental morphological operations, primarily used to find maxima and minima in an image. Dilation is similar to region expansion, extending bright or white areas outwards, resulting in larger bright areas in the dilated image compared to the undilated image. Erosion, on the other hand, is similar to region erosion, shrinking bright or white areas inwards, resulting in smaller bright areas in the eroded image compared to the uneroded image.

[0108] Based on the content of Example 1 above, it can be seen that when the user specifies a facial adjustment requirement as a smooth face (such as making the face shape more rounded), such as Figure 4 As shown, after obtaining the fused image, face segmentation and skin segmentation are first performed on the fused image to obtain face masks and skin masks; then, dilation and erosion are performed on the face masks to obtain dilated masks and eroded masks; finally, the eroded mask is removed from the dilated mask by subtraction to obtain the first residual mask (e.g., ...). Figure 4 The deformed area mask shown is used to remove the face mask from the skin mask by subtraction, resulting in a second residual mask (such as...). Figure 4 The deformed region mask outside the face area is shown; finally, the first residual mask and the second residual mask are merged by addition (e.g., Figure 4 The fusion shown is used to obtain a target mask so that the target mask can accurately represent the deformation characteristics specified by the facial adjustment requirements, thus better meeting the facial adjustment requirements.

[0109] Example 2: When the user-specified facial adjustment requirement is face slimming, the process of determining the target mask may include: determining a face mask and a skin mask from the fused image; eroding the face mask to obtain an eroded mask, such that the face area indicated by the eroded mask is smaller than the face area indicated by the face mask; determining a target mask based on the difference between the face mask and the eroded mask, and the difference between the skin mask and the face mask, so that the target mask can represent the deformed area and the non-deformed area specified by the facial adjustment requirement, thereby enabling the target mask to accurately represent the deformation characteristics specified by the facial adjustment requirement, thus better meeting the facial adjustment requirement.

[0110] As can be seen, when the user specifies a face slimming requirement, after obtaining the fused image, the process first performs face segmentation and skin segmentation on the fused image to obtain a face mask and a skin mask. Then, the face mask is eroded to obtain an eroded mask. Next, the eroded mask is removed from the face mask by subtraction to obtain a third residual mask, and the face mask is removed from the skin mask by subtraction to obtain a fourth residual mask. Finally, the third residual mask and the fourth residual mask are fused by addition to obtain the target mask, so that the target mask can accurately represent the deformation characteristics specified by the face adjustment requirement, thus better meeting the face adjustment requirement.

[0111] Based on the aforementioned image processing methods, the technical solution provided in this application may include: after acquiring the fused image, firstly, by performing certain processing on the fused image (such as semantic segmentation, graphical dilation and erosion, mask elimination, mask fusion, etc.) to obtain a target mask, so that the mask value of the non-deformed region in the target mask is 0 but the mask value of the deformed region is 255, thereby enabling the target mask to accurately indicate the non-deformed region and the deformed region; then, Gaussian blurring is applied to the target mask to ensure that the edge transition between the non-deformed region and the deformed region is sufficiently smooth; finally, normalization (e.g., dividing by 255) is applied to the Gaussian blurred mask to obtain the offset weight (e.g., ...). Figure 3 The deformation offset weight is shown; then, this offset weight is combined with the deformation offset (as shown in Δ above) by...c The offset obtained by lossless transformation to the original space (such as Δ above) o Perform matrix dot product to obtain the weighted offset (e.g., ...). Figure 3 The weighted deformation offset shown is used to ensure that the offset of the non-deformed region is 0, the offset of the transition zone from the non-deformed region to the deformed region gradually increases from 0 to 1, and the offset of the deformed region is 1. This ensures that most of the offset of the deformed region still comes from the predicted deformation offset, thus avoiding obvious discontinuities at the edge between the non-deformed and deformed regions while ensuring that the deformation effect is obvious enough. Finally, the weighted offset is added to the coordinates in the original space to obtain the coordinate values ​​required for grid sampling, so that the coordinate values ​​can be used to perform grid sampling in the fused image to obtain the final effect image to be fed back to the user.

[0112] It is evident that the technical solution provided in this application has the following two advantages: First, after obtaining the fused image, this application first obtains a mask (such as the target mask mentioned above) from the fused image in a certain way that can indicate the deformed and non-deformed regions; then, it performs Gaussian blur and normalization operations on the mask in sequence to obtain the offset weights for edge smoothing. This is beneficial for adaptively controlling the deformed areas of the face based on the face adjustment requirements of the current scene, ensuring that most face areas that do not require deformation do not undergo any deformation, thereby ensuring that the pixels in most face areas that do not require deformation mainly or completely come from the original image, thus significantly improving the clarity of the final image. Secondly, this application obtains the coordinate values ​​required for mesh sampling by first losslessly transforming the predicted deformation offset to the image space of the original image, and then adding the coordinates in the image space of the original image. This ensures that the coordinate values ​​used in mesh sampling are all coordinates in the image space of the original image, thus effectively solving the problem of misalignment between coordinate values ​​and image space. As a result, the clarity of the deformation-free area is completely consistent with the original image, which helps to improve the clarity of the final image.

[0113] Based on the image processing method provided in the embodiments of this application, the embodiments of this application also provide an image processing apparatus, which will be described below in conjunction with... Figure 5 Explanation and clarification will be provided. Among them, Figure 5 This is a schematic diagram of an image processing apparatus provided in an embodiment of this application. It should be noted that for technical details of the image processing apparatus provided in this application embodiment, please refer to the relevant content of the image processing method above.

[0114] like Figure 5 As shown, the image processing apparatus 500 provided in this application embodiment includes:

[0115] The acquisition unit 501 is used to acquire an original image and predict a first image, transparency information, and deformation offset based on the affine transformation result of the original image. The original image belongs to a first space, and the affine transformation result and the first image both belong to a second space. The second space is different from the first space. The transparency information indicates the transparency of each pixel in the first image, and the deformation offset indicates the offset of the coordinates in the second space.

[0116] The fusion unit 502 is used to perform weighted fusion of the inverse affine transformation result of the first image and the original image according to the inverse affine transformation result of the transparency information to obtain a fused image. The inverse affine transformation result of the transparency information indicates the weight of each pixel in the inverse affine transformation result of the first image in the weighted fusion.

[0117] The determining unit 503 is used to determine a second image based on the fused image, the deformation offset, and a target mask determined from the fused image, wherein the target mask indicates the deformation region.

[0118] In one possible implementation, the target mask further indicates a non-deformation region, and the determining unit 503 is specifically configured to: transform the deformation offset from the second space to the first space according to the transformation matrix of the inverse affine transformation to obtain a transformed offset, wherein the transformed offset indicates the offset of the coordinates in the first space; update the transformed offset according to the target mask to obtain an updated offset, wherein the updated offset indicates that the offset of the coordinates located in the non-deformation region is 0; offset the coordinates in the first space according to the updated offset to obtain an offset result; and perform grid sampling on the fused image according to the offset result to obtain the second image.

[0119] In one possible implementation, the determining unit 503 is specifically configured to: normalize the target mask to obtain a normalization result, wherein the normalization result indicates the offset weight corresponding to the coordinates in the first space; and perform weighted processing on the transformed offset according to the normalization result to obtain the updated offset.

[0120] In one possible implementation, the determining unit 503 is specifically used to: perform Gaussian blurring on the target mask to obtain a blurred result; and normalize the blurred result to obtain the normalized result.

[0121] In one possible implementation, the determining unit 503 is specifically used to: transform the deformation offset from the coordinate system of the second space to the coordinate system of the first space according to the transformation matrix of the inverse affine transformation, so as to obtain the transformed offset; and perform an inverse affine transformation on the transformed offset to obtain the transformed offset.

[0122] In one possible implementation, the maximum value of the deformation offset on the first coordinate axis and the maximum value of the deformation offset on the second coordinate axis are both 1; the determining unit 503 is specifically used for: adjusting the range of the deformation offset according to the width and height of the second space to obtain an adjusted offset, wherein the maximum value of the adjusted offset on the first coordinate axis is the width of the second space, and the maximum value of the adjusted offset on the second coordinate axis is the height of the second space; processing the adjusted offset according to the transformation matrix of the inverse affine transformation to obtain a processed offset, wherein the maximum value of the processed offset on the first coordinate axis is the width of the first space, and the maximum value of the processed offset on the second coordinate axis is the height of the first space; and normalizing the processed offset according to the width and height of the first space to obtain the transformed offset, wherein the maximum value of the transformed offset on the first coordinate axis and the maximum value of the transformed offset on the second coordinate axis are both 1.

[0123] In one possible implementation, the process of determining the target mask includes: determining a face mask and a skin mask from the fused image; dilating the face mask to obtain a dilated mask, and eroding the face mask to obtain an eroded mask; and determining the target mask based on the difference between the dilated mask and the eroded mask, and the difference between the skin mask and the face mask.

[0124] In one possible implementation, the process of determining the target mask is based on the face adjustment requirements achieved by the method; and / or, the affine transformation result describes a portion of the original image, the portion of the image including at least the face in the original image.

[0125] Based on the aforementioned content of the image processing device 500, its working principle includes: first, acquiring an original image to ensure it belongs to a first space; then, predicting the first image, transparency information, and deformation offset based on the affine transformation result of the original image, so that both the affine transformation result and the first image belong to a second space different from the first space; furthermore, enabling the transparency information to indicate the transparency of each pixel in the first image, and enabling the deformation offset to indicate the offset of coordinates in the second space, thereby enabling the first image to represent the result of adjusting (e.g., beautifying) the face in the affine transformation result to a certain extent; enabling the transparency information to indicate the accuracy of the prediction of each pixel in the first image, and enabling the deformation offset to indicate the type of deformation applied to the face in the first image; and then, according to the weight indicated by the inverse affine transformation result of the transparency information... The inverse affine transformation result of the first image and the original image are weighted and fused to obtain a fused image, which can represent the result of adjusting the face in the original image to a certain extent. Then, based on the fused image, the deformation offset, and the target mask determined from the fused image that can indicate the deformation area, a second image is determined so that the changes in the second image compared to the fused image are located as close as possible within the deformation area. In this way, the effective range of the deformation indicated by the deformation offset in the fused image can be limited as much as possible within the deformation area by using the target mask, so as to overcome the blurring caused by the inability to ensure that the value of the deformation offset is 0 in the non-deformation area. This can better meet the needs of deforming part of the original image (such as part of the face), and thus help to improve the image processing effect, such as image clarity.

[0126] In addition, this application also provides an electronic device, the device including a processor and a memory: the memory is used to store instructions or computer programs; the processor is used to execute the instructions or computer programs in the memory, so that the electronic device performs any implementation of the image processing method provided in this application.

[0127] See Figure 6 This diagram illustrates a structural schematic of an electronic device 600 suitable for implementing embodiments of the present disclosure. The terminal devices in the embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 6 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0128] like Figure 6 As shown, electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 601, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 602 or a program loaded from storage device 608 into random access memory (RAM) 603. RAM 603 also stores various programs and data required for the operation of electronic device 600. Processing device 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0129] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 608 including, for example, magnetic tapes, hard disks, etc.; and communication devices 609. Communication device 609 allows electronic device 600 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 6 An electronic device 600 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0130] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, it performs the functions defined in the methods of embodiments of this disclosure.

[0131] The electronic device provided in this embodiment belongs to the same inventive concept as the method provided in the above embodiments. Technical details not described in detail in this embodiment can be found in the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.

[0132] This application also provides a computer-readable medium storing instructions or a computer program that, when executed on a device, causes the device to perform any implementation of the image processing method provided in this application.

[0133] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0134] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and end-to-end networks (e.g., ad hoc end-to-end networks), as well as any currently known or future-developed networks.

[0135] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0136] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, enable the electronic device to perform the aforementioned methods.

[0137] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0138] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0139] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units / modules do not necessarily limit the specific unit itself.

[0140] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0141] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0142] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems or apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and relevant parts can be referred to the method section.

[0143] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0144] It should also be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0145] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0146] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image processing method, characterized in that, The method includes: The original image is acquired, and a first image, transparency information, and deformation offset are predicted based on the affine transformation result of the original image. The original image belongs to a first space, and the affine transformation result and the first image both belong to a second space. The second space is different from the first space. The transparency information indicates the transparency of each pixel in the first image, and the deformation offset indicates the offset of the coordinates in the second space. According to the inverse affine transformation result of the transparency information, the inverse affine transformation result of the first image and the original image are weighted and fused to obtain a fused image. The inverse affine transformation result of the transparency information indicates the weight of each pixel in the inverse affine transformation result of the first image in the weighted fusion. A second image is determined based on the fused image, the deformation offset, and a target mask determined from the fused image, wherein the target mask indicates the deformation region.

2. The method according to claim 1, characterized in that, The target mask also indicates non-deformable regions; Determining the second image based on the fused image, the deformation offset, and the target mask determined from the fused image includes: Based on the transformation matrix of the inverse affine transformation, the deformation offset is transformed from the second space to the first space to obtain the transformed offset, which indicates the offset of the coordinates in the first space; The transformed offset is updated according to the target mask to obtain the updated offset, which indicates that the offset of the coordinates located in the non-deformation region is 0; The coordinates in the first space are offset according to the updated offset to obtain the offset result; The fused image is then subjected to grid sampling based on the offset result to obtain the second image.

3. The method according to claim 2, characterized in that, The step of updating the transformed offset based on the target mask to obtain the updated offset includes: The target mask is normalized to obtain a normalization result, and the normalization result indicates the offset weight corresponding to the coordinates in the first space; Based on the normalization result, the transformed offset is weighted to obtain the updated offset.

4. The method according to claim 3, characterized in that, The method further includes: Gaussian blurring is applied to the target mask to obtain a blurred result; The normalization of the target mask to obtain the normalization result includes: The fuzzy result is normalized to obtain the normalized result.

5. The method according to claim 2, characterized in that, The step of transforming the deformation offset from the second space to the first space based on the transformation matrix of the inverse affine transformation to obtain the transformed offset includes: Based on the transformation matrix of the inverse affine transformation, the deformation offset is transformed from the coordinate system of the second space to the coordinate system of the first space to obtain the transformed offset; The transformed offset is obtained by performing an inverse affine transformation on the transformed offset.

6. The method according to claim 5, characterized in that, The maximum value of the deformation offset on the first coordinate axis and the maximum value of the deformation offset on the second coordinate axis are both 1; The step of transforming the deformation offset from the coordinate system of the second space to the coordinate system of the first space based on the transformation matrix of the inverse affine transformation to obtain the transformed offset includes: The deformation offset is adjusted according to the width and height of the second space to obtain the adjusted offset. The maximum value of the adjusted offset on the first coordinate axis is the width of the second space, and the maximum value of the adjusted offset on the second coordinate axis is the height of the second space. The adjusted offset is processed according to the transformation matrix of the inverse affine transformation to obtain the processed offset. The maximum value of the processed offset on the first coordinate axis is the width of the first space, and the maximum value of the processed offset on the second coordinate axis is the height of the first space. Based on the width and height of the first space, the processed offset is normalized to obtain the transformed offset. The maximum value of the transformed offset on the first coordinate axis and the maximum value of the transformed offset on the second coordinate axis are both 1.

7. The method according to claim 1, characterized in that, The process of determining the target mask includes: Determine the face mask and skin mask from the fused image; The face mask is dilated to obtain an dilated mask, and the face mask is eroded to obtain an eroded mask; The target mask is determined based on the difference between the dilated mask and the eroded mask, and the difference between the skin mask and the face mask.

8. The method according to any one of claims 1-7, characterized in that, The process of determining the target mask is based on the facial adjustment requirements achieved through the method. And / or, The affine transformation result describes a portion of the original image, and the portion of the image includes at least the face in the original image.

9. An image processing apparatus, characterized in that, include: An acquisition unit is used to acquire an original image and predict a first image, transparency information, and deformation offset based on the affine transformation result of the original image. The original image belongs to a first space, and the affine transformation result and the first image both belong to a second space. The second space is different from the first space. The transparency information indicates the transparency of each pixel in the first image, and the deformation offset indicates the offset of the coordinates in the second space. The fusion unit is used to perform weighted fusion of the inverse affine transformation result of the first image and the original image according to the inverse affine transformation result of the transparency information to obtain a fused image. The inverse affine transformation result of the transparency information indicates the weight of each pixel in the inverse affine transformation result of the first image in the weighted fusion. The determining unit is configured to determine a second image based on the fused image, the deformation offset, and a target mask determined from the fused image, wherein the target mask indicates the deformation region.

10. An electronic device, characterized in that, The device includes: a processor and a memory; The memory is used to store instructions or computer programs; The processor is configured to execute the instructions or computer program in the memory to cause the electronic device to perform the method according to any one of claims 1-8.

11. A computer-readable medium, characterized in that, The computer-readable medium stores instructions or computer programs that, when executed on the device, cause the device to perform the method according to any one of claims 1-8.

12. A computer program product, characterized in that, It includes a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the method of any one of claims 1-8.

Images