Image processing methods, image processing devices and electronic devices

Abstract

This application discloses an image processing method, an image processing apparatus, and an electronic device, belonging to the field of image processing technology. The image processing method includes: dividing a target image into a target face region and multiple sub-background regions; determining a first constraint function corresponding to the target face region and a second constraint function corresponding to each of the sub-background regions, wherein the second constraint functions corresponding to any two of the sub-background regions are different; correcting the target face region based on the first constraint function, and correcting the sub-background regions corresponding to the second constraint function based on the second constraint function.

Description

Technical Field

[0001] This application belongs to the field of image processing technology, specifically relating to an image processing method, an image processing device, and an electronic device. Background Technology

[0002] To improve image quality, distortion correction is necessary. Related techniques often calibrate images based on facial distortion to eliminate deformation at the face. However, this process can lead to varying degrees of false distortion in the background area, and even cause regular objects to become distorted, thus affecting the correction effect. Summary of the Invention

[0003] The purpose of this application is to provide an image processing method, image processing apparatus, and electronic device that can solve the problem of poor image correction effect.

[0004] In a first aspect, embodiments of this application provide an image processing method, the method comprising:

[0005] The target image is divided into the target face region and multiple sub-background regions;

[0006] Determine the first constraint function corresponding to the target face region and the second constraint function corresponding to the sub-background region respectively, wherein the second constraint functions corresponding to any two sub-background regions are different;

[0007] The target face region is corrected based on the first constraint function, and the sub-background region corresponding to the second constraint function is corrected based on the second constraint function.

[0008] Secondly, embodiments of this application provide an image processing apparatus, the apparatus comprising:

[0009] The first processing module is used to divide the target image into the target face region and multiple sub-background regions;

[0010] The second processing module is used to determine the first constraint function corresponding to the target face region and the second constraint function corresponding to the sub-background region, wherein the second constraint functions corresponding to any two sub-background regions are different.

[0011] The third processing module is used to correct the target face region based on the first constraint function and to correct the sub-background region corresponding to the second constraint function based on the second constraint function.

[0012] Thirdly, embodiments of this application provide an electronic device including a processor and a memory, wherein the memory stores programs or instructions that can run on the processor, and the programs or instructions, when executed by the processor, implement the method as described in the first aspect.

[0013] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the method described in the first aspect.

[0014] Fifthly, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the method as described in the first aspect.

[0015] In a sixth aspect, embodiments of this application provide a computer program product stored in a storage medium, which is executed by at least one processor to implement the method described in the first aspect.

[0016] In this embodiment, by dividing the target image into a target human face region and multiple sub-background regions, and setting different constraint functions for different regions, the method performs different degrees of correction on each region based on the constraint functions corresponding to each region, which significantly improves the accuracy and precision of the correction and achieves smooth correction between regions. This allows the correction effect of both the face region and the background region to be taken into account, thereby improving the quality of the final generated image. In addition, this method is applicable to the correction of any image and has high universality. Attached Figure Description

[0017] Figure 1 This is one of the schematic flowcharts of the image processing method provided in the embodiments of this application;

[0018] Figure 2 This is a second schematic flowchart of the image processing method provided in the embodiments of this application;

[0019] Figure 3 This is a schematic diagram of the interface of the image processing method provided in the embodiments of this application;

[0020] Figure 4 This is a schematic diagram of the structure of the image processing apparatus provided in the embodiments of this application;

[0021] Figure 5 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application;

[0022] Figure 6 This is a hardware schematic diagram of the electronic device provided in the embodiments of this application. Detailed Implementation

[0023] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0024] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0025] In related technologies, there are three methods for image distortion correction:

[0026] One approach is to use global spherical projection on the image to eliminate distortion at the face, but this method causes background objects to bend and makes the face in the center of the image stand out, thus affecting the correction effect.

[0027] Secondly, a mesh optimization method is used to perform local spherical projection on the portrait, and then optimize the transition area between the portrait and the background, stretching the background to a certain extent to achieve the purpose of correcting the portrait. However, using local spherical projection will cause distortion in the area near the face, and this distortion is particularly obvious when there are lines around the face. Therefore, this method has significant limitations.

[0028] Third, the method detects straight line segments as salient regions and reduces the distortion of background lines by constraining these salient regions. However, if there are objects in the background that are not straight line segments, this method cannot eliminate the problem of background distortion. Therefore, this method also has significant limitations.

[0029] The image processing method, image processing apparatus, electronic device, and readable storage medium provided in this application will be described in detail below with reference to the accompanying drawings and through specific embodiments and application scenarios.

[0030] The image processing method can be applied to the terminal, and can be executed by the hardware or software in the terminal.

[0031] The terminal includes, but is not limited to, mobile phones or tablets and other portable communication devices with touch-sensitive surfaces (e.g., touchscreen displays and / or touchpads). It should also be understood that, in some embodiments, the terminal may not be a portable communication device, but rather a desktop computer with touch-sensitive surfaces (e.g., touchscreen displays and / or touchpads).

[0032] The following embodiments describe a terminal including a display and a touch-sensitive surface. However, it should be understood that the terminal may include one or more other physical user interface devices such as a physical keyboard, mouse, and joystick.

[0033] The image processing method provided in this application embodiment can be executed by an electronic device or a functional module or entity in an electronic device that can implement the image processing method. The electronic devices mentioned in this application embodiment include, but are not limited to, mobile phones, tablets, computers, cameras, wearable devices, and vehicle terminals. The image processing method provided in this application embodiment will be described below using an electronic device as the execution subject.

[0034] like Figure 1 As shown, the image processing method includes steps 110, 120 and 130.

[0035] Step 110: Divide the target image into the target face region and multiple sub-background regions;

[0036] In this step, the target image is the image that needs distortion correction.

[0037] The target image can be an image acquired by an image sensor, such as an image captured by a camera or video stream data after frame capture; or the target image can be an image stored in a local or cloud database; or the target image can be an image pulled from a network-side device, such as an image or video stream data frame downloaded from a webpage.

[0038] Based on the content in the target image, the target image can be divided into a face region and multiple sub-background regions.

[0039] The target face region is the area in the target image where the face of the person whose face needs to be corrected is located.

[0040] The number of target face regions can be one or more.

[0041] The sub-background region is a region generated by dividing the target image into regions other than the target face region.

[0042] The number of sub-background regions is at least two, for example, there can be three or four sub-background regions, etc., and this application does not limit it.

[0043] The distances between each sub-background region and the target face region are different, and the degree of correlation between the two is also different.

[0044] When there are two sub-background regions, the multiple sub-background regions include: the first sub-background region and the second sub-background region.

[0045] Among them, the distance between the first sub-background region and the target face region is smaller than the distance between the second sub-background region and the target face region, and the correlation between the first sub-background region and the target face region is higher than the correlation between the second sub-background region and the target face region.

[0046] The target image is simply divided into face and background, and both face and background are corrected. However, after a lot of experiments, the inventors found that the corrected image obtained by this method can only achieve a rough correction of the background. If you look closely at the details of the image, you will find that there is still a certain degree of distortion between different areas of the background, which is difficult to use for high-standard image correction.

[0047] like Figure 2 As shown, in some embodiments of this application, step 110 may include:

[0048] Feature extraction is performed on the target image to generate M element regions, which include the target face region and N background element regions, where M > N > 0;

[0049] Obtain the first distance between the target face region and the background element region;

[0050] Based on the first distance, the N background element regions are divided into multiple sub-background regions.

[0051] In this embodiment, the element region is a feature region in the target image that has a certain semantic meaning. The target image may include one or more element regions.

[0052] For example, when the target image is a travel photo of user A, such as Figure 3 As shown, the element regions in the target image may include user A's face region 311, user A's limbs region 312, and the car region 313 behind user A, building region 314, plant region 315, and sky region 316, etc.

[0053] The target face region is the element region corresponding to the face of the target user. The target user is the user in the target image who needs to be corrected; such as the face region 311 of user A.

[0054] The background element region is the element region other than the target face region in all element regions of the target image; such as the limb region 312 of user A, and the car region 313, building region 314, plant region 315 and sky region 316 behind user A, etc.

[0055] M is a positive integer, and the value of M should not exceed the sum of the element types in the target image.

[0056] N is a positive integer, and the value of N should not exceed M.

[0057] In actual execution, a DeepLab V3+ semantic segmentation model that integrates multi-scale features can be used to extract the element segmentation results, and then refine and label the element regular regions to generate P regions. These P regions include labeled and identified regions and unlabeled and unidentified regions. Unidentified regions can be areas formed by pixels at the boundaries of multiple adjacent element regions.

[0058] For example, the P regions may include: user A's face region 311, user A's limbs region 312, the car region behind user A 313, the building region 314, the plant region 315, the sky region 316, and the unrecognized region, etc.

[0059] Where P is a positive integer, and the value of P should not be lower than the value of M.

[0060] After obtaining P regions, the identified regions are marked, and the connected components of the marked identified regions are calculated. Each connected component is then assigned to the corresponding point set. That is, the pixels in the unidentified regions are assigned to the corresponding marked regions to obtain M element regions.

[0061] For example, the unrecognized area includes multiple pixels such as target face pixels, leaf pixels, and building pixels. By calculating the connected components, the target face pixels can be assigned to user A's face region 311, the leaf pixels to the plant region 315, and the building pixels to the building region 314, thereby generating the final user A's face region 311, user A's limb region 312, and the car region 313, building region 314, plant region 315, and sky region 316 behind user A.

[0062] After generating M element regions, the other element regions in the M element regions, excluding the target face region, are determined as N background element regions.

[0063] Calculate the first distance between the target face region and each background element region.

[0064] The first distance is used to characterize the closest distance between the target face region and the background element region. In actual execution, the centroid of the face region can be determined as the first target point I. face Calculate the first distance O between the background element region and the first target point respectively. face,i .

[0065] For example, the pixel closest to the first target point on the boundary of the background element region can be used as the second target point, and the first distance can be determined by the distance between the first target point and the second target point.

[0066] For example, the center of gravity of the background element area can be used as the second target point, and the first distance can be determined by the distance between the first target point and the second target point. This application does not impose any limitations.

[0067] The first distance can be Euclidean distance or Paradigm distance, etc.

[0068] Based on the first distance, N background element regions can be divided into multiple sub-background regions.

[0069] In actual implementation, a target threshold Os can be set and used as a benchmark for judging the distance between the background element area and the target face area.

[0070] The number and value of the target thresholds can both be customized by the user. By adjusting the number or value of the target thresholds, the correction accuracy of the target image can be adjusted, thereby adjusting the final correction result.

[0071] For example, when there are two target thresholds, the distance between the background element region and the target face region can be divided into three segments: far, medium, and near. Similarly, when there is only one target threshold, the distance between the background element region and the target face region can be divided into two segments: far and near.

[0072] By comparing the target threshold with the first distance, the N background element regions can be divided into different intervals.

[0073] Each interval segment corresponds to a sub-background region.

[0074] The following description uses a target threshold of 1 as an example.

[0075] By comparing a first distance with a target threshold, if the first distance does not exceed the target threshold, and the background element region corresponding to the first distance is determined to be relatively close to the target face region, it is approximately considered that the correlation between the background element region corresponding to the first distance and the target face region is high. Therefore, the background element region corresponding to the first distance is defined as the first sub-background region, and the points in the first sub-background region are set as I. hs,i .

[0076] If the first distance exceeds the target threshold, and the background element region corresponding to the first distance is determined to be far from the target face region, then it is approximately considered that the correlation between the background element region corresponding to the first distance and the target face region is relatively low. Therefore, the background element region corresponding to the first distance is defined as the second sub-background region, and the points in the second sub-background region are set as I. ls,i .

[0077] The following description uses Euclidean distance as an example to illustrate this embodiment.

[0078] For example, by segmenting user A's travel photos, six element regions are generated: user A's face region 311, user A's body region 312, and the car region behind user A 313, building region 314, plant region 315, and sky region 316.

[0079] The calculated Euclidean distances are as follows: a between user A's face region 311 and user A's limb region 312; b between user A's face region 311 and the car region 313 behind user A; c between user A's face region 311 and the building region 314; d between user A's face region 311 and the plant region 315; and e between user A's face region 311 and the sky region 316.

[0080] By comparing the size relationships of a, b, c, d, e and Os, we find that a and d are both less than Os, while b, c, and e are greater than Os. Therefore, the limb area 312 and plant area 315 of user A are divided into the first sub-background area; the car area 313, building area 314 and sky area 316 behind user A are divided into the second sub-background area.

[0081] According to the image processing method provided in the embodiments of this application, feature extraction is performed on the target image to generate multiple element regions. The background element regions are then divided into multiple sub-background regions based on the distance relationship between the background element regions and the face region, so as to finely divide the background region in the target image. Then, the background element regions are divided based on the correlation between each background element region and the target face region, so that different constraint corrections can be achieved according to different correlations in the subsequent process, thereby helping to improve the accuracy and precision of the correction.

[0082] Continue to refer Figure 2 In some embodiments, prior to step 110, the method may further include:

[0083] Detect facial information in a target image;

[0084] If facial information is detected, proceed to step 110.

[0085] In this embodiment, facial recognition technology can be used to extract facial information from the target image.

[0086] If facial information is extracted, and it is determined that the target image contains facial features, then distortion correction is performed on the target image.

[0087] In other embodiments, if no facial information is detected, the target image is not distorted and is directly output.

[0088] Step 120: Determine the first constraint function corresponding to the target face region and the second constraint function corresponding to the sub-background region respectively. The second constraint functions corresponding to any two sub-background regions are different.

[0089] In this step, constraint functions are used to apply correction constraints so that each region achieves the best correction results.

[0090] Each region corresponds to a constraint function, and the number of second constraint functions is the same as the number of sub-background regions.

[0091] It should be noted that the second constraint function is different for any two sub-background regions, that is, the correction constraint term is different for each sub-background region.

[0092] Within the same sub-background region, the weight parameters corresponding to different element background regions may also be different.

[0093] In implementing the technical solution of this application, the inventors also considered that, in the case of simply dividing the target image into face and background, although it is possible to correct the face and background regions separately, especially for some images where the background region occupies a large area of ​​the image, the correction of the background region still appears to be relatively coarse, and the correction effect is actually not ideal.

[0094] In some embodiments, step 120 may include:

[0095] A first constraint function is generated based on the face scaling factor function, perspective projection transformation parameters, and face translation amount; the face translation amount is used to control the first constraint function to limit the deformation direction of the target face region;

[0096] Generate directional gradient histograms for each sub-background region;

[0097] Based on the oriented gradient histogram, the target orientation corresponding to each sub-background region is generated through gradient optimization.

[0098] Based on the target direction corresponding to the sub-background region and the weight parameters corresponding to the sub-background region, a second constraint function is generated. The weight parameters corresponding to the sub-background region are used to control the second constraint function to restrict the deformation direction of the sub-background region.

[0099] In this embodiment, the oriented gradient histogram is used to represent statistical values ​​of the direction information of image gradients in local regions of the image. It can be understood that at each pixel, the gradient has both magnitude and direction.

[0100] By statistically analyzing the gradient magnitude and direction of all pixels within a region, and accumulating the gradient values ​​of all pixels in each channel based on the number of channels for that direction, a histogram of oriented gradients for that region can be generated.

[0101] The target direction is the optimization direction corresponding to the sub-background region.

[0102] It is understandable that the optimization directions corresponding to different sub-background regions may be the same or different.

[0103] Each sub-background region corresponds to a weight parameter, which is used to restrict the deformation direction of each sub-background region. The method for generating the weight parameters will be explained in detail in subsequent embodiments, and will not be elaborated here.

[0104] The face translation amount is used to constrain the deformation direction of the target face region, that is, to control the first constraint function to constrain the deformation direction of the highly saliency region according to the gradient direction.

[0105] The face scaling factor function is used to adjust the face scaling based on the distance between the target face and the center of the target face.

[0106] The perspective projection transformation parameters are the coordinate offset input parameters used in the perspective projection transformation (Warp).

[0107] The first constraint function can be constructed based on the face scaling factor function, perspective projection transformation parameters, pixel coordinates of the face region, and face translation amount.

[0108] The second constraint function can be constructed based on the target direction corresponding to the sub-background region, the weight parameters corresponding to the sub-background region, the pixel coordinates of the sub-background region, and the pixel coordinates of the original image.

[0109] The following explanation will focus on the sub-background area as two separate areas.

[0110] In some embodiments, the target image is divided to generate a target face region, a first sub-background region, and a second sub-background region.

[0111] The first sub-background region is a strongly correlated region with the target face region, and the second sub-background region is a weakly correlated region with the target face region.

[0112] The second constraint function includes a first sub-constraint function and a second sub-constraint function. The first sub-background region corresponds to the first sub-constraint function, and the second sub-background region corresponds to the second sub-constraint function.

[0113] In actual implementation, the following formula can be used:

[0114] E face =(S(r)I face,x +t x -I u,x ) 2 +(S(r)I face,y +t y -I u,y ) 2

[0115] Construct the first constraint function;

[0116] Among them, E face Let S(r) be the first constraint function, and S(r) be the face scaling coefficient function. face,x , I face,y ) represents the pixel coordinates of the face region, i.e., the coordinates of the first target point, (t) x , t y ) is the face translation amount, (I u,x , I u,y(x, y) represents the pixel coordinates of the spherical polar plane projection, i.e., the perspective projection transformation parameters; (x, y) represents the horizontal and vertical coordinates.

[0117] It can be done through the formula:

[0118] E i,hs =(1+|cos(θ) i )|)(I hs,i,x +W x -I 0,x ) 2 +(1+|sin(θ i )|)(I hs,i,y +W y -I 0,y ) 2

[0119] Construct the first sub-constraint function;

[0120] Among them, E i,hs Let θ be the first sub-constraint function. i For the target direction corresponding to the first sub-background region, (I hs,i,x , I hs,i,y ) represents the pixel coordinates of the first sub-background region, (I 0,x , I 0,y ) represents the pixel coordinates of the original image, (W) x W y (x, y) represents the weight parameters corresponding to the first sub-background region; (x, y) represents the horizontal and vertical coordinates.

[0121] It can be done through the formula:

[0122] E i,ls =(1-|cos(θ) i )|)(I ls,i,x +W x -I 0,x ) 2 +(1-|sin(θ i )|)(I ls,i,y +W y -I 0,y ) 2

[0123] Construct the first sub-constraint function;

[0124] Among them, E i,ls For the second sub-constraint function, θ i For the target direction corresponding to the second sub-background region, (I ls,i,x , I ls,i,y ) represents the pixel coordinates of the second sub-background region, (I 0,x , I 0,y ) represents the pixel coordinates of the original image, (W)x , W y (x, y) represents the weight parameters corresponding to the second sub-background region; (x, y) represents the horizontal and vertical coordinates.

[0125] In this embodiment, based on the gradient optimization direction, different constraint functions are constructed for different regions, thereby performing constraint corrections of different degrees on different regions to achieve the best correction effect for each region, which helps to improve the accuracy and precision of image correction results.

[0126] The generation method of the weight parameters will be explained in detail below.

[0127] In some embodiments, the weighting parameters can be determined through the following steps:

[0128] Obtain the second distance between the target face region and the sub-background region;

[0129] Based on the second distance and the face translation amount, the weight parameters of the sub-background region corresponding to the second distance are determined.

[0130] In this embodiment, the second distance is the distance between the target face region and the background element region in the sub-background region.

[0131] It is understandable that the second distance may include one or more first distances.

[0132] The second distance can be Euclidean distance or Paradigm distance, etc.

[0133] Taking the second distance as the Euclidean distance as an example, in actual implementation, it can be achieved using the formula:

[0134] (W x W y )=(W(O face,i,x ,t x ),W(O face,i,y ,t y ))

[0135] Construct the weight parameters corresponding to the sub-background regions; where, (W x W y ) represents the weight parameter corresponding to this sub-background region, (O face,i,x O face,i,y (t) represents the Euclidean distance between the background element region and the target face region within the sub-background region. x , t y ) represents the face translation, and (x, y) represents the horizontal and vertical coordinates.

[0136] Through the formula:

[0137]

[0138] Determine the weight parameter values ​​corresponding to the sub-background regions, where (x0, xf) are parameters in W.

[0139] Continuing with the example of correcting user A's travel photos, the above embodiment will be explained.

[0140] In step 110, by segmenting user A's travel photos, six element regions are generated: user A's face region 311, user A's limbs region 312, and the car region behind user A 313, building region 314, plant region 315, and sky region 316.

[0141] The calculated Euclidean distances are as follows: a between user A's face region 311 and user A's limb region 312; b between user A's face region 311 and the car region 313 behind user A; c between user A's face region 311 and the building region 314; d between user A's face region 311 and the plant region 315; and e between user A's face region 311 and the sky region 316.

[0142] By comparing the size relationships of a, b, c, d, e and Os, we find that a and d are both less than Os, while b, c, and e are greater than Os. Therefore, the limb area 312 and plant area 315 of user A are divided into the first sub-background area; the car area 313, building area 314 and sky area 316 behind user A are divided into the second sub-background area.

[0143] In step 120, the constraint functions corresponding to each region are first constructed.

[0144] Then, the values ​​of a, b, c, d, and e are substituted into the corresponding W(x0, xf) to generate the weight parameter values ​​for each element region. These weight parameter values ​​are then substituted into the corresponding constraint functions to obtain the first constraint function E. face First sub-constraint function E i,hs Second sub-constraint function E i,ls .

[0145] In this embodiment, weight parameters corresponding to different element regions are generated based on the Euclidean distance between the target face region and each background element region. These weight parameters are then substituted into the corresponding second constraint function to further refine the correction effect on the background region, achieve a smooth transition between different background regions, and thus improve image quality.

[0146] Step 130: Correct the target face region based on the first constraint function, and correct the sub-background region corresponding to the second constraint function based on the second constraint function.

[0147] In this step, correction weights can be set for each region, and each region can be corrected using a nonlinear constraint method.

[0148] The following description uses multiple sub-background regions, including a first sub-background region and a second sub-background region, as an example to illustrate this embodiment.

[0149] For example, the correction weights corresponding to the target face region, the first sub-background region, and the second sub-background region are set respectively. Based on the correction weights corresponding to each region, the correction offsets corresponding to the pixels in each region are generated. Then, based on the correction offsets, the target image can be transformed into an image without background distortion through Warp transformation.

[0150] The correction offset for each pixel in each region is the offset of the optimized coordinates of the pixel in that region relative to the original coordinates.

[0151] In some embodiments, step 130 may include:

[0152] Construct an energy function optimization equation based on the first and second constraint functions;

[0153] The nonlinear constraint method is used to solve the energy function optimization equation to generate the first offset of the pixel points in the face region and the second offset of the pixel points in the sub-background region.

[0154] A perspective projection transformation is performed on the target face region based on the first offset; a perspective projection transformation is performed on the sub-background region based on the second offset.

[0155] In this embodiment, the first offset is the offset of the optimized coordinates of the pixel positions in the face region relative to the original coordinates, which is the face translation amount in the above embodiment. The second offset is the offset of the optimized coordinates of the pixel positions in the sub-background region relative to the original coordinates, which is the weight parameter corresponding to the sub-background region in the above embodiment.

[0156] It is understandable that different pixels will have different offsets after optimization, and the correction constraints for different pixels may be the same or different.

[0157] In actual implementation, the energy function optimization equation can be constructed based on the constraint functions and correction weights corresponding to each region, as shown below:

[0158] E=W face 2 ·E face +∑(W i,hs 2 ·Ei,hs +W i,ls 2 ·E i,ls )

[0159] Where E is the energy function optimization equation, E face Let E be the first constraint function. i,hs E is the first sub-constraint function. i,ls W is the second sub-constraint function. face W represents the correction weights corresponding to the target face region. i,hs W represents the correction weight corresponding to the first sub-background region. i,ls This is the correction weight corresponding to the second sub-background region.

[0160] The correction weights can be user-defined, use default values, or be determined through continuous optimization.

[0161] By solving the equation E=0 using the nonlinear constraint method, constraint correction can be achieved for each region of the target image.

[0162] Continuing with the example of multiple sub-background regions, including the first and second sub-background regions, after constructing the energy function optimization equation as described above, the equation E=0 is solved by the nonlinear constraint method to obtain the optimized offset of each pixel in the target image. After obtaining the offset corresponding to each region, Warp transform is performed on the image of each region based on the offset. This allows for the correction of the target face region and multiple sub-background regions with different constraints, thereby generating a corrected target image that takes into account both the face and background regions and is naturally smooth.

[0163] In this step, the energy function equation is solved by an optimization method to generate the optimized coordinates for each region. Based on the optimized coordinates, the pixels in each region are corrected with different constraints to control the deformation of the target face and multiple sub-background regions according to their respective target directions, suppress the deformation between regions, and take into account both the face region and the background region. This achieves face distortion correction while preventing false distortion and distortion in the background region around the face, thus realizing smooth correction between regions.

[0164] After correcting the target face region based on the first constraint function and correcting the sub-background region corresponding to the second constraint function based on the second constraint function, the corrected target image can be output.

[0165] In some embodiments, the calibrated target image can be output to the display interface for display. For example, during the process of a user taking a picture or recording a video, the terminal performs calibration processing on the target image acquired by the image sensor to generate a calibrated target image and displays the calibrated target image on the screen to present the user with a high-quality shooting effect.

[0166] In some embodiments, the corrected target image may also be output to a database for storage, but this application does not limit this.

[0167] The image processing method provided in this application divides the target image into a target human face region and multiple sub-background regions, sets different constraint functions for different regions, and performs different degrees of correction on each region based on the constraint functions corresponding to each region. This significantly improves the accuracy and precision of the correction, achieves smooth correction between regions, and thus can take into account the correction effect of the face region and the background region, thereby improving the quality of the final generated image. In addition, this method is applicable to the correction of any image and has high universality.

[0168] In some embodiments, the method can also be used as a constraint strategy for beautifying and shaping faces. By semantically segmenting the target image, a target face region and multiple sub-background regions are generated, and a constraint function corresponding to each region is constructed to reduce the translation of pixels in the background region, thereby avoiding the problem of background distortion during the face shaping process.

[0169] The image processing method provided in this application can be executed by an image processing device. This application uses an image processing device executing the image processing method as an example to illustrate the image processing apparatus provided in this application.

[0170] This application also provides an image processing apparatus.

[0171] like Figure 4 As shown, the image processing device includes: a first processing module 410, a second processing module 420 and a third processing module 430.

[0172] The first processing module 410 is used to divide the target image into a target face region and multiple sub-background regions;

[0173] The second processing module 420 is used to determine the first constraint function corresponding to the target face region and the second constraint function corresponding to the sub-background region, wherein the second constraint functions corresponding to any two sub-background regions are different.

[0174] The third processing module 430 is used to correct the target face region based on the first constraint function and to correct the sub-background region corresponding to the second constraint function based on the second constraint function.

[0175] The image processing apparatus provided in the embodiments of this application divides the target image into a target human image region and multiple sub-background regions, sets different constraint functions for different regions, and performs different degrees of correction on each region based on the constraint functions corresponding to each region. This significantly improves the accuracy and precision of the correction, achieves smooth correction between regions, and thus can take into account the correction effect of the face region and the background region, thereby improving the quality of the final generated image. In addition, this method is applicable to the correction of any image and has high universality.

[0176] In some embodiments, the first processing module 410 may also be used for:

[0177] Feature extraction is performed on the target image to generate M element regions, which include the target face region and N background element regions, where M > N > 0;

[0178] Obtain the first distance between the target face region and the background element region;

[0179] Based on the first distance, the N background element regions are divided into multiple sub-background regions.

[0180] In some embodiments, the second processing module 420 may also be used for:

[0181] A first constraint function is generated based on the face scaling factor function, perspective projection transformation parameters, and face translation amount; the face translation amount is used to control the first constraint function to limit the deformation direction of the target face region;

[0182] Generate directional gradient histograms for each sub-background region;

[0183] Based on the oriented gradient histogram, the target orientation corresponding to each sub-background region is generated through gradient optimization.

[0184] Based on the target direction corresponding to the sub-background region and the weight parameters corresponding to the sub-background region, a second constraint function is generated. The weight parameters corresponding to the sub-background region are used to control the second constraint function to restrict the deformation direction of the sub-background region.

[0185] In some embodiments, the weighting parameters can be determined through the following steps:

[0186] Obtain the second distance between the target face region and the sub-background region;

[0187] Based on the second distance and the face translation amount, the weight parameters of the sub-background region corresponding to the second distance are determined.

[0188] In some embodiments, the third processing module 430 can also be used for:

[0189] Construct an energy function optimization equation based on the first and second constraint functions;

[0190] The nonlinear constraint method is used to solve the energy function optimization equation to generate the first offset of the pixel points in the face region and the second offset of the pixel points in the sub-background region.

[0191] A perspective projection transformation is performed on the target face region based on the first offset; a perspective projection transformation is performed on the sub-background region based on the second offset.

[0192] The image processing device in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television set (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the device.

[0193] The image processing device in this application embodiment can be a device with an operating system. The operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit the specific operating system.

[0194] The image processing apparatus provided in this application embodiment can achieve... Figures 1 to 3 The various processes implemented in the method implementation examples will not be described again here to avoid repetition.

[0195] Optionally, such as Figure 5 As shown, this application embodiment also provides an electronic device 500, including a processor 501, a memory 502, and a program or instructions stored in the memory 502 and executable on the processor 501. When the program or instructions are executed by the processor 501, they implement the various processes of the above-described image processing method embodiments and achieve the same technical effects. To avoid repetition, they will not be described again here.

[0196] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0197] Figure 6 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this application.

[0198] The electronic device 600 includes, but is not limited to, components such as: radio frequency unit 601, network module 602, audio output unit 603, input unit 604, sensor 605, display unit 606, user input unit 607, interface unit 608, memory 609, and processor 610.

[0199] Those skilled in the art will understand that the electronic device 600 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 610 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 6 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0200] The processor 610 is used for:

[0201] The target image is divided into the target face region and multiple sub-background regions;

[0202] Determine the first constraint function corresponding to the target face region and the second constraint function corresponding to the sub-background region respectively, and the second constraint functions corresponding to any two sub-background regions are different;

[0203] The target face region is corrected based on the first constraint function, and the sub-background region corresponding to the second constraint function is corrected based on the second constraint function.

[0204] The electronic device provided in the embodiments of this application divides the target image into a target human image region and multiple sub-background regions, sets different constraint functions for different regions, and performs different degrees of correction on each region based on the constraint functions corresponding to each region. This significantly improves the accuracy and precision of the correction, achieves smooth correction between regions, and thus can take into account the correction effect of the face region and the background region, thereby improving the quality of the final generated image.

[0205] Optionally, the processor 610 is also used for:

[0206] Feature extraction is performed on the target image to generate M element regions, which include the target face region and N background element regions, where M > N > 0;

[0207] Obtain the first distance between the target face region and the background element region;

[0208] Based on the first distance, the N background element regions are divided into multiple sub-background regions.

[0209] Optionally, the processor 610 is also used for:

[0210] A first constraint function is generated based on the face scaling factor function, perspective projection transformation parameters, and face translation amount; the face translation amount is used to control the first constraint function to limit the deformation direction of the target face region;

[0211] Generate directional gradient histograms for each sub-background region;

[0212] Based on the oriented gradient histogram, the target orientation corresponding to each sub-background region is generated through gradient optimization.

[0213] Based on the target direction corresponding to the sub-background region and the weight parameters corresponding to the sub-background region, a second constraint function is generated. The weight parameters corresponding to the sub-background region are used to control the second constraint function to restrict the deformation direction of the sub-background region.

[0214] Optionally, the processor 610 is also used for:

[0215] Obtain the second distance between the target face region and the sub-background region;

[0216] Based on the second distance and the face translation amount, the weight parameters of the sub-background region corresponding to the second distance are determined.

[0217] Optionally, the processor 610 is also used for:

[0218] Construct an energy function optimization equation based on the first and second constraint functions;

[0219] The nonlinear constraint method is used to solve the energy function optimization equation to generate the first offset of the pixel points in the face region and the second offset of the pixel points in the sub-background region.

[0220] A perspective projection transformation is performed on the target face region based on the first offset; a perspective projection transformation is performed on the sub-background region based on the second offset.

[0221] It should be understood that, in this embodiment, the input unit 604 may include a graphics processing unit (GPU) 6041 and a microphone 6042. The GPU 6041 processes image data of still images or videos obtained by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 606 may include a display panel 6061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like. The user input unit 607 includes at least one of a touch panel 6071 and other input devices 6072. The touch panel 6071 is also called a touch screen. The touch panel 6071 may include two parts: a touch detection device and a touch controller. Other input devices 6072 may include, but are not limited to, a physical keyboard, function keys (such as volume control buttons, power buttons, etc.), a trackball, a mouse, and a joystick, which will not be described in detail here.

[0222] The memory 609 can be used to store software programs and various data. The memory 609 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 609 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 609 in this embodiment includes, but is not limited to, these and any other suitable types of memory.

[0223] Processor 610 may include one or more processing units; optionally, processor 610 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into processor 610.

[0224] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described image processing method embodiments and achieve the same technical effects. To avoid repetition, they will not be described again here.

[0225] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0226] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described image processing method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0227] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0228] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0229] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0230] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. An image processing method, characterized in that, include: The target image is divided into the target face region and multiple sub-background regions; Determine the first constraint function corresponding to the target face region and the second constraint function corresponding to the sub-background region respectively, wherein the second constraint functions corresponding to any two sub-background regions are different; The target face region is corrected based on the first constraint function, and the sub-background region corresponding to the second constraint function is corrected based on the second constraint function. The process of dividing the target image into a target face region and multiple sub-background regions includes: Feature extraction is performed on the target image to generate M element regions, which include the target face region and N background element regions, where M > N > 0; Obtain the first distance between the target face region and the background element region; Based on the first distance, the N background element regions are divided into the plurality of sub-background regions.

2. The image processing method according to claim 1, characterized in that, The step of determining the first constraint function corresponding to the target face region and the second constraint function corresponding to the sub-background region includes: The first constraint function is generated based on the face scaling factor function, perspective projection transformation parameters, and face translation amount; the face translation amount is used to control the first constraint function to limit the deformation direction of the target face region; Generate directional gradient histograms for each of the aforementioned sub-background regions; Based on the directional gradient histogram, the target direction corresponding to each of the sub-background regions is generated through gradient optimization; Based on the target direction corresponding to the sub-background region and the weight parameters corresponding to the sub-background region, a second constraint function is generated. The weight parameters corresponding to the sub-background region are used to control the second constraint function to restrict the deformation direction of the sub-background region.

3. The image processing method according to claim 2, characterized in that, The weighting parameters are determined through the following steps: Obtain the second distance between the target face region and the sub-background region; Based on the second distance and the face translation amount, the weight parameters of the sub-background region corresponding to the second distance are determined.

4. The image processing method according to any one of claims 1-3, characterized in that, The step of correcting the target face region based on the first constraint function and correcting the sub-background region corresponding to the second constraint function based on the second constraint function includes: Construct an energy function optimization equation based on the first constraint function and the second constraint function; The first offset of the pixel points in the face region and the second offset of the pixel points in the sub-background region are generated by solving the energy function optimization equation using a nonlinear constraint method. The target face region is transformed by perspective projection based on the first offset; the sub-background region is transformed by perspective projection based on the second offset.

5. An image processing apparatus, characterized in that, include: The first processing module is used to divide the target image into the target face region and multiple sub-background regions; The second processing module is used to determine the first constraint function corresponding to the target face region and the second constraint function corresponding to the sub-background region, wherein the second constraint functions corresponding to any two sub-background regions are different. The third processing module is used to correct the target face region based on the first constraint function and to correct the sub-background region corresponding to the second constraint function based on the second constraint function. The first processing module is further configured to: Feature extraction is performed on the target image to generate M element regions, which include the target face region and N background element regions, where M > N > 0; Obtain the first distance between the target face region and the background element region; Based on the first distance, the N background element regions are divided into the plurality of sub-background regions.

6. The image processing apparatus according to claim 5, characterized in that, The second processing module is further configured to: The first constraint function is generated based on the face scaling factor function, perspective projection transformation parameters, and face translation amount; the face translation amount is used to control the first constraint function to limit the deformation direction of the target face region; Generate directional gradient histograms for each of the aforementioned sub-background regions; Based on the directional gradient histogram, the target direction corresponding to each of the sub-background regions is generated through gradient optimization; Based on the target direction corresponding to the sub-background region and the weight parameters corresponding to the sub-background region, a second constraint function is generated. The weight parameters corresponding to the sub-background region are used to control the second constraint function to restrict the deformation direction of the sub-background region.

7. An electronic device, characterized in that, It includes a processor and a memory, the memory storing a program or instructions that can run on the processor, the program or instructions being executed by the processor to implement the image processing method as described in any one of claims 1-4.

8. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the image processing method as described in any one of claims 1-4.

Images