Image perspective distortion correction method, device and computer readable storage medium

Abstract

This invention relates to the field of image processing technology, specifically providing an image perspective distortion correction method, device, and computer-readable storage medium. The image perspective distortion correction method includes: acquiring an original face image; performing meshing processing on the original face image to obtain an initial deformation mesh, wherein the meshes in the initial deformation mesh correspond to the pixels in the original face image; converting the original face image into a target distance face image based on a preset target distance; determining the coordinate change vectors of at least some pixels based on the original face image and the target distance face image, and determining the target coordinates of each mesh in the initial deformation mesh; adjusting the pixels in the original face image to the target coordinates of the corresponding meshes to generate a target image. Through the above method, this application can improve the accuracy of image perspective distortion correction.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to methods, devices and computer-readable storage media for correcting perspective distortion in images. Background Technology

[0002] Currently, when shooting close-up portraits, perspective distortion often occurs in the resulting facial images. A typical characteristic of perspective distortion is that the subject's nose appears larger and the facial contours are distorted. This distortion causes the face in the captured image to differ from the actual face, affecting the shooting results.

[0003] However, current methods for correcting perspective distortion in images are not ideal in terms of their effectiveness. Therefore, an effective perspective distortion correction method is needed to improve the accuracy of perspective distortion correction. Summary of the Invention

[0004] In view of this, the present invention provides an image perspective distortion correction method, device and computer-readable storage medium, which can improve the accuracy of image perspective distortion correction.

[0005] To address the aforementioned technical problems, this invention provides an image perspective distortion correction method, comprising: acquiring an original face image; performing meshing processing on the original face image to obtain an initial deformation mesh, wherein the meshes in the initial deformation mesh correspond to the pixels in the original face image; converting the original face image into a target distance face image based on a preset target distance, wherein the target distance face image is the image of the face in the original face image at the target distance, and the target distance is greater than the shooting distance of the original face image; determining the coordinate change vectors of at least some pixels based on the original face image and the target coordinates of each mesh in the initial deformation mesh; adjusting the pixels in the original face image to the target coordinates of the corresponding meshes to generate a target image.

[0006] This embodiment takes into account the influence of shooting distance on the degree of perspective distortion, mapping the original face image taken at close range to a suitable distance to obtain a face image at the target distance. Using this target distance face image as a reference for perspective distortion correction, it achieves precise correction of image perspective distortion, resulting in a target image that more closely resembles the subject's true appearance. Furthermore, this embodiment establishes an initial deformation mesh and determines the target coordinates of pixels in the original face image by referring to the target coordinates of the mesh within the initial deformation mesh. This allows for precise adjustment of pixel positions in the original face image, further improving the accuracy of image perspective distortion correction.

[0007] According to some embodiments of the present invention, the step of converting an original face image into a face image at a target distance based on a preset target distance includes: obtaining an original depth map corresponding to the original face image; performing offset and scaling processing on the original depth map based on the target distance to generate a target distance depth map; and reprojecting the target distance depth map to obtain the face image at the target distance.

[0008] According to some embodiments of the present invention, the step of generating a target distance depth map by offsetting and scaling the original depth map based on the target distance includes: using the difference between the target distance and the shooting distance of the original face image as a depth offset value, and performing depth offset on the pixels in the original depth map according to the depth offset value; performing facial keypoint detection on the original depth map to obtain facial keypoints; determining the depth corresponding to the centroid of all facial keypoints; and determining a scaling factor based on the depth corresponding to the centroid and the depth offset value, and scaling the pixels in the original depth map according to the scaling factor.

[0009] According to some embodiments of the present invention, the step of determining the coordinate change vector of at least some pixels based on the original face image and the target distance face image includes: using the coordinate difference between at least some pixels in the original face image and the target distance face image as the coordinate change vector corresponding to at least some pixels.

[0010] This embodiment determines the target coordinates of all grids in the initial deformed grid by selecting the coordinate change vectors of some pixels. On the one hand, this reduces the amount of computation and improves the correction efficiency; on the other hand, it avoids the errors caused by the small face area and dense pixel sampling, thus improving the correction accuracy.

[0011] According to some embodiments of the present invention, the step of converting an original face image into a face image at a target distance based on a preset target distance includes: inputting the original face image into a GAN inverse mapping model and outputting the face image at the target distance. The GAN inverse mapping model is used to parse the hidden face information in the original face image and construct equivalent camera parameters at the target distance. Based on the hidden face information and the equivalent camera parameters, the original face image is reconstructed to generate the face image at the target distance.

[0012] The GAN inverse mapping model in this embodiment can fill the gaps in the face image by parsing the hidden information of the face, making the target closer to the real face image, thereby improving the accuracy of subsequent correction.

[0013] According to some embodiments of the present invention, the step of converting an original face image into a face image at a target distance based on a preset target distance includes: inputting the original face image into an image diffusion model and outputting a face image at a target distance, wherein the image diffusion model has the ability to adjust the imaging distance of the image.

[0014] This embodiment utilizes an image diffusion model to adjust the imaging distance of the image and generate a target distance face image. This improves both conversion efficiency and conversion accuracy, making the target distance face image closer to a real face, thereby improving the accuracy of subsequent correction.

[0015] According to some embodiments of the present invention, the step of determining the coordinate change vector of at least some pixels based on an original face image and a target distance face image includes: extracting facial key points from the target distance face image to obtain target key points; extracting facial key points from the original face image to obtain original key points; and determining the coordinate difference between the target key points and the corresponding original key points as the coordinate change vector.

[0016] This embodiment extracts facial key points, which can associate key pixels in the target distance face image generated by the GAN inverse mapping model or image diffusion model with the key pixels in the original face image, and then determine the coordinate change vector of the pixel. This can improve the reference value of the coordinate change vector for image correction, thereby ensuring the accuracy of subsequent image correction.

[0017] According to some embodiments of the present invention, the step of determining the target coordinates of each grid in the initial deformed mesh includes: determining the initial target coordinates corresponding to each grid in the initial deformed mesh based on the coordinate change vectors of at least some pixels; optimizing the position of all grids in the initial deformed mesh based on the initial target coordinates corresponding to each grid to obtain an optimized deformed mesh; and using the coordinates of each grid in the optimized deformed mesh as the target coordinates corresponding to each grid in the initial deformed mesh.

[0018] This embodiment improves the accuracy of target coordinates in each grid by using position optimization techniques, which makes the target image obtained after subsequent correction more closely resemble a real human face.

[0019] According to some embodiments of the present invention, the step of determining the initial target coordinates corresponding to each grid in an initial deformed grid based on the coordinate change vectors of at least some pixels includes: determining the initial target coordinates of the grids corresponding to at least some pixels in the initial deformed grid based on the coordinate change vectors of at least some pixels; determining whether there are any missing grids in the initial deformed grid whose initial target coordinates have not been determined; and if so, determining the initial target coordinates of the missing grids based on multiple pixels adjacent to the missing grids.

[0020] According to some embodiments of the present invention, the step of determining the initial target coordinates of a missing grid based on a plurality of pixels adjacent to the missing grid includes: determining a weight value corresponding to each adjacent pixel based on the distance between the original coordinates of the missing grid and the plurality of adjacent pixels; and determining the initial target coordinates of the missing grid by weighted summing of the original coordinates of the missing grid and the displacements of each adjacent pixel based on the weight values ​​corresponding to each adjacent pixel.

[0021] According to some embodiments of the present invention, the method further includes: filling in the missing facial portions in the target image to generate a target image containing a complete facial portion.

[0022] This embodiment generates a target image containing a complete face by filling in the missing parts of the face in the target image, which makes the target image closer to the real face of the photographer.

[0023] According to some embodiments of the present invention, the step of completing the missing facial portion in a target image to generate a target image containing a complete face includes: performing face analysis on the target image to obtain facial cheek edge pixels; removing a predetermined number of pixels near the facial cheek edge pixels, facing the background area in the target image, to generate a first intermediate image; and performing image restoration on the first intermediate image to obtain the target image containing a complete face.

[0024] This embodiment can generate a first intermediate image containing a mask of the area to be repaired by clearing a preset number of pixels, so that the face can be accurately repaired by referring to the mask area in the first intermediate image.

[0025] According to some embodiments of the present invention, the step of filling in missing facial portions in a target image to generate a target image containing a complete face includes: acquiring a target distance face image containing a complete face; stretching the target distance face image containing a complete face to align the face in it with the face in the target image to generate a second intermediate image; determining the non-overlapping region between the second intermediate image and the target image; and performing image inpainting on the non-overlapping region in the target image to generate the target image containing a complete face.

[0026] This embodiment uses a target distance face image containing a complete face as a reference, which can improve the accuracy of face completion. Moreover, by using the non-overlapping region generated by comparing the second intermediate image and the target region as the missing face region, the detection accuracy of the missing face region can be further improved, thereby improving the accuracy of image completion.

[0027] According to some embodiments of the present invention, the step of stretching a target distance face image containing a complete face to align the face in the target image with the face in the target image, thereby generating a second intermediate image, includes: performing face analysis on the target image and finding the pixel points where the cheek edges of the face in the target image are located, as first cheek pixels; performing face analysis on the target distance face image containing a complete face and finding the pixel points where the cheek edges of the face in the target distance face image containing a complete face are located, as second cheek pixels; and stretching the target distance face image containing a complete face based on the first cheek pixels and the second cheek pixels to align the face in the target image with the face in the target image, thereby generating a second intermediate image.

[0028] According to some embodiments of the present invention, the step of stretching a target distance face image containing a complete face based on a first cheek pixel and a second cheek pixel includes: stretching each first cheek pixel to the position of the second cheek pixel that is closest to it.

[0029] This embodiment ensures that by stretching each first cheek pixel to the position of the second cheek pixel closest to it, the average distance between each cheek pixel in the stretched second intermediate image and the second cheek pixel in the original face image is minimized, thereby ensuring the alignment of the face region between the second intermediate image and the original face image.

[0030] Secondly, embodiments of this application provide an electronic device, including: a processor; and a memory, wherein computer program instructions are stored in the memory, wherein when the computer program instructions are executed by the processor, the processor performs the image perspective distortion correction method as described above.

[0031] Thirdly, embodiments of this application provide a computer-readable storage medium storing a computer program. When the computer program is run by a processor, the processor executes the image perspective distortion correction method as described above. Attached Figure Description

[0032] Figure 1(a) shows a scene of taking a selfie with a mobile phone;

[0033] Figure 1(b) is a comparison between a real human face and a photographed human face image showing perspective distortion;

[0034] Figure 2 This is a flowchart of an image perspective distortion correction method according to an embodiment of this application;

[0035] Figure 3(a) is a schematic diagram of the initial deformed mesh with uniform mesh distribution according to an embodiment of this application;

[0036] Figure 3(b) is a schematic diagram of the initial deformed mesh with non-uniform mesh distribution according to an embodiment of this application;

[0037] Figure 4 Here is an example of a camera imaging optical path diagram;

[0038] Figure 5 This is a flowchart of the step in the image perspective distortion correction method of this application, which converts the original face image into a face image at a target distance based on a preset target distance.

[0039] Figure 6 This is a flowchart of the step in the image perspective distortion correction method of this application, which involves offsetting and scaling the original depth map based on the target distance to generate a target distance depth map.

[0040] Figure 7 Here is an example of a camera imaging optical path diagram;

[0041] Figure 8 This is a flowchart illustrating the steps of determining the target coordinates of each grid in the initial deformed grid in the image perspective distortion correction method of this application embodiment;

[0042] Figure 9(a) is a schematic diagram of an example of an original human face image including key points of the eyes according to an embodiment of this application;

[0043] Figure 9(b) is a schematic diagram of another example of an original human face image including key points of the eyes according to an embodiment of this application;

[0044] Figure 10 This is a flowchart illustrating the step of filling in missing facial features in a target image to generate a target image containing a complete face in an image perspective distortion correction method according to an embodiment of this application.

[0045] Figure 11(a) is a schematic diagram of a target image containing pixels at the edge of a human face cheek according to an embodiment of this application;

[0046] Figure 11(b) is a schematic diagram of the first intermediate image of an embodiment of this application;

[0047] Figure 11(c) is a schematic diagram of a target image containing a complete human face according to an embodiment of this application;

[0048] Figure 12 This is a flowchart illustrating the step of filling in missing facial features in a target image to generate a target image containing a complete face in an image perspective distortion correction method according to an embodiment of this application.

[0049] Figure 13 This is a schematic diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0050] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.

[0051] Currently, when taking close-up portraits via selfies or other means, perspective distortion often occurs in the resulting facial images. As shown in Figure 1(a), the photographer holds a mobile phone and uses the front-facing camera (selfie lens) to take a selfie. However, when using the selfie lens to capture a close-up portrait, perspective distortion occurs in the resulting facial image. A typical characteristic of perspective distortion is that the subject's nose appears larger, and the facial contours are distorted. This distortion causes the face in the captured image to differ from the real face, severely affecting the shooting effect. As shown in Figure 1(b), comparing a real face with a face image exhibiting perspective distortion, the nose in the distorted image is larger, the facial contours show distortions such as a pointed and elongated chin, a narrower face, and the ears and hair are also obscured, failing to appear fully in the captured facial image.

[0052] To correct perspective distortion in images, some implementations use a method that fits a depth map to obtain the deformation vector of the face image to be corrected, and then uses this deformation vector to perform stretching, scaling, or other operations on the face image to achieve image correction. However, this method is still not ideal for correcting perspective distortion.

[0053] Therefore, an effective image perspective distortion correction scheme is needed to improve the accuracy of image perspective distortion correction.

[0054] To improve the accuracy of perspective distortion correction, the applicant discovered that perspective distortion often occurs in images captured at close range. Increasing the shooting distance between the face and the camera significantly improves the accuracy of perspective distortion correction. Therefore, based on this principle, the applicant proposes an image perspective distortion correction method that effectively improves the accuracy of perspective distortion correction.

[0055] The image perspective distortion correction method proposed in this application includes: acquiring an original face image; performing meshing processing on the original face image to obtain an initial deformation mesh; then converting the original face image into a target distance face image based on a preset target distance; determining the coordinate change vectors of at least some pixels based on the original face image and the target distance face image, and determining the target coordinates of each mesh in the initial deformation mesh; finally adjusting the pixels in the original face image to the target coordinates of the corresponding mesh to generate a target image.

[0056] The embodiments of this application can take into account the influence of shooting distance on the degree of perspective distortion, map the original face image taken at close range to a suitable distance to obtain a face image at the target distance, and use the face image at the target distance as a reference for perspective distortion correction, thereby realizing the correction of image perspective distortion and obtaining a target image that is closer to the real appearance of the subject.

[0057] Furthermore, by establishing an initial deformation mesh and referring to the target coordinates of the mesh in the initial deformation mesh to determine the target coordinates of the pixels in the original face image, the position of the pixels in the original face image can be precisely adjusted, thereby further improving the accuracy of image perspective distortion correction.

[0058] The image perspective distortion correction method proposed in the embodiments of this application will be described in detail below.

[0059] like Figure 2 As shown, the image perspective distortion correction method of this application embodiment can be applied to electronic devices such as computer terminals, mobile terminals (such as mobile phones, personal computers, tablet computers, wearable devices), virtual reality (VR) devices, or augmented reality (AR) devices.

[0060] like Figure 2 The flowchart shown illustrates the image perspective distortion correction method according to this application, which includes the following steps.

[0061] Step 110: Obtain the original face image.

[0062] A raw face image refers to a face image that exhibits perspective distortion. Raw face images can be obtained through self-portraits or photos taken by others, and the method of capturing the raw face image is not limited here. Raw face images are stored on electronic devices in commonly used image formats such as Joint Photographic Experts Group (JPG), Portable Network Graphics (PNG), RGB, High Efficiency Image File Format (HEIF), and RAW.

[0063] Step 120: Perform meshing on the original face image to obtain the initial deformed mesh.

[0064] In one embodiment, the original face image can be meshed according to preset mesh parameters to obtain an initial deformed mesh. The mesh parameters may include the size, number, and distribution pattern of the meshes. The size and number of meshes can be set according to the size of the face region in the original face image. For example, if the face region is large, a larger size and fewer meshes can be set; if the face region is small, a smaller size and more meshes can be set. This method facilitates accurate subsequent adjustments to the face region, thereby ensuring the subsequent correction effect.

[0065] The grid distribution pattern can include uniform and non-uniform distribution. In some examples, the initial deformed grid can be uniformly distributed, meaning the grid density is uniform, such as the uniformly distributed initial deformed grid shown in Figure 3(a), where the same grid size is used throughout the entire image. In other examples, the initial deformed grid can be non-uniformly distributed, meaning the grid density is not uniform, such as the non-uniformly distributed initial deformed grid shown in Figure 3(b). Understandably, the specific values ​​of the grid parameters can be set according to the actual situation and are not limited here.

[0066] In some examples, a non-uniformly distributed initial deformation mesh can be randomly generated within the original face image; that is, the sparse distribution of the mesh in the initial deformation mesh is randomly generated. In other examples, a dense mesh can be set for the face region in the original face image, and a sparse mesh can be set for the background region (or non-face region) in the original face image. This allows the correction focus to be concentrated on the face region, reducing the computational load on the background region while ensuring the face image correction effect.

[0067] It should be noted that the grid sizes in Figures 3(a) and 3(b) are for illustrative purposes only to show different grid distribution patterns, and the grid sizes presented in the figures do not limit the grid settings in the embodiments of this application.

[0068] The grids in the initial deformed mesh correspond to the pixels in the original face image. In one embodiment, the grids in the initial deformed mesh and the pixels in the original face image can have a one-to-one, one-to-many, or hybrid correspondence. A hybrid correspondence means that some grids have a one-to-one correspondence with pixels, while others have a one-to-many correspondence. For example, when the original face image is uniformly divided into grids based on pixels, the grids in the initial deformed mesh can correspond one-to-one with the pixels. Or, in some implementations, one grid corresponds to multiple pixels, such as 4, 9, or 16. On the other hand, the correspondence between grids and pixels can also be hybrid. For example, when the original face image is non-uniformly divided into grids, in some image regions of the initial deformed mesh, the grids are set to correspond one-to-one with pixels, while in other image regions, one grid corresponds to multiple pixels.

[0069] Understandably, the specific correspondence between grids and pixels can be obtained from the actual grid distribution in the initial deformed grid, and is not limited here.

[0070] In one embodiment, projection algorithms such as spherical projection or cylindrical projection can be used to mesh the original face image to obtain an initial deformed mesh. Spherical projection refers to the mapping of a sphere onto a plane, while cylindrical projection refers to the process of projecting a plane onto a cylinder.

[0071] Specifically, the original face image can first be meshed according to preset grid parameters to obtain the original grid. Then, projection transformations such as spherical projection or cylindrical projection are used to project the original grid to generate the initial deformed grid.

[0072] This embodiment utilizes a projection algorithm to transform the initial deformed mesh, which can perform preliminary correction on the initial deformed mesh, allowing it to maintain the realism of the face to a certain extent, thereby improving the correction effect of image perspective distortion correction in this embodiment.

[0073] Step 130: Based on the preset target distance, convert the original face image into a face image at the target distance.

[0074] The process converts a raw face image into a face image at a target distance. The target distance face image is an image of the face from the raw face image rendered at a target distance, which is greater than the original image's shooting distance. In other words, a raw face image captured at close range can be converted into a face image captured at a distant distance at the target distance.

[0075] The target distance can be set at the factory default according to actual needs. It only needs to ensure that the facial image captured at the target distance has virtually no perspective distortion. This embodiment does not limit the specific value of the target distance. For example, the most natural and comfortable distance for a user's selfie is 35cm-40cm; however, this close shooting distance can cause perspective distortion of the face. Experiments have shown that images captured at a distance of 60cm have virtually no perspective distortion. Therefore, the target distance can be set to 60cm, or it can be set to other values ​​greater than 60cm. It is understood that the aforementioned 60cm or other values ​​greater than 60cm are merely examples of target distance values, and this embodiment is not limited to these.

[0076] Perspective distortion in original facial images is generally caused by close-up photography. Therefore, compared to the original facial image, the degree of perspective distortion in facial images captured at the target distance is lower. Thus, by acquiring a facial image at the target distance and then using it as a reference to perform subsequent correction operations on the original facial image, a foundation can be laid for accurate correction of image perspective distortion. It is worth noting that the target distance facial image obtained by adjusting the imaging distance of the original facial image may still exhibit distortion, or facial details may change. Therefore, simply changing the image imaging distance to improve the facial shape cannot achieve accurate correction. This embodiment uses the target distance facial image as a reference to further perform correction operations on the original facial image, ensuring the accuracy of image perspective distortion correction.

[0077] In one embodiment, an image conversion method capable of changing the image imaging distance can be used to convert the original face image into a face image at the target distance. For example, image similarity transformation or image conversion models with imaging distance adjustment capabilities can be used, and no limitation is made here.

[0078] Step 140: Based on the original face image and the target distance face image, determine the coordinate change vector of at least some pixels, and determine the target coordinates of each grid in the initial deformed grid.

[0079] After obtaining the original face image and the target distance face image, the differences between them can be determined. The coordinate positions of some or all pixels in the two images will change, and the coordinate change vectors of these pixels can be obtained by comparing the two images. The "coordinate change vector" refers to the vector displacement of a pixel, representing the magnitude and direction of its movement from the original coordinates to the target coordinates. By obtaining the coordinate change vectors corresponding to pixels in the original face image and the target distance face image, the correction direction and displacement amount for pixels in the original face image can be provided, thus making face distortion correction more accurate.

[0080] Generally, the coordinate change vector can be obtained by using the difference between pixels in the original face image and the target face image. For example, if the coordinates of pixel A in the original face image are (x1, y1), and the coordinates of pixel A in the target face image are (x2, y2), then the coordinate change vector for pixel A is (x2-x1, y2-y1). In other examples, common vector displacement methods such as decomposition and geometric methods can also be used to determine the coordinate change vector, which are not limited here.

[0081] The following is based on Figure 4 Using the camera imaging optical path diagram shown as an example, the coordinate transformation vector is explained. The imaging distance of the original face image is Z1, and the nose of the subject in the face image is marked as A. Nose A is projected onto the imaging plane through the optical center O of the camera (i.e., the convex lens), generating image A1, which can also be denoted as pixel coordinate A1. The imaging distance of the target from the face image is Z2. At this time, nose A is projected onto the imaging plane through the optical center O of the camera, generating image A2, which can also be denoted as pixel coordinate A2. Therefore, the coordinate transformation vector corresponding to the pixel point where nose A is located is the coordinate difference between A1 and A2.

[0082] After determining the coordinate change vectors of at least some pixels, the target coordinates of each grid in the initial deformed mesh can be determined based on these vectors. By determining the target coordinates of each grid in the initial deformed mesh, the coordinate change vectors can be converted into intuitive target coordinates, facilitating subsequent adjustments to the pixel positions in the original face image. Furthermore, by combining interpolation or mesh optimization algorithms, more accurate and comprehensive grid target coordinates can be determined using the coordinate change vectors of some pixels, thereby improving the accuracy and efficiency of face correction. The interpolation or mesh optimization algorithms are described in detail in the following embodiments and will not be elaborated upon here.

[0083] The target coordinates of the grid refer to the target coordinates of the grid vertices, the target coordinates of the grid center point, or the target coordinates of other feature points of the grid, and are not limited here.

[0084] In one embodiment, the coordinate change vectors corresponding to all pixels can be determined one by one first, and then the target coordinates of each grid in the initial deformed grid can be determined based on the coordinate change vectors corresponding to all pixels.

[0085] In one embodiment, coordinate change vectors of some pixels in the original face image and the target distance face image can be obtained. Then, based on these coordinate change vectors, the target coordinates of each grid in the initial deformed mesh are determined. In one example, coordinate change vectors of facial key points (such as eyes, nose, and mouth) in the original face image and the target distance face image can be obtained. The target coordinates of the corresponding grids in the initial deformed mesh are determined using the coordinate change vectors of the facial key points. Then, interpolation processing is performed on the remaining grids to determine their target coordinates. The specific interpolation method is described in detail in the following embodiments and will not be elaborated here.

[0086] "Partial pixels" can refer to facial landmarks extracted using facial landmark detection technology, pixels selected within the face region according to preset selection rules, or pixels randomly selected within the face region. The preset selection rules can include the number of pixels selected and the selection spacing. These preset selection rules can be set according to actual needs and are not limited here.

[0087] This embodiment determines the target coordinates of all grids in the initial deformed grid by selecting the coordinate change vectors of some pixels. On the one hand, this reduces the amount of computation and improves the correction efficiency; on the other hand, it avoids the errors caused by the small face area and dense pixel sampling, thus improving the correction accuracy.

[0088] Step 150: Adjust the pixels in the original face image to the target coordinates of the corresponding grid to generate the target image.

[0089] After determining the target coordinates of each grid in the initial deformation mesh, the pixels in the original face image can be adjusted to the target coordinates of the corresponding grid to generate the target image. In other words, by referring to the target coordinates of each grid in the initial deformation mesh, the positions of the corresponding pixels in the original face image can be adaptively adjusted (i.e., the pixel positions are moved to the target coordinates of the corresponding grid), thereby correcting the perspective distortion of the original face image and generating a target image that reflects a realistic face.

[0090] This embodiment maps a close-up, original face image to a suitable distance to obtain a target-distance face image. This target-distance face image is then used as a reference for perspective distortion correction, enabling precise correction of image perspective distortion and resulting in a target image that more closely resembles the subject's true appearance. Furthermore, this embodiment establishes an initial deformation mesh and uses the target coordinates of the meshes within that mesh to determine the target coordinates of pixels in the original face image. This allows for precise adjustment of pixel positions in the original face image, further improving the accuracy of perspective distortion correction. Additionally, by selecting the coordinate change vectors of a subset of pixels to determine the target coordinates of all meshes in the initial deformation mesh, the computational load can be reduced, improving correction efficiency. Furthermore, this approach avoids errors caused by small face areas and dense pixel sampling, thus enhancing correction accuracy.

[0091] The following are examples of... Figure 2 The process of converting the original face image into a face image at a target distance based on a preset target distance explained in step 130, and the process of determining the coordinate change vector of at least some pixels and the target coordinates of each grid in the initial deformed grid based on the original face image and the face image at the target distance in step 140, will be further explained.

[0092] In one embodiment, such as Figure 2 Step 130 above, which is the step of converting the original face image into a face image at the target distance based on a preset target distance, may specifically include, as follows: Figure 5 The following steps are shown.

[0093] Step 131: Obtain the original depth map corresponding to the original face image.

[0094] The process involves obtaining the original depth map corresponding to the original face image. The original depth map reflects the depth information or depth values ​​of each pixel in the original face image, with the depth values ​​representing the distance between the pixel and the camera. The original face image refers to a color image of the face, which can be captured by a color camera. There is a one-to-one relationship between the pixels in the original depth map and the original face image.

[0095] In one embodiment, existing depth estimation algorithms can be used to extract depth information from the original face image to obtain an original depth map. The depth estimation algorithm may include, but is not limited to, monocular depth estimation technology and deep learning-based depth estimation models.

[0096] In one embodiment, the original depth map may also be a depth map obtained by using a depth camera under the same shooting conditions (such as shooting time, object, and scene) as the original face image. The depth camera may include, but is not limited to, an RGBD camera, a binocular depth camera, a structured light depth camera, and a ToF (Time of Flight) depth camera. Understandably, in one example, the original face image and the original depth map can be acquired simultaneously using an RGBD camera.

[0097] In one embodiment, the raw depth map obtained using an RGBD depth camera needs to be aligned with the corresponding raw face image to unify the pixel coordinates in the raw depth map and the raw face image. The following example illustrates the alignment step one, using a raw face image captured by an RGB camera and the raw depth map captured by a depth camera.

[0098] In this example, we first obtain the camera's intrinsic parameters and extrinsic parameters. The intrinsic parameters describe the camera's internal properties, such as focal length and principal point position. The extrinsic parameters describe the camera's position and orientation in the world coordinate system, and may include the rotation matrix (R) and translation vector (T) between the two cameras. Each pixel coordinate and its corresponding depth in the original depth map is converted into 3D point coordinates in the depth camera coordinate system. Then, using the extrinsic parameters, the 3D point coordinates are transformed from the depth camera coordinate system to the RGB camera coordinate system, and finally projected onto the RGB pixel coordinate system via the RGB camera, thus aligning the original depth map and the original face image.

[0099] In one embodiment, camera intrinsic parameters may be calibrated before using the camera to capture raw face images or raw depth maps.

[0100] The following example, using an elliptical array calibration board, illustrates a sample of the camera calibration process. First, the central convex hull of the calibration image is calculated. For a given set of points, the convex hull is the smallest convex polygon or polyhedron containing that set of points. In other words, the convex hull contains all points in the set and is the shape with the smallest area (or volume) capable of containing all points. The central convex hull of the calibration image refers to the smallest convex polygon or polyhedron containing the center points of all ellipses in the calibration image. Then, by filtering by the angles of adjacent points, the four corner points of the calibration image are selected, where these four corner points correspond to the centers of the four corners of the calibration board. An affine transformation is performed on each pixel in the calibration image to align the centers of each elliptical feature point in the calibration image with the centers of the ellipses on the calibration board. The elliptical feature points in the calibration image are then sorted by row and column to obtain the correct order of all ellipses in the calibration image. Finally, by combining the real coordinates of each elliptical feature point in the calibration image in physical space, the camera intrinsic parameters are calculated using the OpenCV calibration function, thus completing the camera intrinsic parameter calibration.

[0101] Understandably, in other embodiments, a checkerboard or dot array calibration board may be used for camera calibration, or other camera calibration methods may be used, which are not limited here.

[0102] After calibrating the camera intrinsic parameters using the above method, the calibrated camera can be used to capture the original face image and the original depth map. Then, the original depth map is aligned with the corresponding original face image to unify the pixel coordinates in the original depth map and the original face image.

[0103] Step 132: Based on the target distance, offset and scale the original depth map to generate a target distance depth map.

[0104] The offset and scaling factor required to generate the target distance depth map can be determined based on the target distance. Then, the original depth map is offset based on the offset and scaled based on the scaling factor to generate the target distance depth map. A detailed explanation of step 132 will be provided later.

[0105] Step 133: Reproject the target distance depth map to obtain the target distance face image.

[0106] "Reprojecting the target distance depth map" refers to projecting the pixel coordinates in the target distance depth map back onto the coordinate system of the original face image to obtain the target distance face image.

[0107] In one example, step 133 may include: converting the coordinates and depth of each pixel in the target distance depth map into a 3D point in the camera coordinate system. Then, the 3D point coordinates are converted to the world coordinate system, and the corresponding RGB pixel values ​​of each pixel in the world coordinate system are mapped to the 3D point to generate a target distance face image.

[0108] In one embodiment, such as Figure 5 The process described in step 132, which involves offsetting and scaling the original depth map based on the target distance to generate a target distance depth map, can be specifically described as follows: Figure 6 The steps shown are as follows.

[0109] Step 1321: Use the difference between the target distance and the shooting distance of the original face image as the depth offset value, and perform depth offset on the pixels in the original depth map according to the depth offset value.

[0110] For example, if the original face image is captured at a distance of 45cm and the target distance is 60cm, then the depth offset value can be set to 15cm. Then, the depth of each pixel in the original depth map is shifted by 15cm in a direction parallel to the camera's optical axis and away from the camera.

[0111] Step 1322: Perform facial landmark detection on the original depth map to obtain facial landmarks.

[0112] Facial landmarks refer to key feature points on a face, used to represent the characteristics of key facial features such as facial features and face shape. Facial landmarks generally include facial contours, eyebrows, eyes, nose contours, lips, etc. Facial landmark detection can be achieved using any facial landmark detection algorithm in the field of image processing technology, such as Active Shape Model (ASM), Cascadedpose regression (CPR), or deep learning-based detection algorithms, etc., and understandably, this application is not limited to these.

[0113] Step 1323: Determine the depth corresponding to the centroid of all facial landmarks.

[0114] Obtain the coordinates of all facial key points, calculate the coordinates of the centroid of the polygon formed by all facial key points using geometric methods, and then determine the depth value at the centroid.

[0115] For example, keypoints for the eyes, nose, and mouth are collected in the original depth map. These three keypoints form a triangle. Three medians are then determined from the three sides of this triangle, and the pixel where these medians intersect is identified as the centroid. Finally, by comparing the depth information of each pixel in the original depth map, the depth value at the centroid is determined.

[0116] Step 1324: Based on the depth and depth offset value corresponding to the centroid point, determine the scaling factor and scale the pixels in the original depth map according to the scaling factor.

[0117] In one example, the scaling factor can be determined using the following formula (1) based on the depth and depth offset value corresponding to the centroid point:

[0118]

[0119] In equation (1) above, s refers to the scaling factor, d refers to the depth corresponding to the centroid, and offset refers to the depth offset value.

[0120] After determining the scaling factor, the coordinates of each pixel can be multiplied by the scaling factor to complete the scaling of the pixels in the original depth map.

[0121] In one embodiment, before performing step 132 above, or before offsetting and scaling the original depth map to generate the target distance depth map, face parsing can be performed on the original depth map to obtain a face parsing mask or face mask, wherein the face parsing mask contains the segmented parts of the face and their corresponding identifiers. Then, the face region is segmented in the original depth map using the face parsing mask, thereby offsetting and scaling the face region.

[0122] Face parsing refers to the decomposition of a person's head, including facial features such as eyes, nose, mouth, cheeks, eyebrows, and facial contours. Face parsing algorithms can be used to perform face parsing on the original depth map, such as semantic segmentation, which are not limited here.

[0123] Although the above explanation illustrates the process of obtaining a target range-depth map by offsetting and scaling the original depth map, and then reprojecting the target range-depth map, this is not the actual implementation. Figure 2 Step 130 in the diagram refers to the step of converting the original face image into a face image at the target distance based on a preset target distance. In some other implementations, this process can also be achieved using existing GAN inverse mapping models.

[0124] Specifically, step 130 may also include: inputting the original face image into the GAN inversion model and outputting a face image at the target distance. The GAN inversion model can generate images under different camera parameters. In this embodiment, the GAN inversion model can be used to convert the image shooting distance and generate a face image at the target distance.

[0125] The GAN inverse mapping model is used to parse the hidden facial information in the original face image and construct the equivalent camera parameters at the target distance. Based on the hidden facial information and the equivalent camera parameters, the original face image is reconstructed to generate the face image at the target distance. The hidden facial information can also be understood as face latent code, which contains detailed image information of the face that cannot be observed in the original face image, such as parts not imaged due to the shooting angle or image information of parts like ears or hair hidden due to perspective distortion. By parsing the hidden facial information, gaps in the face image can be filled, making the face image at the target distance closer to the real face, thereby improving the accuracy of subsequent correction.

[0126] In one example, the equivalent camera parameters may include object distance and focal length. That is, the GAN inverse mapping model in this embodiment can adjust the object distance in the virtual camera to the target distance and adjust the focal length adaptively to a reasonable value, thereby generating a target distance face image based on the adjusted focal length.

[0127] The following is combined with, for example Figure 7 The camera imaging optical path diagram shown illustrates the principle of how the GAN inverse mapping model constructs the focal length at the target distance.

[0128] The target P is imaged by projecting its optical center O onto the imaging plane through the camera (i.e., the convex lens), resulting in P'. Wherein... Figure 7 On the optical axis shown, F refers to the focal point, f refers to the focal length, z refers to the object distance, X refers to the image height, and AO refers to the phase distance. The parameters on the optical axis satisfy the following relationship: X / u = z / AO = z / (AF+f) = f / AF.

[0129] With the image height X constant, X / u is a constant, and from the above correlation, we know that z / (X / u+1)f is also a constant. Therefore, if we want to increase the object distance z to the target distance while keeping the image height X constant, we need to enlarge f proportionally to construct the focal length at the target distance.

[0130] The above explains how to achieve this using the GAN inverse mapping model. Figure 2Step 130 in the diagram refers to the step of converting the original face image into a face image at the target distance based on a preset target distance. In some other implementations, this process can also be achieved using existing image diffusion models.

[0131] Specifically, step 130 may also include: inputting the original face image into an image diffusion model and outputting a target distance face image, wherein the image diffusion model has the ability to adjust the imaging distance of the image. The image diffusion model may include deep learning text-to-image generation models (Stable Diffusion, SD), diffusion probability models (DPMs), or denoising diffusion probability models (DDPMs), etc., and this application is not limited to these.

[0132] The following describes in detail the step 140 above, which involves determining the coordinate change vectors of at least some pixels based on the original face image and the target distance face image, using the three embodiments described above for generating target distance face images.

[0133] like Figure 2 In step 140, which explains the step of determining the coordinate change vectors of at least some pixels based on the original face image and the target distance face image, in the case where the target distance face image is generated based on a depth map (such as combining...), Figure 5 The step 140, which involves determining the coordinate change vector of at least some pixels based on the original face image and the target distance face image (explained in steps 131 to 133), may include: using the coordinate difference between at least some pixels in the original face image and the target distance face image as the coordinate change vector corresponding to at least some pixels.

[0134] In other words, the pixel coordinates of the original face image and the new pixel coordinates of the target distance in the face image after reprojection in step 133 can be obtained. Then, the difference between the pixel coordinates before and after is calculated to obtain the coordinate change vector. Understandably, in this embodiment, the coordinate change vector corresponding to each pixel can be determined, or the coordinate change vector corresponding to some pixels can be determined, which will not be elaborated here.

[0135] In cases where the target distance face image is generated using a GAN inverse mapping model or an image diffusion model, step 140, which determines the coordinate change vector of at least some pixels based on the original face image and the target distance face image, may include: extracting facial key points from the target distance face image to obtain target key points; extracting facial key points from the original face image to obtain original key points; and determining the coordinate difference between the target key points and the corresponding original key points as the coordinate change vector.

[0136] For example, a facial landmark detection algorithm can be used to extract facial landmarks corresponding to the eyes, nose, and mouth from a target face image, resulting in target eye landmarks, target nose landmarks, and target mouth landmarks, respectively. The algorithm is then used to extract the original eye landmarks, nose landmarks, and mouth landmarks from the original face image, resulting in original eye landmarks, original nose landmarks, and original mouth landmarks, respectively. The coordinate difference between the target eye landmark and the original eye landmark is defined as the coordinate change vector corresponding to the eye landmark. Similarly, the coordinate difference between the target nose landmark and the original nose landmark is defined as the coordinate change vector corresponding to the nose landmark. The coordinate difference between the target mouth landmark and the original mouth landmark is also defined as the coordinate change vector corresponding to the mouth landmark. It is understood that this example only illustrates the extraction of three facial landmarks corresponding to the eyes, nose, and mouth; the embodiments of this application are not limited to this, and the number and location of facial landmarks can be selected according to the actual situation.

[0137] Facial landmark detection can be achieved using any facial landmark detection algorithm in the field of image processing technology, such as Active Shape Model (ASM), Cascadedpose regression (CPR), or deep learning-based detection algorithms. Understandably, the embodiments of this application are not limited thereto.

[0138] This embodiment extracts facial key points, which can associate key pixels in the target distance face image generated by the GAN inverse mapping model or image diffusion model with the key pixels in the original face image, and then determine the coordinate change vector of the pixel. This can improve the reference value of the coordinate change vector for image correction, thereby ensuring the accuracy of subsequent image correction.

[0139] The following is about... Figure 2 The process of determining the target coordinates of each grid in the initial deformed grid, as explained in step 140, will be described in detail.

[0140] In one embodiment, such as Figure 2 The step of determining the target coordinates of each grid in the initial deformed grid, as explained in step 140, may specifically include, for example: Figure 8 The following steps are shown.

[0141] Step 141: Determine the initial target coordinates of each grid in the initial deformed grid based on the coordinate change vectors of at least some pixels.

[0142] In one embodiment, the initial target coordinates of the grids corresponding to at least a portion of the pixels in the initial deformed grid can be determined first using the coordinate change vectors of at least a portion of the pixels. Then, it is determined whether there are any missing grids in the initial deformed grid whose initial target coordinates are not determined. If there are missing grids whose initial target coordinates are not determined, the initial target coordinates of the missing grids are determined based on multiple pixels adjacent to the missing grids. If there are no missing grids whose initial target coordinates are not determined, the initial target coordinates corresponding to all grids in the initial deformed grid are generated. A detailed explanation of the process for determining the initial target coordinates of missing grids based on multiple pixels adjacent to the missing grids, as explained in step 141, will be further explained later.

[0143] Step 142: Based on the initial target coordinates corresponding to each grid, perform position optimization on all grids in the initial deformed grid to obtain the optimized deformed grid.

[0144] "Position optimization" can be understood as a further adjustment of the initial target coordinates to make the target coordinates of each grid more accurate. Specifically, "position optimization" can be performed by referring to the spatial correlation between grids. "Spatial correlation" refers to the fact that when the position of a certain grid moves, it may cause the position of surrounding grids to shift. Improving the accuracy of the target coordinates of each grid through position optimization can make the target image obtained after subsequent correction closer to the real human face.

[0145] In one embodiment, a similarity transformation can be performed on the initial target coordinates corresponding to each grid to optimize the position of all grids in the initial deformed grid. For example, an energy term corresponding to the face region can be constructed, and then a similarity transformation can be performed on the grids of the face region based on the energy term and a loss function. The data term of the loss function is the error of the grid vertices, that is, the L2 norm of the Euclidean distance between the current grid vertex position and the target coordinates on the pixel plane is calculated in each iteration. In addition, an energy term corresponding to the background region can also be constructed to ensure that straight lines are scaled proportionally rather than distorted. That is, the direction vectors between the grids in the background region and their adjacent grids remain straight lines before and after optimization.

[0146] Step 143: Use the coordinates of each grid in the optimized deformed grid as the target coordinates of each grid in the initial deformed grid.

[0147] The following is a detailed explanation of the step 141 above, which involves determining the initial target coordinates of the missing grid based on multiple pixels adjacent to the missing grid.

[0148] In one embodiment, the step of determining the initial target coordinates of the missing grid based on multiple pixels adjacent to it may include: determining a weight value corresponding to each adjacent pixel based on the distance between the original coordinates of the missing grid and the original coordinates of the multiple adjacent pixels; and determining the initial target coordinates of the missing grid by weighted summing of the original coordinates of the missing grid and the displacements of each adjacent pixel based on the weight values ​​of each adjacent pixel. Here, the displacement of each adjacent pixel refers to the displacement generated by the adjacent pixel between the original face image and the target distance face image, which can be calculated using the distance formula between two points.

[0149] In one example, the weight value corresponding to each adjacent pixel can be determined using the following formula (2):

[0150]

[0151] In equation (2) above, λ k This refers to the weight value corresponding to the adjacent pixel k, |nE| refers to the distance between the adjacent pixel n and the original coordinates of the missing grid, and m refers to the number of adjacent pixels. Here, "original coordinates of the missing grid" refers to the coordinates of the missing grid in the initial deformed grid, and "original coordinates of the pixel" refers to the coordinates of the pixel in the original face image.

[0152] The initial target coordinates of the missing grid can be determined using the following formula (3):

[0153]

[0154] In equation (3) above, E' refers to the target coordinates of the missing grid, and |n'n| refers to the displacement of the adjacent pixel n.

[0155] As shown in Figure 9(a), which is a schematic diagram of an example of an original face image containing eye key points, the following description, in conjunction with Figure 9(a) and taking the eye key points AD extracted using face key point detection technology as an example, will illustrate the example of determining the initial target coordinates of the missing grid in this embodiment.

[0156] As shown in Figure 9(a), the missing mesh E is adjacent to the eye key point AD. The original coordinates of the eye key point AD in the initial deformed mesh can be obtained first. Then, using equation (2) above, the weight value corresponding to the eye key point AD is determined based on the distance between the missing mesh E and the eye key point AD, as shown in equation (4) below:

[0157]

[0158] Obtain the coordinate change vector of the eye key point AD in the face image from the target distance. Based on the weight value corresponding to the eye key point AD, the displacement of the eye key point AD and the original coordinates of the missing grid are weighted and summed to determine the target coordinates of the missing grid, as shown in the following formula (5):

[0159] E'=E+|A'A|*λ A +|B'B|*λ B +|C'C|*λ C +|D'D|*λ D Equation (5)

[0160] In equation (5) above, E' refers to the target coordinates of the missing grid, |A'A| refers to the displacement of the adjacent pixel A, |B'B| refers to the displacement of the adjacent pixel B, |C'C| refers to the displacement of the adjacent pixel C, and |D'D| refers to the displacement of the adjacent pixel D. The relative positions of the target coordinates E' of the missing grid and the target coordinates A'-D' of the adjacent pixel AD are shown in Figure 9(b).

[0161] In one embodiment, the initial target coordinates of the remaining meshes in the initial deformed mesh can also be completed using an interpolation algorithm. The interpolation algorithm may include at least one of the following: Inverse Distance Weighting (IDW), Kriging, Linear Interpolation, or Nearest Neighbor Interpolation. The principle of Inverse Distance Weighting is that the value of an unknown point is obtained by weighting the values ​​of its nearest known points, where the weight is inversely proportional to the reciprocal of the distance. Kriging interpolation establishes an inter-correlation model between the values ​​of known points and then uses this model to estimate the value of the unknown point. Linear interpolation estimates the value based on the two nearest known points to the unknown point in the one-dimensional data sequence. Nearest Neighbor Interpolation sets the value of the unknown point to the value of the nearest known point. It is understood that the interpolation algorithm used in this embodiment is not limited to these.

[0162] In one embodiment, the image perspective distortion correction method of this application further includes: completing the missing facial portion in the target image to generate a target image containing a complete face, thereby making the target image closer to the photographer's real face. The missing facial portion may include ears, hair, or other hidden parts of the face. It is worth noting that, for ease of description, the following embodiments use the missing ear in the target image as an example for detailed explanation, but this application is not limited to this.

[0163] In one embodiment, such as Figure 10As shown, the process of filling in the missing facial parts in the target image to generate a target image containing a complete face can specifically include:

[0164] Step 210: Perform face analysis on the target image to obtain the edge pixels of the face cheeks.

[0165] By utilizing the facial features and corresponding identifiers in the face resolution mask generated by face analysis, the position of the cheek edge is determined, thereby obtaining the cheek edge pixels. No limitation is made on the type of face analysis algorithm used here.

[0166] Taking the target image shown in Figure 11(a) as an example, the target image in Figure 11(a) is marked with the edge pixels of the face cheek. In Figure 11(a), the edge pixels of the face cheek are indicated by white dots.

[0167] Step 220: Facing the background area in the target image, clear a preset number of pixels near the edge pixels of the face cheek to generate the first intermediate image.

[0168] Facing the background area in the target image, a predetermined number of pixels are removed near the edge pixels of the face cheek to generate a first intermediate image. Removing this predetermined number of pixels is to generate a first intermediate image containing a mask of the area to be repaired, so that subsequent face repair can be accurately performed by referring to the mask area in this first intermediate image.

[0169] Taking the target image shown in Figure 11(a) as an example, near the edge pixels of the face cheek, towards the background area in the target image, a preset number of pixels are cleared to generate the first intermediate image shown in Figure 11(b). In Figure 11(b), the white area is the mask area where the preset number of pixels have been cleared.

[0170] Understandably, the number and location of the pixels removed can be reasonably set according to the size and shape of the ear, and are not limited here. Understandably, when it is necessary to complete other missing parts besides the ear, the pixels in the areas where the other missing parts are located can be adaptively removed, and then the following steps can be used to complete them.

[0171] Step 230: Perform image restoration on the first intermediate image to obtain the target image containing a complete human face.

[0172] Taking Figure 11(b) as an example, the first intermediate image shown in Figure 11(b) is inpainted using an image inpainting algorithm. This algorithm can inpaint the masked part of the image and complete the ear part of the face, generating the target image containing the complete face shown in Figure 11(c).

[0173] The image inpainting algorithms used in this embodiment may include, but are not limited to, mask-aware transformer (MAT), generative landmark-guided face inpaintor (LaFIn), DeepFill v2 (Free-Form Image Inpainting with Gated Convolution), and large mask inpainting model (LaMa).

[0174] In other embodiments, methods including, can also be used. Figure 12 The following steps are shown to complete the missing facial parts in the target image and generate a target image containing a complete face.

[0175] Step 310: Obtain the target distance face image containing the complete face.

[0176] A "target distance face image containing a complete face" can be generated by processing the original face image using a GAN inverse mapping model or an image diffusion model. Understandably, target distance face images generated using GAN inverse mapping models or image diffusion models have higher face completeness, but facial details may deviate from the original face image. Therefore, a target distance face image containing a complete face generated by a GAN inverse mapping model or image diffusion model can be used as a reference to achieve accurate face completion.

[0177] Step 320: Stretch the target distance face image containing the complete face so that the face in the target image is aligned with the face in the target image to generate a second intermediate image.

[0178] The target distance containing the complete face can be stretched to the face region in the face image to align the faces in the two images.

[0179] In one embodiment, step 320 may include: performing face analysis on the target image and finding the pixel points where the cheek edges of the face are located in the target image, as the first cheek pixel points; performing face analysis on the target distance face image containing the complete face and finding the pixel points where the cheek edges of the face are located in the target distance face image containing the complete face, as the second cheek pixel points; then, based on the first cheek pixel points and the second cheek pixel points, stretching the target distance face image containing the complete face so that the face in it is aligned with the face in the target image, generating a second intermediate image. The type of face analysis algorithm is not limited here.

[0180] In one example, the process of stretching a target distance face image containing the complete face based on the first cheek pixels and the second cheek pixels can be implemented using the following steps: stretching each first cheek pixel to the position of the second cheek pixel that is closest to it. By stretching each first cheek pixel to the position of the second cheek pixel that is closest to it, the average distance between each cheek pixel in the stretched second intermediate image and the second cheek pixel in the original face image can be minimized, thereby ensuring the alignment of the face region between the second intermediate image and the original face image.

[0181] In another example, the process of stretching a target distance face image containing a complete face based on first cheek pixels and second cheek pixels can also be implemented using the following steps: Determine second cheek pixels adjacent to each first cheek pixel. Calculate the average distance between each first cheek pixel and its corresponding adjacent second cheek pixel. If the average distance is less than a preset distance threshold, stretch the first cheek pixels to their corresponding adjacent second cheek pixels to generate a second intermediate image. If the average distance is greater than or equal to the preset threshold, return to the step of determining the second cheek pixels adjacent to each first cheek pixel until the average distance is less than the preset distance threshold. The preset distance threshold can be determined based on the average distance between each first cheek pixel and its nearest second cheek pixel, or it can be customized according to actual needs, which is not limited here.

[0182] Step 330: Determine the non-overlapping region between the second intermediate image and the target image.

[0183] The second intermediate image is an image containing a complete face that is aligned with the target image. The "non-overlapping region" between the second intermediate image and the target image refers to the region in the target image where the missing part of the face is located. By using the non-overlapping region generated by comparing the second intermediate image and the target region as the region of the missing face, the detection accuracy of the missing face can be further improved, thereby improving the accuracy of image completion.

[0184] Generally, the target image may have missing ears and hair. Therefore, the "non-overlapping area" between the second intermediate image and the target image usually refers to the part of the second intermediate image that extends beyond the face area of ​​the target image, namely the ear and hair parts.

[0185] Step 340: Perform image inpainting on the non-overlapping regions in the target image to generate a target image containing a complete human face.

[0186] In one example, pixels in non-overlapping regions can be eliminated from the target image first. Then, an image inpainting algorithm can be used to fill in the non-overlapping regions by referencing a target image containing a complete face, thereby generating a target image containing a complete face. The type of image inpainting algorithm is not limited here.

[0187] One embodiment of the present invention also provides an electronic device, which includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, the at least one instruction or the at least one program being loaded and executed by the processor to implement the image perspective distortion correction method provided in the above method embodiments.

[0188] Memory is used to store software programs and modules. The processor executes these stored software programs and modules to perform various functional applications and data processing. Memory can primarily consist of a program storage area and a data storage area. The program storage area stores the operating system, application programs required for functionality, etc.; the data storage area stores data created based on device usage, etc. Furthermore, memory can include high-speed random access memory (RAM) and non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, memory can also include a memory controller to provide the processor with access to the memory.

[0189] In one specific embodiment Figure 13 The diagram shows a structural schematic of an electronic device used to implement the embodiments of this application. The electronic device can be a computer terminal, a mobile terminal (such as a mobile phone, personal computer, tablet computer, wearable device), or a virtual reality device, augmented reality device, etc.

[0190] like Figure 13 As shown, this application provides an electronic device, including a processor 1301 and a memory 1302. The memory 1302 stores computer program instructions, wherein when the computer program instructions are executed by the processor, the processor 1301 performs the image perspective distortion correction method as described in the above embodiments.

[0191] Furthermore, such as Figure 13 As shown, the electronic device also includes a network interface 1303, an input device 1304, a hard disk 1305, and a display device 1306. The display device 1306 can display the results obtained by the processor 1301 executing instructions.

[0192] Memory 1302 is used to store programs and data necessary for the operation of the operating system, as well as intermediate results during the calculation process of processor 1301. In some embodiments, memory 1302 stores the following elements, executable modules or data structures, or subsets thereof, or extended sets thereof: operating system 13021 and application programs 13022.

[0193] When the processor 1301 calls and executes the application program and data stored in the memory 1302, specifically the program or instructions stored in the application program 13022, it executes the image perspective distortion correction method as described in the above embodiments.

[0194] The methods disclosed in the above embodiments of the present invention can be applied to, or implemented by, processor 1301. Processor 1301 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above methods can be completed by integrated logic circuits in the hardware of processor 1301 or by instructions in software form. The processor 1301 may be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components, and can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application.

[0195] In addition, this application also provides a computer-readable storage medium storing a computer program. When the computer program is run by a processor, the processor performs the image perspective distortion correction method as described in the above embodiments.

[0196] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0197] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can be physically comprised separately, or two or more units can be integrated into one unit. The integrated unit described above can be implemented in hardware or in the form of hardware plus software functional units.

[0198] The integrated units implemented as software functional units described above can be stored in a computer-readable storage medium. These software functional units, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute some steps of the transmission and reception methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0199] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for correcting perspective distortion in images, characterized in that, Includes the following steps: Obtain the original face image; The original face image is meshed to obtain an initial deformed mesh, wherein the mesh in the initial deformed mesh corresponds to the pixel in the original face image, and the mesh density of the face region in the original face image is greater than the mesh density of the non-face region in the original face image. Based on a preset target distance, the original face image is converted into a target distance face image, wherein the target distance face image is an image of the face in the original face image at the target distance, and the target distance is greater than the shooting distance of the original face image; Based on the original face image and the target distance face image, determine the coordinate change vector of at least some pixels; Based on the coordinate change vectors of the at least some pixels, determine the initial target coordinates of the grids corresponding to the at least some pixels in the initial deformed grid; Determine whether there are any missing meshes in the initial deformation mesh whose initial target coordinates have not been determined; If so, then based on the distance between the missing grid and the original coordinates of multiple adjacent pixels, determine the weight value corresponding to each adjacent pixel, and based on the weight value corresponding to each adjacent pixel, perform a weighted summation of the original coordinates of the missing grid and the displacement of each adjacent pixel to determine the initial target coordinates of the missing grid. Based on the initial target coordinates corresponding to each grid, the position of all grids in the initial deformable grid is optimized to obtain the optimized deformable grid. The coordinates of each grid in the optimized deformable mesh are used as the target coordinates of each grid in the initial deformable mesh; The pixels in the original face image are adjusted to the target coordinates of the corresponding grid to generate the target image.

2. The image perspective distortion correction method according to claim 1, characterized in that, The step of converting the original face image into a face image at a target distance based on a preset target distance includes: Obtain the original depth map corresponding to the original face image; Based on the target distance, the original depth map is offset and scaled to generate a target distance depth map; The target distance depth map is reprojected to obtain the target distance face image.

3. The image perspective distortion correction method according to claim 2, characterized in that, The step of offsetting and scaling the original depth map based on the target distance to generate a target distance depth map includes: The difference between the target distance and the shooting distance of the original face image is used as the depth offset value, and the pixels in the original depth map are subjected to the depth offset value. Facial landmarks are detected by performing facial landmark detection on the original depth map to obtain facial landmarks; Determine the depth corresponding to the centroid of all facial keypoints; Based on the depth corresponding to the centroid and the depth offset value, a scaling factor is determined, and the pixels in the original depth map are scaled according to the scaling factor.

4. The image perspective distortion correction method according to claim 2, characterized in that, The step of determining the coordinate change vectors of at least some pixels based on the original face image and the target distance face image includes: The coordinate difference between at least a portion of the pixels in the original face image and the target distance face image is used as the coordinate change vector corresponding to the at least a portion of the pixels.

5. The image perspective distortion correction method according to claim 1, characterized in that, The step of converting the original face image into a face image at a target distance based on a preset target distance includes: The original face image is input into the GAN inverse mapping model, and the target distance face image is output. The GAN inverse mapping model is used to parse the hidden face information in the original face image and construct the equivalent camera parameters at the target distance; based on the hidden face information and the equivalent camera parameters, the original face image is reconstructed to generate the face image at the target distance.

6. The image perspective distortion correction method according to claim 1, characterized in that, The step of converting the original face image into a face image at a target distance based on a preset target distance includes: The original face image is input into an image diffusion model, which outputs the target distance face image, wherein the image diffusion model has the ability to adjust the imaging distance of the image.

7. The image perspective distortion correction method according to claim 5 or 6, characterized in that, The step of determining the coordinate change vectors of at least some pixels based on the original face image and the target distance face image includes: Extract the facial key points from the target distance face image to obtain the target key points; Extract facial key points from the original face image to obtain the original key points; The coordinate difference between the target key point and the corresponding original key point is determined as the coordinate change vector.

8. The image perspective distortion correction method according to claim 1, characterized in that, The method further includes: The missing facial features in the target image are filled in to generate a target image containing a complete face.

9. The image perspective distortion correction method according to claim 8, characterized in that, The step of filling in the missing facial portions in the target image to generate a target image containing a complete face includes: The target image is subjected to face analysis to obtain the edge pixels of the face cheeks; Towards the background area in the target image, near the edge pixels of the human face cheek, a preset number of pixels are cleared to generate a first intermediate image; Image restoration is performed on the first intermediate image to obtain a target image containing a complete human face.

10. The image perspective distortion correction method according to claim 8, characterized in that, The step of filling in the missing facial portions in the target image to generate a target image containing a complete face includes: Obtain a target distance face image containing a complete human face; Stretch the target distance face image containing a complete face so that the face in the image is aligned with the face in the target image to generate a second intermediate image; Determine the non-overlapping region between the second intermediate image and the target image; Image restoration is performed on the non-overlapping regions in the target image to generate a target image containing a complete human face.

11. The image perspective distortion correction method according to claim 10, characterized in that, The step of stretching the target distance face image containing a complete face, aligning the face in the target image with the face in the target image, and generating a second intermediate image includes: The target image is subjected to face analysis, and the pixel point where the edge of the face cheek is located in the target image is found as the first cheek pixel point; Face analysis is performed on the target distance face image containing a complete face, and the pixel point where the edge of the face cheek is located in the target distance face image containing a complete face is found as the second cheek pixel point; Based on the first cheek pixels and the second cheek pixels, the target distance face image containing the complete face is stretched so that the face in it is aligned with the face in the target image, thereby generating the second intermediate image.

12. The image perspective distortion correction method according to claim 11, characterized in that, The step of stretching the target distance face image containing the complete face based on the first cheek pixels and the second cheek pixels includes: Stretch each first cheek pixel to the position of the second cheek pixel that is closest to it.

13. An electronic device, characterized in that, include: processor; and a memory, in which computer program instructions are stored. When the computer program instructions are executed by the processor, the processor performs the image perspective distortion correction method as described in any one of claims 1-12.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions that, when executed by a processor, cause the processor to perform the image perspective distortion correction method as described in any one of claims 1-12.

Images