Image distortion correction method and device, storage medium and program product

Abstract

The application relates to a correction method and device for image distortion, a storage medium and a program product. The method comprises the following steps: acquiring a distorted image; performing image detection on the distorted image to obtain structure object detection information corresponding to the distorted image; the structure object detection information is used for representing the related information of a planar object and a non-planar object in the distorted image; performing corresponding image content structure analysis on the distorted image based on the object geometric attribute in the structure object detection information to obtain structure information; determining a correction strategy corresponding to the distorted image based on the object geometric attribute; and performing corresponding correction processing on the distorted image based on the correction strategy and the structure information to obtain a target corrected image. The method can improve the adaptability between the correction process and the distortion characteristics of the distorted image, so that the restored degree of the corrected image is high.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method, apparatus, computer device, computer-readable storage medium, and computer program product for correcting image distortion. Background Technology

[0002] With the development of image processing technology, due to factors such as camera lens optical distortion, perspective shift during shooting, and environmental interference, the original images acquired often have varying degrees of distortion. To restore the true geometric shape of the image, image correction methods have emerged.

[0003] Traditional methods primarily obtain the geometric deformation parameters of an image by detecting key feature points in the distorted image, and then perform a one-time correction on the distorted image based on these geometric deformation parameters.

[0004] However, traditional methods employ the same fixed and uniform correction process and transformation parameters for different types and degrees of image distortion, failing to adaptively differentiate and process the distortion based on its actual characteristics. This correction strategy results in low compatibility between the correction process and the distorted image, making it difficult to obtain accurate correction results. Consequently, there remains a significant discrepancy between the corrected image and the actual scene being captured. Summary of the Invention

[0005] Therefore, it is necessary to provide a method, apparatus, computer device, computer-readable storage medium, and computer program product for image distortion correction that can improve the adaptability between the correction process and the distortion characteristics of the distorted image, thereby making the corrected image closer to the real shooting scene.

[0006] In a first aspect, this application provides a method for correcting image distortion, comprising:

[0007] Acquire distorted images;

[0008] Image detection is performed on the distorted image to obtain structural object detection information corresponding to the distorted image; the structural object detection information is used to characterize the relevant information of planar objects and non-planar objects in the distorted image;

[0009] Based on the geometric attributes of the objects in the structural object detection information, the distorted image is subjected to corresponding image content structure analysis to obtain structural information;

[0010] A correction strategy corresponding to the distorted image is determined based on the geometric properties of the object.

[0011] Based on the correction strategy and the structural information, the distorted image is subjected to corresponding correction processing to obtain the target corrected image.

[0012] Secondly, this application also provides a correction device for image distortion, comprising:

[0013] The acquisition module is used to acquire distorted images;

[0014] The structure detection module is used to perform image detection on distorted images and obtain structural object detection information corresponding to the distorted images; the structural object detection information is used to characterize the relevant information of planar regions within the distorted images;

[0015] The structural analysis module is used to perform corresponding image content structural analysis on distorted images based on the geometric attributes of objects in the structural object detection information, and to obtain structural information.

[0016] The strategy determination module is used to determine the correction strategy corresponding to the distorted image based on the geometric properties of the object.

[0017] The correction module is used to correct distorted images based on correction strategies and structural information to obtain the target corrected image.

[0018] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps described in the above method embodiments.

[0019] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps described in the above method embodiments.

[0020] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps described in the above method embodiments.

[0021] The aforementioned image distortion correction methods, apparatuses, computer devices, computer-readable storage media, and computer program products obtain structural object detection information corresponding to the distorted image by performing image detection on the distorted image. Since the structural object detection information characterizes the relevant information of planar and non-planar objects within the distorted image, a correction strategy matching the distorted image can be determined based on the structural object detection information. Thus, during the corresponding correction processing of the distorted image based on the correction strategy, the corresponding correction process can be executed according to the structural characteristics within the distorted image. Therefore, compared to applying a uniform correction process to all distorted images, it is possible to implement differentiated correction processes for distorted images. In addition, before correcting the distorted image, corresponding image content structure analysis is performed on the distorted image to obtain structural information corresponding to the distorted image. During correction, the key structures characterized by the structural information can improve the image correction efficiency. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart illustrating a method for correcting image distortion in one embodiment;

[0024] Figure 2 This is a flowchart illustrating a method for correcting image distortion in another embodiment;

[0025] Figure 3 This is a structural block diagram of an image distortion correction method apparatus in one embodiment;

[0026] Figure 4 This is an internal structural diagram of a computer device in one embodiment;

[0027] Figure 5 This is a diagram of the internal structure of a computer device in another embodiment. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0029] In one embodiment, such as Figure 1As shown, a method for correcting image distortion is provided. This embodiment illustrates the application of this method to a terminal. It is understood that this method can also be applied to a server, or to a system including both a terminal and a server, and implemented through interaction between the terminal and the server. The terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, projection devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc. Head-mounted devices can be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. The server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0030] In this embodiment, the method includes the following steps:

[0031] Step S102: Obtain the distorted image.

[0032] Distorted images refer to distorted images caused by optical defects in the imaging system or the influence of perspective, resulting in the geometric shape, proportions, or straight-line characteristics of objects in the image deviating from the original appearance of the real scene, and exhibiting unexpected deformations such as bending, stretching, or compression.

[0033] For example, a scene is captured in real time using an image acquisition device (such as a camera with a wide-angle lens) to obtain a distorted image.

[0034] Alternatively, the recorded image data can be read from a preset storage medium to obtain the distorted image.

[0035] It is understood that if this embodiment is applied to a terminal, distorted images can be acquired based on an image acquisition device. If this embodiment is applied to a server, distorted images can be acquired from data interacted with by the terminal.

[0036] Step S104: Perform image detection on the distorted image to obtain structural object detection information corresponding to the distorted image.

[0037] Among them, structural object detection information is used to characterize the relevant information of planar objects and non-planar objects within the distorted image. Planar objects refer to objects that appear as flat surfaces in real three-dimensional space. Non-planar objects refer to objects other than planar objects, such as objects with curved surfaces, solid shapes, or irregular shapes in real three-dimensional space, such as human faces.

[0038] For example, after acquiring the distorted image, image detection is first performed on the distorted image to obtain structural object detection information corresponding to the distorted image. Structural object detection information can include object semantic attributes and object geometric attributes. Object semantic attributes are used to determine the overall content attributes corresponding to the distorted image, such as whether the distorted image belongs to the document, screen, building, or portrait category. Object geometric attributes are used to determine the geometric morphological features of each local region within the distorted image. Therefore, based on the object geometric attributes, it can be determined whether the object corresponding to each local region within the image belongs to a planar object or a non-planar object. Planar objects include simple objects such as document pages, screens, and posters. Non-planar objects include objects in complex scenes such as faces, buildings, interiors, and goods.

[0039] Furthermore, geometric features can also include object area. Based on object area, key planar objects, i.e., principal planes, can be identified in the distorted image. For example, the object occupying the largest area and exceeding a preset area threshold is considered a key object. Subsequently, correction strategies can be determined based on information about key objects in the distorted image. Therefore, structural object detection information can also include whether a principal plane exists within the distorted image and related information about the principal plane.

[0040] Step S106: Based on the geometric attributes of the objects in the structural object detection information, perform corresponding image content structure analysis on the distorted image to obtain structural information.

[0041] Among them, object geometric properties are a set of parameters used to characterize the physical morphological features of each object in an image. For example, they include flatness or stiffness, and may also include principal plane information, etc.

[0042] Structural information can vary depending on the object's geometric attributes. For example, if the image is determined to contain primarily planar objects based on their geometric attributes, keypoints and keylines of these planar objects can be extracted to obtain structural information. If the image contains non-planar objects or the non-planar objects have large areas, structural information such as depth maps, normal maps, and edge maps corresponding to the distorted image can be extracted. Additionally, structural information can be obtained by extracting planar region masks from the distorted image. This facilitates subsequent correction of planar objects in the distorted image based on these planar region masks. Alternatively, structural information can be independent of the object's geometric attributes. That is, regardless of the object's geometric attributes, structural information such as depth maps, normal maps, edge maps, and planar region masks corresponding to the distorted image can be extracted. In subsequent correction processes, target information matching the geometric attributes of the object being corrected can be further extracted from this structural information for correction.

[0043] For example, the principal plane information in a distorted image can be determined based on the object geometric attributes in the structural object detection information. For instance, if a compliant principal plane object exists in the distorted image, its edge details can be preserved during the preprocessing stage for extracting structural information. If the principal object in the distorted image is determined to be an irregular structural object based on its geometric attributes, or if a compliant principal plane object does not exist in the distorted image, the distorted image can be smoothed during the preprocessing stage for extracting structural information to reduce interference from texture noise in subsequent structural information extraction.

[0044] Step S108: Determine the correction strategy corresponding to the distorted image based on the geometric properties of the object.

[0045] Step S110: Based on the correction strategy and structural information, perform corresponding correction processing on the distorted image to obtain the target corrected image.

[0046] The correction strategy can include a first strategy and a second strategy. The first strategy uses geometric transformations for global correction, while the second strategy, in addition to geometric transformations for global correction, also utilizes a generative model for residual processing. Furthermore, during the geometric transformation for global correction, the corresponding global correction strategy can be determined based on the object's geometric properties. For example, for images with a defined principal plane, homography matrices can be used for global correction; for complex scene images, global deformation mesh estimation can be used for global correction.

[0047] For example, if the correction strategy determined based on the object's geometric properties corresponds to the first strategy, i.e., there is a definite principal plane in the distorted image, then the distorted image can be corrected based on the previously extracted structural information. If the correction strategy determined based on the object's geometric properties corresponds to the second strategy, i.e., there is no definite principal plane in the distorted image, then the distorted image can be globally corrected first based on the structural information to obtain a globally corrected image, and then local residual processing can be performed on the globally corrected image to obtain the final target corrected image.

[0048] In the aforementioned image distortion correction method, image detection is performed on the distorted image to obtain structural object detection information corresponding to the distorted image. Since structural object detection information represents the relevant information of planar and non-planar objects within the distorted image, a correction strategy matching the distorted image can be determined based on the structural object detection information. Thus, during the corresponding correction processing of the distorted image based on the correction strategy, the corresponding correction process can be executed according to the structural characteristics within the distorted image. Therefore, compared to applying a uniform correction process to all distorted images, a differentiated correction process can be implemented for distorted images. In addition, before correcting the distorted image, corresponding image content structure analysis is performed to obtain structural information corresponding to the distorted image. During correction, the key structures represented by the structural information can improve the image correction efficiency.

[0049] First, in an exemplary embodiment, such as Figure 2 As shown, after acquiring the distorted image, it is detected to obtain structural object detection information. Specifically, structural object detection information can include relevant information about objects in the image, such as object semantic attributes and object geometric attributes. Based on the object semantic attributes, the image type corresponding to the distorted image is determined, and based on the object geometric attributes, the principal plane information in the distorted image is determined. For example, whether a principal plane exists. In addition, correction process information can be further determined based on principal plane information or object semantic attributes. For example, if there is no principal plane that meets the preset conditions, or if the image type is determined to be a complex scene type based on object semantic attributes, then local residual repair processing needs to be performed after global correction. For example, if the system determines that there is a clear principal plane in the image, global perspective estimation can be performed as the focus; if it determines that the image contains a face or main subject, content protection needs to be strengthened in subsequent corrections; if it determines that there is local residual deformation in the image, local compensation and generative refinement need to be performed after coarse correction. Therefore, detecting the image first to obtain structural object detection information can clarify the subsequent correction process, rather than blindly stretching the entire image.

[0050] After obtaining the structural object detection information, such as Figure 2As shown, geometric prior extraction can be performed on the distorted image based on the structural object detection information to obtain the corresponding structural information. For example, the structural information may include depth maps, normal maps, edge maps, and planar region masks. Based on this information, foreground and background in the distorted image can be distinguished, and planar regions, key boundaries, and key structural lines in the image can be determined during subsequent correction, thereby ensuring the quality of the corrected image. In some exemplary embodiments, image content structure analysis is performed on the distorted image based on structural object detection information to obtain structural information, including: preprocessing the distorted image based on structural object detection information to obtain a preprocessed image; performing depth information analysis on the preprocessed image to obtain a depth map corresponding to the preprocessed image; obtaining the normal information of each pixel in the preprocessed image based on the depth map to obtain a normal map corresponding to the preprocessed image; identifying target objects in the preprocessed image and extracting the edge lines corresponding to the target objects to obtain an edge map corresponding to the preprocessed image; performing geometric consistency analysis based on the depth map and normal map to obtain planar region information in the distorted image; the planar region information includes the main plane region mask in the distorted image and the confidence level corresponding to the main plane region; and obtaining structural information based on the depth map, normal map, edge map, and planar region information.

[0051] Among these, the preprocessed image refers to the intermediate image obtained after denoising, texture enhancement, or color normalization of the original distorted image, facilitating subsequent feature extraction. The depth map is a grayscale map representing the relative distances of pixels in the distorted image, reflecting the three-dimensional spatial depth of the scene. The normal map is a vector map representing the orientation of the surface of each pixel in the distorted image, reflecting the geometric pose of the scene surface. Planar region information refers to a dataset describing the location and reliability of flat regions in the distorted image, obtained based on geometric consistency analysis. For example, it can be represented using a planar region mask.

[0052] Specifically, firstly, the distorted image is preprocessed based on structural object detection information. Different preprocessing procedures are performed for different types of images (such as complex scenes with high noise or low-contrast documents). For example, if the principal plane information in the structural object detection information indicates the presence of a target principal plane (e.g., a clear principal plane) in the distorted image, then during preprocessing, the edge information of the target principal plane is extracted to preserve more edge details. If the principal plane information in the structural object detection information indicates the absence of a target principal plane in the distorted image, or if the object semantic attributes in the structural object detection information indicate that the distorted image belongs to the human portrait category or other complex scene category, then during preprocessing, the smoothing of the distorted image is enhanced to reduce the interference of texture noise on subsequent edge extraction.

[0053] For example, the preprocessing also includes scale normalization and basic enhancement processing. For instance, the distorted image is scaled to a uniform long-side resolution, preserving the original aspect ratio, and the original distorted image dimensions are recorded simultaneously so that the obtained structural information can be unified to the original image coordinate system later. For example, for distorted images with low brightness, insufficient contrast, or blurred edges, brightness normalization and sharpening can be performed to conduct basic enhancement processing.

[0054] Subsequently, multi-dimensional structural information is extracted from the preprocessed image. First, depth information of each pixel in the preprocessed image is extracted to obtain a depth map. Based on the foreground and background represented by the depth map, the subject region and environment region in the distorted image are determined. This allows for the differentiation of the regions occupied by each object during subsequent correction, avoiding the forced straightening of objects located at different depths as if they were on the same plane. Furthermore, planar regions in the distorted image can also be determined based on the depth map. Optionally, the principal plane information in the structural object detection information can also be verified based on the depth map to determine the principal plane region information.

[0055] Next, after obtaining the depth map, the normal map can be calculated. The normal map can be represented in a three-channel format, with each pixel corresponding to a direction vector. For the main plane region, the normal direction is usually relatively consistent; for edges, corners, and non-planar regions, the normal direction will change significantly. Therefore, the normal map is useful for plane detection and perspective direction estimation.

[0056] For example, a preprocessed image can be converted into a 3D point cloud based on a depth map. The surface normals corresponding to each 3D point are estimated using neighborhood difference estimation, thus obtaining a normal map. For instance, suppose the 3D coordinates of a pixel in the preprocessed image are... Then the normal can be calculated using the deflection vectors in the two local directions, referring to the following formula:

[0057]

[0058] in, This represents the normal vector corresponding to a 3D point.

[0059] Alternatively, in order to obtain the normal map, the gradient can be calculated on the depth map using the Sobel operator, and then the corresponding normal vector can be approximately constructed based on the gradient.

[0060] After obtaining the normal map, the target objects in the preprocessed image can be identified, and their corresponding edge lines can be extracted to obtain an edge map corresponding to the preprocessed image. First, a global nested edge detection algorithm is applied to the input distorted image to obtain the main contours and structural boundaries, outputting a basic edge map. This step outlines the overall contours of the image content, preserving the approximate shape and main boundary information of objects. Next, after obtaining the basic edge map, a line segment detector or Hough line transform algorithm is used to specifically extract long straight line structures in the image, outputting a structural line map. This step can filter out linear features that are crucial for perspective correction, such as building edges, document edges, and screen outlines, while filtering out irregular texture noise. Finally, the basic edge map and structural line map obtained in the above steps are fused to generate the final geometric edge control map, i.e., the edge map. This fusion... Figure 1 On the one hand, it preserves the outline information of the target object, and on the other hand, it highlights the key linear geometric structure, thus providing structural guidance for subsequent perspective estimation that includes both the overall outline and precise geometric constraints.

[0061] Finally, based on the depth map, normal map, and edge map obtained above, the structural information corresponding to the distorted image can be obtained.

[0062] In this embodiment, by introducing a preprocessing mechanism, the problem of feature extraction failure caused by poor image quality (such as blurring or reflection) is effectively avoided, ensuring that subsequent steps still run stably in complex environments. By introducing depth maps and normal maps, it is possible to accurately distinguish between foreground occlusion and background planes, avoiding miscorrection.

[0063] In some embodiments, in addition to the depth map, normal map, and edge map described above, structural information can also be obtained based on planar region information. For example, geometric consistency analysis combined with semantic prior assistance can be used to obtain planar region information corresponding to the preprocessed image, in order to determine the regions in the distorted image that are most planar and most worthy of global perspective correction. Exemplarily, geometric consistency analysis is performed based on the depth map and normal map to obtain planar region information in the distorted image; the planar region information includes the main planar region mask in the distorted image and the confidence level corresponding to the main planar region; structural information is obtained based on the depth map, normal map, edge map, and planar region information.

[0064] Specifically, local region clustering is performed based on depth maps and normal maps to obtain planar region information. That is, planar regions in the preprocessed image with depth variations within a preset depth range, normal directions within a preset direction range, and boundary closure within a preset closure range are identified, thus obtaining planar region information.

[0065] In some embodiments, if it is determined that a principal planar object exists in the distorted image based on structural object detection information, and the type of the principal planar object is determined, such as a document page, screen, front of product packaging, or building facade, then object detection can be performed on the preprocessed image to obtain various bounding boxes. The regions enclosed by each bounding box are then verified based on depth maps and normal maps to determine planar region information. For example, for document pages, screens, front of product packaging, and building facades, object detection or semantic segmentation can be further introduced as auxiliary constraints. For instance, document and screen images can be verified by detecting candidate rectangular regions and then combining depth and normal maps; building facades can be verified by facade detection or salient rectangular region detection and then combined with geometric consistency filtering; product images can first locate the main body of the product or the front of the packaging, and then combine normal consistency to filter the most likely principal planar region.

[0066] In addition to the planar region mask mentioned above, the planar region information can also include the planar confidence level corresponding to the planar region. Thus, during subsequent correction, if the planar confidence level corresponding to the planar region is greater than the first preset confidence level, a global perspective estimation using the homography matrix can be used for correction; if the planar confidence level is greater than the second preset confidence level but less than or equal to the first preset confidence level, a global grid estimation can be used for global correction; if the planar confidence level is less than or equal to the second preset confidence level, no correction is performed to avoid bias in the master plane judgment leading to overstretching or miscorrection.

[0067] In some embodiments, after obtaining the above structural information, the various pieces of information can be processed uniformly. For example, the spatial dimensions, coordinate system, and numerical range can be unified to facilitate subsequent correction using a unified, standardized geometric control map. Specifically, the depth map can be normalized to the range of 0 to 1. The normal map can maintain a three-channel directional representation. The edge map can be output as a single-channel binary map or a probability map, and the structure line map can also be used as an additional layer. The planar region mask can be output as a single-channel soft mask.

[0068] Optionally, for ease of subsequent use and improved correction efficiency, a multi-channel geometric control tensor can be generated based on a combination of depth map, normal map, edge map, and planar region mask map. This facilitates subsequent input into the generated view model for processing. An example of a multi-channel geometric control tensor is... .in, Represents a depth map. They represent the normals respectively. Figure 3 One directional channel, Represents the edge map. This represents a planar region mask.

[0069] It is understood that the information about planar objects and non-planar objects in the structured object detection information may be relatively coarse. Based on this embodiment, the information about planar objects and non-planar objects can be refined.

[0070] In this embodiment, the principal plane region information obtained through geometric consistency analysis can be used to determine which plane to use as a reference for straightening, thereby avoiding image distortion or incorrect correction direction due to lack of a correction target.

[0071] In addition to obtaining structural information, semantic key regions can be extracted from distorted images to obtain semantic key region information. For example, semantic key region information can be obtained by performing target detection on the distorted image to obtain the semantic information corresponding to each target object in the distorted image. Then, based on this semantic information, target semantic information with high semantic value in the distorted image is determined, and semantic key region information is determined based on the image region occupied by the target object corresponding to this target semantic information. Semantic key regions include, for example, human body regions and subject regions. Correspondingly, semantic key region information includes, for example, human body region masks and subject region masks. This semantic key region information can be used in subsequent correction processes to constrain the global correction process and the local residual repair process, so as to avoid significant deformation of the semantic key regions.

[0072] After obtaining the aforementioned structural information, structural object detection information, and semantic key region information, a correction process can be executed. In an exemplary embodiment, the correction strategy includes a first strategy and a second strategy. Based on the correction strategy, structural object detection information, and structural information, corresponding correction processing is performed on the distorted image to obtain a target corrected image. This includes: if the correction strategy corresponds to the first strategy, then the target plane in the distorted image is determined based on the structural object detection information, and the target plane is globally corrected based on the structural information to obtain the target corrected image; if the correction strategy corresponds to the second strategy, then the distorted image is globally corrected based on the structural information to obtain a global corrected image; and local residual repair is performed on the global corrected image based on the structural information to obtain the target corrected image.

[0073] Specifically, if the structural object detection information determines that a target principal plane exists in the distorted image, such as a document page, screen area, poster area, product packaging front, or building facade, key points (e.g., boundary points, corner points) or key lines (e.g., key structural lines) can be extracted based on the structural information. Geometric transformation information (e.g., homography matrix) can then be obtained based on these key points or key lines. Finally, this geometric transformation information is used to correct the distorted image, resulting in the corrected target image.

[0074] If the structural object detection information determines that there is no target principal plane in the distorted image (e.g., the boundary of the principal plane is incomplete) or there are multi-plane structures, local non-rigid deformations, wide-angle edge stretching, commodity side compression, etc., then the distorted image can be coarsely corrected based on the structural information first, and then constrained repair of local residual problems based on the coarse correction results.

[0075] In practical applications, when a target principal plane exists in a distorted image, after global correction based on structural information, the globally corrected image can be input into a generative model for residual repair, making the final target corrected image closer to the actual situation.

[0076] In this embodiment, by applying different correction strategies to images with different structural characteristics, a first strategy can be used for simple planar scenes to reduce complex computational overhead and achieve fast and accurate rigid correction; a second strategy can be used for complex scenes to reduce severe distortion caused by forced stretching. Furthermore, in the second strategy, through cascaded processing of global correction and local repair, the correctness of the overall perspective relationship of the image can be ensured (such as straightening building lines), while local residual repair can compensate for details that geometric transformations cannot handle (such as filling black edges and repairing minor facial deformations), thereby greatly improving the visual realism of the final output image.

[0077] In some embodiments, the structural information includes planar region information. Global correction of the distorted image based on the structural information includes: if the structural object detection information corresponds to a target plane, then obtaining target geometric points based on the planar region information, obtaining a target homography matrix based on the target geometric points, and performing global correction of the distorted image based on the target homography matrix; if the structural object detection information does not correspond to a target plane, then obtaining a global deformation mesh estimate based on the structural information, and performing global correction of the distorted image based on the global deformation mesh estimate.

[0078] In this context, target geometric points refer to key feature points extracted from planar region information used to calculate the transformation matrix, such as corner points and boundary points of the plane. The target homography matrix is ​​a transformation matrix used to describe the one-to-one correspondence between two planes, mapping an inclined target plane to a normal viewing angle. Global deformation mesh estimation refers to a data structure that divides an image into several mesh cells and describes the overall deformation field of the image by optimizing the positions of the mesh vertices; it can express more complex nonlinear distortions than the homography matrix.

[0079] Specifically, the obtained structural information is first aligned and normalized to ensure that the depth map, normal map, edge map, and planar region mask maintain the same resolution as the original distorted image. Then, candidate principal plane regions are extracted based on the planar region mask, and depth continuity information and normal consistency information corresponding to these candidate principal plane regions are obtained based on the depth map and normal map. This allows for the determination of whether the candidate principal plane region can undergo global perspective correction. If, based on the depth continuity information and normal consistency information, the depth variation is within a preset depth range and the normal variation is within a preset normal range, or the normals are essentially consistent, then the candidate principal plane region can be determined to be suitable for global perspective correction. For the edge map within the planar region, only structural edges related to the principal plane object are retained, while texture edges, shadow edges, and background interference edges are filtered out.

[0080] When an image is determined to contain a clearly defined principal plane, such as a document page, screen area, poster area, product packaging front, or building facade, a homography matrix can be used for coarse correction. Specifically, boundary points, corner points, or main structural lines corresponding to the principal plane area can be obtained based on structural information, and the mapping relationship from the original image plane to the target frontal plane can be estimated based on these key geometric points. This mapping relationship can be expressed as:

[0081]

[0082] in, Represents the pixel coordinates in the original image. Indicates the target coordinates in the globally corrected image. It is a homography matrix. is the scale factor. After obtaining the homography matrix, a globally corrected image can be generated through inverse mapping and interpolation resampling.

[0083] When a distorted image is determined to lack a complete principal plane, or to exhibit multi-planar structures, local non-rigid deformations, wide-angle edge stretching, or product side compression, a global warp grid is used for coarse correction. This involves constructing a regular control grid on the image and estimating the displacement of each grid control point based on edge direction, depth variations, normal direction, and planar region distribution. Compared to homography matrices, the global warp grid is more flexible, capable of representing overall perspective changes and mild non-linear deformations, while maintaining global continuity and smoothness, avoiding local breaks or unnatural distortions.

[0084] During Warp Grid estimation, several types of constraints can be used to control the deformation results. Structural line constraints are used to restore the main edges, building lines, document boundaries, or product outlines within the master plane to a reasonable orientation; planar consistency constraints are used to maintain deformation continuity within the same master plane; depth and normal constraints are used to prevent foreground subjects or non-target areas from being erroneously stretched; and smoothness constraints are used to limit abrupt displacements between adjacent grid points, preventing local folding and structural breaks.

[0085] In other words, this embodiment selects different correction strategies based on different structural object detection information. For planar targets such as documents, screens, and posters, homography matrix is ​​used to complete fast coarse correction; for complex scenes such as buildings, interiors, products, and portraits, global warp grid is used for more flexible overall correction; for images with weak distortion or low confidence in the principal plane, only weak correction can be performed or global correction can be skipped to avoid overstretching the main subject area.

[0086] In some embodiments, after global correction of the distorted image, the effective region and residual region in the globally corrected image can be identified to obtain the effective region mask and residual region hint information, which can be used to mark the black edges, holes, stretched regions or local structural discontinuities generated after coarse correction, and serve as input conditions for subsequent generative models.

[0087] In this embodiment, the homography matrix or deformation mesh is intelligently selected as the correction tool based on whether the image contains a principal plane, achieving dynamic optimization of the correction model. Furthermore, for simple planar scenes, using a homography matrix significantly reduces computational complexity, achieving millisecond-level correction; for complex scenes, mesh estimation ensures the accuracy of correction for local details. This mechanism allows the same system process to seamlessly adapt to both regular planar and complex scene images, automatically choosing the optimal algorithm without manual intervention, thus significantly improving the robustness and applicability of the solution.

[0088] In some embodiments, when processing severe distortion or large-angle correction, the edges of the geometrically transformed image often exhibit problems such as missing black borders, unnatural texture stretching, and local structural breaks. Simple geometric interpolation cannot repair these content-level defects; therefore, generative models can be used for residual inpainting. For example, performing local residual inpainting on a globally corrected image based on structural information to obtain a target corrected image includes: acquiring a target generative model; acquiring local residual image information corresponding to the globally corrected image; inputting the distorted image, the globally corrected image, the structural information, and the local residual image information into the generative model; encoding the globally corrected image through the generative model to obtain latent features; processing the latent features based on the local residual image information, the distorted image, and the structural information to obtain generated features; and decoding the generated features to obtain the target corrected image.

[0089] Among them, the target generative model is based on a neural network model (such as a diffusion model or generative adversarial network) trained by deep learning and capable of image content understanding and generation, used for semantic-level inpainting and redrawing of images. Local residual image information refers to data used to characterize defective regions (such as black borders and holes) and the locations to be repaired in the globally corrected image, usually presented in the form of masks or cue maps. Latent features refer to the high-dimensional compressed representation of the image in the model's latent space, containing semantic and structural priors of the image, and serve as the intermediate carrier for the model's generative processing. Generative features refer to the latent space feature representation after model processing and fusion of repaired content, and are the direct basis for decoding and generating the final image.

[0090] Specifically, the various inputs are first preprocessed and aligned. The original distorted image, the globally corrected image, the depth map, the normal map, the edge map, the planar region mask, and the residual cue map are unified to the same resolution, and a local refinement region mask is determined in the globally corrected image based on the above information. This local refinement region mask mainly covers the black edges, holes, texture stretching, structural breaks, local blurring, and slight subject edge deformation areas generated after coarse correction. Other regions, except for the local refinement region mask, retain the global correction result to avoid unnecessary redrawing of the entire image by the generative model.

[0091] Subsequently, the above information is input into the target generative model, where the globally corrected image serves as the primary editing base map. The original distorted image serves as the content reference map, used to preserve the original texture, subject appearance, and local details. The structural information map tells the model which areas should maintain structural consistency, which edges should be as continuous as possible, and which planar directions should not be disrupted. Local residual deformation prompts clearly indicate the areas that need compensation and their approximate repair directions. Text instruction prompts provide global editing goals, such as "Keep the subject unchanged, repair edge holes and local stretching after perspective correction, so that the structure is natural, lines are continuous, and textures are complete."

[0092] Within the model, the globally corrected image is first encoded and mapped to the latent space to obtain latent features. Subsequently, the model uses the texture information of the original distorted image as a reference in the latent space, and under the geometric constraints of structural information, performs targeted feature completion and reconstruction for the regions indicated by the local residual image information to obtain the repaired generated features.

[0093] Finally, the decoder maps the generated features back from the latent space to the pixel space, outputting the final target-corrected image. This image retains the upright geometric structure of the global correction and fills in the content gaps through generative inpainting, presenting a complete visual effect.

[0094] In this embodiment, by leveraging the semantic understanding capabilities of generative models, black borders and holes that geometric transformations cannot handle can be intelligently filled, and unnatural texture stretching can be repaired, making image details more realistic and natural. Furthermore, by using structural information as a conditional input, the generative model is guided to strictly adhere to the original geometric skeleton of the image (such as straight lines and plane orientation) during the repair process, avoiding the common "illusions" or random fabrications found in generative models, ensuring the straightness of lines and the stability of the structure. Introducing the original distorted image as a condition ensures that the content of the repaired area (such as skin tone and texture details) remains highly consistent with the original image, preventing color banding or stylistic differences between the repaired area and its surroundings, greatly improving the visual quality of the final image.

[0095] In addition to the above inputs, semantic key region information can also be input into the generative model to protect the semantic key regions from significant deformation. For example, correcting a distorted image based on a correction strategy and structural information to obtain a target corrected image includes: performing target detection on the distorted image to obtain semantic key region information in the distorted image; correcting the distorted image based on the correction strategy, structural information, and semantic key region information to obtain the target corrected image; and ensuring that the difference between the image region information corresponding to the semantic key region information within the target corrected image and the semantic key region information meets the target condition.

[0096] Among them, semantic key region information refers to the location mask, bounding box, or key point coordinate information of regions with high semantic value (such as faces, product subjects, core objects) in the image, extracted through object detection or segmentation algorithms. Target conditions refer to the evaluation criteria used to constrain the consistency between the corrected image content and the original content, which can be specifically expressed as the degree of regional deformation being less than a preset threshold, the key point displacement distance being within the allowable range, or the structural similarity being greater than a preset value.

[0097] Specifically, the distorted image, global correction image, depth map, normal map, edge map, planar region mask map, semantic key region mask map, and residual cue map are unified to the same resolution.

[0098] Subsequently, the distorted image with uniform resolution, the globally corrected image, the depth map, the normal map, the edge map, the planar region mask map, the semantic key region mask map, and the residual cue map are input into the generative editing model. The semantic key region mask map is used to protect content such as faces, subjects, key product regions, text, or logos. Finally, the target corrected image is obtained based on the output of the generative editing model.

[0099] In this embodiment, by detecting semantically sensitive regions, the deformation of areas such as faces and subjects can be limited during the global perspective correction stage to prevent key content from being widened, elongated, or proportionally abnormal due to overall straightening. On the other hand, during the residual repair stage using a generative model, the model strives to maintain the identity of faces, the outline of subjects, the texture of goods, and key structures as much as possible when repairing edge holes, local stretching, and residual distortions, thereby providing semantic protection boundaries for subsequent correction processes.

[0100] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0101] In one exemplary embodiment, such as Figure 2 As shown, the input to the entire process is an original image with perspective distortion, such as a portrait taken at an angle, a tilted building, a distorted product image, or a misaligned document or screen image. The final output is a corrected image, which should be visually more upright, with more natural proportions, more stable local structures, while preserving as much of the original image's face, subject, texture, and key content as possible.

[0102] Module 1 is the distortion diagnosis and correction mode selection module. The input to this module is the original image (i.e., the distorted image), and the output is structured diagnostic information (corresponding to structural object detection information). This module determines the type of the image, such as a portrait, product image, document image, architectural image, or ordinary scene image; it also determines what distortions exist in the image, such as obvious perspective tilt, wide-angle stretching, subject deformation, or local residual distortion. In addition, this module also determines whether there is a master plane in the image that can be globally corrected, such as a document page, screen plane, product packaging front, or building facade, and further determines whether global perspective correction and subsequent local residual repair are needed. By first determining the problem type and then deciding on the subsequent processing method, the same fixed correction strategy is avoided for all images.

[0103] The structured diagnostic information output by Module 1 will serve as control information for subsequent modules. For example, if the system determines that a clear principal plane exists in the image, it can focus on global perspective estimation; if it determines that the image contains a face or a main subject, it needs to strengthen content protection in subsequent corrections; if it determines that the image has local residual deformation, it needs to continue with local compensation and generative refinement after coarse correction. The advantage of this is that it gives subsequent algorithms a clear objective, rather than blindly stretching the entire image.

[0104] Module two is the geometric prior extraction module. The input to this module is also the original image, and the output is a multi-source geometric prior, including a depth map, normal map, edge map, and planar region mask map. The depth map describes the distance relationships between different regions in the image, helping the model understand which regions are further forward and which are further back. The normal map expresses the orientation of local surfaces in the image, such as whether a plane faces left, right, or directly towards the camera. The edge map highlights contours, boundaries, and structural lines in the image, such as building edges, document boundaries, and product outlines. The planar region mask map marks the main planar regions in the image that may need to be straightened.

[0105] Module 2's function is to explicitly extract the implicit spatial structure from the original image. Traditional methods often rely solely on edge detection or the four corner points, which tends to fail with complex backgrounds, occlusions, blurriness, reflections, or unclear boundaries. This embodiment combines depth, normals, edges, and planar regions to form more complete geometric control information. Module 2 can be understood as further specifying "where a plane resembles in the original image, where this plane faces, which structural lines may need to be straightened, and which regions are spatially more forward or backward." This information is then fed into the subsequent global perspective estimation module and generative retouching module to guide the correction direction.

[0106] Module three is the semantic prior extraction module. The input to this module is the original image, and the output is a semantically sensitive region map, mainly including a face mask map and a subject saliency map. The face mask map is used to locate face regions in the image, and the subject saliency map is used to locate the most important subject regions in the image, such as people, goods, main objects, or the core objects of the image. The role of this module is not to directly straighten the image, but rather to tell the subsequent correction module which content must not be corrupted.

[0107] Module 3 ensures that important elements in the image remain undistorted. For example, in portrait images, edge stretching can be appropriately corrected, but the face should not be elongated, widened, or have its proportions altered; in product images, packaging perspective can be corrected, but the main product outline and key textures should not be changed; in general scene images, background structure can be straightened, but the main subject should not be unnaturally distorted. Therefore, the semantic prior extraction module essentially adds a "protected area" to the subsequent correction process, letting the algorithm know which areas require careful handling and which areas can undergo stronger geometric adjustments.

[0108] Module 4 is the global perspective estimation and coarse correction module. Its input primarily comes from the geometric priors output by Module 2, and can also be guided by diagnostic information from Module 1. Its output is the coarse correction result. Specifically, Module 4 estimates the global perspective transformation relationships in the image based on the depth map, normal map, edge map, and planar region mask map, including homography matrix and global deformation mesh estimation.

[0109] For example, if a document photo is skewed overall, Module 4 is responsible for roughly restoring the page to a rectangular shape; if a building photo is noticeably tilted, Module 4 is responsible for restoring the main structural orientation to a reasonable state; if a product image has overall perspective compression, Module 4 is responsible for first restoring the approximate proportions of the main product. This stage does not aim to perfect every detail, but rather to first resolve the most significant and obvious global perspective distortions, providing a stable foundation for subsequent fine-tuning.

[0110] It's understandable that the key aspect of the interface between Module 2 and Module 4 is "supporting global perspective estimation." If the planar region mask map output by Module 2 has high confidence and clear structural boundaries exist in the edge map, then Module 4 can prioritize extracting candidate corner points or fitting target rectangles from these regions before estimating the homography matrix. If the image is not strictly uniplane, or the planar region mask only covers part of the main region, then Module 4 can combine the depth map and normal map to estimate a more flexible global deformation mesh. In this case, the depth map is used to distinguish between front and back layers, the normal map is used to determine the main direction, the edge map is used to stabilize structural lines, and the planar region mask is used to limit the main area of ​​action of the global transformation.

[0111] The key interface between Modules 2 and 5 lies in "supporting local residual compensation and generative refinement." For generative models, the geometric priors output by Module 2 are not intended for direct calculation of the perspective matrix, but rather serve as conditional control information to constrain the model for local refining. For example, edge maps constrain which contours cannot be destroyed during generation, normal maps and planar region masks constrain which regions should maintain planar consistency during generation, and depth maps ensure the model pays attention to the hierarchical relationship between different regions during generation, preventing the foreground and background from being mixed together. Therefore, in Module 5, these geometric control maps can be used as inputs to independent encoding branches or injected into the generative model by encoding them together with the coarse-corrected image to guide fine-grained refining.

[0112] Module 5 is the Local Residual Compensation and Generative Refinement module. Its inputs include the original image, the coarsely corrected image output from Module 4, the geometric control map output from Module 2, the semantically sensitive region map output from Module 3, local residual deformation cues, and text instruction prompts. Its output is the final corrected image.

[0113] After global perspective coarse correction, unnatural areas may still exist in the image, such as black borders, holes, stretched textures, local blurring, broken lines, and slight deformation of the edges of people or subjects. Module 5 combines the original image and the coarsely corrected image, using geometric control maps to identify areas that need to maintain structural consistency, semantically sensitive region maps to protect faces and subject areas, and local residual deformation cues and text instructions to guide the generative model to perform natural restoration of local areas. Simply put, Module 5 is responsible for making the "roughly corrected image" more natural, more complete, and more usable.

[0114] In other words, the original image enters the main correction process through modules one, two, and four. Module three provides semantic protection information, while the structured diagnostic information output by module one influences the generative refinement in module five through control constraints. This design enables the system to process images not merely linearly, but to dynamically adjust subsequent processing strategies based on the preceding judgment results.

[0115] In this embodiment, the geometric structure is inferred from the image's own depth map, normal map, edge map, and planar region mask map. This is then combined with global deformation mesh estimation or homography matrix for coarse correction. This approach can work under weak or even no calibration conditions, making it more applicable than pure calibration methods. Furthermore, by introducing global deformation mesh estimation and local residual deformation cues, local residual deformation can be processed after the main perspective correction is completed, making it more suitable for complex real-world scenes.

[0116] Based on the same inventive concept, this application also provides an image distortion correction device for implementing the image distortion correction method described above. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations of one or more image distortion correction device embodiments provided below can be found in the limitations of the image distortion correction method described above, and will not be repeated here.

[0117] In one exemplary embodiment, such as Figure 3 As shown, a correction device 300 for image distortion is provided, including: an acquisition module 301, a structure detection module 302, a structure analysis module 303, a strategy determination module 304, and a correction module 305, wherein:

[0118] Acquisition module 301 is used to acquire distorted images;

[0119] The structure detection module 302 is used to perform image detection on the distorted image to obtain structural object detection information corresponding to the distorted image; the structural object detection information is used to characterize the relevant information of planar regions within the distorted image.

[0120] The structural analysis module 303 is used to perform corresponding image content structural analysis on the distorted image based on the geometric attributes of the object in the structural object detection information to obtain structural information;

[0121] The strategy determination module 304 is used to determine the correction strategy corresponding to the distorted image based on the geometric properties of the object.

[0122] The correction module 305 is used to correct the distorted image based on the correction strategy and structural information to obtain the target corrected image.

[0123] In some embodiments, the correction strategy includes a first strategy and a second strategy. When the distorted image is corrected based on the correction strategy, structural object detection information, and structural information to obtain a target corrected image, the correction module 305 is specifically used to: if the correction strategy corresponds to the first strategy, determine the target plane in the distorted image based on the structural object detection information, and perform global correction on the target plane based on the structural information to obtain the target corrected image; if the correction strategy corresponds to the second strategy, perform global correction on the distorted image based on the structural information to obtain a global corrected image; and perform local residual repair on the global corrected image based on the structural information to obtain the target corrected image.

[0124] In some embodiments, the structural information includes planar region information. When performing global correction on the distorted image based on the structural information, the correction module 305 is specifically used to: if the structural object detection information corresponds to the existence of a target plane, then obtain the target geometric points corresponding to the target plane based on the planar region information, obtain the target homography matrix based on the target geometric points, and perform global correction on the target plane based on the target homography matrix; if the structural object detection information does not correspond to the target plane, then obtain a global deformation mesh estimate based on the structural information, and perform global correction on the distorted image based on the global deformation mesh estimate.

[0125] In some embodiments, when a target corrected image is obtained by performing local residual repair on a global corrected image based on structural information, the correction module 305 is specifically used to: obtain a target generative model; obtain local residual image information corresponding to the global corrected image; input the distorted image, the global corrected image, the structural information, and the local residual image information into the generative model; encode the global corrected image through the generative model to obtain latent features; process the latent features based on the local residual image information, the distorted image, and the structural information to obtain generated features; and decode the generated features to obtain the target corrected image.

[0126] In some embodiments, when a distorted image is corrected based on a correction strategy and structural information to obtain a target corrected image, the correction module 305 is specifically used to: perform target detection on the distorted image to obtain semantic key region information in the distorted image; correct the distorted image based on the correction strategy, structural information, and semantic key region information to obtain a target corrected image; and ensure that the difference between the image region information corresponding to the semantic key region information and the semantic key region information in the target corrected image meets the target conditions.

[0127] In some embodiments, when performing image content structure analysis on a distorted image based on structural object detection information to obtain structural information, the structure analysis module 303 is specifically used for: preprocessing the distorted image based on structural object detection information to obtain a preprocessed image; performing depth information analysis on the preprocessed image to obtain a depth map corresponding to the preprocessed image; obtaining the normal information of each pixel in the preprocessed image based on the depth map to obtain a normal map corresponding to the preprocessed image; performing key edge line recognition on the preprocessed image to obtain an edge map corresponding to the preprocessed image; and obtaining structural information based on the depth map, normal map, and edge map.

[0128] In some embodiments, when performing image content structure analysis on a distorted image based on structural object detection information to obtain structural information, the structure analysis module 303 is specifically used to: perform geometric consistency analysis based on depth map and normal map to obtain planar region information in the distorted image; the planar region information includes the main plane region mask in the distorted image and the confidence level corresponding to the main plane region; and obtain structural information based on depth map, normal map, edge map, and planar region information.

[0129] The modules in the aforementioned image distortion correction device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0130] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and databases. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a method for correcting image distortion.

[0131] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 5As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When executed by the processor, the computer program implements a method for correcting image distortion. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0132] Those skilled in the art will understand that Figure 4 or Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0133] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0134] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0135] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0136] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0137] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0138] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0139] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for correcting image distortion, characterized in that, The method includes: Acquire distorted images; Image detection is performed on the distorted image to obtain structural object detection information corresponding to the distorted image; the structural object detection information is used to characterize the relevant information of planar objects and non-planar objects in the distorted image; Based on the geometric attributes of the objects in the structural object detection information, the distorted image is subjected to corresponding image content structure analysis to obtain structural information; A correction strategy corresponding to the distorted image is determined based on the geometric properties of the object. Based on the correction strategy and the structural information, the distorted image is subjected to corresponding correction processing to obtain the target corrected image.

2. The method according to claim 1, characterized in that, The correction strategy includes a first strategy and a second strategy. The step of performing corresponding correction processing on the distorted image based on the correction strategy, the structural object detection information, and the structural information to obtain a target corrected image includes: If the correction strategy corresponds to the first strategy, then the target plane in the distorted image is determined based on the structural object detection information, and the target plane is globally corrected based on the structural information to obtain the target corrected image; If the correction strategy corresponds to the second strategy, then the distorted image is globally corrected based on the structural information to obtain a globally corrected image; and the globally corrected image is locally residual repaired based on the structural information to obtain a target corrected image.

3. The method according to claim 2, characterized in that, The structural information includes planar region information, and the global correction of the distorted image based on the structural information includes: If the detected structural object information corresponds to the existence of a target plane, then the target geometric point corresponding to the target plane is obtained based on the plane region information, the target homography matrix is ​​obtained based on the target geometric point, and the target plane is globally corrected based on the target homography matrix; If the structure object detection information does not correspond to the target plane, then a global deformation mesh estimate is obtained based on the structure information, and the distorted image is globally corrected based on the global deformation mesh estimate.

4. The method according to claim 2, characterized in that, The step of performing local residual repair on the global corrected image based on the structural information to obtain the target corrected image includes: Obtain the target generative model; Obtain the local residual image information corresponding to the global correction image; The distorted image, the global corrected image, the structural information, and the local residual image information are input into the generative model. The global corrected image is encoded by the generative model to obtain latent features. The latent features are processed based on the local residual image information, the distorted image, and the structural information to obtain generated features. The generated features are then decoded to obtain the target corrected image.

5. The method according to claim 2, characterized in that, The step of correcting the distorted image based on the correction strategy and the structural information to obtain the target corrected image includes: Target detection is performed on the distorted image to obtain semantic key region information in the distorted image; The distorted image is corrected based on the correction strategy, the structural information, and the semantic key region information to obtain a target corrected image; the difference between the image region information corresponding to the semantic key region information and the semantic key region information in the target corrected image meets the target condition.

6. The method according to claim 1, characterized in that, The step of performing image content structure analysis on the distorted image based on the structured object detection information to obtain structural information includes: The distorted image is preprocessed based on the structural object detection information to obtain a preprocessed image; The preprocessed image is subjected to depth information analysis to obtain a depth map corresponding to the preprocessed image; Based on the depth map, the normal information of each pixel in the preprocessed image is obtained to obtain the normal map corresponding to the preprocessed image; Key edge lines are identified in the preprocessed image to obtain an edge map corresponding to the preprocessed image; The structural information is obtained based on the depth map, the normal map, and the edge map.

7. The method according to claim 6, characterized in that, The step of performing image content structure analysis on the distorted image based on the structured object detection information to obtain structural information includes: Geometric consistency analysis is performed based on the depth map and the normal map to obtain planar region information in the distorted image; the planar region information includes the main plane region mask in the distorted image and the confidence level corresponding to the main plane region; The structural information is obtained based on the depth map, the normal map, the edge map, and the planar region information.

8. A device for correcting image distortion, characterized in that, The device includes: The acquisition module is used to acquire distorted images; A structure detection module is used to perform image detection on the distorted image to obtain structural object detection information corresponding to the distorted image; the structural object detection information is used to characterize relevant information of planar regions within the distorted image; The structural analysis module is used to perform corresponding image content structural analysis on the distorted image based on the object geometric attributes in the structural object detection information to obtain structural information; The strategy determination module is used to determine a correction strategy corresponding to the distorted image based on the geometric properties of the object. The correction module is used to correct the distorted image based on the correction strategy and the structural information to obtain the target corrected image.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images