A method and system for camera image perspective correction and distortion rectification

Abstract

This invention discloses a camera image perspective correction and distortion correction method and system, relating to the field of computer vision technology. It includes simultaneously capturing images of the same target object using cameras from different viewpoints, capturing initial images of the target object, correcting distortion in the initial images, outputting a distortion-corrected image, analyzing the distortion-corrected image to calculate the three-dimensional geometric information of the target object plane in the world's three-dimensional space, and reprojecting the distortion-corrected image onto the target object plane to generate an orthogonal view. This invention pre-stores lens parameters before leaving the factory and performs parameter-based distortion correction in the early stages of image processing. It utilizes simultaneous shooting with cameras from different viewpoints, combines epipolar geometry and triangulation to accurately calculate the three-dimensional geometric information of the target plane, constructs a virtual orthogonal camera and precisely sets its pose, and uses a homography matrix containing relative pose and plane normal vectors for reprojection, thus meeting the stringent requirements of high-precision application scenarios.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method and system for camera image perspective correction and distortion correction. Background Technology

[0002] In recent years, in the fields of digital image processing and computer vision, perspective correction and distortion correction of camera images have been key technologies for improving image quality. Because ordinary cameras are easily affected by the optical characteristics of the lens and the shooting angle during the shooting process, the images often have varying degrees of distortion and perspective deformation. Especially in application scenarios that require precise geometric information, such as document scanning, object measurement, and industrial inspection, this distortion will seriously affect the accuracy of subsequent analysis and processing.

[0003] Currently, Chinese invention patent application number CN202411539767.1 discloses a confocal microscope camera calibration method based on high-precision motion control, including: establishing four coordinate systems; placing a checkerboard calibration plate on a high-precision platform and adjusting it to the center of the camera's field of view; and capturing and recording the image coordinates of the calibration points. Based on the imaging relationship between the camera's optical center and the object point, the mapping relationship between each coordinate system is determined, thereby establishing a preliminary imaging model. Subsequently, nonlinear correction is performed on the imaging model to eliminate lens distortion, and the corrected image coordinates are converted into pixel coordinates. The nonlinear correction parameters are optimized using multi-view data, and the mapping relationship between the image plane and the platform's physical plane is further calculated. This mapping relationship is then optimized using the least squares method to improve calibration accuracy. Finally, the pixel coordinates in the image are converted into the physical coordinates of the platform through the optimized matrix, which supports the autofocus process of the confocal microscope. In the existing technology, the single-camera method cannot fundamentally eliminate the inherent distortion of the lens, and the perspective correction accuracy is limited by the accuracy of feature point extraction, which is prone to image stretching or information loss. Multi-camera systems often ignore the reliability of synchronous shooting and calibration parameters, resulting in the accumulation of errors in the calculation of three-dimensional geometric information. At the same time, the reprojection process does not fully consider the pose optimization of the virtual orthogonal camera, resulting in geometric distortion or uneven resolution in the generated orthogonal view, which is difficult to meet the requirements of high-precision application scenarios. Summary of the Invention

[0004] The technical problem solved by this invention is that single-camera methods cannot fundamentally eliminate the inherent distortion of the lens, and the accuracy of perspective correction is limited by the accuracy of feature point extraction, which can easily lead to image stretching or information loss. Multi-camera systems often ignore the reliability of synchronous shooting and calibration parameters, resulting in the accumulation of errors in the calculation of three-dimensional geometric information. At the same time, the reprojection process does not fully consider the pose optimization of the virtual orthogonal camera, resulting in geometric distortion or uneven resolution in the generated orthogonal view, which is difficult to meet the requirements of high-precision application scenarios.

[0005] To solve the above-mentioned technical problems, the present invention provides the following technical solution: a method for camera image perspective correction and distortion correction, comprising the following steps:

[0006] Step S1: Simultaneously capture the same target object using cameras from different viewpoints to obtain the initial image of the target object;

[0007] Step S2: Perform distortion correction on the initial image and output the distortion-corrected image;

[0008] Step S3: Calculate the three-dimensional geometric information of the target object plane in the three-dimensional space of the world by analyzing the distortion-corrected image;

[0009] Step S4: Based on the three-dimensional geometric information, the target object plane is reprojected onto the virtual orthogonal imaging plane to generate an orthogonal view.

[0010] As a preferred embodiment of the camera image perspective correction and distortion correction method of the present invention, step S1 includes the following sub-steps:

[0011] Step S101: Receive the shutter command triggered by the user through the user interface;

[0012] Step S102: In response to the shutter command, the mobile terminal's processor simultaneously sends image capture trigger signals to cameras at different viewpoints.

[0013] The cameras with different viewpoints include a main camera and a secondary camera;

[0014] Step S103: Using cameras at different viewpoints, the same target object is exposed and photographed within a preset synchronization error range to capture the initial image of the target object.

[0015] As a preferred embodiment of the camera image perspective correction and distortion correction method of the present invention, step S1 further includes:

[0016] When capturing an initial image of a target object, the target object is placed on a planar background that includes identifiable features, such as edge contour features, artificial corner features, and natural texture features.

[0017] As a preferred embodiment of the camera image perspective correction and distortion correction method of the present invention, the planar background with identifiable features includes a solid color background.

[0018] As a preferred embodiment of the camera image perspective correction and distortion correction method of the present invention, step S2 includes the following sub-steps:

[0019] Step S201: Obtain the preset lens parameters of the camera that captures the initial image;

[0020] The preset lens parameters include radial distortion coefficient and tangential distortion coefficient;

[0021] Step S202: Based on the preset lens parameters, establish a positive distortion mapping function. The positive distortion mapping function is used to calculate and map the ideal pixel coordinates to the actual pixel coordinates according to the radial distortion coefficient and the tangential distortion coefficient.

[0022] Step S203: Initialize the image to be output, which is a solid color background image. Traverse each ideal pixel coordinate in the image to be output, obtain the actual pixel coordinates of each ideal pixel coordinate in the initial image by solving the forward distortion mapping function in reverse, and calculate the pixel value of the actual pixel coordinates using a bilinear interpolation algorithm. Fill the current pixel of the image to be output to generate a distortion-corrected image.

[0023] As a preferred embodiment of the camera image perspective correction and distortion correction method of the present invention, the preset lens parameters are obtained by pre-calibrating the camera using the Zhang Zhengyou calibration method before the mobile terminal leaves the factory, and the calibration results are encrypted and stored in the memory of the mobile terminal as firmware parameters.

[0024] As a preferred embodiment of the camera image perspective correction and distortion correction method of the present invention, step S3 includes the following sub-steps:

[0025] Step S301: Extract and match feature points in the distortion-corrected image to obtain the same feature points in each initial image;

[0026] The corresponding feature points include the corner points of the target object, the intersection points of the planar background and the target object, and the surface texture feature points of the target object;

[0027] Step S302: Based on the pixel coordinates of the corresponding feature points in each initial image, and combined with the built-in parameters of the cameras at different viewpoints, calculate the relative rotation matrix and translation vector between the cameras at different viewpoints using epipolar geometry constraints, and determine the planar pose of the corresponding feature points relative to the cameras at different viewpoints.

[0028] The built-in parameters include focal length, principal point coordinates, and distortion coefficients;

[0029] Step S303: Based on the planar pose and the pixel coordinates, calculate the three-dimensional coordinates of the corresponding feature points in the world three-dimensional space using triangulation.

[0030] Based on the three-dimensional coordinates, a plane is fitted using the least squares method to obtain the target fitting plane. According to the normal vector of the target fitting plane and the coordinates of the known points of the target fitting plane, a three-dimensional spatial equation of the target object plane is constructed, and the three-dimensional spatial equation is used as three-dimensional geometric information.

[0031] As a preferred embodiment of the camera image perspective correction and distortion correction method of the present invention, step S4 includes the following sub-steps:

[0032] Step S401: Based on the three-dimensional geometric information of the target object plane, set the pose of the preset virtual orthogonal camera, make the optical axis of the preset virtual orthogonal camera parallel to the normal vector of the target object plane, and take the imaging plane of the preset virtual orthogonal camera as the target orthogonal imaging plane.

[0033] Step S402: Use one of the cameras with different viewpoints as the capture camera, obtain the pose of the capture camera in the world coordinate system, and calculate the homography matrix from the image plane of the capture camera to the target orthogonal imaging plane based on the pose of the capture camera and the pose of the preset virtual orthogonal camera.

[0034] Step S403: Use the homography matrix to perform perspective transformation on the distortion-corrected image to generate an orthogonal view located on the target orthogonal imaging plane;

[0035] The calculation of the homography matrix from the captured camera image plane to the target orthogonal imaging plane includes:

[0036] The pose of the captured camera is decomposed into a first rotation matrix and a first translation vector, and the pose of the preset virtual orthogonal camera is decomposed into a second rotation matrix and a second translation vector.

[0037] Based on the first rotation matrix, the first translation vector, the second rotation matrix, and the second translation vector, the mathematical expressions for calculating the relative rotation matrix and relative translation vector of the capturing camera relative to the preset virtual orthogonal camera are as follows:

[0038] Relative rotation matrix: ;

[0039] Relative translation vector: ;

[0040] in, Represents the relative rotation matrix. Represents a relative translation vector. Denotes the first rotation matrix. Denotes the first translation vector. Denotes the second rotation matrix. Indicates the second translation vector;

[0041] Using the relative rotation matrix, the normal vector of the target object plane is transformed from the world coordinate system to the camera coordinate system of the capturing camera, thus obtaining the normal vector in the camera coordinate system;

[0042] Based on the relative rotation matrix, relative translation vector, normal vector in the camera coordinate system, and intrinsic parameter matrix of the capturing camera, a homography matrix from the image plane of the capturing camera to the orthogonal imaging plane of the target is constructed according to the preset homography matrix calculation formula.

[0043] The mathematical expression for the preset homography matrix calculation formula is:

[0044] ;

[0045] in, This represents the preset homography matrix. Represents the relative rotation matrix. Represents a relative translation vector. This represents the intrinsic parameter matrix of the camera. This represents the normal vector in the camera coordinate system. This represents the distance from the target plane to the optical center of the capturing camera. This represents the inverse of the intrinsic parameter matrix of the captured camera.

[0046] As a preferred embodiment of the camera image perspective correction and distortion correction method of the present invention, the step of performing perspective transformation on the distortion-corrected image using the homography matrix specifically includes:

[0047] The homography matrix is ​​used as a coordinate transformation operator to map the coordinates of each pixel in the distortion-corrected image, and the corresponding coordinates of each pixel coordinate in the target orthogonal imaging plane are calculated. Based on the calculated corresponding coordinates, the pixel values ​​corresponding to the corresponding coordinates are extracted from the distortion-corrected image and filled into the corresponding positions in the target orthogonal imaging plane.

[0048] A camera image perspective correction and distortion correction system is applied in a camera image perspective correction and distortion correction method, including a capture module, a correction module, an analysis module and a generation module;

[0049] The capture module is used to simultaneously capture the same target object using cameras from different viewpoints, thereby capturing the initial image of the target object.

[0050] The correction module is used to correct the distortion of the initial image and output a distortion-corrected image;

[0051] The analysis module is used to calculate the three-dimensional geometric information of the target object plane in the three-dimensional space of the world by analyzing the distortion-corrected image;

[0052] The generation module is used to reproject the target object plane onto a virtual orthogonal imaging plane based on the three-dimensional geometric information to generate an orthogonal view.

[0053] The beneficial effects of this invention are as follows: By pre-storing lens parameters before leaving the factory and performing parameter-based distortion correction in the early stages of image processing, this invention eliminates inherent lens distortion and avoids image deformation caused by the inability of single-camera methods to handle distortion. By using cameras with different viewpoints to shoot simultaneously and combining epipolar geometry and triangulation to accurately calculate the three-dimensional geometric information of the target plane, the results are more robust and accurate, effectively avoiding image stretching and information loss. By constructing a virtual orthogonal camera and accurately setting its pose, and using a homography matrix containing relative pose and plane normal vectors for reprojection, the geometric orthogonality of the generated orthogonal view is ensured. This overcomes the view distortion and resolution unevenness caused by neglecting pose optimization in multi-camera systems, and meets the stringent requirements of high-precision application scenarios. Attached Figure Description

[0054] Figure 1 This is a schematic diagram of the basic process of a camera image perspective correction and distortion correction method provided in one embodiment of the present invention.

[0055] Figure 2 This is a schematic diagram of the basic process of a camera image perspective correction and distortion correction system provided in one embodiment of the present invention. Detailed Implementation

[0056] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0057] Reference Figure 1 As an embodiment of the present invention, a method for camera image perspective correction and distortion correction is provided, comprising the following steps:

[0058] Step S1: Simultaneously capture the same target object using cameras from different viewpoints to obtain the initial image of the target object;

[0059] Step S2: Perform distortion correction on the initial image and output the distortion-corrected image;

[0060] Step S3: Calculate the three-dimensional geometric information of the target object plane in the three-dimensional space of the world by analyzing the distortion-corrected image;

[0061] Step S4: Based on the three-dimensional geometric information, reproject the target object plane onto the virtual orthogonal imaging plane to generate an orthogonal view.

[0062] A complete technical loop was constructed, from multi-viewpoint image capture to orthogonal view generation. By using multi-viewpoint synchronous shooting, the limitations of insufficient information in a single viewpoint were overcome. Subsequently, distortion correction and 3D geometric information calculation were treated as independent but closely linked steps, ensuring that the input data for subsequent viewpoint correction had high geometric fidelity. Finally, reprojection was performed based on accurate 3D information, which fundamentally solved the geometric distortion problem caused by the lack of 3D spatial constraints in traditional perspective correction methods, and achieved a breakthrough in the accuracy of viewpoint correction.

[0063] Step S1 includes the following sub-steps:

[0064] Step S101: Receive the shutter command triggered by the user through the user interface;

[0065] Step S102: In response to the shutter command, the mobile terminal's processor simultaneously sends image capture trigger signals to cameras at different viewpoints.

[0066] Cameras with different viewpoints include main cameras and secondary cameras;

[0067] Step S103: Using cameras at different viewpoints, the same target object is exposed and photographed within a preset synchronization error range to capture the initial image of the target object.

[0068] The preset synchronization error range is set to the microsecond level through a hardware interrupt mechanism to ensure that the relative displacement of the target object in different images is less than one pixel, thereby providing a strict time alignment basis for epipolar geometry calculation.

[0069] The image capture process has been refined, and the synchronous trigger signal of the processor ensures that the exposure of multiple cameras is within a very small error range, providing a strict time alignment basis for subsequent 3D calculations. This hardware-level synchronization mechanism can more effectively eliminate motion blur and parallax misalignment, and ensure the spatial consistency of the same feature points in different images, thereby significantly improving the accuracy and robustness of epipolar geometric constraint solution and laying a solid data foundation for high-precision 3D reconstruction.

[0070] Step S1 also includes:

[0071] When capturing the initial image of the target object, the target object is placed on a planar background that includes identifiable features, such as edge contour features, artificial corner features, and natural texture features.

[0072] The algorithm uses the Cannyl operator to extract edge contour features, employs H-corner detection to identify artificial corners, and extracts and matches natural texture features using the ORB descriptor. In subsequent steps, the identifiable features are used as corresponding feature points and matched under epipolar geometric constraints to jointly solve the relative pose between cameras. Feature points at the intersection of the background and the target can accurately define the boundary of the target plane, while the rich feature points on the target surface and the background provide robust geometric constraints, thereby more accurately fitting the overall pose and spatial equation of the target plane.

[0073] Placing the target object on a planar background with identifiable features greatly enhances the reliability of feature point extraction and matching. By utilizing the edge contours, intersections, and texture features of the background and the target, this method constructs a rich and stable set of feature points, enabling the system to perform accurate planar pose calculations even when the target object features are sparse or the texture is simple. This broadens the applicability of the method and improves the stability and accuracy of 3D geometric information calculation.

[0074] Recognizable flat backgrounds include solid color backgrounds.

[0075] By limiting the planar background to a solid color, the boundary features between the target object and the background are highlighted by simplifying the background information. This makes the extraction of edge contours and intersection features more efficient and accurate. The solid color background provides a low computational complexity and high reliability implementation method, which is especially suitable for application scenarios such as document scanning that require clear definition of the target contour. This ensures the accuracy of planar fitting and thus optimizes the edge quality of the final orthogonal view.

[0076] Step S2 includes the following sub-steps:

[0077] Step S201: Obtain the preset lens parameters of the camera that captures the initial image;

[0078] Preset lens parameters include radial distortion coefficient and tangential distortion coefficient;

[0079] Step S202: Based on preset lens parameters, establish a positive distortion mapping function. The positive distortion mapping function is used to calculate and map the ideal pixel coordinates to the actual pixel coordinates according to the radial distortion coefficient and the tangential distortion coefficient.

[0080] Step S203: Initialize the image to be output. The image to be output is a solid color background image. Traverse the ideal pixel coordinates in the image to be output. Obtain the actual pixel coordinates of each ideal pixel coordinate in the initial image by solving the forward distortion mapping function in reverse. Calculate the pixel value of the actual pixel coordinates using a bilinear interpolation algorithm and fill it into the current pixel of the image to be output to generate a distortion-corrected image.

[0081] The forward distortion mapping function is based on the Brown-Conrady distortion model. This model uses preset radial and tangential distortion coefficients to nonlinearly correct the ideal normalized coordinates, thereby calculating the actual pixel coordinates caused by lens distortion. For the reverse solution process, to balance accuracy and efficiency, a combination of pre-calculated lookup table and iterative approximation method is used. That is, the initial estimate is first obtained through the lookup table, and then the Newton-Raphson method is used for fast convergence to accurately obtain the actual pixel coordinates. Bilinear interpolation is chosen instead of nearest neighbor interpolation because the target pixel coordinates are usually non-integer after coordinate transformation. Bilinear interpolation can calculate the pixel value by weighted average of the surrounding four pixels, effectively avoiding the jagged and blocky effects produced by nearest neighbor interpolation, thus ensuring the smoothness and visual quality of the image after distortion correction.

[0082] By employing inverse mapping based on preset lens parameters and bilinear interpolation, high-precision correction of lens distortion is achieved. By solving the forward distortion mapping function in reverse, this method avoids pixel holes and overlap problems that may occur in forward mapping, while bilinear interpolation ensures the smoothness of the corrected image. Utilizing pre-calibrated physical parameters, the correction process is more accurate and stable, providing ideal distortion-free input for all subsequent geometric calculations, which is the cornerstone of the high-precision realization of the entire method.

[0083] The preset lens parameters are obtained by pre-calibrating the camera using the Zhang Zhengyou calibration method before the mobile terminal leaves the factory, and the calibration results are encrypted and stored in the mobile terminal's memory as firmware parameters.

[0084] Zhang Zhengyou's calibration method includes: preparing a high-precision printed black and white checkerboard as the calibration object, with specifications of 10×8 internal corner points, and the physical size of each square accurate to 1 mm to ensure the accuracy of the reference. Subsequently, a six-axis robotic arm is controlled by a program to guide the camera to be calibrated to take pictures of the checkerboard from depth and tilt angles, capturing a total of 20 images. This step aims to ensure that the checkerboard can cover the center, edges and four corners of the camera's field of view and provide rich and diverse geometric constraints. The calibration algorithm automatically detects the corner coordinates in each image and uses the correspondence between these points in the image plane and the checkerboard world coordinate system to obtain the initial estimates of the camera intrinsic parameters and distortion coefficients through homography matrix decomposition. It then enters the global nonlinear optimization stage, with the optimization objective of minimizing the reprojection error of all corner points. The final preset lens parameters are accurately solved through iterative calculation.

[0085] To ensure the security of these critical firmware parameters, the calibration results are encrypted using the AES-256 encryption algorithm and securely stored in the hardware security module of the mobile terminal. This hardware-level security isolation mechanism ensures that the parameters cannot be read or tampered with by the operating system or other malicious software, fundamentally guaranteeing the accuracy and integrity of the initial input of the entire algorithm process.

[0086] The acquisition and storage methods for lens parameters were clearly defined. Pre-calibration using Zhang Zhengyou's calibration method and encrypted firmware storage before shipment ensured the authority and immutability of the camera's intrinsic parameters. This pre-calibration mode avoids the burden of complex calibration for users during use. At the same time, encrypted storage ensures the security and consistency of the parameters. The provided parameters have higher accuracy and reliability, fundamentally guaranteeing the accuracy of distortion correction and 3D geometric calculation, and improving the overall performance of the system.

[0087] Step S3 includes the following sub-steps:

[0088] Step S301: Extract and match feature points on the distortion-corrected image to obtain the same feature points in each initial image;

[0089] Feature points with the same name include the corner points of the target object, the intersection points of the planar background and the target object, and the surface texture feature points of the target object;

[0090] Step S302: Based on the pixel coordinates of the same feature points in each initial image, and combined with the built-in parameters of the cameras at different viewpoints, calculate the relative rotation matrix and translation vector between the cameras at different viewpoints using epipolar geometry constraints, and determine the planar pose of the same feature points relative to the cameras at different viewpoints.

[0091] Built-in parameters include focal length, principal point coordinates, and distortion coefficients;

[0092] Step S303: Based on the planar pose and pixel coordinates, calculate the three-dimensional coordinates of the corresponding feature points in the world three-dimensional space using triangulation.

[0093] Plane fitting using the least squares method includes: based on the fundamental principles of spatial analytic geometry, utilizing normal vectors... and known points Construct the three-dimensional spatial equations. The mathematical expression of the three-dimensional spatial equations is as follows:

[0094] ;

[0095] This equation accurately and unambiguously describes the position and orientation of the target object plane in the three-dimensional world, providing a crucial mathematical basis for subsequent viewpoint correction and reprojection.

[0096] In least squares plane fitting, the plane is an abstract two-dimensional infinitely extending plane.

[0097] Feature point extraction includes: employing multiple feature extraction operators to adapt to different scenarios; for edge contours in the image, the Canny operator is used for edge detection to obtain macroscopic structural information of the target; for manually marked corner points, Harris corner detection is used for precise localization; and for regions with rich natural textures, the ORB local invariant feature descriptor is used to extract key points and their descriptors that are scale- and rotation-invariant.

[0098] Feature point matching includes: after acquiring feature points from different images, performing feature point matching to find corresponding points; performing a fast nearest neighbor search on the descriptor of each feature point to find the most similar candidate matching point in another image; performing preliminary screening by calculating the distance between descriptors and setting a proportional threshold to exclude fuzzy matches; and using epipolar geometric constraints and random sampling consensus algorithms to remove all outliers that do not conform to geometric relationships, thereby obtaining a set of accurate and reliable corresponding feature point matching pairs, providing a robust data foundation for subsequent camera pose estimation and 3D reconstruction.

[0099] The relative rotation matrix and translation vector between cameras at different viewpoints are calculated using epipolar geometry constraints. All matching feature point pairs are combined into a feature point pair set. The feature point pair set is iteratively solved using a random sampling consensus algorithm to obtain the fundamental matrix F. Combined with camera intrinsic parameters, the fundamental matrix F is transformed into the essential matrix E. The relative rotation matrix R and translation vector T between cameras are decomposed from the essential matrix E through singular value decomposition. The fundamental matrix F can effectively eliminate erroneous matching pairs caused by occlusion or duplicate textures. The fundamental matrix F is used to filter interior points that meet the epipolar geometry constraints.

[0100] The mathematical expression for transforming the fundamental matrix F into the essential matrix E is:

[0101] ;

[0102] in, and These represent the intrinsic parameter matrices of the left and right cameras, respectively.

[0103] Based on planar pose and pixel coordinates, the three-dimensional coordinates of corresponding feature points in the world three-dimensional space are calculated by triangulation. The triangulation method includes: when a three-dimensional space point P is observed simultaneously by two cameras at different positions, the three-dimensional space point P will form a pixel on each camera image. For each pixel on the image, the pixel coordinates are converted into a direction vector in the camera coordinate system by inverting the built-in parameter matrix. Based on the rotation matrix and translation vector, the direction vector and the ray origin are respectively converted to the world coordinate system to obtain a three-dimensional ray originating from the camera optical center and passing through the pixel. For corresponding feature points, two rays are constructed in three-dimensional space. The construction rule is that, ideally, the two rays should intersect at one point. This intersection point is the true three-dimensional position of the feature point in the world coordinate system.

[0104] The coordinates of the two ray equations are uniformly transformed to the world coordinate system. The initial coordinates of the intersection point are obtained by solving a linear system of equations consisting of the two ray equations. The sum of squares of the distances between the initial coordinates of all rays constructed in 3D space and the pixels on the camera image is calculated. When the sum of squares is minimized, the 3D coordinates of the feature point are obtained.

[0105] The calculation process for 3D geometric information is disclosed, which organically combines epipolar geometric constraints, triangulation, and least squares plane fitting to achieve high-precision solution of the 3D equation of the target plane. It not only accurately recovers the 3D coordinates of feature points using multi-viewpoint information, but also effectively suppresses the influence of noise and outliers through least squares fitting, thereby obtaining robust plane normal vectors. It provides a more objective and accurate geometric description based on real 3D measurements, and provides key spatial constraints for the subsequent generation of distortion-free orthogonal views.

[0106] Step S4 includes the following sub-steps:

[0107] Step S401: Based on the three-dimensional geometric information of the target object plane, set the pose of the preset virtual orthogonal camera, make the optical axis of the preset virtual orthogonal camera parallel to the normal vector of the target object plane, and use the imaging plane of the preset virtual orthogonal camera as the target orthogonal imaging plane.

[0108] Step S402: Use one of the cameras with different viewpoints as the capture camera, obtain the pose of the capture camera in the world coordinate system, and calculate the homography matrix from the capture camera image plane to the target orthogonal imaging plane based on the pose of the capture camera and the pose of the preset virtual orthogonal camera.

[0109] Step S403: Use the homography matrix to perform perspective transformation on the distortion-corrected image to generate an orthogonal view located on the target orthogonal imaging plane;

[0110] Calculating the homography matrix from the captured camera image plane to the target orthogonal imaging plane includes:

[0111] The pose of the captured camera is decomposed into a first rotation matrix and a first translation vector, and the pose of the preset virtual orthogonal camera is decomposed into a second rotation matrix and a second translation vector.

[0112] Based on the first rotation matrix, the first translation vector, the second rotation matrix, and the second translation vector, the mathematical expressions for calculating the relative rotation matrix and relative translation vector of the capturing camera relative to the preset virtual orthogonal camera are as follows:

[0113] Relative rotation matrix: ;

[0114] Relative translation vector: ;

[0115] in, Represents the relative rotation matrix. Represents a relative translation vector. Denotes the first rotation matrix. Denotes the first translation vector. Denotes the second rotation matrix. Indicates the second translation vector;

[0116] Using a relative rotation matrix, the normal vector of the target object plane is transformed from the world coordinate system to the camera coordinate system of the capturing camera, thus obtaining the normal vector in the camera coordinate system.

[0117] Based on the relative rotation matrix, relative translation vector, normal vector in the camera coordinate system, and intrinsic parameter matrix of the capturing camera, a homography matrix from the image plane of the capturing camera to the orthogonal imaging plane of the target is constructed according to the preset homography matrix calculation formula.

[0118] The mathematical expression for the predefined homography matrix calculation formula is:

[0119] ;

[0120] in, This represents the preset homography matrix. Represents the relative rotation matrix. Represents a relative translation vector. This represents the intrinsic parameter matrix of the camera. This represents the normal vector in the camera coordinate system. This represents the distance from the target plane to the optical center of the capturing camera. This represents the inverse of the intrinsic parameter matrix of the captured camera.

[0121] Preset homography matrix calculation formula The physical significance lies in the fact that it precisely encodes the rigid transformation of a three-dimensional plane in the real world into a projection transformation between two-dimensional images. and A complete 3D rigid transformation from the perspective of the capturing camera to that of the virtual camera is defined, namely rotation and translation, while key terms... or The spatial constraint information of the target plane is incorporated into the transformation process. It corrects the perspective distortion caused by the movement of the viewpoint. This is the core mechanism for achieving orthogonal reprojection instead of ordinary perspective transformation. Before calculating this transformation, a capture camera is selected from the multi-camera array. The selection strategy is usually to give priority to the main camera whose viewpoint is most directly facing the target plane or has the highest resolution, so as to ensure that the source image used for reprojection has the best clarity and the least perspective distortion.

[0122] Setting the pose of the virtual orthogonal camera includes: setting the camera's projection model as an orthogonal projection and forcing its optical axis direction to be mathematically aligned with the normal vector of the target plane. The camera obtains two translational degrees of freedom and one rotational degree of freedom out of six degrees of freedom in a plane perpendicular to its optical axis. According to the preset composition rules, the two translational degrees of freedom and one rotational degree of freedom are numerically solved to calculate the optimal position and orientation of the camera in the target plane, thus determining the pose of the virtual orthogonal camera in three-dimensional space.

[0123] By setting the pose of a virtual orthogonal camera and constructing a homography matrix containing plane normals, a precise transformation from arbitrary viewpoints to strictly orthogonal projection is achieved. The mathematical expression of the homography matrix integrates the relative poses between cameras and the spatial geometric information of the target plane, ensuring the geometric rigor of the reprojection process. This overcomes the shortcomings of traditional perspective transformation, which can only distort two-dimensional images and cannot guarantee three-dimensional orthogonality, and can generate views that are geometrically perpendicular to the target plane.

[0124] The perspective transformation of distorted images using the homography matrix specifically includes:

[0125] The homography matrix is ​​used as a coordinate transformation operator to convert the pixel coordinates in the distortion-corrected image into homogeneous coordinates. The homogeneous coordinate vector is multiplied by the homography matrix to obtain the transformed coordinates in the target image. The result is divided by the last component to obtain the two-dimensional pixel coordinates. The two-dimensional pixel coordinates accurately indicate the corresponding position of the source image pixels in the target view. The corresponding coordinates of each pixel coordinate in the target orthogonal imaging plane are calculated. Based on the calculated corresponding coordinates, the pixel values ​​corresponding to the corresponding coordinates are extracted from the distortion-corrected image and filled into the corresponding positions in the target orthogonal imaging plane.

[0126] The process of coordinate mapping and pixel value filling involves traversing the pixel coordinates of each pixel in the target orthogonal imaging plane. For each target pixel, the inverse transformation of the pre-calculated homography matrix is ​​used to calculate its corresponding floating-point coordinates in the source distortion-corrected image. Bilinear interpolation is used to determine the pixel value of the floating-point coordinates. Centered on the floating-point coordinates, the four nearest integer pixels around the floating-point coordinates in the source image are located. The pixel value of the target pixel is calculated by weighted averaging based on the distance relationship between the floating-point coordinates and these four points.

[0127] The implementation details of perspective transformation are further clarified. The complex homography matrix is ​​used as an efficient coordinate transformation operator. Through pixel-by-pixel mapping and interpolation filling, the image on the orthogonal imaging plane of the target is directly generated. This direct transformation method has a clear process and high computational efficiency. It can completely preserve the pixel information of the original image and avoid the accuracy loss that may be introduced by multi-step processing. While ensuring high accuracy, it has lower computational overhead and faster processing speed, making it more suitable for resource-constrained platforms such as mobile terminals.

[0128] Reference Figure 2 This is another embodiment of the present invention. Unlike the first embodiment, this embodiment provides a camera image perspective correction and distortion correction system, including a capture module, a correction module, an analysis module, and a generation module.

[0129] The capture module is used to simultaneously capture images of the same target object using cameras from different viewpoints, thereby capturing the initial image of the target object;

[0130] The correction module is used to correct distortion in the initial image and output a distortion-corrected image.

[0131] The analysis module is used to calculate the three-dimensional geometric information of the target object plane in the world's three-dimensional space by analyzing the distortion-corrected image;

[0132] The generation module is used to reproject the target object plane onto a virtual orthogonal imaging plane based on the three-dimensional geometric information to generate an orthogonal view.

[0133] This invention eliminates inherent lens distortion by pre-storing lens parameters before shipment and performing parameter-based distortion correction in the early stages of image processing. This avoids image deformation caused by the inability of single-camera methods to handle distortion. By using cameras with different viewpoints to shoot simultaneously and combining epipolar geometry and triangulation to accurately calculate the three-dimensional geometric information of the target plane, the results are more robust and accurate, effectively avoiding image stretching and information loss. By constructing a virtual orthogonal camera and accurately setting its pose, and using a homography matrix containing relative pose and plane normal vectors for reprojection, the geometric orthogonality of the generated orthogonal view is ensured. This overcomes the view distortion and resolution unevenness caused by neglecting pose optimization in multi-camera systems, meeting the stringent requirements of high-precision application scenarios.

[0134] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0135] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for camera image perspective correction and distortion rectification, the method comprising: receiving a camera image; determining a camera image perspective correction and distortion rectification transformation; and applying the camera image perspective correction and distortion rectification transformation to the camera image. Includes the following steps: Step S1: Simultaneously capture the same target object using cameras from different viewpoints to obtain the initial image of the target object; Step S2: Perform distortion correction on the initial image and output the distortion-corrected image; Step S3: Calculate the three-dimensional geometric information of the target object plane in the three-dimensional space of the world by analyzing the distortion-corrected image; Step S4: Based on the three-dimensional geometric information, reproject the target object plane onto the virtual orthogonal imaging plane to generate an orthogonal view; Step S4 includes the following sub-steps: Step S401: Based on the three-dimensional geometric information of the target object plane, set the pose of the preset virtual orthogonal camera, make the optical axis of the preset virtual orthogonal camera parallel to the normal vector of the target object plane, and take the imaging plane of the preset virtual orthogonal camera as the target orthogonal imaging plane. Step S402: Use one of the cameras with different viewpoints as the capture camera, obtain the pose of the capture camera in the world coordinate system, and calculate the homography matrix from the image plane of the capture camera to the target orthogonal imaging plane based on the pose of the capture camera and the pose of the preset virtual orthogonal camera. Step S403: Use the homography matrix to perform perspective transformation on the distortion-corrected image to generate an orthogonal view located on the target orthogonal imaging plane; The specific steps of performing perspective transformation on the distortion-corrected image using the homography matrix include: The homography matrix is ​​used as a coordinate transformation operator to map the coordinates of each pixel in the distortion-corrected image, and the corresponding coordinates of each pixel in the target orthogonal imaging plane are calculated. Based on the calculated corresponding coordinates, the pixel values ​​corresponding to the corresponding coordinates are extracted from the distortion-corrected image and filled into the corresponding positions in the target orthogonal imaging plane.

2. The method of claim 1, wherein, Step S1 includes the following sub-steps: Step S101: Receive the shutter command triggered by the user through the user interface; Step S102: In response to the shutter command, the mobile terminal's processor simultaneously sends image capture trigger signals to cameras at different viewpoints. The cameras with different viewpoints include a main camera and a secondary camera; Step S103: Using cameras at different viewpoints, the same target object is exposed and photographed within a preset synchronization error range to capture the initial image of the target object.

3. The method of claim 2, wherein, Step S1 further includes: When capturing an initial image of a target object, the target object is placed on a planar background that includes identifiable features, such as edge contour features, artificial corner features, and natural texture features.

4. The camera image perspective correction and distortion correction method as described in claim 3, characterized in that, The planar background with identifiable features includes a solid color background.

5. The camera image perspective correction and distortion correction method as described in claim 4, characterized in that, Step S2 includes the following sub-steps: Step S201: Obtain the preset lens parameters of the camera that captures the initial image; The preset lens parameters include radial distortion coefficient and tangential distortion coefficient; Step S202: Based on the preset lens parameters, establish a positive distortion mapping function. The positive distortion mapping function is used to calculate and map the ideal pixel coordinates to the actual pixel coordinates according to the radial distortion coefficient and the tangential distortion coefficient. Step S203: Initialize the image to be output, which is a solid color background image. Traverse each ideal pixel coordinate in the image to be output, obtain the actual pixel coordinates of each ideal pixel coordinate in the initial image by solving the forward distortion mapping function in reverse, and calculate the pixel value of the actual pixel coordinates using a bilinear interpolation algorithm. Fill the current pixel of the image to be output to generate a distortion-corrected image.

6. The camera image perspective correction and distortion correction method as described in claim 5, characterized in that, The preset lens parameters are obtained by pre-calibrating the camera using the Zhang Zhengyou calibration method before the mobile terminal leaves the factory, and the calibration results are encrypted and stored in the memory of the mobile terminal as firmware parameters.

7. The camera image perspective correction and distortion correction method as described in claim 6, characterized in that, Step S3 includes the following sub-steps: Step S301: Extract and match feature points in the distortion-corrected image to obtain the same feature points in each initial image; The corresponding feature points include the corner points of the target object, the intersection points of the planar background and the target object, and the surface texture feature points of the target object; Step S302: Based on the pixel coordinates of the corresponding feature points in each initial image, and combined with the built-in parameters of the cameras at different viewpoints, calculate the relative rotation matrix and translation vector between the cameras at different viewpoints using epipolar geometry constraints, and determine the planar pose of the corresponding feature points relative to the cameras at different viewpoints. The built-in parameters include focal length, principal point coordinates, and distortion coefficients; Step S303: Based on the planar pose and the pixel coordinates, calculate the three-dimensional coordinates of the corresponding feature points in the world three-dimensional space using triangulation. Based on the three-dimensional coordinates, a plane is fitted using the least squares method to obtain the target fitted plane. According to the normal vector of the target fitted plane and the coordinates of the known points of the target fitted plane, a three-dimensional spatial equation of the target object plane is constructed, and the three-dimensional spatial equation is used as three-dimensional geometric information.

8. The camera image perspective correction and distortion correction method as described in claim 7, characterized in that, The calculation of the homography matrix from the captured camera image plane to the target orthogonal imaging plane includes: The pose of the captured camera is decomposed into a first rotation matrix and a first translation vector, and the pose of the preset virtual orthogonal camera is decomposed into a second rotation matrix and a second translation vector. Based on the first rotation matrix, the first translation vector, the second rotation matrix, and the second translation vector, the mathematical expressions for calculating the relative rotation matrix and relative translation vector of the capturing camera relative to the preset virtual orthogonal camera are as follows: Relative rotation matrix: ; Relative translation vector: ; in, Represents the relative rotation matrix. Represents a relative translation vector. Denotes the first rotation matrix. Denotes the first translation vector. Denotes the second rotation matrix. Indicates the second translation vector; Using the relative rotation matrix, the normal vector of the target object plane is transformed from the world coordinate system to the camera coordinate system of the capturing camera, thus obtaining the normal vector in the camera coordinate system; Based on the relative rotation matrix, relative translation vector, normal vector in the camera coordinate system, and intrinsic parameter matrix of the capturing camera, a homography matrix from the image plane of the capturing camera to the orthogonal imaging plane of the target is constructed according to the preset homography matrix calculation formula. The mathematical expression for the preset homography matrix calculation formula is: ； in, This represents the preset homography matrix. Represents the relative rotation matrix. Represents a relative translation vector. This represents the intrinsic parameter matrix of the camera. This represents the normal vector in the camera coordinate system. This represents the distance from the target plane to the optical center of the capturing camera. This represents the inverse of the intrinsic parameter matrix of the captured camera.

9. A camera image perspective correction and distortion correction system, applied in a camera image perspective correction and distortion correction method as described in any one of claims 1-8, characterized in that, It includes a capture module, a correction module, an analysis module, and a generation module; The capture module is used to simultaneously capture the same target object using cameras from different viewpoints, thereby capturing the initial image of the target object. The correction module is used to correct the distortion of the initial image and output a distortion-corrected image; The analysis module is used to calculate the three-dimensional geometric information of the target object plane in the three-dimensional space of the world by analyzing the distortion-corrected image; The generation module is used to reproject the target object plane onto a virtual orthogonal imaging plane based on the three-dimensional geometric information to generate an orthogonal view.

Images