Simplified Field of View Method for Accelerating Image Processing Based on a Rotating Double Prism Imaging System

Abstract

The present invention relates to a simplified field of view method for accelerating image processing based on a rotating double prism imaging system, including: building a rotating double prism imaging system to obtain an original distorted image; correcting the imaging distortion caused by the camera lens; constructing the normal vectors of each surface of the double prism and the boundary vectors of the original distorted image, and calculating the positions of the deflected field of view boundaries in the undistorted grid; obtaining the maximum inscribed rectangular boundary of the field of view boundary in the undistorted grid, obtaining the coordinates of all integer points to be filled inside it, and then constructing the reverse incident light vector; successively calculating the four refractions of the incident light to obtain the positions of the integer points in the undistorted grid in the distorted grid, and calculating the gray values of the integer points in the undistorted grid; filling the obtained gray values into the corresponding undistorted grid one by one to achieve the correction of the imaging distortion of the rotating double prism. This method is beneficial to improving the efficiency of image processing, providing a good basis for subsequent processing, and saving time costs.

Description

Technical Field

[0001] The invention belongs to the technical field of imaging systems and image processing, and in particular relates to a simplified field of view method based on a rotating dual-prism imaging system to accelerate image processing. Background Art

[0002] As a typical refractive beam scanning device, the rotating biprism has been widely used in free-space optical communication, infrared countermeasures, laser radar and other beam control equipment due to its powerful beam deflection ability and control accuracy. With the further in-depth study of the rotating biprism system, the rotating biprism imaging system using imaging detectors as working parts came into being. The rotating biprism imaging system has a powerful field of view expansion capability and excellent dynamic performance, and is constantly expanding its application prospects in the direction of large field of view imaging. A large field of view means that it contains more spatial area information, which has important value in wide-area search and rescue, land surveying, military reconnaissance and other fields.

[0003] However, the nonlinear deflection of light by the prism causes serious imaging distortion in the rotating dual-prism imaging system. The existence of imaging distortion seriously destroys the morphology and spatial position relationship of the objects in the image, making it difficult for the original images collected by the system to be used for further target position detection, large field of view stitching, super-resolution reconstruction, etc. In order to overcome the imaging distortion of the rotating dual-prism system and obtain images with good appearance and can be used for further data processing, a large number of related studies have been carried out and achieved good results. For example, there are distortion correction methods based on first-order paraxial approximation and inverse ray tracing distortion correction methods based on the law of vector refraction. In addition, there are also methods that analyze the error sources and influences of the distortion correction of the inverse ray tracing method, and propose corresponding error parameter identification methods to further improve the accuracy of distortion correction. However, although the above methods can achieve distortion correction, none of them consider the efficiency of distortion correction.

[0004] The prior art (V. Lavigne and B. Ricard, "Fast Risley prisms camera steering system: calibration and image distortions correction through the use of a three-dimensional refraction model," Opt. Eng. 46, 043201 (2007).) proposed a distortion correction method based on inverse ray tracing. This method performs ray tracing based on first-order paraxial approximation and uses homography transformation for correction, but the correction accuracy is not high.

[0005] The inverse ray tracing distortion correction method based on the vector refraction law proposed by the prior art (Y. Zhou, S. Fan, G. Liu, Y. Chen, and D. Fan, "Image distortions caused by rotational double prisms and their correction," Acta Opt. Sin. 35(9), 143–150 (2015)). This method overcomes the shortcomings of previous linear transformations and inaccurate solutions, has high calculation accuracy, and the correction results are more reliable. It is the most commonly used algorithm for correcting the imaging distortion of rotational double prisms. Although the above two methods solve the problem of imaging distortion, the irregular image boundary after correction introduces new problems: the process of implementing back-projection interpolation on the entire corrected image to find its internal integer points will greatly increase the computational complexity and seriously affect the efficiency of the image processing process. Summary of the Invention

[0006] The object of the present invention is to provide a simplified field of view method for accelerating image processing based on a rotational double prism imaging system, which is beneficial to improving the efficiency of image processing, providing a good basis for subsequent processing, and saving time costs.

[0007] To achieve the above object, the technical solution adopted by the present invention is: a simplified field of view method for accelerating image processing based on a rotational double prism imaging system, including:

[0008] Step S1: Build a rotational double prism imaging system to obtain the original distorted image;

[0009] Step S2: Read in the original distorted image and correct the imaging distortion caused by the camera lens using the pre-calibrated camera parameters;

[0010] Step S3: Read in the rotation angle of the double prism, construct the normal vectors of each surface of the double prism and the boundary vectors of the original distorted image, and calculate the position of the deflected field of view boundary in the undistorted grid;

[0011] Step S4: Obtain the maximum inscribed rectangular boundary of the field of view boundary in the undistorted grid, obtain the coordinates of all integer points to be filled inside according to the rectangular boundary, and construct the reverse incident ray vector based on the integer point coordinates;

[0012] Step S5: Calculate the four refractions of the incident ray in sequence according to the inverse ray tracing method to obtain the position of the integer point in the undistorted grid in the distorted grid, and obtain the gray value of the integer point in the undistorted grid using bilinear interpolation;

[0013] Step S6: Fill the calculated gray values into the corresponding undistorted grids one by one to achieve the correction of the imaging distortion of the rotational double prism.

[0014] Furthermore, the rotary double prism imaging system includes a single camera and a pair of identical wedge prisms; the two wedge prisms are coaxially placed at a set distance and can be independently rotated respectively driven by a driving system.

[0015] Furthermore, in the step S2, the correction formula for correcting the imaging distortion caused by the camera lens by using the pre-calibrated camera parameters is:

[0016]

[0017] where x' and y' are the horizontal and vertical coordinates of the pixel after correction, and x and y are the horizontal and vertical coordinates of the pixel before correction, a 1 and a 2 are the radial distortion coefficients of the camera lens respectively, b 1 and b 2 are the tangential distortion coefficients of the camera lens respectively, and the parameters a 1 and a 2 and b 1 and b 2 are obtained by camera calibration.

[0018] Furthermore, the step S3 specifically includes the following steps:

[0019] Step S31: According to the size of the camera pixel array and the equivalent focal length, establish the incident vector S 0 =(x 0 , y 0 , f) of the light ray emitted from the edge of the pixel array to the prism, and convert it into a unit vector;

[0020] Step S32: Calculate the boundary after the edge light ray is deflected by the rotary double prism according to the law of refraction of vectors, and the formula is:

[0021]

[0022] In the formula, i∈[1,4] is the number of times the light beam passes through the prism surface, S i-1 and S i respectively represent the direction vectors of the light beam before and after refraction, n i-1 and n i respectively represent the refractive indices of the incident end and the refraction end media, and N i represents the normal vector of the refraction surface.

[0023] Furthermore, the step S4 specifically includes the following steps:

[0024] Step S41: Construct the maximum inscribed rectangle boundary according to the extreme values of the four field-of-view boundaries in the undistorted grid;

[0025] Step S42: Obtain all integer point sets A within the boundary based on the constructed maximum inscribed rectangle boundary;

[0026] Step S43: Construct a reverse incident light vector S 0 ' = (-x', -y', -f) from the object plane to the image plane, and convert it into a unit vector.

[0027] Furthermore, step S5 specifically includes the following steps:

[0028] Step S51: Use S 0 ' calculated in step S43. According to formula (3), reverse map the integer point A(x', y') in the undistorted grid to the distorted grid to obtain a non-integer point F(x, y);

[0029]

[0030] Step S52: Based on the bilinear interpolation formula, use the gray information at the integer points of the distorted grid to obtain the gray information of the non-integer point F(x, y);

[0031]

[0032] In the formula, k and j are the integer parts of the non-integer point coordinates x and y respectively; u and v are the decimal parts of the non-integer point coordinates x and y respectively. F(k, j), F(k, j + 1), F(k + 1, j), and F(k + 1, j + 1) respectively represent the gray information of the four integer points closest to the non-integer point in the distorted grid.

[0033] Furthermore, step S6 is specifically:

[0034] Assign the gray information of the non-integer point F(x, y) in the distorted grid obtained in step S52 to the integer point A(x', y') in the undistorted grid one by one to achieve the correction of the rotational double prism imaging distortion.

[0035] Compared with the prior art, the present invention has the following beneficial effects: A simplified field of view method for accelerating image processing based on a rotating double prism imaging system is provided. This method replaces the irregular boundary with a rectangular boundary based on the image after simplified field of view correction, discards the area with too low resolution, and has a higher information entropy, which is beneficial to the storage and transmission of information. The finally corrected rectangular boundary image not only reduces the occupation of storage space, but also provides a good basis for image processing tasks such as image stitching because the regular image obtained by cutting off the black edges due to field of view simplification. The simplified field of view method for accelerating image processing based on the rotating double prism imaging system of the present invention greatly improves the image processing efficiency and saves time costs for subsequent further processing based on the corrected image, such as image stitching, super-resolution and other tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 is the schematic diagram of the rotating double prism imaging system in the embodiment of the present invention.

[0037] Figure 2 is the flowchart of the method implementation in the embodiment of the present invention.

[0038] Figure 3 is the schematic diagram of the simplified field of view with different prism rotation angles in the embodiment of the present invention.

[0039] Figure 4 is the distortion correction diagram under different prism rotation angles in the embodiment of the present invention. The first row is the original distorted image, the second row is the corrected image based on the complete field of view, and the third row is the corrected image based on the simplified field of view. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0040] The present invention will be further described below in conjunction with the drawings and embodiments.

[0041] It should be noted that the following detailed description is exemplary and is intended to provide further description of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which the present application belongs.

[0042] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0043] This embodiment provides a simplified field of view method for accelerating image processing based on a rotating double prism imaging system. This method uses a rotating double prism imaging system to accelerate the process of obtaining all internal points of the simplified field of view by simplifying the complete field of view, thereby achieving fast image processing based on the simplified field of view. As Figure 2 shown, the method specifically includes the following steps:

[0044] Step S1: Build a rotating double prism imaging system to obtain the original distorted image;

[0045] Step S2: Read in the original distorted image and correct the imaging distortion caused by the camera lens using the pre-calibrated camera parameters;

[0046] Step S3: Read in the rotation angle of the double prism, construct the normal vectors of each face of the double prism and the boundary vectors of the original distorted image, and calculate the position of the deflected field of view boundary in the undistorted grid;

[0047] Step S4: Obtain the maximum inscribed rectangular boundary of the field of view boundary in the undistorted grid, obtain the integer point coordinates of all points to be filled inside it based on the rectangular boundary, and construct the reverse incident light vector based on the integer point coordinates;

[0048] Step S5: Calculate the four refractions of the incident light successively according to the inverse ray tracing method to obtain the position of the integer points in the undistorted grid in the distorted grid, and obtain the gray value of the integer points in the undistorted grid using bilinear interpolation;

[0049] Step S6: Fill the calculated gray values into the corresponding undistorted grid one by one to correct the imaging distortion of the rotating double prism.

[0050] As Figure 1 shown, the rotating double prism imaging system includes a single camera and a pair of identical wedge prisms. The two wedge prisms are coaxially placed at a set distance apart. When the two prisms are installed, their coaxiality and spacing are ensured through a mechanical structure, and they can be independently rotated respectively driven by a drive system. To facilitate the description of the rotation states and relative position relationships of the two prisms, the rotation angles of the two prisms are defined as the counterclockwise angles between their main sections and the positive x-axis, and are represented by (θ 1 , θ 2 ) respectively.

[0051] In this embodiment, the two prisms have relatively large apex angles and the camera itself has a relatively large field of view angle. Set the parameters of the rotating double prism imaging system as follows: the apex angles of the two prisms α 1 = α 2 = 14°51′, the prism material is N-BK7, and the refractive index is taken as n 1 = n 2= 1.515. The pixel array size of the camera is 640×480 pixels, and the equivalent focal length f = 667 pixels.

[0052] In this embodiment, in the step S2, the correction formula for correcting the imaging distortion caused by the camera lens by using the pre-calibrated camera parameters is:

[0053]

[0054] where x' and y' are the horizontal and vertical coordinates of the pixels after correction, and x and y are the horizontal and vertical coordinates of the pixels before correction, a 1 and a 2 are the radial distortion coefficients of the camera lens respectively, b 1 and b 2 are the tangential distortion coefficients of the camera lens respectively, and the parameters a 1 and a 2 and b 1 and b 2 are obtained by camera calibration.

[0055] In this embodiment, the step S3 specifically includes the following steps:

[0056] Step S31: According to the camera pixel array size and the equivalent focal length (in this embodiment, the camera pixel array size is 640×480 pixels, and the equivalent focal length f = 677 pixels), establish the incident vector S 0 =(x 0 , y 0 , f) of the light ray emitted from the edge of the pixel array to the prism, and convert it into a unit vector;

[0057] Step S32: Calculate the boundary of the edge light ray deflected by the rotating double prism according to the law of refraction of vectors, and the formula is:

[0058]

[0059] In the formula, i∈[1,4] is the number of times the light beam passes through the prism surface, S i-1 and S i respectively represent the direction vectors of the light beam before and after refraction, n i-1 and n i respectively represent the refractive indices of the incident end and the refraction end media, and N i represents the normal vector of the refraction surface.

[0060] In this embodiment, the step S4 specifically includes the following steps:

[0061] Step S41: As Figure 3As shown, according to the extreme values of the four field - of - view boundaries in the undistorted grid, construct the maximum inscribed rectangular boundary;

[0062] Step S42: Based on the constructed maximum inscribed rectangular boundary, obtain all the integer - point sets A within the boundary;

[0063] Step S43: According to the integer - point set A, construct the reverse incident light - ray vector S 0 ' = (-x', -y', -f), and convert it into a unit vector.

[0064] In this embodiment, the step S5 specifically includes the following steps:

[0065] Step S51: Using S 0 ' calculated in step S43, according to formula (3), inversely map the integer point A(x', y') in the undistorted grid to the distorted grid to obtain the non - integer point F(x, y);

[0066]

[0067] Step S52: Based on the bilinear interpolation formula, use the gray - scale information at the integer points of the distorted grid to obtain the gray - scale information of the non - integer point F(x, y);

[0068]

[0069] In the formula, k and j are the integer parts of the non - integer point coordinates x and y respectively; u and v are the fractional parts of the non - integer point coordinates x and y respectively, and F(k, j), F(k, j + 1), F(k + 1, j), F(k + 1, j + 1) respectively represent the gray - scale information of the four integer points closest to the non - integer point in the distorted grid.

[0070] In this embodiment, the step S6 is specifically:

[0071] Assign the gray - scale information of the non - integer point F(x, y) in the distorted grid obtained in step S52 to the integer point A(x', y') in the undistorted grid one by one to achieve the correction of the rotational double - prism imaging distortion, as Figure 4 shown.

[0072] In this embodiment, in step S4, according to the extreme values of the four field - of - view boundaries in the undistorted grid, construct the maximum inscribed rectangular boundary. Based on the constructed maximum inscribed rectangular boundary, directly obtain all the integer - point sets within the boundary, which greatly improves the overall efficiency of the algorithm. When θ 1 = θ 2 = 90°, the average time taken to correct the complete imaging field of view is 2.1446 s, while the simplified field of view only requires 0.120 s; when θ 1 = θ2 When θ = 0°, the average time required to correct the entire imaging field of view increased to 2.559 s, while only 0.136 s was needed for the simplified field of view. Further, we calculated the correction time required for any combination of prism rotation angles: the time required to correct the entire imaging field of view was 1.019 - 3.832 s, while the time required to correct the simplified field of view was only 0.113 - 0.137 s. Therefore, using the simplified field of view can reduce the distortion correction time by 88.9 - 96.9% on average.

[0073] The above embodiments show that the method proposed by the present invention can be based on a rotating double prism imaging system. By simplifying the field of view, all internal integer points of the simplified field of view can be directly obtained, greatly improving the image processing speed and saving time costs.

[0074] The above are only the preferred embodiments of the present invention, and are not intended to limit the present invention in other forms. Any person skilled in the art may use the technical content disclosed above to make changes or modifications into equivalent embodiments with equivalent changes. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the technical solution content of the present invention still fall within the protection scope of the technical solution of the present invention.

Claims

1. A simplified field of view method for accelerating image processing based on a rotating double prism imaging system, characterized in that, it includes: Step S1: Build a rotating double prism imaging system to obtain an original distorted image; Step S2: Read in the original distorted image and correct the imaging distortion caused by the camera lens using pre-calibrated camera parameters; Step S3: Read in the rotation angle of the double prism, construct the normal vectors of each face of the double prism and the boundary vectors of the original distorted image, and calculate the positions of the deflected field of view boundaries in the undistorted grid; Step S4: Obtain the maximum inscribed rectangular boundary of the field of view boundary in the undistorted grid, obtain the integer point coordinates of all points to be filled inside it according to the rectangular boundary, and construct the reverse incident light vector based on the integer point coordinates; Step S5: Calculate the four refractions of the incident light successively according to the inverse ray tracing method to obtain the positions of the integer points in the undistorted grid in the distorted grid, and use bilinear interpolation to obtain the gray values of the integer points in the undistorted grid; Step S6: Fill the calculated gray values into the corresponding undistorted grid one by one to correct the imaging distortion of the rotating double prism.

2. The simplified field of view method for accelerating image processing based on a rotating double prism imaging system according to claim 1, characterized in that, the rotating double prism imaging system includes a single camera and a pair of identical wedge prisms; the two wedge prisms are coaxially placed at a set distance and can be independently rotated respectively under the drive of a drive system.

3. The simplified field of view method for accelerating image processing based on a rotating double prism imaging system according to claim 2, characterized in that, in the step S2, the correction formula for correcting the imaging distortion caused by the camera lens using pre-calibrated camera parameters is: Among them, x' and y' are the horizontal and vertical coordinates of the pixel after correction, and x and y are the horizontal and vertical coordinates of the pixel before correction, a 1 and a 2 are the radial distortion coefficients of the camera lens respectively, and b 1 and b 2 are the tangential distortion coefficients of the camera lens respectively. The parameters a 1 and a 2 and b 1 and b 2 are obtained through camera calibration.

4. The simplified field of view method for accelerating image processing based on a rotating double prism imaging system according to claim 2, characterized in that, the step S3 specifically includes the following steps: Step S31: Establish the incident vector S of the light rays emitted from the edge of the pixel array to the prism according to the size of the camera pixel array and the equivalent focal length. 0 =(x 0 , y 0 , f), and convert it into a unit vector. Step S32: Calculate the boundary of the marginal ray deflected by the rotating double prism according to the law of vector refraction, and the formula is: where \(i\in[1,4]\) is the number of times the light beam passes through the prism surface, \(S\) i-1 and \(S\) i respectively represent the direction vectors of the light beam before and after refraction, \(n\) i-1 and \(n\) i respectively represent the refractive indices of the media at the incident end and the refraction end, \(N\) i represents the normal vector of the refraction surface.

5. The simplified field of view method for accelerating image processing based on a rotating double prism imaging system according to claim 3, characterized in that, the step S4 specifically includes the following steps: Step S41: Construct the maximum inscribed rectangular boundary according to the extreme values of the 4 field of view boundaries in the undistorted grid; Step S42: Obtain all the integer point sets A inside the boundary according to the constructed maximum inscribed rectangular boundary; Step S43: Construct a reverse incident light ray vector S 0 ' = (-x', -y', -f) from the object plane to the image plane according to the integer point set A, and convert it into a unit vector.

6. The simplified field of view method for accelerating image processing based on a rotating double prism imaging system according to claim 5, characterized in that, the step S5 specifically includes the following steps: Step S51: Using S 0 ' calculated in step S43, according to formula (3), the integer point A(x', y') in the undistorted grid is inversely mapped to the distorted grid to obtain the non-integer point F(x, y); Step S52: Based on the bilinear interpolation formula, use the gray information of the integer points on the distorted grid to obtain the gray information of the non-integer point F(x, y); In the formula, k and j are the integer parts of the non-integer point coordinates x and y respectively; u and v are the decimal parts of the non-integer point coordinates x and y respectively, and F(k, j), F(k, j + 1), F(k + 1, j), F(k + 1, j + 1) respectively represent the gray information of the four integer points closest to the non-integer point in the distorted grid.

7. The simplified field of view method for accelerating image processing based on a rotating double prism imaging system according to claim 6, characterized in that, the specific step S6 is as follows: The gray scale information of the non-integer point F(x, y) in the distorted grid obtained in step S52 is assigned to the integer point A(x', y') in the undistorted grid one by one to achieve the correction of the rotation double prism imaging distortion.

Images