Real-time processing method, apparatus, fisheye camera and storage medium for fisheye images

Abstract

This application relates to the field of image processing technology, providing a real-time fisheye image processing method, apparatus, fisheye camera, and storage medium. When the fisheye camera starts, this application initializes a coordinate transformation table from spherical coordinates to image coordinates based on a distortion table / distortion coefficients. It determines the region of interest (ROI) based on a preset field of view and initializes a first spherical coordinate table based on the ROI's image coordinates and the distortion table / distortion coefficients. As the fisheye camera's pose changes, a rotation matrix is ​​calculated based on the real-time pose, and the first spherical coordinate table is rotated inversely using the rotation matrix to obtain a second spherical coordinate table. The second spherical coordinate table is then converted to a target image coordinate table based on the coordinate transformation table. Finally, the acquired original fisheye image is remapped using the target image coordinate table to obtain the target fisheye image. This application enables real-time distortion correction of fisheye images based on their pose, improving the image quality of fisheye images.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a real-time fisheye image processing method, apparatus, fisheye camera, and storage medium. Background Technology

[0002] Electronic pan-tilt units are widely used in high-end network cameras. Through program settings, the camera's viewing angle and zoom can be controlled internally, thereby enabling image capture and transmission.

[0003] However, when using an electronic gimbal mounted on an unmanned imaging platform (such as a drone, other flying equipment, or an underwater robot) in conjunction with a fisheye lens for aerial or underwater photography, the fisheye image distortion problem caused by changes in the attitude of the unmanned imaging platform is frequently encountered. This distortion affects the quality of the captured images. Summary of the Invention

[0004] In view of the above, it is necessary to propose a real-time fisheye image processing method, device, fisheye camera, and computer-readable storage medium that can perform real-time correction of fisheye images according to the posture of an underwater robot to improve image quality.

[0005] A first aspect of this application provides a real-time fisheye image processing method, the method comprising:

[0006] When the fisheye camera is started, a coordinate transformation table from spherical coordinates to image coordinates is initialized based on the distortion table / distortion coefficients stored in the fisheye camera.

[0007] The region of interest is determined based on a preset field of view, and a first spherical coordinate table is obtained by initializing the image coordinates of the region of interest and the distortion table / distortion coefficients.

[0008] When the attitude change of the fisheye camera is detected, a rotation matrix is ​​calculated based on the real-time attitude of the fisheye camera, and the first spherical coordinate table is rotated in reverse according to the rotation matrix to obtain the second spherical coordinate table.

[0009] According to the coordinate transformation table, the second spherical coordinate table is converted into the target image coordinate table;

[0010] A real-time raw fisheye image is acquired, and the raw fisheye image is remapped according to the target image coordinate table to obtain the target fisheye image.

[0011] Optionally, the method further includes:

[0012] Determine if a pixel angle comparison table provided by the fisheye lens manufacturer exists;

[0013] When the pixel angle lookup table exists, the distortion table is obtained according to the pixel angle lookup table;

[0014] When the pixel angle lookup table is not available, a first black and white square image is captured, and distortion correction is performed on the first black and white square image to obtain the distortion coefficient.

[0015] Optionally, the method further includes:

[0016] Take a second black and white square image;

[0017] The distortion table / distortion coefficients are used to correct the distortion of the second black and white grid image to obtain a corrected black and white grid image;

[0018] The correction of the black and white grid image is verified.

[0019] When the calibration verification passes, the distortion table / distortion coefficients are confirmed to be correct.

[0020] If the calibration verification fails, the third black and white square image is re-captured, and a new distortion coefficient is obtained based on the third black and white square image, until the new distortion coefficient is correct.

[0021] Optionally, the step of initializing and establishing a coordinate transformation table from spherical coordinates to image coordinates based on the distortion table / distortion coefficients stored in the fisheye camera includes:

[0022] Obtain the focal length of the fisheye lens and the pixel value corresponding to the focal length;

[0023] Establish spherical coordinates based on the pixel values;

[0024] Obtain the projection mode of the fisheye camera;

[0025] Based on the spherical coordinates, the projection mode, the distortion table / distortion coefficients, and the sensor target image coordinates, an initial coordinate transformation table from spherical coordinates to image coordinates is established.

[0026] Optionally, the preset field of view includes a preset horizontal field of view and a preset vertical field of view, and determining the region of interest based on the preset field of view includes:

[0027] Using the principle of trigonometric functions, the width of the region of interest is calculated based on the focal length and the preset horizontal field of view.

[0028] Using the principle of trigonometric functions, the height of the region of interest is calculated based on the focal length and the preset vertical field of view.

[0029] The region of interest is determined based on the width and the height.

[0030] Optionally, the step of initializing the first spherical coordinate table based on the image coordinates of the region of interest and the distortion table / distortion coefficients includes:

[0031] An image coordinate table is initialized and generated based on the image coordinates of the region of interest;

[0032] The first spherical coordinate table is initialized based on the image coordinate table and the distortion table / distortion coefficients.

[0033] Optionally, the step of remapping the original fisheye image according to the target image coordinate table to obtain the target fisheye image includes:

[0034] For each pixel in the original fisheye image, find the corresponding target coordinates in the target image coordinate table;

[0035] Identify the key coordinates in the target coordinates, and represent the key coordinates and their corresponding interpolation weights in floating-point format;

[0036] The calculations involving coordinate rotation during the remapping process are represented in floating-point format;

[0037] Generate the target fisheye image based on the floating-point represented coordinates.

[0038] A second aspect of this application provides a real-time fisheye image processing apparatus, the apparatus comprising:

[0039] The first initialization module is used to initialize and establish a coordinate transformation table from spherical coordinates to image coordinates based on the distortion table / distortion coefficients stored in the fisheye camera when the fisheye camera is started.

[0040] The second initialization module is used to determine the region of interest according to the preset field of view angle, and to initialize the first spherical coordinate table according to the image coordinates of the region of interest and the distortion table / distortion coefficients.

[0041] The attitude calculation module is used to calculate a rotation matrix based on the real-time attitude of the fisheye camera when the attitude change of the fisheye camera is detected, and to perform a reverse rotation calculation on the first spherical coordinate table based on the rotation matrix to obtain a second spherical coordinate table.

[0042] The coordinate transformation module is used to convert the second spherical coordinate table into the target image coordinate table according to the coordinate transformation relationship table;

[0043] The remapping processing module is used to acquire the real-time raw fisheye image, and remap the raw fisheye image according to the target image coordinate table to obtain the target fisheye image.

[0044] A third aspect of this application provides a fisheye camera, which has a built-in electronic gimbal and a memory. The electronic gimbal is used to execute a computer program stored in the memory to implement the real-time fisheye image processing method.

[0045] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned fisheye image real-time processing method.

[0046] This application, upon startup of the fisheye camera, initializes a coordinate transformation table from spherical coordinates to image coordinates based on a distortion table / distortion coefficients. It determines the region of interest (ROI) based on a preset field of view and initializes a first spherical coordinate table using the ROI's image coordinates and the distortion table / distortion coefficients. As the fisheye camera's attitude changes, a rotation matrix is ​​calculated based on the real-time attitude, and the first spherical coordinate table is then rotated inversely using this matrix to obtain a second spherical coordinate table. The second spherical coordinate table is then converted to a target image coordinate table based on the coordinate transformation table. Finally, the acquired original fisheye image is remapped using the target image coordinate table to obtain the target fisheye image. This application can process fisheye images more accurately, achieving distortion correction and attitude correction. The establishment and use of the coordinate transformation table efficiently achieves the mapping from spherical coordinates to image coordinates. The application of real-time attitude calculation ensures that the target fisheye image reflects the camera's true attitude information. The final generated target fisheye image will have better visual effects and higher accuracy, suitable for various application scenarios requiring fisheye image processing. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 A flowchart illustrating the offline calibration method for fisheye cameras provided in this application embodiment;

[0049] Figure 2 A flowchart of a real-time fisheye image processing method provided in an embodiment of this application;

[0050] Figure 3 This is a schematic diagram of the structure of the real-time fisheye image processing device provided in the embodiments of this application;

[0051] Figure 4 This is a schematic diagram of the structure of a fisheye camera provided in an embodiment of this application. Specific Implementation

[0052] To better understand the above-mentioned objectives, features, and advantages of this application, the application will be described in detail below with reference to the accompanying drawings and specific embodiments. Unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0053] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein in the specification of this application is for the purpose of describing an embodiment in one alternative implementation and is not intended to be limiting of the application.

[0054] Example 1

[0055] Figure 1 This is a flowchart of an offline calibration method for a fisheye camera provided in an embodiment of this application. The offline calibration method for a fisheye camera specifically includes the following steps.

[0056] S11, determine if a pixel angle comparison table provided by the fisheye lens manufacturer exists.

[0057] Fisheye cameras use special lenses, which result in imaging methods different from traditional pinhole cameras. Fisheye cameras can capture images with a wider field of view, but objects in the image will be distorted.

[0058] When the fisheye camera is offline, check if the internal storage contains a pixel angle lookup table provided by the fisheye lens manufacturer. The pixel angle lookup table contains the mapping relationship between each pixel position (usually two-dimensional coordinates, such as (x, y)) and the corresponding incident ray angle (which may be a polar angle θ and an azimuth angle φ). The incident ray angle represents the direction of light before it enters the fisheye lens and is mapped to the pixel position.

[0059] If the pixel angle lookup table exists, execute S12; if the pixel angle lookup table does not exist, execute S13.

[0060] S12, Obtain the distortion table based on the pixel angle lookup table.

[0061] If the fisheye camera's internal storage contains a pixel angle lookup table, then the distortion table can be directly used to construct the distortion table, thus simplifying the calibration process.

[0062] In practice, each pixel coordinate (x, y) in the image is traversed. For each coordinate, a pixel angle lookup table is consulted to obtain the corresponding incident ray angle (θ, φ). The incident ray angle (θ, φ) and the distortion model of the fisheye lens are used to calculate the pixel coordinates to which the actual light rays should be mapped due to lens distortion. The polar angle θ and azimuth angle φ are converted to Cartesian coordinates (e.g., X', Y'), and a distortion correction model (e.g., isometric projection, isostatic projection, etc.) is used to obtain the corrected coordinates. For each pixel coordinate (x, y), the corrected coordinates and any other possible correction parameters (e.g., scaling factor, offset, etc.) are stored in the distortion table. This distortion table allows for quick retrieval of corrected pixel coordinates when needed, avoiding the need to calculate complex distortion correction models at runtime and improving the speed of real-time image processing.

[0063] S13, capture a first black and white grid image, and perform distortion correction on the first black and white grid image to obtain the distortion coefficient.

[0064] If the fisheye camera's internal storage does not contain a pixel angle reference table provided by the fisheye lens manufacturer, then the distortion coefficient needs to be obtained through camera calibration.

[0065] In practice, a fisheye camera is used to photograph the black-and-white grid calibration board, obtaining a black-and-white grid image. For ease of description below, this black-and-white grid image is referred to as the first black-and-white grid image. Corner points on the calibration board are detected from the first black-and-white grid image. The positions of the corner points in the image (pixel coordinates) and their positions in three-dimensional space (three-dimensional corner coordinates) are known. Using the known three-dimensional corner coordinates and corresponding pixel coordinates, along with the imaging model of the fisheye camera, an optimization algorithm (such as least squares method) is used to estimate the camera's intrinsic parameters (such as focal length and principal point coordinates) and distortion coefficients.

[0066] To verify and correct the accuracy of the distortion table or distortion coefficients, in an optional implementation, the method further includes:

[0067] Take a second black and white square image;

[0068] The distortion table / distortion coefficients are used to correct the distortion of the second black and white grid image to obtain a corrected black and white grid image;

[0069] The correction of the black and white grid image is verified.

[0070] When the calibration verification passes, the distortion table / distortion coefficients are confirmed to be correct.

[0071] If the calibration verification fails, the third black and white square image is re-captured, and a new distortion coefficient is obtained based on the third black and white square image, until the new distortion coefficient is correct.

[0072] A fisheye camera was used to photograph the black and white grid calibration board to obtain a second black and white grid image, which was used to verify the accuracy of the distortion table or distortion coefficients.

[0073] Based on the distortion table or distortion coefficients, distortion correction is performed on the second black-and-white grid image to obtain the corrected black-and-white grid image. The quality of the corrected black-and-white grid image is verified by measuring the straightness and angular accuracy of the black and white squares to determine if the distortion correction was successful. If the correction verification passes, meaning the quality of the corrected black-and-white grid image meets the requirements, then the distortion table or distortion coefficients can be confirmed as correct. If the correction verification fails, meaning the quality of the corrected black-and-white grid image does not meet the requirements, then a new black-and-white grid image (the third black-and-white grid image) needs to be captured to recalculate the distortion coefficients.

[0074] Repeat the above steps to extract corner points from the third black-and-white grid image and calculate new distortion coefficients using a camera calibration algorithm. Use these new distortion coefficients to correct the distortion of the new black-and-white grid image and verify the correction again. If the verification passes, the new distortion coefficients are confirmed to be correct; if the verification fails, continue repeating this process until the correct distortion coefficients are found.

[0075] It should be understood that determining the distortion table or distortion coefficients is an iterative process. Through continuous verification and correction, it can be ensured that the final distortion table or distortion coefficients are accurate and can be used to effectively correct distortion in images captured by fisheye cameras.

[0076] Example 2

[0077] Figure 2 This is a flowchart of a real-time fisheye image processing method provided in an embodiment of this application. The real-time fisheye image processing method specifically includes the following steps.

[0078] S21, when the fisheye camera is started, a coordinate transformation table from spherical coordinates to image coordinates is initialized based on the distortion table / distortion coefficients stored in the fisheye camera.

[0079] When the fisheye camera is detected to be running, in order to extract useful information from the image captured by the fisheye camera and perform further analysis or processing, it is necessary to load a distortion table or distortion coefficients from its storage space (such as internal memory or external memory card), and establish a transformation relationship from the spherical coordinates obtained from the fisheye camera to the pixel coordinates (image coordinates) on the image plane based on the distortion table / distortion coefficients.

[0080] Spherical coordinates are a three-dimensional coordinate system used to represent the position of a point relative to a reference sphere. Spherical coordinates include three components: radial distance (distance to the center of the sphere), polar angle (an angle measured from the positive z-axis), and azimuth angle (an angle measured from the positive x-axis in the xy-plane). Image coordinates are a coordinate system used to represent the position of pixels on a two-dimensional image plane.

[0081] In an optional implementation, the initialization of the coordinate transformation table from spherical coordinates to image coordinates based on the distortion table / distortion coefficients stored in the fisheye camera includes:

[0082] Obtain the focal length of the fisheye lens and the pixel value corresponding to the focal length;

[0083] Establish spherical coordinates based on the pixel values;

[0084] Obtain the projection mode of the fisheye camera;

[0085] Based on the spherical coordinates, the projection mode, the distortion table / distortion coefficients, and the sensor target image coordinates, an initial coordinate transformation table from spherical coordinates to image coordinates is established.

[0086] First, obtain the basic parameters of the fisheye lens, including the focal length and the corresponding pixel value. Focal length is a key parameter of a fisheye lens, describing the lens's ability to focus light and determining the sharpness and angle of view of the image. It can be obtained from the technical specifications provided by the fisheye lens manufacturer; the focal length is usually measured in millimeters (mm).

[0087] The pixel value corresponding to the focal length is the projected size of the focal length on the image sensor of the fisheye camera, which is usually related to the resolution and pixel size of the camera sensor.

[0088] Based on the acquired pixel values, a spherical coordinate system can be constructed or simulated. In the imaging model of a fisheye camera, due to the special design of the lens, light rays projected from three-dimensional space onto the image sensor undergo a special mapping relationship, similar to mapping a point in three-dimensional space onto a sphere. Therefore, establishing spherical coordinates using pixel values ​​can simulate this mapping relationship.

[0089] The projection mode of a fisheye camera determines how light is mapped from three-dimensional space to a two-dimensional image. Projection modes can include equidistant projection, equisolid angle projection, and others. The projection mode describes how light is distributed on a sphere and how spherical coordinates are mapped to image coordinates. Different projection modes of a fisheye camera use different trigonometric functions.

[0090] Sensor target surface image coordinates are a two-dimensional coordinate system of pixels on an image sensor, used to describe the position of each pixel in an image. Sensor target surface image coordinates are typically established based on the physical size and pixel density of the image sensor. Each pixel has a unique set of coordinates used for pixel location and access during image processing.

[0091] After obtaining the spherical coordinates, projection mode, distortion table / distortion coefficients, and sensor target image coordinates, we can begin establishing a coordinate transformation table from spherical coordinates to image coordinates. In practice, for a given spherical coordinate, it is converted to intermediate coordinates (e.g., normalized coordinates or perspective coordinates) according to the projection mode. The intermediate coordinates are then corrected using the distortion table or distortion coefficients to eliminate coordinate offsets or distortions caused by lens distortion. The corrected intermediate coordinates are then matched with the sensor target image coordinates to establish a coordinate transformation table from spherical coordinates to image coordinates. All spherical coordinates and their corresponding image coordinates are stored in the coordinate transformation table for quick lookup and use later.

[0092] The above-described optional implementation, by obtaining the focal length of the fisheye lens and its corresponding pixel values, can more accurately establish the mapping relationship between spherical coordinates and image coordinates. Distortion correction using a distortion table and / or distortion coefficients can eliminate image distortion caused by the fisheye lens, further improving the accuracy of coordinate transformation. Because distortion correction and projection modes are considered, the established coordinate transformation table can better adapt to the imaging characteristics of fisheye cameras under different scenes and conditions, enhancing robustness and adaptability. By establishing an accurate coordinate transformation table, pixel coordinates in images captured by a fisheye camera can be converted to spherical coordinates in three-dimensional space, or spherical coordinates can be converted to image coordinates.

[0093] S22, determine the region of interest according to the preset field of view angle, and initialize the first spherical coordinate table according to the image coordinates of the region of interest and the distortion table / distortion coefficient.

[0094] Fisheye lenses, due to their unique design, can capture an extremely wide field of view, containing much information that is unnecessary or irrelevant to focus on. By identifying the region of interest (ROI), the image size can be reduced, thereby decreasing the computational load and time consumption of subsequent processing steps. The ROI is typically much smaller than the original image, making it easier to store and transmit.

[0095] Furthermore, the original fisheye image may contain noise and interference information unrelated to the region of interest (ROI), which can affect the accuracy of subsequent processing. By identifying the ROI, irrelevant noise and interference can be removed, improving the accuracy of the processing results.

[0096] The preset field of view includes a preset horizontal field of view and a preset vertical field of view, defining the boundary of the region of interest (ROI) in the image. The horizontal and vertical field of view are input by the user to the electronic pan-tilt unit. Based on the preset field of view and the camera's internal parameters, the boundary coordinates of the ROI on the image plane are calculated. The boundary coordinates define the four corner points or edges of the ROI. The ROI region can be marked on the image, and visualized by drawing rectangles or other shapes on the image.

[0097] In an optional implementation, determining the region of interest based on a preset field of view includes:

[0098] Using the principle of trigonometric functions, the width of the region of interest is calculated based on the focal length and the preset horizontal field of view.

[0099] Using the principle of trigonometric functions, the height of the region of interest is calculated based on the focal length and the preset vertical field of view.

[0100] The region of interest is determined based on the width and the height.

[0101] Assuming the image sensor width is W_sensor (in pixels), the preset horizontal field of view is HFOV (in degrees), the focal length is f (in millimeters), and that the image sensor and lens are matched (i.e., there is no additional scaling factor), the tangent function from trigonometric functions is used to calculate the angle corresponding to each pixel in the horizontal direction. This angle, along with the preset horizontal field of view, is used to calculate the width of the ROI, Crop_w (in pixels).

[0102] Assuming the image sensor width W_sensor = 1920 pixels, the horizontal field of view HFOV = 70 degrees, and the focal length f = 2.8 mm, then the horizontal angle corresponding to each pixel is pixel_angle_horizontal = math.atan(0.5*W_sensor / f)*(180 / math.pi). Assuming the ROI occupies the entire horizontal field of view, the ROI width Crop_w = int(W_sensor*(scaled_HFOV / HFOV)). Similarly, the ROI height Crop_h is calculated using the vertical field of view (VFOV) and the image sensor height (H_sensor, in pixels).

[0103] Assuming the image sensor height H_sensor = 1080 pixels, vertical field of view VFOV = 60 degrees, and focal length f = 2.8 mm, then the vertical angle corresponding to each pixel is pixel_angle_vertical = math.atan(0.5*H_sensor / f)*(180 / math.pi). Assuming the ROI occupies the entire vertical field of view, the height of the ROI is Crop_h = int(H_sensor*(scaled_VFOV / VFOV)).

[0104] Determining the width and height of the Region of Interest (ROI) in pixels determines its position on the image. The ROI will be located in the center of the image, but it can also be placed in other locations as needed.

[0105] Using the center of the image sensor as the center, the sensor's X-axis as the X-axis, and the sensor's Y-axis as the Y-axis, a rectangular region with a width of Crop_w and a height of Crop_h is extracted from the center of the sensor target image. This rectangular region is the region of interest.

[0106] In an optional implementation, the step of initializing the first spherical coordinate table based on the image coordinates of the region of interest and the distortion table / distortion coefficients includes:

[0107] An image coordinate table is initialized and generated based on the image coordinates of the region of interest;

[0108] The first spherical coordinate table is initialized based on the image coordinate table and the distortion table / distortion coefficients.

[0109] Obtain the image coordinates of all pixels in the region of interest (ROI). Based on the image coordinates of all pixels, an initialized image coordinate table (CROP-2D coordinate table) can be obtained. Each element in the image coordinate table is a two-dimensional coordinate point, representing a pixel position within the ROI.

[0110] For each coordinate point in the image coordinate table, distortion correction is performed using an intrinsic parameter matrix established using a distortion table or distortion coefficients. This correction process transforms the image coordinates into corrected image coordinates to eliminate or mitigate the effects of distortion.

[0111] After obtaining the corrected image coordinates, the two-dimensional coordinate points are converted into three-dimensional spherical coordinates using a fisheye camera projection model (such as equidistant projection or equal-area projection). This involves mapping the two-dimensional coordinates to points on a three-dimensional unit sphere. The calculated spherical coordinates are stored in a table, resulting in the first spherical coordinate table (CROP-3D coordinate table). Each element in the first spherical coordinate table is a three-dimensional coordinate point, representing the position of a point within the ROI in three-dimensional spherical space.

[0112] S23, when the attitude change of the fisheye camera is detected, the rotation matrix is ​​calculated based on the real-time attitude of the fisheye camera, and the first spherical coordinate table is rotated in reverse according to the rotation matrix to obtain the second spherical coordinate table.

[0113] Due to changes in the attitude of the unmanned imaging platform, the relative position between the fisheye lens and the image sensor also changes, causing a shift in the angle and position of the light received by the image sensor, thus resulting in image distortion. Therefore, when the fisheye camera's attitude changes, to maintain the accuracy and consistency of image processing, the coordinate system related to the camera's attitude needs to be transformed accordingly. That is, a rotation matrix needs to be calculated based on the fisheye camera's real-time attitude (usually expressed as a rotation angle or quaternion), and then this rotation matrix is ​​used to reverse the previously determined first spherical coordinate table to eliminate the rotation introduced by the attitude change of the unmanned imaging platform, thereby obtaining a second spherical coordinate table that matches the current camera attitude.

[0114] Sensors (such as IMUs, gyroscopes, etc.) are used to acquire the real-time attitude of the unmanned imaging platform (e.g., drones, other flying devices, or underwater robots). The real-time attitude includes pitch, yaw, and roll.

[0115] Based on the acquired real-time attitude, a 3x3 rotation matrix can be calculated. This rotation matrix describes how a point is rotated from the world coordinate system (or camera coordinate system) to the current attitude coordinate system of the unmanned imaging platform. These three rotations are applied in the ZYX order (yaw, pitch, and roll). Each rotation can be represented by a separate rotation matrix.

[0116] Yaw angle (rotation about the Z-axis):

[0117] Pitch (rotation about the Y-axis):

[0118] Roll angle (rotation about the X-axis):

[0119] Multiplying the above three together, we can obtain the final 3x3 rotation matrix.

[0120] Next, a rotation matrix is ​​used to perform a reverse rotation calculation on each coordinate point in the first spherical coordinate table. "Reverse rotation" refers to transforming the coordinate point from its current position in the camera's pose back to its initial position to maintain consistency in subsequent processing. This reverse rotation can be achieved by multiplying each coordinate point in the first spherical coordinate table by the rotation matrix.

[0121] By performing reverse rotation calculations on all coordinate points in the first spherical coordinate table, a new spherical coordinate table, the second spherical coordinate table, can be obtained. The coordinate points in the second spherical coordinate table have been adjusted according to the real-time attitude of the camera, thus maintaining consistency with the current camera attitude in subsequent processing.

[0122] S24, according to the coordinate transformation table, convert the second spherical coordinate table into the target image coordinate table.

[0123] Converting the second spherical coordinate table (the corrected CROP-3D coordinate table) to the target image coordinate table (New-2D_tab) refers to the mapping from three-dimensional spherical coordinates to two-dimensional image coordinates.

[0124] The image coordinates corresponding to each coordinate point in the coordinate transformation table and the second spherical coordinate table are found and stored in a new table to obtain the target image coordinate table. Each element in the target image coordinate table is a two-dimensional coordinate point, representing the position of the original spherical coordinate point on the target image.

[0125] S25, acquire the real-time raw fisheye image, and remap the raw fisheye image according to the target image coordinate table to obtain the target fisheye image.

[0126] Since the target image coordinate table (New-2D_tab) contains the coordinate mapping relationship from the fisheye image to the target image (usually the corrected image), after capturing the real-time raw fisheye image using a fisheye camera, each pixel of the raw fisheye image is traversed, and for each pixel, its corresponding coordinates in the target image coordinate table are found. An interpolation algorithm (such as bilinear interpolation, cubic spline interpolation, etc.) is used to calculate the pixel value at the corresponding coordinates in the target image. All the interpolated pixel values ​​are combined to form the target fisheye image. The target fisheye image is then encoded and output in the required format.

[0127] Because the edge view pixels of fisheye images are relatively sparse, edge view interpolation has jagged edges, and fine interpolation is computationally expensive. Therefore, in order to balance the accuracy and computational power of remapping processing, in an optional implementation, the remapping processing of the original fisheye image according to the target image coordinate table to obtain the target fisheye image includes:

[0128] For each pixel in the original fisheye image, find the corresponding target coordinates in the target image coordinate table;

[0129] Identify the key coordinates in the target coordinates, and represent the key coordinates and their corresponding interpolation weights in floating-point format;

[0130] The calculations involving coordinate rotation during the remapping process are represented in floating-point format;

[0131] Generate the target fisheye image based on the floating-point represented coordinates.

[0132] Edge detection algorithms, such as Canny, Sobel, or Laplacian, can be used to identify edge regions in the original fisheye image. The target coordinates within these edge regions are used as key coordinates. Representing these key coordinates and their corresponding interpolation weights in floating-point format increases interpolation accuracy and reduces jagged edges caused by interpolation at edge views. Because distortion and perspective effects may be more pronounced at edge views, higher computational accuracy is needed to ensure the smoothness and accuracy of the interpolation results. Bilinear interpolation, cubic spline interpolation, or higher-order interpolation algorithms can be applied to the edge regions to further reduce jagged edges.

[0133] Using floating-point numbers for key steps in rotation calculations can reduce the precision loss caused by rotation.

[0134] During the conversion of the second spherical coordinates to the target image coordinates, the coordinate points may be mapped to locations outside the target fisheye image or to non-integer pixel locations. Check if the coordinates are within the boundaries of the original fisheye image. For coordinate points mapped outside the fisheye image, you can choose to ignore them or map them to the fisheye image boundary.

[0135] After completing the remapping and interpolation processing, the result is converted from a double-precision floating-point value back to an image data type suitable for storage or display (such as an 8-bit or 16-bit unsigned integer). The processed image data is then encoded into an appropriate image format (such as JPEG, PNG, TIFF, etc.) for storage or transmission. The encoded target fisheye image is saved to a file on disk or displayed on the screen via a graphical user interface (GUI).

[0136] The above-described optional implementation can significantly improve the visual quality of the remapped fisheye image by using double-precision floating-point values ​​and floating-point increments for interpolation calculations throughout the entire processing, especially in the edge view regions. This helps to reduce artifacts and jagged edges caused by interpolation, making the image look smoother and more realistic.

[0137] It should be noted that in the above embodiments, the offline calibration of the fisheye camera (S11-S13) only needs to be performed once. After the fisheye camera is started, the initialization process (S21-S22) is executed once first. When the attitude of the fisheye camera changes, S23-S24 is executed, and the remapping difference process (S25) is continuously executed and output.

[0138] The principle behind this embodiment is that by accurately correcting fisheye image distortion caused by changes in the posture of the unmanned imaging platform, the quality and accuracy of fisheye images can be significantly improved. Furthermore, by combining offline calibration with online remapping, not only are complex operational procedures simplified, but the practicality and stability of the system are also greatly enhanced.

[0139] Example 3

[0140] Figure 3 This is a structural diagram of the fisheye image real-time processing device provided in the embodiments of this application.

[0141] In some embodiments, the real-time fisheye image processing device 30 may include a plurality of functional modules composed of computer program segments. The computer program for each program segment in the real-time fisheye image processing device 30 may be stored in the memory of the fisheye camera and executed by at least one processor to perform (see details). Figure 2 (Description) Functionality for real-time processing of fisheye images.

[0142] In this embodiment, the fisheye image real-time processing device 30 can be divided into multiple functional modules according to its functions. These functional modules may include: an offline calibration module 301, a first initialization module 302, a second initialization module 303, a pose calculation module 304, a coordinate transformation module 305, and a remapping processing module 306. The term "module" in this application refers to a series of computer program segments that can be executed by at least one processor and perform a fixed function, and which are stored in memory. In this embodiment, the functions of each module will be detailed in subsequent embodiments.

[0143] The offline calibration module 301 is used to obtain the distortion table / distortion coefficients.

[0144] The first initialization module 302 is used to initialize and establish a coordinate transformation table from spherical coordinates to image coordinates based on the distortion table / distortion coefficients stored in the fisheye camera when the fisheye camera is started.

[0145] The second initialization module 303 is used to determine the region of interest according to a preset field of view, and initialize the first spherical coordinate table according to the image coordinates of the region of interest and the distortion table / distortion coefficients.

[0146] The attitude calculation module 304 is used to calculate a rotation matrix based on the real-time attitude of the fisheye camera when the attitude change of the fisheye camera is detected, and to perform a reverse rotation calculation on the first spherical coordinate table based on the rotation matrix to obtain a second spherical coordinate table.

[0147] The coordinate transformation module 305 is used to convert the second spherical coordinate table into a target image coordinate table according to the coordinate transformation relationship table;

[0148] The remapping processing module 306 is used to acquire the real-time original fisheye image, and perform remapping processing on the original fisheye image according to the target image coordinate table to obtain the target fisheye image.

[0149] It should be understood that the various variations and specific embodiments of the fisheye image real-time processing method provided in the above embodiments are also applicable to the fisheye image real-time processing device in this embodiment. Through the detailed description of the fisheye image real-time processing method described above, those skilled in the art can clearly understand the implementation process of the fisheye image real-time processing device in this embodiment. For the sake of brevity, it will not be described in detail here.

[0150] Example 3

[0151] This embodiment provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the steps described in the above-described real-time fisheye image processing method embodiment, for example... Figure 1 S11-S13 and / or shown Figure 2 S21-S25 are shown.

[0152] S21, when the fisheye camera is started, a coordinate transformation table from spherical coordinates to image coordinates is initialized based on the distortion table / distortion coefficients stored in the fisheye camera.

[0153] S22, determine the region of interest according to the preset field of view angle, and initialize the first spherical coordinate table according to the image coordinates of the region of interest and the distortion table / distortion coefficient.

[0154] S23, when the attitude change of the fisheye camera is detected, the rotation matrix is ​​calculated based on the real-time attitude of the fisheye camera, and the first spherical coordinate table is rotated in reverse according to the rotation matrix to obtain the second spherical coordinate table.

[0155] S24, according to the coordinate transformation table, convert the second spherical coordinate table into the target image coordinate table.

[0156] S25, acquire the real-time raw fisheye image, and remap the raw fisheye image according to the target image coordinate table to obtain the target fisheye image.

[0157] Alternatively, when the computer program is executed by the processor, it implements the functions of each module / unit in the above-described device embodiments, for example... Figure 3 Modules 301-306 in the document.

[0158] The offline calibration module 301 is used to obtain the distortion table / distortion coefficients.

[0159] The first initialization module 302 is used to initialize and establish a coordinate transformation table from spherical coordinates to image coordinates based on the distortion table / distortion coefficients stored in the fisheye camera when the fisheye camera is started.

[0160] The second initialization module 303 is used to determine the region of interest according to a preset field of view, and initialize the first spherical coordinate table according to the image coordinates of the region of interest and the distortion table / distortion coefficients.

[0161] The attitude calculation module 304 is used to calculate a rotation matrix based on the real-time attitude of the fisheye camera when the attitude change of the fisheye camera is detected, and to perform a reverse rotation calculation on the first spherical coordinate table based on the rotation matrix to obtain a second spherical coordinate table.

[0162] The coordinate transformation module 305 is used to convert the second spherical coordinate table into a target image coordinate table according to the coordinate transformation relationship table;

[0163] The remapping processing module 306 is used to acquire the real-time original fisheye image, and perform remapping processing on the original fisheye image according to the target image coordinate table to obtain the target fisheye image.

[0164] Example 4

[0165] See Figure 4 The diagram shown is a structural schematic of a fisheye camera provided in an embodiment of this application. In a preferred embodiment of this application, the fisheye camera 40 includes a memory 401, an electronic pan-tilt unit 402, and at least one communication bus 403.

[0166] Those skilled in the art should understand that Figure 4 The structure of the fisheye camera shown does not constitute a limitation of the embodiments of this application. It can be a bus structure or a star structure. The fisheye camera 40 may also include more or fewer other hardware or software than shown, or different component arrangements.

[0167] In some embodiments, the fisheye camera 40 is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions. Its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), programmable gate arrays (FPGAs), digital processors, and embedded devices. The fisheye camera 40 may also include a client device, which includes, but is not limited to, any electronic product capable of human-computer interaction with a client via a keyboard, mouse, remote control, touchpad, or voice control device, such as a personal computer, tablet computer, smartphone, or digital camera.

[0168] The fisheye camera 40 described is merely an example. Other existing or future electronic products that are suitable for this application should also be included within the scope of protection of this application and are incorporated herein by reference.

[0169] In some embodiments, the memory 401 stores a computer program that, when executed by the electronic pan-tilt unit 402, implements all or part of the steps in the real-time fisheye image processing method described above. The memory 401 includes a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0170] Furthermore, the computer-readable storage medium may primarily include a program storage area and a data storage area, wherein the program storage area may store the operating system, at least one application required for a function, etc.; and the data storage area may store data created based on the use of blockchain nodes, etc.

[0171] In some embodiments, the electronic gimbal 402 is the control unit of the fisheye camera 40, connecting each component of the fisheye camera 40 via various interfaces and lines. It executes programs or modules stored in the memory 401 and calls data stored in the memory 401 to perform various functions and process data of the fisheye camera 40. For example, when the electronic gimbal 402 executes the computer program stored in the memory, it implements all or part of the steps of the real-time fisheye image processing method described in this application embodiment; or it implements all or part of the functions of the real-time fisheye image processing device. The electronic gimbal 402 can be composed of integrated circuits, such as a single-packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips.

[0172] In some embodiments, the at least one communication bus 403 is configured to enable communication between the memory 401 and the electronic gimbal 402, etc.

[0173] Although not shown, the fisheye camera 40 may also include a power supply (such as a battery) to power each component. Preferably, the power supply can be logically connected to the electronic gimbal 402 through a power management device, thereby enabling functions such as charging, discharging, and power consumption management. The power supply may also include one or more DC or AC power supplies, recharging devices, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The fisheye camera 40 may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0174] The integrated unit implemented as a software functional module described above can be stored in a computer-readable storage medium. This software functional module, stored in a storage medium, includes several instructions to cause a fisheye camera (which may be a personal computer, a fisheye camera, or a network device, etc.) or a processor to execute portions of the methods described in each embodiment of this application.

[0175] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0176] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0177] Furthermore, in each embodiment of this application, the functional modules can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0178] It will be apparent to those skilled in the art that this application is not limited to the details of the exemplary embodiments described above, and that it can be implemented in other specific forms without departing from the spirit or essential characteristics of this application. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of this application is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within this application. No reference numerals in the claims should be construed as limiting the scope of the claims. Furthermore, it is clear that the word "comprising" does not exclude other elements or, and the singular does not exclude the plural. Multiple elements or devices recited in the specification may also be implemented by a single element or device through software or hardware. The terms "first," "second," etc., are used to indicate names and do not indicate any particular order.

[0179] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit it. Although this application has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of this application without departing from the spirit and scope of the technical solutions of this application.

Claims

1. A real-time fisheye image processing method, characterized in that, The method includes: When the fisheye camera is started, a coordinate transformation table from spherical coordinates to image coordinates is initialized based on the distortion table / distortion coefficients stored in the fisheye camera. The region of interest is determined based on a preset field of view, and a first spherical coordinate table is obtained by initializing the image coordinates of the region of interest and the distortion table / distortion coefficients. When the attitude change of the fisheye camera is detected, a rotation matrix is ​​calculated based on the real-time attitude of the fisheye camera, and the first spherical coordinate table is rotated in reverse according to the rotation matrix to obtain the second spherical coordinate table. The rotation matrix is ​​used to rotate a point from the camera coordinate system to the current attitude coordinate system. According to the coordinate transformation table, the second spherical coordinate table is converted into the target image coordinate table; A real-time raw fisheye image is acquired, and the raw fisheye image is remapped according to the target image coordinate table to obtain the target fisheye image.

2. The real-time fisheye image processing method as described in claim 1, characterized in that, The method further includes: Determine if a pixel angle comparison table provided by the fisheye lens manufacturer exists; When the pixel angle lookup table exists, the distortion table is obtained according to the pixel angle lookup table; When the pixel angle lookup table is not available, a first black and white square image is captured, and distortion correction is performed on the first black and white square image to obtain the distortion coefficient.

3. The real-time fisheye image processing method as described in claim 2, characterized in that, The method further includes: Take a second black and white square image; The distortion table / distortion coefficients are used to correct the distortion of the second black and white grid image to obtain a corrected black and white grid image; The correction of the black and white grid image is verified. When the calibration verification passes, the distortion table / distortion coefficients are confirmed to be correct. If the calibration verification fails, the third black and white square image is re-captured, and a new distortion coefficient is obtained based on the third black and white square image, until the new distortion coefficient is correct.

4. The real-time fisheye image processing method as described in claim 1, characterized in that, The process of initializing and establishing a coordinate transformation table from spherical coordinates to image coordinates based on the distortion table / distortion coefficients stored in the fisheye camera includes: Obtain the focal length of the fisheye lens and the pixel value corresponding to the focal length; Establish spherical coordinates based on the pixel values; Obtain the projection mode of the fisheye camera; Based on the spherical coordinates, the projection mode, the distortion table / distortion coefficients, and the sensor target image coordinates, an initial coordinate transformation table from spherical coordinates to image coordinates is established.

5. The real-time fisheye image processing method as described in claim 4, characterized in that, The preset field of view includes a preset horizontal field of view and a preset vertical field of view, and determining the region of interest based on the preset field of view includes: Using the principle of trigonometric functions, the width of the region of interest is calculated based on the focal length and the preset horizontal field of view. Using the principle of trigonometric functions, the height of the region of interest is calculated based on the focal length and the preset vertical field of view. The region of interest is determined based on the width and the height.

6. The real-time fisheye image processing method according to any one of claims 1 to 4, characterized in that, The process of initializing the first spherical coordinate table based on the image coordinates of the region of interest and the distortion table / distortion coefficients includes: An image coordinate table is initialized and generated based on the image coordinates of the region of interest; The first spherical coordinate table is initialized based on the image coordinate table and the distortion table / distortion coefficients.

7. The real-time fisheye image processing method as described in claim 1, characterized in that, The step of remapping the original fisheye image according to the target image coordinate table to obtain the target fisheye image includes: For each pixel in the original fisheye image, find the corresponding target coordinates in the target image coordinate table; Identify the key coordinates in the target coordinates, and represent the key coordinates and their corresponding interpolation weights in floating-point format; The calculations involving coordinate rotation during the remapping process are represented in floating-point format; Generate the target fisheye image based on the floating-point represented coordinates.

8. A real-time fisheye image processing device, characterized in that, The device includes: The first initialization module is used to initialize and establish a coordinate transformation table from spherical coordinates to image coordinates based on the distortion table / distortion coefficients stored in the fisheye camera when the fisheye camera is started. The second initialization module is used to determine the region of interest according to the preset field of view angle, and to initialize the first spherical coordinate table according to the image coordinates of the region of interest and the distortion table / distortion coefficients. The attitude calculation module is used to calculate a rotation matrix based on the real-time attitude of the fisheye camera when the attitude change of the fisheye camera is detected, and to perform a reverse rotation calculation on the first spherical coordinate table based on the rotation matrix to obtain a second spherical coordinate table. The rotation matrix is ​​used to rotate a point from the camera coordinate system to the current attitude coordinate system. The coordinate transformation module is used to convert the second spherical coordinate table into the target image coordinate table according to the coordinate transformation relationship table; The remapping processing module is used to acquire the real-time raw fisheye image, and remap the raw fisheye image according to the target image coordinate table to obtain the target fisheye image.

9. A fisheye camera, characterized in that, The fisheye camera has a built-in electronic gimbal and a memory. The electronic gimbal is used to execute the computer program stored in the memory to implement the fisheye image real-time processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is executed by the processor, it implements the real-time fisheye image processing method according to any one of claims 1 to 7.

Images