Fisheye image real-time processing method and device, fisheye camera and storage medium

Abstract

The invention relates to the technical field of image processing, and provides a fisheye image real-time processing method and device, a fisheye camera and a storage medium. When the fisheye camera is started, a coordinate conversion relation table from spherical coordinates to image coordinates is initialized and established according to a distortion table / distortion coefficient, a region of interest is determined according to a preset field angle, and a first spherical coordinate table is obtained according to the image coordinates of the region of interest and the distortion table / distortion coefficient; when the posture of the fisheye camera changes, a rotation matrix is obtained through calculation according to the real-time posture, a second spherical coordinate table is obtained through reverse rotation calculation of the first spherical coordinate table according to the rotation matrix, and the second spherical coordinate table is converted into a target image coordinate table according to the coordinate conversion relation table; and performing remapping processing on the obtained original fisheye image according to the target image coordinate table to obtain a target fisheye image. According to the invention, distortion correction can be carried out on the fisheye image in real time according to the attitude, and the image quality of the fisheye image is improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method and device for real-time processing of fisheye images, a fisheye camera and a storage medium. Background Art

[0002] Electronic PTZ has been widely used in high-end network cameras. Through program settings, the viewing angle and zoom of the lens are controlled inside the camera, thereby realizing image capture and transmission.

[0003] However, when using an electronic gimbal mounted on an unmanned shooting platform (e.g., a drone, other flying equipment, or an underwater robot, etc.) in combination with a fisheye lens for aerial or underwater shooting, the fisheye image distortion problem caused by the posture change of the unmanned shooting platform is often encountered. This distortion affects the quality of the captured image. Summary of the invention

[0004] In view of the above, it is necessary to propose a real-time fisheye image processing method, device, fisheye camera and computer-readable storage medium, which can correct the fisheye image in real time according to the posture of the underwater robot to improve the image quality.

[0005] A first aspect of the present application provides a method for real-time processing of fisheye images, the method comprising:

[0006] When the fisheye camera is started, a coordinate conversion relationship table from spherical coordinates to image coordinates is initialized and established according to the distortion table / distortion coefficient stored in the fisheye camera;

[0007] Determine a region of interest according to a preset field of view angle, and obtain a first spherical coordinate table by initializing the image coordinates of the region of interest and the distortion table / distortion coefficient;

[0008] When a change in the posture of the fisheye camera is monitored, a rotation matrix is ​​calculated according to the real-time posture of the fisheye camera, and a reverse rotation calculation is performed on the first spherical coordinate table according to the rotation matrix to obtain a second spherical coordinate table;

[0009] According to the coordinate conversion relationship table, converting the second spherical coordinate table into a target image coordinate table;

[0010] A real-time original fisheye image is acquired, and a remapping process is performed on the original fisheye image according to the target image coordinate table to obtain a target fisheye image.

[0011] Optionally, the method further includes:

[0012] Determine whether there is a pixel angle comparison table provided by the fisheye lens manufacturer;

[0013] When the pixel angle comparison table exists, obtaining the distortion table according to the pixel angle comparison table;

[0014] When the pixel angle comparison table does not exist, a first black and white checkered image is photographed, and distortion correction is performed on the first black and white checkered image to obtain the distortion coefficient.

[0015] Optionally, the method further includes:

[0016] Taking a second black and white checkered image;

[0017] Using the distortion table / distortion coefficient, the second black and white checkered image is subjected to distortion correction to obtain a corrected black and white checkered image;

[0018] Performing correction verification on the corrected black and white checkered image;

[0019] When the calibration verification passes, it is determined that the distortion table / distortion coefficient is correct;

[0020] When the correction verification fails, the third black and white checkered image is re-photographed, and a new distortion coefficient is obtained based on the third black and white checkered image until the new distortion coefficient is correct.

[0021] Optionally, the initializing and establishing a coordinate conversion relationship table from spherical coordinates to image coordinates according to the distortion table / distortion coefficient stored in the fisheye camera includes:

[0022] Obtaining the focal length of the fisheye lens and the pixel value corresponding to the focal length;

[0023] Establishing spherical coordinates according to the pixel values;

[0024] Obtaining a projection mode of the fisheye camera;

[0025] A coordinate conversion relationship table from spherical coordinates to image coordinates is initialized and established according to the spherical coordinates, the projection mode, the distortion table / distortion coefficient and the sensor target surface image coordinates.

[0026] Optionally, the preset field of view angle includes a preset horizontal field of view angle and a preset vertical field of view angle, and determining the region of interest according to the preset field of view angle includes:

[0027] Using the trigonometric principle, the width of the region of interest is calculated according to the focal length and the preset horizontal field of view angle;

[0028] Using the trigonometric principle, the height of the region of interest is calculated according to the focal length and the preset vertical field of view angle;

[0029] A region of interest is determined according to the width and the height.

[0030] Optionally, the initializing the first spherical coordinate table according to the image coordinates of the region of interest and the distortion table / distortion coefficients includes:

[0031] Initialize and generate an image coordinate table according to the image coordinates of the region of interest;

[0032] A first spherical coordinate table is initialized according to the image coordinate table and the distortion table / distortion coefficient.

[0033] Optionally, remapping the original fisheye image according to the target image coordinate table to obtain the target fisheye image includes:

[0034] For each pixel in the original fisheye image, searching for the corresponding target coordinates in the target image coordinate table;

[0035] Identify key coordinates in the target coordinates, and represent the key coordinates and corresponding interpolation weights in floating point form;

[0036] The calculations involving coordinate rotation in the remapping process are expressed in floating point format;

[0037] Generate a target fisheye image based on the floating-point coordinates.

[0038] A second aspect of the present application provides a fisheye image real-time processing device, the device comprising:

[0039] A first initialization module is used to initialize and establish a coordinate conversion relationship table from spherical coordinates to image coordinates according to the distortion table / distortion coefficient stored in the fisheye camera when the fisheye camera is started;

[0040] A second initialization module is used to determine a region of interest according to a preset field of view angle, and to initialize a first spherical coordinate table according to the image coordinates of the region of interest and the distortion table / distortion coefficient;

[0041] a posture calculation module, for calculating a rotation matrix according to the real-time posture of the fisheye camera when a posture change of the fisheye camera is detected, and performing reverse rotation calculation on the first spherical coordinate table according to the rotation matrix to obtain a second spherical coordinate table;

[0042] A coordinate conversion module, used for converting the second spherical coordinate table into a target image coordinate table according to the coordinate conversion relationship table;

[0043] The remapping processing module is used to obtain a real-time original fisheye image, and remap the original fisheye image according to the target image coordinate table to obtain a target fisheye image.

[0044] A third aspect of the present application provides a fisheye camera, wherein the fisheye camera has a built-in electronic pan / tilt platform and a memory, and the electronic pan / tilt platform is used to implement the real-time fisheye image processing method when executing a computer program stored in the memory.

[0045] A fourth aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the real-time fisheye image processing method is implemented.

[0046] When the fisheye camera is started, the application establishes a coordinate conversion relationship table from spherical coordinates to image coordinates according to the distortion table / distortion coefficient initialization, determines the region of interest according to the preset field of view angle, and obtains the first spherical coordinate table according to the image coordinates of the region of interest and the distortion table / distortion coefficient initialization. When the posture of the fisheye camera changes, the rotation matrix is ​​calculated according to the real-time posture, and the first spherical coordinate table is reversely rotated according to the rotation matrix to obtain the second spherical coordinate table, and the second spherical coordinate table is converted into a target image coordinate table according to the coordinate conversion relationship table, and the original fisheye image obtained is remapped according to the target image coordinate table to obtain the target fisheye image. The application can process fisheye images more accurately and realize distortion correction and posture correction. By establishing and using the coordinate conversion relationship table, the mapping of spherical coordinates to image coordinates can be efficiently realized. The application of real-time posture calculation enables the target fisheye image to reflect the real posture information of the camera. The target fisheye image finally generated will have better visual effects and higher accuracy, and is suitable for various application scenarios that require processing fisheye images. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0048] Figure 1 A flow chart of a fisheye camera offline calibration method provided in an embodiment of the present application;

[0049] Figure 2 A flowchart of a method for real-time processing of fisheye images provided in an embodiment of the present application;

[0050] Figure 3 A schematic diagram of the structure of a real-time fisheye image processing device provided in an embodiment of the present application;

[0051] Figure 4 A schematic diagram of the structure of a fisheye camera provided in an embodiment of the present application. Specific embodiments

[0052] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the present application is described in detail below in conjunction with the accompanying drawings and specific embodiments. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other.

[0053] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present application belongs. The terms used herein in the specification of the present application are only for the purpose of describing the embodiments in an optional embodiment and are not intended to limit the present application.

[0054] Embodiment 1

[0055] Figure 1 1 is a flow chart of a fisheye camera offline calibration method provided in an embodiment of the present application. The fisheye camera offline calibration method specifically includes the following steps.

[0056] S11, determining whether there is a pixel angle comparison table provided by a fisheye lens manufacturer.

[0057] Fisheye cameras use special lenses that are different from traditional pinhole cameras in that they can capture images with a wider viewing angle, but objects in the image will be distorted.

[0058] When the fisheye camera is offline, check whether the internal storage space has a pixel angle comparison table provided by the fisheye lens manufacturer. The pixel angle comparison table contains the mapping relationship between each pixel position (usually a two-dimensional coordinate, such as (x, y) and the corresponding incident light angle (which may be a polar angle θ and an azimuth angle φ). The incident light angle represents the direction of the light before it enters the fisheye lens and is mapped to the pixel position.

[0059] When the pixel angle comparison table exists, execute S12; when the pixel angle comparison table does not exist, execute S13.

[0060] S12, obtaining a distortion table according to the pixel angle comparison table.

[0061] If a pixel angle comparison table is stored in the internal storage space of the fisheye camera, the pixel angle comparison table can be directly used to construct a distortion table, thereby simplifying the calibration process.

[0062] In specific implementation, each pixel coordinate (x, y) in the image is traversed, and for each coordinate, the pixel angle comparison table is searched to obtain the corresponding incident light angle (θ, φ). The incident light angle (θ, φ) and the distortion model of the fisheye lens are used to calculate the pixel coordinates to which the actual light should be mapped due to lens distortion. The polar angle θ and the azimuth angle φ are converted into coordinates in a Cartesian coordinate system (such as X', Y'), and a distortion correction model (such as equidistant projection, equistereoscopic projection, etc.) is used to obtain the corrected coordinates. For each pixel coordinate (x, y), the corrected coordinates and other correction parameters that may be required (such as scaling factor, offset, etc.) are stored in the distortion table. A distortion table is constructed, and the corrected pixel coordinates can be quickly obtained by looking up the distortion table when needed, avoiding the calculation of complex distortion correction models at runtime and improving the speed of real-time image processing.

[0063] S13, photographing a first black and white checkered image, and performing distortion correction on the first black and white checkered image to obtain a distortion coefficient.

[0064] If the internal storage space of the fisheye camera does not store the pixel angle comparison table provided by the fisheye lens manufacturer, then the distortion coefficient needs to be obtained through camera calibration.

[0065] In the specific implementation, a fisheye camera is used to shoot a black and white checkered calibration plate to obtain a black and white checkered image. For the convenience of the following description, the black and white checkered image here is referred to as the first black and white checkered image. The corner points on the calibration plate are detected from the first black and white checkered image, and the positions of the corner points in the image (pixel coordinates) and in the three-dimensional space (three-dimensional corner point coordinates) are known. Using the known three-dimensional corner point coordinates and the corresponding pixel coordinates, as well as the imaging model of the fisheye camera, the camera's intrinsic parameters (such as focal length, principal point coordinates) and distortion coefficients are estimated through an optimization algorithm (such as the least squares method).

[0066] In order to verify and correct the accuracy of the distortion table or the distortion coefficient, in an optional embodiment, the method further includes:

[0067] Taking a second black and white checkered image;

[0068] Using the distortion table / distortion coefficient, the second black and white checkered image is subjected to distortion correction to obtain a corrected black and white checkered image;

[0069] Performing correction verification on the corrected black and white checkered image;

[0070] When the calibration verification passes, it is determined that the distortion table / distortion coefficient is correct;

[0071] When the correction verification fails, the third black and white checkered image is re-photographed, and a new distortion coefficient is obtained based on the third black and white checkered image until the new distortion coefficient is correct.

[0072] Use a fisheye camera to shoot the black and white square calibration plate to obtain a second black and white square image, which is used to verify the accuracy of the distortion table or distortion coefficient.

[0073] According to the distortion table or distortion coefficient, the second black and white checkered image is subjected to distortion correction to obtain a corrected black and white checkered image. The straightness and angle accuracy of the black and white checkered in the corrected black and white checkered image are measured to verify the quality of the corrected black and white checkered image to determine whether the distortion correction is successful. If the correction verification passes, that is, the quality of the corrected black and white checkered image meets the requirements, then it can be determined that the distortion table or distortion coefficient is correct. If the correction verification fails, that is, the quality of the corrected black and white checkered image does not meet the requirements, then it is necessary to retake a new black and white checkered image (the third black and white checkered image) to recalculate the distortion coefficient.

[0074] Repeat the above steps, extract corner points from the third black and white checkered image, and calculate new distortion coefficients through the camera calibration algorithm. Use the new distortion coefficients to perform distortion correction on the new black and white checkered image, and perform correction verification again. If the verification passes, it is determined that the new distortion coefficients are correct; if the verification fails, continue to repeat this process until the correct distortion coefficients are found.

[0075] It should be understood that the determination of the distortion table or distortion coefficient is an iterative process. Through continuous verification and correction, it can be ensured that the final distortion table or distortion coefficient is accurate and can be used to effectively correct the distortion of images taken by the fisheye camera.

[0076] Embodiment 2

[0077] Figure 2 1 is a flow chart of a method for real-time processing of fisheye images provided in an embodiment of the present application. The method for real-time processing of fisheye images specifically comprises the following steps.

[0078] S21, when the fisheye camera is started, a coordinate conversion relationship table from spherical coordinates to image coordinates is initialized and established according to the distortion table / distortion coefficient stored in the fisheye camera.

[0079] When the fisheye camera is detected to be started, in order to extract useful information from the image captured by the fisheye camera and perform further analysis or processing, it is necessary to load the distortion table or distortion coefficient from its storage space (such as internal memory or external memory card), and establish a conversion relationship from the spherical coordinates obtained by the fisheye camera to the pixel coordinates (image coordinates) on the image plane according to the distortion table / distortion coefficient.

[0080] Spherical coordinates are a three-dimensional coordinate system used to express the position of a point relative to the center of a reference sphere. Spherical coordinates consist of three components: radial distance (the distance from the center of the sphere), polar angle (the angle measured from the positive z-axis), and azimuth angle (the angle measured from the positive x-axis in the xy plane). Image coordinates are a coordinate system that expresses the location of pixels on a two-dimensional image plane.

[0081] In an optional implementation, the initialization of establishing a coordinate conversion relationship table from spherical coordinates to image coordinates according to the distortion table / distortion coefficient stored in the fisheye camera includes:

[0082] Obtaining the focal length of the fisheye lens and the pixel value corresponding to the focal length;

[0083] Establishing spherical coordinates according to the pixel values;

[0084] Obtaining a projection mode of the fisheye camera;

[0085] A coordinate conversion relationship table from spherical coordinates to image coordinates is initialized and established according to the spherical coordinates, the projection mode, the distortion table / distortion coefficient and the sensor target surface image coordinates.

[0086] First, obtain the basic parameters of the fisheye lens, including the focal length and the pixel value corresponding to the focal length. Focal length is a key parameter of the fisheye lens, which describes the lens's ability to focus light and determines the clarity and viewing angle of the fisheye camera's imaging. It can be obtained from the technical specifications provided by the fisheye lens manufacturer. The focal length value is usually in millimeters (mm).

[0087] The pixel value corresponding to the focal length is the projected size of the focal length on the image sensor of the fisheye camera, which is usually related to the resolution and pixel size of the camera sensor.

[0088] Based on the acquired pixel values, a spherical coordinate system can be constructed or simulated. In the imaging model of the fisheye camera, due to the particularity of the lens design, when the light is projected from the three-dimensional space to the image sensor, it will experience a special mapping relationship, which is similar to mapping a point in the three-dimensional space onto a sphere. Therefore, by establishing spherical coordinates through pixel values, this mapping relationship can be simulated.

[0089] The projection mode of a fisheye camera determines how light is mapped from three-dimensional space to a two-dimensional image. Projection modes can include equidistant projection, equisolid angle projection, etc. The projection mode describes how light is distributed on the sphere and how spherical coordinates are mapped to image coordinates. Different projection modes of fisheye cameras use different trigonometric functions.

[0090] Sensor image coordinates are a two-dimensional coordinate system of pixels on an image sensor that describe the location of each pixel in an image. Sensor image coordinates are typically established based on the physical size and pixel density of the image sensor. Each pixel has a unique coordinate that is used to locate and access the pixel during image processing.

[0091] After obtaining the spherical coordinates, projection mode, distortion table / distortion coefficients and sensor target surface image coordinates, you can start to establish a coordinate conversion relationship table from spherical coordinates to image coordinates. In specific implementation, for a given spherical coordinate, convert it into an intermediate coordinate (for example, a normalized coordinate or a perspective coordinate) according to the projection mode. Use the distortion table or distortion coefficient to correct the intermediate coordinates to eliminate the coordinate offset or distortion caused by lens distortion. Match the corrected intermediate coordinates with the sensor target surface image coordinates to establish a coordinate conversion relationship table from spherical coordinates to image coordinates. Store all spherical coordinates and their corresponding image coordinates in the coordinate conversion relationship table for subsequent quick search and use.

[0092] The above optional implementation manner can more accurately establish the mapping relationship between spherical coordinates and image coordinates by obtaining the focal length of the fisheye lens and the corresponding pixel value. Using the distortion table and / or distortion coefficient for distortion correction can eliminate the image distortion caused by the fisheye lens and further improve the accuracy of coordinate conversion. Since the distortion correction and projection mode are taken into consideration, the established coordinate conversion relationship table can better adapt to the imaging characteristics of the fisheye camera under different scenes and conditions, and enhance the robustness and adaptability. By establishing an accurate coordinate conversion relationship table, the pixel coordinates in the image taken by the fisheye camera can be converted into spherical coordinates in three-dimensional space, or the spherical coordinates can be converted into image coordinates.

[0093] S22, determining a region of interest according to a preset field of view angle, and initializing a first spherical coordinate table according to the image coordinates of the region of interest and the distortion table / distortion coefficient.

[0094] Due to its unique design, the fisheye lens can capture an extremely wide field of view, which contains a lot of information that is not needed or needed. By determining the region of interest, the size of the image can be reduced, thereby reducing the amount of calculation and time consumption of subsequent processing steps. The region of interest is usually much smaller than the original image, so it is easier to store and transmit.

[0095] In addition, the original fisheye image may contain noise and interference information that is irrelevant to the region of interest, which may affect the accuracy of subsequent processing. By determining the region of interest, the irrelevant noise and interference information can be removed, thereby improving the accuracy of the processing results.

[0096] The preset field of view includes a preset horizontal field of view and a preset vertical field of view, which define the boundary of the region of interest (ROI) in the image. The horizontal field of view and the vertical field of view are input into the electronic pan / tilt by the user. Based on the preset field of view and the internal parameters of the camera, the boundary coordinates of the region of interest ROI on the image plane are calculated. The boundary coordinates define the four corner points or edges of the ROI. The ROI area can be marked on the image, and the ROI can be visualized by drawing a rectangular box or other shapes on the image.

[0097] In an optional implementation, determining the region of interest according to a preset field of view angle includes:

[0098] Using the trigonometric principle, the width of the region of interest is calculated according to the focal length and the preset horizontal field of view angle;

[0099] Using the trigonometric principle, the height of the region of interest is calculated according to the focal length and the preset vertical field of view angle;

[0100] A region of interest is determined according to the width and the height.

[0101] Assuming the width of the image sensor is W_sensor (in pixels), the preset horizontal field of view is HFOV (in degrees), the focal length is f (in millimeters), and assuming that the image sensor and lens are matched, that is, there is no additional scaling factor, use the tangent (tan) function in trigonometric functions to calculate the angle corresponding to each pixel in the horizontal direction. Use this angle and the preset horizontal field of view to calculate the width of the ROI Crop_w (in pixels).

[0102] Assuming that the width of the image sensor W_sensor = 1920 pixels, the horizontal field of view HFOV = 70 degrees, and the focal length f = 2.8 mm, the horizontal angle corresponding to each pixel is pixel_angle_horizontal = math.atan(0.5*W_sensor / f)*(180 / math.pi). Assuming that the ROI occupies the entire horizontal field of view, the width of the ROI Crop_w = int(W_sensor*(scaled_HFOV / HFOV)). Similarly, the height of the ROI Crop_h is calculated using the vertical field of view (VFOV) and the height of the image sensor (H_sensor, in pixels).

[0103] Assuming that the height of the image sensor H_sensor = 1080 pixels, the vertical field of view VFOV = 60 degrees, and the focal length f = 2.8 mm, the vertical angle corresponding to each pixel is pixel_angle_vertical = math.atan(0.5*H_sensor / f)*(180 / math.pi). Assuming that the ROI occupies the entire vertical field of view, the height of the ROI Crop_h = int(H_sensor*(scaled_VFOV / VFOV)).

[0104] By specifying the width and height of the ROI in pixels, you can determine the location of the ROI on the image. The ROI will be located in the center of the image, but you can place it elsewhere in the image if you want.

[0105] With the center of the image sensor sensor as the center, the sensor X axis as the X axis, and the sensor Y axis as the Y axis, a rectangular area with a width of Crop_w and a height of Crop_h is intercepted from the center of the sensor target image. The rectangular area is the region of interest.

[0106] In an optional implementation, the initializing the first spherical coordinate table according to the image coordinates of the region of interest and the distortion table / distortion coefficients includes:

[0107] Initialize and generate an image coordinate table according to the image coordinates of the region of interest;

[0108] A first spherical coordinate table is initialized according to the image coordinate table and the distortion table / distortion coefficient.

[0109] Get the image coordinates of all pixels in the region of interest, and get an initialized image coordinate table (CROP-2D coordinate table) based on the image coordinates of all pixels. Each element in the image coordinate table is a two-dimensional coordinate point, representing a pixel position in the ROI.

[0110] For each coordinate point in the image coordinate table, distortion correction is performed using the internal parameter matrix established by the distortion table or distortion coefficients. This correction process converts the image coordinates into corrected image coordinates to eliminate or reduce the effects of distortion.

[0111] After obtaining the corrected image coordinates, the projection model of the fisheye camera (such as equidistant projection, equal-area projection, etc.) is used to convert the two-dimensional coordinate points into three-dimensional spherical coordinates. That is, the two-dimensional coordinates are mapped to points on the three-dimensional unit sphere. The calculated spherical coordinates are stored in a table to obtain the first spherical coordinate table (CROP-3D coordinate table). Each element in the first spherical coordinate table is a three-dimensional coordinate point, representing the position of a point in the ROI in the three-dimensional spherical space.

[0112] S23, when a posture change of the fisheye camera is monitored, a rotation matrix is ​​calculated according to the real-time posture of the fisheye camera, and a reverse rotation calculation is performed on the first spherical coordinate table according to the rotation matrix to obtain a second spherical coordinate table.

[0113] Due to the change in the posture of the unmanned shooting platform, the relative position between the fisheye lens and the image sensor will also change, resulting in the angle and position of the light received by the image sensor to shift, thereby causing image distortion. Therefore, when the posture of the fisheye camera changes, in order to maintain the accuracy and consistency of image processing, the coordinate system related to the camera posture needs to be converted accordingly. That is, it is necessary to calculate a rotation matrix based on the real-time posture of the fisheye camera (usually expressed as a rotation angle or quaternion), and then use this rotation matrix to reversely rotate the first spherical coordinate table previously determined to eliminate the rotation introduced by the posture change of the unmanned shooting platform, thereby obtaining a second spherical coordinate table that matches the current camera posture.

[0114] Use sensors (such as IMU, gyroscope, etc.) to obtain the real-time attitude of the unmanned shooting platform (for example, drone, other flying equipment or underwater robot, etc.). The real-time attitude includes: pitch angle (Pitch), yaw angle (Yaw) and roll angle (Roll).

[0115] Based on the real-time attitude acquired, a 3*3 rotation matrix can be calculated. The rotation matrix describes how to rotate a point from the world coordinate system (or camera coordinate system) to the current attitude coordinate system of the unmanned shooting platform. Apply these three rotations in the order of ZYX (i.e. yaw first, then pitch, and finally roll). Each rotation can be represented by a separate rotation matrix.

[0116] Yaw angle (rotation around the Z axis):

[0117] Pitch angle (rotation around the Y axis):

[0118] Roll angle (Roll, rotation around the X axis):

[0119] Multiplying the above three together, we can get the final 3*3 rotation matrix.

[0120] The rotation matrix is ​​then used to perform a reverse rotation calculation on each coordinate point in the first spherical coordinate table. "Reverse rotation" means converting the coordinate point from the position in the current camera pose back to the position in the initial pose so that it remains consistent in subsequent processing. Reverse rotation can be achieved by multiplying each coordinate point in the first spherical coordinate table with the rotation matrix.

[0121] After all the coordinate points in the first spherical coordinate table are reversely rotated, a new spherical coordinate table, namely the second spherical coordinate table, can be obtained. The coordinate points in the second spherical coordinate table have been adjusted according to the real-time posture of the camera, so they can maintain consistency with the current camera posture in subsequent processing.

[0122] S24, converting the second spherical coordinate table into a target image coordinate table according to the coordinate conversion relationship table.

[0123] Converting the second spherical coordinate table (corrected CROP-3D coordinate table) to the target image coordinate table (New-2D_tab) refers to mapping from three-dimensional spherical coordinates to two-dimensional image coordinates.

[0124] Find the image coordinates corresponding to each coordinate point in the second spherical coordinate table in the coordinate conversion relationship table, store the found image coordinates in a new table, and obtain the target image coordinate table. Each element in the target image coordinate table is a two-dimensional coordinate point, representing the position of the original spherical coordinate point on the target image.

[0125] S25, acquiring a real-time original fisheye image, and remapping the original fisheye image according to the target image coordinate table to obtain a target fisheye image.

[0126] Since the target image coordinate table (New-2D_tab) contains the coordinate mapping relationship from the fisheye image to the target image (usually the corrected image), after using the fisheye camera to capture the real-time original fisheye image, traverse each pixel of the original fisheye image, and for each pixel, find its corresponding coordinate in the target image coordinate table. Use an interpolation algorithm (such as bilinear interpolation, cubic spline interpolation, etc.) to calculate the pixel value of the corresponding coordinate on the target image. Combine all pixel values ​​obtained by interpolation calculation into the target fisheye image. Encode the target fisheye image and output it in the required format.

[0127] Since the edge perspective pixels of the fisheye image are relatively sparse, there are burrs in the edge perspective interpolation, and the refined interpolation consumes high computing power. Therefore, in order to balance the accuracy and computing power of the remapping process, in an optional implementation, the remapping process is performed on the original fisheye image according to the target image coordinate table to obtain the target fisheye image, including:

[0128] For each pixel in the original fisheye image, searching for the corresponding target coordinates in the target image coordinate table;

[0129] Identify key coordinates in the target coordinates, and represent the key coordinates and corresponding interpolation weights in floating point form;

[0130] The calculations involving coordinate rotation in the remapping process are expressed in floating point format;

[0131] Generate a target fisheye image based on the floating-point coordinates.

[0132] Edge detection algorithms, such as Canny edge detection, Sobel edge detection, or Laplacian edge detection, can be used to identify edge regions in the original fisheye image, where the target coordinates in the edge region are key coordinates. Floating-point representation of the key coordinates and the corresponding interpolation weights can increase interpolation accuracy and reduce glitches caused by edge perspective interpolation. Because distortion and perspective effects may be more obvious at edge perspectives, higher calculation accuracy is required to ensure the smoothness and accuracy of the interpolation results. Bilinear interpolation, cubic spline interpolation, or higher-order interpolation algorithms can be applied to edge regions to further reduce glitches.

[0133] Using floating point numbers for key steps in rotation calculation can reduce the precision loss caused by rotation.

[0134] Since the coordinate points may be mapped to positions outside the target fisheye image or to non-integer pixel positions during the process of converting the second spherical coordinates to the target image coordinates, check whether the coordinates are within the boundary of the original fisheye image. For coordinate points mapped outside the fisheye image, you can choose to ignore them or map them to the boundary of the fisheye image.

[0135] After the remapping and interpolation process is completed, the result is converted from a double-precision floating point value back to an image data type suitable for storage or display (such as an 8-bit or 16-bit unsigned integer). The processed image data is encoded into an appropriate image format (such as JPEG, PNG, TIFF, etc.) for storage or transmission. Save the encoded target fisheye image to a file on disk or display it on the screen through a graphical user interface (GUI).

[0136] The above optional implementation, by using double-precision floating-point values ​​and floating-point increased precision for interpolation calculations throughout the entire processing process, can significantly improve the visual quality of the remapped fisheye image, especially in the edge viewing area, and help reduce artifacts and glitches caused by interpolation, making the image look smoother and more realistic.

[0137] It should be noted that the fisheye camera offline calibration (S11-S13) in the above embodiment only needs to be performed once. After the fisheye camera is started, the initialization process (S21-S22) is first performed once, and when the posture of the fisheye camera changes, S23-S24 is performed, and the remapping difference process (S25) is always executed and output in a loop.

[0138] The implementation principle of this embodiment is that by accurately correcting the fisheye image distortion caused by the posture change of the unmanned shooting platform, the quality and accuracy of the fisheye image can be significantly improved. At the same time, by combining offline calibration with online remapping, not only the complex operation process is simplified, but also the practicality and stability of the system are greatly enhanced.

[0139] Embodiment 3

[0140] Figure 3 It is a structural diagram of a real-time fisheye image processing device provided in an embodiment of the present application.

[0141] In some embodiments, the fisheye image real-time processing device 30 may include a plurality of functional modules composed of computer program segments. The computer program of each program segment in the fisheye image real-time processing device 30 may be stored in the memory of the fisheye camera and executed by at least one processor to perform (see Figure 2 Description) Function for real-time processing of fisheye images.

[0142] In this embodiment, the fisheye image real-time processing device 30 can be divided into multiple functional modules according to the functions it performs. The functional modules may include: an offline calibration module 301, a first initialization module 302, a second initialization module 303, a posture calculation module 304, a coordinate conversion module 305, and a remapping processing module 306. The module referred to in this application refers to a series of computer program segments that can be executed by at least one processor and can complete fixed functions, which are stored in a memory. In this embodiment, the functions of each module will be described in detail in subsequent embodiments.

[0143] The offline calibration module 301 is used to obtain a distortion table / distortion coefficient.

[0144] The first initialization module 302 is used to initialize and establish a coordinate conversion relationship table from spherical coordinates to image coordinates according to the distortion table / distortion coefficient stored in the fisheye camera when the fisheye camera is started;

[0145] The second initialization module 303 is used to determine the region of interest according to a preset field of view angle, and initialize the first spherical coordinate table according to the image coordinates of the region of interest and the distortion table / distortion coefficient;

[0146] The posture calculation module 304 is used to calculate a rotation matrix according to the real-time posture of the fisheye camera when a posture change of the fisheye camera is monitored, and perform reverse rotation calculation on the first spherical coordinate table according to the rotation matrix to obtain a second spherical coordinate table;

[0147] The coordinate conversion module 305 is used to convert the second spherical coordinate table into a target image coordinate table according to the coordinate conversion relationship table;

[0148] The remapping processing module 306 is used to obtain a real-time original fisheye image, and remap the original fisheye image according to the target image coordinate table to obtain a target fisheye image.

[0149] It should be understood that the various variations and specific embodiments of the fisheye image real-time processing method provided in the above embodiments are also applicable to the fisheye image real-time processing device in this embodiment. Through the detailed description of the above-mentioned fisheye image real-time processing method, those skilled in the art can clearly understand the implementation process of the fisheye image real-time processing device in this embodiment. For the sake of brevity of the specification, it will not be described in detail here.

[0150] Embodiment 3

[0151] This embodiment provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned fisheye image real-time processing method embodiment are implemented, such as Figure 1 S11-S13 and / or Figure 2 S21-S25 shown.

[0152] S21, when the fisheye camera is started, a coordinate conversion relationship table from spherical coordinates to image coordinates is initialized and established according to the distortion table / distortion coefficient stored in the fisheye camera.

[0153] S22, determining a region of interest according to a preset field of view angle, and initializing a first spherical coordinate table according to the image coordinates of the region of interest and the distortion table / distortion coefficient.

[0154] S23, when a posture change of the fisheye camera is monitored, a rotation matrix is ​​calculated according to the real-time posture of the fisheye camera, and a reverse rotation calculation is performed on the first spherical coordinate table according to the rotation matrix to obtain a second spherical coordinate table.

[0155] S24, converting the second spherical coordinate table into a target image coordinate table according to the coordinate conversion relationship table.

[0156] S25, acquiring a real-time original fisheye image, and remapping the original fisheye image according to the target image coordinate table to obtain a target fisheye image.

[0157] Alternatively, when the computer program is executed by a processor, the functions of each module / unit in the above-mentioned device embodiment are realized, for example Figure 3 Modules 301-306 in.

[0158] The offline calibration module 301 is used to obtain a distortion table / distortion coefficient.

[0159] The first initialization module 302 is used to initialize and establish a coordinate conversion relationship table from spherical coordinates to image coordinates according to the distortion table / distortion coefficient stored in the fisheye camera when the fisheye camera is started;

[0160] The second initialization module 303 is used to determine the region of interest according to a preset field of view angle, and initialize the first spherical coordinate table according to the image coordinates of the region of interest and the distortion table / distortion coefficient;

[0161] The posture calculation module 304 is used to calculate a rotation matrix according to the real-time posture of the fisheye camera when a posture change of the fisheye camera is monitored, and perform reverse rotation calculation on the first spherical coordinate table according to the rotation matrix to obtain a second spherical coordinate table;

[0162] The coordinate conversion module 305 is used to convert the second spherical coordinate table into a target image coordinate table according to the coordinate conversion relationship table;

[0163] The remapping processing module 306 is used to obtain a real-time original fisheye image, and remap the original fisheye image according to the target image coordinate table to obtain a target fisheye image.

[0164] Embodiment 4

[0165] See also Figure 4 , which is a schematic diagram of the structure of a fisheye camera provided in an embodiment of the present application. In a preferred embodiment of the present application, the fisheye camera 40 includes a memory 401 , an electronic pan / tilt platform 402 , and at least one communication bus 403 .

[0166] Those skilled in the art should understand that Figure 4 The structure of the fisheye camera shown does not constitute a limitation of the embodiments of the present application, and can be either a bus structure or a star structure. The fisheye camera 40 can also include more or less other hardware or software than shown in the figure, or a different component arrangement.

[0167] In some embodiments, the fisheye camera 40 is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application-specific integrated circuits, programmable gate arrays, digital processors, and embedded devices. The fisheye camera 40 may also include a client device, which includes but is not limited to any electronic product that can interact with a client through a keyboard, mouse, remote control, touchpad, or voice control device, such as a personal computer, tablet computer, smart phone, digital camera, etc.

[0168] The fisheye camera 40 is only an example. Other existing or future electronic products that are suitable for the present application should also be included in the protection scope of the present application and are included here by reference.

[0169] In some embodiments, the memory 401 stores a computer program, and when the computer program is executed by the electronic pan / tilt 402, all or part of the steps in the real-time fisheye image processing method are implemented. The memory 401 includes a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable rewritable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0170] Furthermore, the computer-readable storage medium may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of the blockchain node, etc.

[0171] In some embodiments, the electronic pan / tilt 402 is the control core (ControlUnit) of the fisheye camera 40, and uses various interfaces and lines to connect each component of the entire fisheye camera 40, and executes various functions and processes data of the fisheye camera 40 by running or executing the program or module stored in the memory 401, and calling the data stored in the memory 401. For example, when the electronic pan / tilt 402 executes the computer program stored in the memory, it implements all or part of the steps of the real-time fisheye image processing method described in the embodiment of the present application; or implements all or part of the functions of the real-time fisheye image processing device. The electronic pan / tilt 402 can be composed of an integrated circuit, for example, it can be composed of a single packaged integrated circuit, or it can be composed of multiple integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and a combination of various control chips.

[0172] In some embodiments, the at least one communication bus 403 is configured to implement connection and communication between the memory 401 and the electronic pan / tilt platform 402 , etc.

[0173] Although not shown, the fisheye camera 40 may also include a power source (such as a battery) for powering each component. Preferably, the power source may be logically connected to the electronic pan / tilt 402 through a power management device, so that the power management device can manage charging, discharging, and power consumption. The power source may also include one or more DC or AC power sources, recharging devices, power failure detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The fisheye camera 40 may also include a variety of sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0174] The above-mentioned integrated unit implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, and includes a number of instructions for enabling a fisheye camera (which can be a personal computer, a fisheye camera, or a network device, etc.) or a processor to execute a part of the method described in each embodiment of the present application.

[0175] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.

[0176] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, and may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0177] In addition, each functional module in each embodiment of the present application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0178] It is obvious to those skilled in the art that the present application is not limited to the details of the above exemplary embodiments, and that the present application can be implemented in other specific forms without departing from the spirit or basic features of the present application. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-restrictive, and the scope of the present application is limited by the attached claims rather than the above description, so it is intended to include all changes that fall within the meaning and scope of the equivalent elements of the claims in the present application. Any figure mark in the claims should not be regarded as limiting the claims involved. In addition, it is obvious that the word "including" does not exclude other units or, and the singular does not exclude the plural. Multiple units or devices stated in the specification can also be implemented by one unit or device through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any particular order.

[0179] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present application and are not intended to limit it. Although the present application has been described in detail with reference to the preferred embodiments, a person of ordinary skill in the art should understand that the technical solution of the present application may be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present application.

Claims

1. A real-time fisheye image processing method, characterized in that: The method comprises: When the fisheye camera is started, a coordinate conversion relationship table from spherical coordinates to image coordinates is initialized and established according to the distortion table / distortion coefficient stored in the fisheye camera; Determine a region of interest according to a preset field of view angle, and obtain a first spherical coordinate table by initializing the image coordinates of the region of interest and the distortion table / distortion coefficient; When a change in the posture of the fisheye camera is monitored, a rotation matrix is ​​calculated according to the real-time posture of the fisheye camera, and a reverse rotation calculation is performed on the first spherical coordinate table according to the rotation matrix to obtain a second spherical coordinate table; According to the coordinate conversion relationship table, converting the second spherical coordinate table into a target image coordinate table; A real-time original fisheye image is acquired, and a remapping process is performed on the original fisheye image according to the target image coordinate table to obtain a target fisheye image.

2. The real-time fisheye image processing method according to claim 1, characterized in that: The method further comprises: Determine whether there is a pixel angle comparison table provided by the fisheye lens manufacturer; When the pixel angle comparison table exists, obtaining the distortion table according to the pixel angle comparison table; When the pixel angle comparison table does not exist, a first black and white checkered image is photographed, and distortion correction is performed on the first black and white checkered image to obtain the distortion coefficient.

3. The real-time fisheye image processing method according to claim 2, characterized in that: The method further comprises: Taking a second black and white checkered image; Using the distortion table / The distortion coefficient is used to perform distortion correction on the second black and white checkered image to obtain a corrected black and white checkered image; Performing correction verification on the corrected black and white checkered image; When the correction verification passes, the distortion table is determined / The distortion coefficient is correct; When the correction verification fails, the third black and white checkered image is re-photographed, and a new distortion coefficient is obtained based on the third black and white checkered image until the new distortion coefficient is correct.

4. The real-time fisheye image processing method according to claim 1, characterized in that: The initialization of establishing a coordinate conversion relationship table from spherical coordinates to image coordinates according to the distortion table / distortion coefficient stored in the fisheye camera includes: Obtaining the focal length of the fisheye lens and the pixel value corresponding to the focal length; Establishing spherical coordinates according to the pixel values; Obtaining a projection mode of the fisheye camera; According to the spherical coordinates, the projection mode, the distortion table / Distortion coefficients and sensor target surface image coordinates are initialized to establish a coordinate conversion relationship table from spherical coordinates to image coordinates.

5. The real-time fisheye image processing method according to claim 4, characterized in that: The preset field of view angle includes a preset horizontal field of view angle and a preset vertical field of view angle, and determining the region of interest according to the preset field of view angle includes: Using the trigonometric principle, the width of the region of interest is calculated according to the focal length and the preset horizontal field of view angle; Using the trigonometric principle, the height of the region of interest is calculated according to the focal length and the preset vertical field of view angle; A region of interest is determined according to the width and the height.

6. The real-time fisheye image processing method according to any one of claims 1 to 4, characterized in that: Initializing the first spherical coordinate table according to the image coordinates of the region of interest and the distortion table / distortion coefficients includes: Initialize and generate an image coordinate table according to the image coordinates of the region of interest; A first spherical coordinate table is initialized according to the image coordinate table and the distortion table / distortion coefficient.

7. The real-time fisheye image processing method according to claim 1, characterized in that: The remapping process of the original fisheye image according to the target image coordinate table to obtain the target fisheye image comprises: For each pixel in the original fisheye image, searching for the corresponding target coordinates in the target image coordinate table; Identify key coordinates in the target coordinates, and represent the key coordinates and corresponding interpolation weights in floating point form; The calculations involving coordinate rotation in the remapping process are expressed in floating point format; Generate a target fisheye image based on the floating-point coordinates.

8. A real-time fisheye image processing device, characterized in that: The device comprises: A first initialization module is used to initialize and establish a coordinate conversion relationship table from spherical coordinates to image coordinates according to the distortion table / distortion coefficient stored in the fisheye camera when the fisheye camera is started; A second initialization module is used to determine a region of interest according to a preset field of view angle, and to initialize a first spherical coordinate table according to the image coordinates of the region of interest and the distortion table / distortion coefficient; a posture calculation module, for calculating a rotation matrix according to the real-time posture of the fisheye camera when a posture change of the fisheye camera is detected, and performing reverse rotation calculation on the first spherical coordinate table according to the rotation matrix to obtain a second spherical coordinate table; A coordinate conversion module, used for converting the second spherical coordinate table into a target image coordinate table according to the coordinate conversion relationship table; The remapping processing module is used to obtain a real-time original fisheye image, and remap the original fisheye image according to the target image coordinate table to obtain a target fisheye image.

9. A fisheye camera, characterized in that: The fisheye camera has an electronic pan / tilt and a memory built therein, and the electronic pan / tilt is used to implement the real-time fisheye image processing method according to any one of claims 1 to 7 when executing a computer program stored in the memory.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the real-time fisheye image processing method according to any one of claims 1 to 7 is implemented.

Images