High-precision fast camera calibration method and system based on large-field-of-view fisheye lens

Abstract

The present invention provides a high-precision fast camera calibration method and system based on a large-field fisheye lens, belonging to the field of computer vision and image processing. In order to solve the problems of low calibration accuracy and single evaluation index in the existing large-field fisheye lens calibration method. The present invention integrates improved methods of image super-resolution, image segmentation, corner point detection, corner point sorting, calibration model, and objective evaluation index, which can greatly shorten the time required for shooting and taking pictures, reduce the requirements for image quality while improving the robustness of the overall method, reduce the hardware requirements for calibration reference objects, improve calibration accuracy, and reduce the time required for model solution.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and image processing, and in particular to a high-precision fast camera calibration method and system based on a large-field-of-view fisheye lens. Background Art

[0002] Fisheye lenses have a wide field of view and can capture a wider range of environmental information. They are widely used in automobile manufacturing, autonomous driving, drones, remote sensing and other fields. Therefore, high-efficiency and high-precision fisheye camera calibration technology has become a research hotspot in these fields. Traditional fisheye camera calibration methods usually require rotation and translation of the calibration reference object, and then feature point detection based on the captured image to achieve internal reference calibration. The main problems of traditional methods include:

[0003] 1. It takes too long to take images: The traditional camera calibration method requires rotating and translating the calibration reference object before taking the image. This process is extremely time-consuming and labor-intensive, and it is easy to cause wear and tear on the calibration reference object.

[0004] 2. Too high image quality requirements: The camera calibration process mainly relies on the feature point information in the image, and the image quality will directly affect the final calibration result. Therefore, if there is one or several low-quality images in the captured images, the calibration may fail and the image needs to be retaken.

[0005] 3. Low calibration accuracy: The traditional fisheye camera calibration model is generally the KB (Kannala-Brandt) model, which has certain limitations in calculation accuracy. Especially when calibrating cameras with a field of view angle of more than 180°, calculation disorder is prone to occur.

[0006] 4. Single evaluation index: Traditional fisheye camera calibration technology generally uses the reprojection error (Root Mean Square, RMS) as an objective evaluation index to measure the accuracy of the calibration results. However, using only this single index has limitations, and it cannot reflect the accuracy of depth information and the uniformity of spatial errors.

[0007] With the development of fisheye camera calibration technology, more and more researchers are devoted to the development of calibration technology to overcome the problems in traditional methods. These methods include fisheye camera calibration method based on deep learning, fisheye camera calibration method based on coded calibration plate, camera calibration method based on Ocam model, wide-angle camera calibration method based on Mei, etc. The above methods have successfully improved the calibration accuracy and efficiency, but there is still room for improvement.

[0008] Existing patents include camera calibration equipment and calibration method for single-shot images of wide-angle fisheye lenses (application number: CN202311372090.2), in which the camera calibration equipment for single-shot images of wide-angle fisheye lenses is a fisheye box, which is an open rectangular parallelepiped with five faces. Each inner surface of the fisheye box is painted with black and white square chessboards with regular n×m rows and columns. The outer edge of the chessboard is spliced ​​around a circle of black and white chessboard grids. A closed black frame is provided outside the chessboard grids, and the black frame does not intersect with the chessboard grids; a gap is set between two adjacent black frames of the fisheye box, and each gap is located at the edge or edge of two adjacent inner surfaces of the fisheye box; two groups of perpendicular line segments are drawn at the center position of the center surface of the fisheye box to form a crosshair; the chessboards on the left and right inner surfaces of the fisheye box are respectively drawn at the middle position of a row of chessboard grids close to the center surface. The provided fisheye box lens calibration saves time and effort, has high calculation accuracy and good robustness. However, there is a black border around each chessboard pattern in the fisheye box, which places too high demands on image quality and environmental factors in the image segmentation step, resulting in hidden dangers in the stability and robustness of the method. Summary of the invention

[0009] The technical problems to be solved by the present invention are:

[0010] In order to solve the problems of low calibration accuracy, single evaluation index, and poor stability and robustness caused by the black border around the fisheye box in the existing large field of view fisheye lens calibration method.

[0011] The present invention adopts the following technical solutions to solve the above technical problems:

[0012] The present invention provides a high-precision fast camera calibration method based on a large-field-of-view fisheye lens, comprising the following steps:

[0013] S100, super-resolution reconstruction is performed on the single-shot fisheye box image based on a multi-scale spatial attention mechanism to obtain a high-quality image;

[0014] S200, performing image segmentation on the high-quality image using a segmentation method based on an adaptive threshold of convex polygon fitting, and obtaining five images containing black and white checkerboards;

[0015] S300, performing corner point detection and sub-pixel precision processing on the five images containing black and white checkerboards obtained in step S200 based on the sub-pixel corner point detection method of ROI gradient screening, and generating a disordered set of two-dimensional coordinates of corner points;

[0016] S400, performing matrix sorting on the disordered two-dimensional corner point coordinate set obtained in step S300 based on the corner point sorting method of bidirectional expansion of feature points to obtain a two-dimensional corner point coordinate set in a regular order, and generating a three-dimensional corner point coordinate initial value set in a regular order based on matrix distribution;

[0017] S500, based on the regular order of the corner point two-dimensional coordinate set and the regular order of the corner point three-dimensional coordinate initial value set obtained in step S400, a six-parameter division model based on the LM optimization algorithm is used to perform high-precision fast fisheye calibration to obtain a high-precision fisheye lens internal parameter and an objective evaluation index RMS;

[0018] S600, based on the high-precision fisheye lens internal parameters obtained in step S500 and the regularly ordered set of two-dimensional coordinates of corner points obtained in step S400, three-dimensional reconstruction verification is performed to obtain a regularly ordered set of three-dimensional coordinates of corner points; the real three-dimensional corner point spacing is used as an additional objective evaluation indicator to verify the accuracy of the calibration result and the uniformity of the spatial error.

[0019] Furthermore, in step S100, it specifically includes:

[0020] S110. According to the channel attention mechanism, set is the input feature map, where are the height, width, and number of input channels of the input feature map, respectively. is a real number; Global average pooling operator for channels for:

[0021] ;

[0022] in, For the The channel height is i Width is j The feature map of

[0023] No. The attention weight of each channel It is expressed by the following formula:

[0024] ;

[0025] in, is the corrected linear unit, and All are fully connected layers. r Represents the dimension space of the matrix; is the activation function;

[0026] S120, based on the multi-scale spatial attention mechanism, is divided into two sub-modules: SA sub-module and MA sub-module. The SA sub-module uses the spatial attention mechanism to calculate the spatial attention feature map by calculating the spatial relationship within the feature map space; the spatial attention weight It is expressed by the following formula:

[0027] ;

[0028] Among them, the input feature map , perform global average pooling and global maximum pooling on the channel dimension, compress the channel size, and get two Feature map and :

[0029] ;

[0030] ;

[0031] The results of global maximum pooling and global average pooling are concatenated according to the channels, and the concatenated results are convolved to obtain the output feature map. :

[0032] ;

[0033] ;

[0034] in, represents the convolution operation, is the activation function; Indicates channel multiplication;

[0035] The S130 and MA submodules use multi-scale convolution kernels to make the network produce different spatial resolutions and depths in a multi-branch manner, which is used to extract spatial information of different scales on each channel of the feature map;

[0036] Assume that the input channel dimension of each branch is , then each different scale The feature maps have the same channel dimension ,in, ; Can be S divisibility;

[0037] A group convolution algorithm is introduced in the convolution kernel. The algorithm designs a new group convolution size selection criterion. The relationship between the multi-scale convolution kernel size and the group convolution size is expressed by the following formula:

[0038] ;

[0039] in,K is the size of the convolution kernel, G is the size of the group convolution;

[0040] Multi-scale feature map generation function It is expressed by the following formula:

[0041] ;

[0042] Among them, i The size of the convolution kernel , No. i The size of the group convolution ,set up , the complete multi-scale feature preprocessing graph is obtained by concatenation, as shown in the following formula:

[0043] ;

[0044] in, is the obtained multi-scale feature map;

[0045] By extracting the channel attention weight information from the multi-scale preprocessed feature map, we get the attention weight vectors of different scales:

[0046] ;

[0047] in, is the attention weight;

[0048] Use the channel weight algorithm to obtain attention weights from input feature maps of different scales , and then the entire multi-scale channel attention vector is obtained in series:

[0049] ;

[0050] in, For the connection operator, for The attention value, Z is the multi-scale attention weight vector;

[0051] The soft attention method is used to adaptively select different spatial scales across channels. i Soft attention weights It is expressed by the following formula:

[0052] ;

[0053] Among them, use To obtain the recalibrated weights of the multi-scale channels , which contains the spatial location information and the attention weight in the channel;

[0054] Then, the feature-recalibrated channel attentions are fused and concatenated in series to obtain the entire channel attention vector:

[0055] ;

[0056] in, Represents the multi-scale channel weights after attention interaction; the recalibrated multi-scale channel attention weights are compared with the corresponding scale Multiplying the feature map of gives the following formula:

[0057] ;

[0058] in, represents channel multiplication, Represents the feature map obtained using multi-scale channel attention weights;

[0059] By embedding the SMA module into the super-resolution reconstruction network and using the checkerboard image dataset for weight training, super-resolution reconstruction of fisheye box images can be achieved.

[0060] Furthermore, in step S200, it specifically includes:

[0061] S210, using the Scharr operator to perform edge detection on the fisheye box image reconstructed by super-resolution in step S100; based on the change of pixel gradient, the blank gaps between the surfaces are used as the outer frame contour points of the black and white chessboard;

[0062] S220, performing convex polygon fitting on the outer frame contour points of the black and white chessboard, and connecting the contour points to ensure that the black and white chessboard of each face is within the convex polygon fitting area;

[0063] S230, segmenting the original image according to the convex polygon fitting area obtained in step S220, and obtaining five preliminary segmentation effect images;

[0064] S240, performing grayscale processing on each preliminary segmentation effect image, and performing adaptive threshold processing on the grayscale image to obtain a binary image;

[0065] S250, performing pixel threshold screening on the binary image, screening out blank gaps, and using them as segmentation boundaries to perform fine segmentation on the original image.

[0066] Furthermore, in step S300, it specifically includes:

[0067] The second-order matrix of image gradient is expressed as follows:

[0068] ;

[0069] in, is the window size, and is the image grayscale and The gradient in direction, , ;

[0070] First, use the Harris algorithm as the initial corner detection method to perform preliminary detection of corners in the image and calculate the corner response value. R :

[0071] ;

[0072] in, is a matrix A scalar feature of The area scaling factor of ; is the sum of the main diagonal elements of the matrix, indicating the overall strength of the matrix; is the empirical coefficient;

[0073] Set the corner point response value The points are screened as preliminary corner points, where is the response threshold;

[0074] Secondly, for each preliminary corner point Extract a size of The square window as :

[0075] ;

[0076] Again, in Calculate the grayscale gradient of pixels in the region; for grayscale images , use the Sobel operator to calculate the horizontal gradient and the vertical gradient :

[0077] ;

[0078] Based on the horizontal gradient and the vertical gradient , calculate the gradient magnitude of each pixel and direction :

[0079] ;

[0080] Set the gradient amplitude threshold ,like , then the point is regarded as the effective response corner point;

[0081] Finally, after gradient screening, In the area, the quadratic surface fitting method is used to achieve sub-pixel corner location; the corner response value Approximately a quadratic function in a local region:

[0082] ;

[0083] in, a , b , c , d , e , v are the coefficients of the general form of the quadratic function.

[0084] Furthermore, in step S400, it specifically includes:

[0085] S410, find the sorting feature point to achieve bidirectional expansion sorting, the sorting feature point refers to the corner point used as the sorting starting point, which needs to meet the following characteristics:

[0086] (1) The sorting feature points are located at the edge of the corner point matrix, which is curved rather than straight.

[0087] (2) The sorting feature points are located in the middle of the edge of the corner point matrix;

[0088] (3) For the fisheye box, the sorting feature points of mutually symmetric surfaces are in an axisymmetric relationship;

[0089] S420, calculating the center point coordinates of the detected corner point array, for the right side, the sorting feature points that meet the conditions are located on the left side of the center point, and locating the sorting feature points on the right side through the following steps:

[0090] (1) Calculate the coordinates of the center point C 1( x 1, y 1);

[0091] (2) From all detected corner points, select C 1 The ordinate is closest to n Corner points, n At least one column of corner points;

[0092] (3) From n Select the corner point with the smallest horizontal coordinate from the corner points as the sorting feature point on the right side;

[0093] S430, after locating the sorting feature point on the right side, use the point as the sorting starting point to sort the corner points, and generate an initial set of three-dimensional coordinates corresponding to the corner points one by one. The corner point sorting process is as follows:

[0094] (1) Sorting feature points on the right side O As the sorting starting point, set its initial three-dimensional coordinate set to (0,0,0);

[0095] (2) Find the distance from all detected corner points O Point to the three corner points with the smallest Euclidean distance, and then select the corner point with the largest horizontal coordinate as the point B (1,0,0); Next, from the remaining two corner points, select the one with the smallest ordinate and smaller than point O The corner points are C (0,-1,0), and select the point with the largest ordinate that is greater than O The corner points are A (0,1,0);

[0096] (3) Point C For the new starting point, find the distance point C Then, select the three corner points with the smallest Euclidean distance and the point with the smallest ordinate that is smaller than C The corner points are D (0,-2,0), repeat this process until no corner point that meets the conditions is found, and return the point A ;

[0097] (4) From point A Start sorting downwards and find the distance point A Then, select the three corner points with the smallest Euclidean distance, and then select the point with the largest ordinate that is greater than A The corner points are E (0,2,0); Repeat this process until no corner point that meets the conditions is found, and return the point B , complete the sorting of the first column of the corner point matrix, and generate an initial set of three-dimensional coordinates corresponding to each one;

[0098] (5) Excluding the sorted corner points, B As the new starting point, follow the same method to find the distance point B The three corner points with the smallest Euclidean distance, choose the corner point with the largest horizontal coordinate as the point G (2,0,0), and then from the remaining two corner points, select the one with the smallest ordinate and smaller than point B The corner points are H (0,-2,0), and select the point with the largest ordinate that is greater than B The corner points are F (0,2,0);

[0099] (6) Similarly, a new starting point is set for bidirectional expansion sorting. After each column of corner points is sorted, the next column is sorted. This continues until all corner points are sorted and the corresponding initial set of three-dimensional coordinates is generated.

[0100] Set point K ( h ,0,0) is the last corner point of the row with the least number of corner points. When executing the corner point sorting algorithm based on bidirectional expansion of feature points, when at point K After the column is sorted, the remaining corner points are sorted. First, the remaining corner points are arranged from small to large according to the ordinate, and then the corner point with the smallest ordinate is selected. L As a starting point, and look for the point K The distance from the corner point in the column L Nearest corner M ( h , l ,0), finally, point L ( , ,0) is inherited from point M , calculate the initial set of three-dimensional coordinates by the following formula:

[0101] ;

[0102] Similarly, all remaining corner points on the right side are sorted, and the same method is used to sort the corner points on other sides and generate the corresponding initial set of three-dimensional coordinates.

[0103] Furthermore, in step S500, it specifically includes:

[0104] set up are the coordinates of the image distortion point, is the coordinate of the undistorted point of the image, and the six-parameter DM model expression is:

[0105] ;

[0106] in, and Distortion points and the undistorted point To the center of distortion P The Euclidean distance of is the radial distortion parameter, and the single-parameter DM model is adopted:

[0107] ;

[0108] Set the distortion center P The coordinates of , then the coordinates of the undistorted points in the image are:

[0109] ;

[0110] in, is the image distortion point and distortion center The distance satisfies:

[0111] ;

[0112] Let the equation of the line ,in k is the slope, b is the intercept, that is, the equation of the undistorted line is , substituting into the above formula, we can get:

[0113] ;

[0114] From the above formula, it can be concluded that under the single-parameter DM model, the ideal straight line is distorted into a circular arc curve:

[0115] ;

[0116] in, A , B , C The following conditions must be met:

[0117] ;

[0118] Solve the parameters by taking three points from the image A , B , C and the distortion center, based on A , B , C The relationship between them is:

[0119] ;

[0120] Obtained by solving A, B, C and the distortion center, and obtain the distortion parameters ;

[0121] The LM algorithm is used to optimize and solve the six-parameter DM model;

[0122] In the six-parameter model based on LM optimization, To optimize the parameters, the objective function is:

[0123] ;

[0124] in, are the coordinates of the distortion point, are the coordinates of the distortion correction points calculated according to the six-parameter division model;

[0125] The iterative formula of the LM algorithm is:

[0126] ;

[0127] in, is the Jacobian matrix; is the identity matrix, is a non-negative number, is regarded as the damping coefficient;

[0128] when When the optimal solution is missed, the increase Thereby reducing the step size; when When the step length is insufficient, reduce Thereby speeding up the convergence;

[0129] Use a first-order Taylor expansion to construct a local approximation model of the objective function:

[0130] ;

[0131] in, The radius is The trust domain of

[0132] The ratio of the actual error reduction to the local model predicted reduction As a measure of trust region update, it is shown in the following formula:

[0133] ;

[0134] in, represents the initial state of Q;

[0135] like If it is greater than 1, then decrease ;like If it is less than 1, it will increase ;like If it is close to 1, it indicates that the operation is relatively accurate. Based on the characteristic of large local distortion of the fisheye box, the DM model is solved into six distortion parameters to more accurately describe the distortion degree of the fisheye box image.

[0136] Furthermore, in step S600, it specifically includes:

[0137] By reversing the model, the distortion correction of the two-dimensional coordinate set of the corner points is achieved, and the undistorted points are obtained. ; Restore the normalized camera coordinates through coordinate system transformation:

[0138] ;

[0139] in, is the normalized coordinate in the camera coordinate system, is the inverse matrix of the camera intrinsic parameter matrix;

[0140] Using the camera's extrinsic matrix ,in, is the rotation matrix and is the translation vector, normalizing the camera coordinates Convert to three-dimensional coordinates in the world coordinate system :

[0141] ;

[0142] The Euclidean distance between each point is calculated according to the three-dimensional coordinates of the corner points after three-dimensional reconstruction, and the average of the Euclidean distance is compared with the actual distance to measure the accuracy of the calibration result.

[0143] Furthermore, the fisheye box is a rectangular structure with an open top, and each inner surface thereof is painted with black and white square checkerboards in regular n×m rows and columns, and the outer edges of the black and white square checkerboards are spliced ​​around a circle of black and white checkerboard grids; blank gaps are set between two adjacent surfaces; two groups of mutually perpendicular line segments are drawn at the center position of the center surface of the fisheye box to form a crosshair; circular marks are drawn on the middle position of a row of checkerboards of the black and white square checkerboards on the left and right inner surfaces of the fisheye box close to the center surface.

[0144] A high-precision fast camera calibration system based on a large-field fisheye lens, the system has a program module corresponding to the above steps, and executes the steps in the high-precision fast camera calibration method based on a large-field fisheye lens when running.

[0145] A computer-readable storage medium stores a computer program, wherein the computer program is configured to implement the steps of a high-precision fast camera calibration method based on a large-field-of-view fisheye lens when called by a processor.

[0146] Compared with the prior art, the present invention has the following beneficial effects:

[0147] 1. The present invention optimizes and upgrades the existing large-field-of-view fisheye camera calibration equipment "Fisheye Box", named "Fisheye Box 2.0", and proposes a high-precision and fast camera calibration method based on a large-field-of-view fisheye lens. It can greatly shorten the calibration time of the fisheye lens and improve the calibration accuracy and efficiency of the large-field-of-view fisheye camera.

[0148] 2. The present invention proposes an image super-resolution reconstruction method based on a multi-scale spatial attention mechanism (SMA). This method can improve the quality of captured images, reduce the interference of environmental factors such as light intensity, image ghosting, mechanical vibration, etc. on the image, reduce the quality requirements of the input image, and improve the robustness of the overall method.

[0149] 3. The present invention proposes an adaptive threshold segmentation method based on convex polygon fitting. Compared with the traditional image segmentation method, this method no longer deliberately requires the image to contain segmentation contours, and can adaptively locate the edges of the segmentation target, thereby reducing the hardware requirements for calibration reference objects.

[0150] 4. The present invention proposes a six-parameter division model based on the LM optimization algorithm. This model is different from the traditional camera distortion model. It adopts a division model that is more suitable for describing large distortion characteristics, and uses the LM algorithm for optimization and solution on this basis, which improves the calibration accuracy while reducing the time required for model solution.

[0151] 5. The present invention proposes a high-precision and fast camera calibration method and system based on a large-field fisheye lens. The method integrates improved methods of image super-resolution, image segmentation, corner point detection, corner point sorting, calibration model, and objective evaluation indexes, which can greatly shorten the time required for shooting and image acquisition, reduce the requirements for image quality, and improve the computational accuracy, robustness, stability, and efficiency of fisheye calibration. Compared with traditional methods, the RMS of this method is reduced by 9.13% and the DIS is reduced by 7.65%. BRIEF DESCRIPTION OF THE DRAWINGS

[0152] Figure 1 Schematic diagram of a fisheye box 2.0 in an embodiment of the present invention;

[0153] Figure 2 This is a flow chart of a high-precision fast camera calibration method based on a large-field-of-view fisheye lens in an embodiment of the present invention;

[0154] Figure 3 This is a network structure diagram of an image super-resolution reconstruction method based on a multi-scale spatial attention mechanism in an embodiment of the present invention;

[0155] Figure 4 Schematic diagram of image super-resolution reconstruction based on a multi-scale spatial attention mechanism in an embodiment of the present invention;

[0156] Figure 5 It is a flow chart of a segmentation method based on adaptive threshold of convex polygon fitting in an embodiment of the present invention;

[0157] Figure 6 It is a schematic diagram after segmentation based on adaptive threshold of convex polygon fitting in an embodiment of the present invention;

[0158] Figure 7 Schematic diagram of sub-pixel corner point detection based on ROI gradient screening in an embodiment of the present invention;

[0159] Figure 8 It is a schematic diagram of the sorting of corner points based on the bidirectional expansion of feature points in an embodiment of the present invention;

[0160] Fig. 9 Schematic diagram of three-dimensional distance measurement of corner points based on camera intrinsic parameters after verification in an embodiment of the present invention. DETAILED DESCRIPTION

[0161] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0162] A high-precision and fast camera calibration method based on a large-field-of-view fisheye lens uses a "Fisheye Box 2.0" as a calibration reference. The Fisheye Box 2.0 has the same meaning as the Fisheye Box, and is used to achieve the use of a large-field-of-view fisheye camera to obtain images containing rich multi-scale spatial corner point information in a single shot, reducing the time required to shoot 11-27 images in the traditional method. The Fisheye Box 2.0 is an open rectangular parallelepiped with five sides, each inner surface of which is painted with a black and white square chessboard with a regular n×m rows and columns. The outer edges of the black and white square chessboard are spliced ​​around a circle of black and white chessboard grids; blank gaps are set between the two adjacent surfaces of the Fisheye Box 2.0; two groups of mutually perpendicular line segments are drawn at the center position of the center surface of the fisheye box to form a crosshair; circular marks are drawn on the black and white square chessboards on the left and right inner surfaces of the fisheye box, respectively, near the middle of a row of chessboard grids on the center surface.

[0163] Combination Figure 1 As shown, the central surface size of the fisheye box 2.0 is 500mm×500mm, and the inside thereof contains a checkerboard pattern with 14×14 rows and columns; the size of the surrounding surfaces of the calibration box is 500mm×300mm, and each surface contains a checkerboard pattern with 14×8 rows and columns; the checkerboard size is 33mm×33mm, and the size of the small grid at the edge of the checkerboard is 15mm; relative to the fisheye box in the camera calibration device and calibration method for single-shot image shooting of a wide-angle fisheye lens (application number: CN202311372090.2), the present invention deletes the black border and updates the image segmentation method, thereby eliminating the influence of the black frame on the image quality and other black components in the image during the image segmentation process, thereby eliminating the potential stability and robustness risks in the algorithm; the calibration box of this specification has a moderate size, and even if a 200° fisheye lens is used for shooting, relatively complete feature information can still be retained, and the edge information of each surface will not be confused due to the close distance;

[0164] Specific implementation plan 1: Combine Figures 1 to 9 As shown, the present invention provides a high-precision rapid camera calibration method based on a large-field fisheye lens, comprising the following steps:

[0165] S100, performs super-resolution reconstruction on a single-shot fisheye box 2.0 image based on a multi-scale spatial attention mechanism (SMA) to obtain a high-quality image;

[0166] In order to eliminate the negative impact of environmental factors such as light intensity and mechanical vibration on image quality, it is necessary to reconstruct the single-shot fisheye box 2.0 image with super resolution. This paper proposes an image super-resolution reconstruction method based on a multi-scale spatial attention mechanism (SMA), and embeds the SMA module into an image super-resolution network with a deep residual structure, named RSMAN network; thereby improving the reconstruction quality, the network structure is as follows Figure 3 As shown; the image super-resolution reconstruction method based on the multi-scale spatial attention mechanism (SMA) is:

[0167] S110, channel attention mechanism can help the network selectively weight each channel and thereby distinguish the importance of different channels; set is the input feature map, where the number H , W , C Respectively represent its height, width, and number of input channels, is a real number; C Global average pooling operator for channels It can be calculated by the following formula:

[0168] ;

[0169] in, For the C The channel height is i Width is j The feature map of

[0170] Global average pooling can generate channel statistics and embed global spatial information into channel descriptors. C The attention weight of each channel It can be expressed by the following formula:

[0171] ;

[0172] in, is the rectified linear unit (ReLU), and Both represent fully connected (FC) layers, rRepresents the dimensional space of the matrix, that is, a matrix over the real number field; through two fully connected layers, the linear information between channels can be combined more effectively, which is conducive to the interaction of high and low channel dimensional information; Represents the activation function, generally using the S-type function; using the activation function, it is possible to assign values ​​to channels after channel interaction, thereby extracting information more effectively;

[0173] In order to learn the multi-scale features of images more efficiently, the present invention proposes a Spatial Multi-scale Attention mechanism (SMA), which is mainly divided into two sub-modules: SA module and MA module: in the SA sub-module, first, the input feature map is average pooled and maximum pooled to obtain two pooling results; second, the two pooling results are spliced, and convolution and Sigmoid activation function operations are performed to obtain spatial attention weights; finally, the attention weights are applied to the input feature map to obtain a weighted output feature map A; in the MA module, first, a multi-branch algorithm is used to obtain a multi-scale feature map in the channel direction; second, the channel attention weight algorithm is used to extract the attention of feature maps of different scales to obtain the channel attention weights; third, the channel-level attention vector is recalibrated using Softmax to obtain the weights of the multi-scale channels; finally, an element-by-element product operation is applied to the recalibrated weights and the output feature map B; the output map A and the output map B obtained by the above two modules are added together to obtain a refined feature map with richer multi-scale spatial feature information as output;

[0174] S120 and SA modules use the spatial attention mechanism to calculate the spatial attention feature map by calculating the spatial relationship within the feature map space. Different from channel attention, spatial attention focuses on the location of valid information on the feature map. It can be expressed by the following formula:

[0175] ;

[0176] Among them, the input feature map , perform global average pooling and global maximum pooling on the channel dimension, compress the channel size, and get two Feature map and :

[0177]

[0178] ;

[0179] The results of global maximum pooling and global average pooling are concatenated according to the channels, and the concatenated results are convolved to obtain the output feature map. :

[0180]

[0181] ;

[0182] in, represents the convolution operation, As the excitation function, Sigmoid is generally used as the excitation function; express Indicates channel multiplication;

[0183] S130 and MA modules use multi-scale convolution kernels to enable the network to produce different spatial resolutions and depths in a multi-branch manner, which can effectively extract spatial information of different scales on each channel of the feature map; suppose the input channel dimension of each branch is C , then each different scale The feature maps have the same channel dimension ,in ; C Can be S divisible; for each branch, it can independently learn multi-scale spatial information and establish local cross-channel links; however, as the kernel size increases, the number of parameters will also increase; in order to process input tensors at different kernel scales without increasing the computational cost, the present invention introduces a group convolution algorithm in the convolution kernel, which designs a new group convolution size selection criterion. The relationship between the multi-scale convolution kernel size and the group convolution size can be expressed by the following formula:

[0184] ;

[0185] in, K is the size of the convolution kernel, G is the size of the group convolution;

[0186] Multi-scale feature map generation function It is expressed by the following formula:

[0187] ;

[0188] Among them, i The size of the convolution kernel , No. i The size of the group convolution ,set up , the complete multi-scale feature preprocessing graph can be obtained by a series connection method, as shown in the following formula:

[0189] ;

[0190] in, is the obtained multi-scale feature map;

[0191] By extracting the channel attention weight information from the multi-scale preprocessed feature map, we can obtain the attention weight vectors of different scales. Mathematically, the vector of attention weights can be expressed as:

[0192] ;

[0193] in, is the attention weight;

[0194] Use the channel weight algorithm to obtain attention weights from input feature maps of different scales , so that the high-level feature map can get better pixel-level attention; further, in order to realize the interaction of attention information and fuse the cross-dimensional vector without destroying the original channel attention vector, the present invention obtains the entire multi-scale channel attention vector in a series manner:

[0195] ;

[0196] in, For the connection operator, for The attention value, Z is the multi-scale attention weight vector;

[0197] The soft attention method is used to adaptively select different spatial scales across channels. i Soft attention weights It can be expressed by the following formula:

[0198] ;

[0199] Among them, use To obtain the recalibrated weights of the multi-scale channels , which contains the spatial position information and the attention weight in the channel; realizes the interaction between local and global channel attention; then, the feature-recalibrated channel attention is fused and concatenated in series to obtain the entire channel attention vector:

[0200] ;

[0201] in, Represents the multi-scale channel weights after attention interaction; the recalibrated multi-scale channel attention weights are compared with the corresponding scale Multiplying the feature map of gives the following formula:

[0202] ;

[0203] in, represents channel multiplication, Represents the feature map obtained using multi-scale channel attention weights; the concatenation operator is more effective than simple summation, which can completely maintain the feature representation without destroying the original feature mapping information;

[0204] By embedding the SMA module into the super-resolution reconstruction network and using the checkerboard image dataset for weight training, super-resolution reconstruction of the fisheye box 2.0 image can be achieved. The specific implementation effect is as follows: Figure 4 As shown;

[0205] S200, performing image segmentation on the high-quality image using a segmentation method based on an adaptive threshold of convex polygon fitting, and obtaining five images containing only black and white checkerboards;

[0206] The fisheye box 2.0 image contains five checkerboard corner points, which are rich in multi-scale spatial information. If the corner points are detected directly, the obtained two-dimensional coordinate set is extremely chaotic and difficult to distinguish. The present invention proposes a segmentation method based on an adaptive threshold of convex polygon fitting to segment the image into five separate checkerboard corner point images, thereby eliminating the influence. The segmentation method process is as follows: Figure 5 The specific implementation effect is as shown in Figure 6 As shown, including:

[0207] S210, using the Scharr operator to perform edge detection on the fisheye box 2.0 image super-resolution reconstructed in step S100; based on the change of pixel gradient, the blank gaps between the surfaces are used as the outer frame contour points of the black and white chessboard;

[0208] S220, performing convex polygon fitting on the outer frame contour points of the black and white chessboard, connecting the contour points, and ensuring that the black and white chessboard of each face is within the convex polygon fitting area;

[0209] S230, segmenting the original image according to the convex polygon fitting area to obtain five preliminary segmentation effect images;

[0210] S240, performing grayscale processing on each preliminary segmentation effect image, and performing adaptive threshold processing on the grayscale image to obtain a binary image;

[0211] S250, performing pixel threshold screening on the binary image, screening out blank gaps, and using them as segmentation boundaries to perform fine segmentation on the original image;

[0212] S300, performing corner point detection and sub-pixel precision processing on the five images containing only black and white checkerboards obtained in step S200 based on a sub-pixel corner point detection method of ROI gradient screening, and generating a disordered set of two-dimensional coordinates of corner points;

[0213] Traditional corner detection methods can detect images with a field of view of less than 180°. However, fisheye lenses with a field of view of 200° are widely used in the industry. Therefore, a method that can achieve stable corner detection under larger distortion is needed. The present invention proposes a sub-pixel corner detection method based on ROI gradient screening. The specific implementation effect is as follows: Figure 7 As shown, including:

[0214] The second-order matrix of image gradient can be expressed as follows:

[0215] ;

[0216] in, is the window size, and is the image grayscale and The gradient in direction, , ;

[0217] First, use the Harris algorithm as the initial corner detection method to perform preliminary detection of corners in the image and calculate the corner response value. R :

[0218] ;

[0219] in, is a matrix A scalar feature of The area scaling factor reflects the degree of change in regional gradient intensity; is the sum of the main diagonal elements of the matrix, indicating the overall strength of the matrix; is the empirical coefficient, usually taken ;

[0220] Set the corner point response value The points are screened as preliminary corner points, where is the response threshold;

[0221] Secondly, for each preliminary corner point Extract a size of The square window as , Generally, 9 is selected to concentrate on processing local areas and lock the interval of interest, which is convenient for subsequent corner point detection:

[0222] ;

[0223] Again, in Calculate the grayscale gradient of pixels in the region; for grayscale images , use the Sobel operator to calculate the horizontal gradient and the vertical gradient :

[0224] ;

[0225] Based on the horizontal gradient and the vertical gradient , calculate the gradient magnitude of each pixel and direction :

[0226] ;

[0227] Set the gradient amplitude threshold ,like , then the point is taken as the effective response corner point, thereby effectively removing noise and weak edge interference and retaining the real corner point features;

[0228] Finally, after gradient screening, In the region, the quadratic surface fitting method is used to achieve sub-pixel corner location;

[0229] Corner point response value In the local area, it can be approximated as a quadratic function:

[0230] ;

[0231] Among them, av represents the coefficient of the quadratic function (quadratic curve);

[0232] S400, performing matrix sorting on the disordered two-dimensional corner point coordinate set obtained in step S300 based on the corner point sorting method of bidirectional expansion of feature points to obtain a regular order of the two-dimensional corner point coordinate set, and setting a regular order of the three-dimensional corner point coordinate initial value set based on the matrix distribution. The specific implementation effect is as follows: Figure 8 As shown;

[0233] S410, find the sorting feature point to achieve bidirectional expansion sorting, the sorting feature point refers to the corner point used as the sorting starting point, which needs to meet the following characteristics:

[0234] (1) The sorting feature points are located at the edge of the corner point matrix, which is curved rather than straight.

[0235] (2) The sorting feature point is located in the middle of the above edge;

[0236] (3) For the fisheye box 2.0, the sorting feature points of mutually symmetric surfaces are in an axisymmetric relationship;

[0237] S420, calculating the center point coordinates of the detected corner point array, for the right side, the sorting feature points that meet the conditions are located on the left side of the center point, and locating the sorting feature points on the right side through the following steps:

[0238] (1) Calculate the coordinates of the center point C 1( x 1, y 1);

[0239] (2) From all detected corner points, select C 1 The ordinate is closest to n Corner points, n At least one column of corner points;

[0240] (3) From this n Among the corner points, select the corner point with the smallest horizontal coordinate as the sorting feature point of the right side;

[0241] S430, after locating the sorting feature point, use the point as the sorting starting point to sort the corner points, and generate an initial set of three-dimensional coordinates corresponding to the corner points one by one. The corner point sorting process is as follows:

[0242] (1) Sorting feature points on the right side O As the sorting starting point, set its initial three-dimensional coordinate set to (0,0,0);

[0243] (2) Find the distance from all detected corner points O Point to the three corner points with the smallest Euclidean distance, and then select the corner point with the largest horizontal coordinate as the point B (1,0,0); Next, from the remaining two corner points, select the one with the smallest ordinate and smaller than point O The corner points are C (0,-1,0), and select the point with the largest ordinate that is greater than O The corner points are A (0,1,0);

[0244] (3) Point C For the new starting point, find the distance point C Then, select the three corner points with the smallest Euclidean distance and the point with the smallest ordinate that is smaller than C The corner points are D (0,-2,0), repeat this process until no corner point that meets the conditions is found, and return the point A ;

[0245] (4) From point A Start sorting downwards and find the distance point A Then, select the three corner points with the smallest Euclidean distance, and then select the point with the largest ordinate that is greater than A The corner points are E (0,2,0); Repeat this process until no corner point that meets the conditions is found, and return the point B , complete the sorting of the first column of the corner point matrix, and generate an initial set of three-dimensional coordinates corresponding to each one;

[0246] (5) Excluding the sorted corner points, B As the new starting point, follow the same method to find the distance point B The three corner points with the smallest Euclidean distance, choose the corner point with the largest horizontal coordinate as the point G (2,0,0), and then from the remaining two corner points, select the one with the smallest ordinate and smaller than point B The corner points are H (0,-2,0), and select the point with the largest ordinate that is greater than B The corner points are F (0,2,0);

[0247] (6) Similarly, a new starting point is set for bidirectional expansion sorting. After each column of corner points is sorted, the next column is sorted. This continues until all corner points are sorted and the corresponding initial set of three-dimensional coordinates is generated.

[0248] Set point K ( h ,0,0) is the last corner point of the row with the least number of corner points. When executing the corner point sorting algorithm based on bidirectional expansion of feature points, when at point K After the column is sorted, the remaining corner points need to be sorted. First, the remaining corner points are arranged from small to large according to the ordinate, and then the corner point with the smallest ordinate is selected. L As a starting point, look for the point K The distance from the corner point in the column L Nearest corner M ( h , l ,0), finally, point L ( , ,0) is inherited from point M , calculate the initial set of three-dimensional coordinates by the following formula:

[0249] ;

[0250] Similarly, all remaining corner points on the right side are sorted, and the same method is used to sort the corner points on other sides and generate the corresponding initial set of three-dimensional coordinates;

[0251] Since the center plane is a corner point matrix of standard rows and columns, the traditional corner point sorting method is used to sort the center plane and generate the corresponding initial set of three-dimensional coordinates;

[0252] S500, based on the above regular order of the two-dimensional coordinate set of the corner points and the regular order of the three-dimensional coordinate initial value set of the corner points, a six-parameter division model based on the LM optimization algorithm is used to perform high-precision fast fisheye calibration to obtain the high-precision fisheye lens internal parameters and the objective evaluation index RMS;

[0253] After executing this, the sorted two-dimensional coordinate set of corner points and the initial three-dimensional coordinate set of corner points have been obtained. The data can be input into the camera distortion model to calculate the calibration result. The traditional method generally uses a polynomial distortion model to realize fisheye camera calibration, but its mathematical modeling does not conform to the fisheye lens with a larger field of view. Therefore, the present invention adopts a six-parameter division model as distortion, and proposes a six-parameter division model based on the LM optimization algorithm to realize fisheye camera calibration.

[0254] set up are the coordinates of the image distortion point, is the coordinate of the undistorted point of the image, and the six-parameter DM model expression is:

[0255] ;

[0256] in, and Distortion points and the undistorted point To the center of distortion P The Euclidean distance of is the radial distortion parameter, and the single-parameter DM model is adopted:

[0257] ;

[0258] Set the distortion center P The coordinates of , then the coordinates of the undistorted points in the image are:

[0259] ;

[0260] in, is the image distortion point and distortion center satisfy:

[0261] ;

[0262] Let the equation of the line ,in is the slope, is the intercept, that is, the equation of the undistorted line is , substituting into the above formula, we can get:

[0263] ;

[0264] It can be seen from the above formula that under the single-parameter DM model, the ideal straight line is distorted into a circular arc curve. After sorting it out, the following equation is obtained:

[0265] ;

[0266] in, A , B , C The following conditions must be met:

[0267] ;

[0268] It solves the parameters by obtaining three points from the image A , B , C and the distortion center, based on A , B , C The relationship between them is:

[0269] ;

[0270] Obtained by solving A, B, C and the distortion center, and obtain the distortion parameters ;However, due to the excessive number of parameters and equation orders involved in the six-parameter DM model, the LM algorithm (Levenberg-Marquardt) was used to optimize the solution;

[0271] In the six-parameter model based on LM optimization, To optimize the parameters, the objective function is:

[0272] ;

[0273] in, are the coordinates of the distortion point, are the coordinates of the distortion correction points calculated according to the six-parameter division model;

[0274] The iterative formula of the LM algorithm is:

[0275] ;

[0276] in, is the Jacobian matrix; is the identity matrix, is a non-negative number, It can be regarded as a damping coefficient, which is responsible for controlling the direction and step size of the iteration; when When the optimal solution is missed, the increase Thereby reducing the step size; when When the step length is insufficient, reduce Thereby speeding up the convergence;

[0277] The LM algorithm belongs to the trust region method. In each iteration, a local "trust region" is defined, that is, an area near the current point of the optimization problem. In this area, an approximate model is used to replace the real objective function for optimization, thereby reducing the computational cost of directly optimizing the complex objective function. The present invention uses a first-order Taylor expansion to construct a local approximate model of the objective function:

[0278] ;

[0279] in, The radius is The trust domain of

[0280] The ratio of the actual error reduction to the local model predicted reduction As a measure of trust region update, it is shown in the following formula:

[0281] ;

[0282] in, represents the initial state of Q;

[0283] like If it is greater than 1, then decrease ;like If it is less than 1, it will increase ;like If it is close to 1, it indicates that the operation is relatively accurate. Based on the characteristic that the fisheye box has large local distortion, the present invention adopts the above method to solve the DM model into six distortion parameters, thereby more accurately describing the degree of distortion of the fisheye box 2.0 image.

[0284] The above method is used to iterate and optimize the solution with the reprojection error (RMS) as the objective evaluation index;

[0285] S600, perform 3D reconstruction verification based on the high-precision fisheye lens internal parameters and the regular order of the corner point 2D coordinate set, and obtain the regular order of the corner point 3D coordinate set; use the real 3D corner point distance (Distance, DIS) as an additional objective evaluation index to verify the accuracy of the calibration result and the uniformity of the spatial error;

[0286] The 3D reconstruction experiment can verify the accuracy of the calibration results (camera intrinsic parameters, distortion parameters, camera extrinsic parameters, etc.), and is mainly used in the reverse projection process of corner point coordinates from the pixel coordinate system to the world coordinate system. The specific implementation effect is as follows: Fig. 9 As shown;

[0287] The basic form of the division model takes into account radial distortion, which corrects the image coordinates as a function of radial distance; therefore, the distortion correction of the two-dimensional coordinate set of the corner points can be achieved by inversely solving the model to obtain the undistorted point ; Restore the normalized camera coordinates through coordinate system transformation:

[0288] ;

[0289] in, is the normalized coordinate in the camera coordinate system, is the inverse matrix of the camera intrinsic parameter matrix;

[0290] Using the camera's extrinsic matrix (Rotation matrix and translation vector ) will normalize the camera coordinates Convert to three-dimensional coordinates in the world coordinate system :

[0291] ;

[0292] The Euclidean distance between each point is calculated based on the three-dimensional coordinates of the corner points after three-dimensional reconstruction, and the average value of the Euclidean distance is compared with the actual spacing to measure the accuracy of the calibration result. The specific embodiment has the following effects: Figure 8 shown.

[0293] The other combinations and connection relationships of this embodiment are the same as those of the first embodiment.

[0294] Simulation experiment

[0295] In order to verify the accuracy of the high-precision fast camera calibration method based on a large-field-of-view fisheye lens proposed in the present invention, five vehicle-mounted fisheye lenses with a field-of-view angle of 200° were used to conduct shooting experiments under a fixed light intensity. The experimental results are shown in Table 1.

[0296] Table 1 Accuracy experimental test data table

[0297]

[0298] Table 1 shows that the high-precision fast camera calibration method based on a large-field fisheye lens proposed in the present invention performs shooting experiments with 5 lenses under fixed light intensity conditions. The RMS is less than 0.2, and the DIS is close to the actual physical distance 33, which has good calculation accuracy.

[0299] In order to verify the robustness of the high-precision fast camera calibration method based on a large-field-of-view fisheye lens proposed in the present invention, five shooting experiments were carried out under different light intensities using a vehicle-mounted fisheye lens with a field-of-view of 200°. The experimental results are shown in Table 2.

[0300] Table 2 Robustness experimental test data table

[0301]

[0302] Table 2 shows that the high-precision fast camera calibration method based on a large-field fisheye lens proposed in the present invention performs a single-lens shooting experiment under conditions of varying light intensity. The RMS is less than 0.2, the DIS is close to the actual physical distance of 33, and the parameter jump of the same lens is less than 0.1, maintaining high precision while having good robustness.

[0303] In order to verify the superiority of the high-precision fast camera calibration method based on a large-field-of-view fisheye lens proposed in the present invention, a vehicle-mounted fisheye lens with a field-of-view angle of 200° was used to conduct two comparative experiments with the traditional calibration method (Zhang’s calibration method) under fixed light intensity. The experimental results are shown in Table 3.

[0304] Table 3 Superiority Experiment Test Data Table

[0305]

[0306] Table 3 shows that the high-precision fast camera calibration method based on a large-field fisheye lens proposed in the present invention is compared with the traditional calibration method under fixed light intensity conditions using a single fisheye lens. Compared with the traditional method, the RMS of the method proposed in the present invention is reduced by 9.77%, the accuracy of DIS is improved by 0.33%, and the algorithm time consumption is reduced by 28.58%, which has excellent superiority.

[0307] The present invention proposes a high-precision and fast camera calibration method and system based on a large-field fisheye lens. The method includes an image super-resolution reconstruction method based on a multi-scale spatial attention mechanism (SMA), a segmentation method based on an adaptive threshold of convex polygon fitting, a sub-pixel corner point detection method based on ROI gradient screening, a corner point sorting method based on bidirectional expansion of feature points, a six-parameter division model based on the LM optimization algorithm, and a corner point three-dimensional ranging inspection method based on camera intrinsic parameters. Compared with traditional calibration methods, the main advantages of the present invention are as follows:

[0308] (1) Only one fisheye box 2.0 image needs to be taken, which greatly shortens the time required for taking pictures and improves efficiency.

[0309] (2) The image quality during shooting is easily affected by environmental factors, resulting in the image quality not meeting the calibration requirements. The present invention can improve the image quality and reduce the difficulty of shooting through a new image super-resolution reconstruction algorithm.

[0310] (3) The six-parameter division model based on the LM optimization algorithm is adopted, the distortion model is more consistent and the calibration result is more accurate.

[0311] (4) A new objective evaluation index DIS and verification method are proposed to make the calibration results more convincing.

[0312] (5) When using different lenses under the same light intensity, the RMS of the calibration results is less than 0.2, and the DIS is close to the actual physical distance 33. The calibration results show good accuracy.

[0313] (6) Using the same lens under different light intensities, the RMS of the calibration results is less than 0.2, the DIS is close to the actual physical distance 33, and the parameter jump is less than 0.1, showing good robustness.

[0314] (7) Using the same lens under the same light intensity, compared with the traditional fisheye calibration method, the RMS is reduced by 9.77%, the accuracy of DIS is improved by 0.33%, and the algorithm time is reduced by 28.58%, showing outstanding superiority.

[0315] Therefore, the high-precision rapid camera calibration method and system based on a large-field-of-view fisheye lens proposed in the present invention show excellent performance in terms of calibration accuracy, calibration efficiency, stability, robustness, etc., and can meet the calibration requirements of various currently popular fisheye lenses.

[0316] Although the present invention is disclosed as above, the protection scope of the present invention is not limited thereto. Those skilled in the art may make various changes and modifications without departing from the spirit and scope of the present invention, and these changes and modifications will fall within the protection scope of the present invention.

Claims

1. A high-precision fast camera calibration method based on a large-field-of-view fisheye lens, characterized in that: The following steps are involved: S100, super-resolution reconstruction is performed on the single-shot fisheye box image based on a multi-scale spatial attention mechanism to obtain a high-quality image; S200, performing image segmentation on the high-quality image using a segmentation method based on an adaptive threshold of convex polygon fitting, and obtaining five images containing black and white checkerboards; S300, performing corner point detection and sub-pixel precision processing on the five images containing black and white checkerboards obtained in step S200 based on the sub-pixel corner point detection method of ROI gradient screening, and generating a disordered set of two-dimensional coordinates of corner points; S400, performing matrix sorting on the disordered two-dimensional corner point coordinate set obtained in step S300 based on the corner point sorting method of bidirectional expansion of feature points to obtain a two-dimensional corner point coordinate set in a regular order, and generating a three-dimensional corner point coordinate initial value set in a regular order based on matrix distribution; S500, based on the regular order of the corner point two-dimensional coordinate set and the regular order of the corner point three-dimensional coordinate initial value set obtained in step S400, a six-parameter division model based on the LM optimization algorithm is used to perform high-precision fast fisheye calibration to obtain a high-precision fisheye lens internal parameter and an objective evaluation index RMS; S600, based on the high-precision fisheye lens internal parameters obtained in step S500 and the regularly ordered set of two-dimensional coordinates of corner points obtained in step S400, three-dimensional reconstruction verification is performed to obtain a regularly ordered set of three-dimensional coordinates of corner points; the real three-dimensional corner point spacing is used as an additional objective evaluation indicator to verify the accuracy of the calibration result and the uniformity of the spatial error.

2. The high-precision rapid camera calibration method based on a large-field-of-view fisheye lens according to claim 1, characterized in that: In step S100, specifically including: S110. According to the channel attention mechanism, set is the input feature map, where are the height, width, and number of input channels of the input feature map, respectively. is a real number; C Global average pooling operator for channels for: ； in, For the C The channel height is i Width is j The feature map of No. C The attention weight of each channel It is expressed by the following formula: ； in, is the corrected linear unit, and All are fully connected layers. Represents the dimension space of the matrix; is the activation function; S120, based on the multi-scale spatial attention mechanism, is divided into two sub-modules: SA sub-module and MA sub-module. The SA sub-module uses the spatial attention mechanism to calculate the spatial attention feature map by calculating the spatial relationship within the feature map space; the spatial attention weight It is expressed by the following formula: ； Among them, the input feature map , perform global average pooling and global maximum pooling on the channel dimension, compress the channel size, and obtain two Feature map and : ； ； The results of global maximum pooling and global average pooling are concatenated according to the channels, and the concatenated results are convolved to obtain the output feature map. : ； ； in, represents the convolution operation, is the activation function; Indicates channel multiplication; The S130 and MA submodules use multi-scale convolution kernels to make the network produce different spatial resolutions and depths in a multi-branch manner, which is used to extract spatial information of different scales on each channel of the feature map; Assume that the input channel dimension of each branch is , then each different scale The feature maps have the same channel dimension ,in, ; Can be divisibility; A group convolution algorithm is introduced in the convolution kernel. The algorithm designs a new group convolution size selection criterion. The relationship between the multi-scale convolution kernel size and the group convolution size is expressed by the following formula: ； in, K is the size of the convolution kernel, G is the size of the group convolution; Multi-scale feature map generation function It is expressed by the following formula: ； Among them, i The size of the convolution kernel , No. i The size of the group convolution ,set up , the complete multi-scale feature preprocessing graph is obtained by concatenation, as shown in the following formula: ； in, is the obtained multi-scale feature map; By extracting the channel attention weight information from the multi-scale preprocessed feature map, we get the attention weight vectors of different scales: ； in, is the attention weight; Use the channel weight algorithm to obtain attention weights from input feature maps of different scales , and then the entire multi-scale channel attention vector is obtained in series: ； in, For the connection operator, for The attention value, Z is the multi-scale attention weight vector; The soft attention method is used to adaptively select different spatial scales across channels. Soft attention weights It is expressed by the following formula: ； Among them, use To obtain the recalibrated weights of the multi-scale channels , which contains the spatial location information and the attention weight in the channel; Then, the feature-recalibrated channel attentions are fused and concatenated in series to obtain the entire channel attention vector: ； in, Represents the multi-scale channel weights after attention interaction; the recalibrated multi-scale channel attention weights are compared with the corresponding scale Multiplying the feature map of gives the following formula: ； in, represents channel multiplication, Represents the feature map obtained using multi-scale channel attention weights; By embedding the SMA module into the super-resolution reconstruction network and using the checkerboard image dataset for weight training, super-resolution reconstruction of fisheye box images can be achieved.

3. The high-precision rapid camera calibration method based on a large-field-of-view fisheye lens according to claim 1, characterized in that: In step S200, specifically including: S210, using the Scharr operator to perform edge detection on the fisheye box image reconstructed by super-resolution in step S100; based on the change of pixel gradient, the blank gaps between the surfaces are used as the outer frame contour points of the black and white chessboard; S220, performing convex polygon fitting on the outer frame contour points of the black and white chessboard, and connecting the contour points to ensure that the black and white chessboard of each face is within the convex polygon fitting area; S230, segmenting the original image according to the convex polygon fitting area obtained in step S220, and obtaining five preliminary segmentation effect images; S240, performing grayscale processing on each preliminary segmentation effect image, and performing adaptive threshold processing on the grayscale image to obtain a binary image; S250, performing pixel threshold screening on the binary image, screening out blank gaps, and using them as segmentation boundaries to perform fine segmentation on the original image.

4. The high-precision rapid camera calibration method based on a large-field-of-view fisheye lens according to claim 1, characterized in that: In step S300, specifically including: The second-order matrix of image gradient is expressed as follows: ； in, is the window size, and is the image grayscale and The gradient in direction, , ; First, use the Harris algorithm as the initial corner detection method to perform preliminary detection of corners in the image and calculate the corner response value. : ； in, is a matrix A scalar feature of The area scaling factor of ; is the sum of the main diagonal elements of the matrix, indicating the overall strength of the matrix; is the empirical coefficient; Set the corner point response value The points are screened as preliminary corner points, where is the response threshold; secondly, for each preliminary corner point Extract a size of The square window as : ； Again, in Calculate the grayscale gradient of pixels in the region; for grayscale images , use the Sobel operator to calculate the horizontal gradient and the vertical gradient : ； Based on the horizontal gradient and the vertical gradient , calculate the gradient magnitude of each pixel and direction : ； Set the gradient amplitude threshold ,like , then this point is regarded as the effective response corner point; Finally, after gradient screening, In the area, the quadratic surface fitting method is used to achieve sub-pixel corner location; the corner response value Approximately a quadratic function in a local region: ； in, a , b , c , d , e , v are the coefficients of the general form of the quadratic function.

5. The high-precision rapid camera calibration method based on a large-field-of-view fisheye lens according to claim 1, characterized in that: In step S400, specifically including: S410, find the sorting feature point to achieve bidirectional expansion sorting, the sorting feature point refers to the corner point used as the sorting starting point, which needs to meet the following characteristics: (1) The sorting feature points are located at the edge of the corner point matrix, which is curved rather than straight. (2) The sorting feature points are located in the middle of the edge of the corner point matrix; (3) For the fisheye box, the sorting feature points of mutually symmetric surfaces are in an axisymmetric relationship; S420, calculating the center point coordinates of the detected corner point array, for the right side, the sorting feature points that meet the conditions are located on the left side of the center point, and locating the sorting feature points on the right side through the following steps: (1) Calculate the coordinates of the center point C 1( x 1, y 1); (2) From all detected corner points, select C 1 The ordinate is closest to n Corner points, n At least one column of corner points; (3) From n Select the corner point with the smallest horizontal coordinate from the corner points as the sorting feature point on the right side; S430, after locating the sorting feature point on the right side, use the point as the sorting starting point to sort the corner points, and generate an initial set of three-dimensional coordinates corresponding to the corner points one by one. The corner point sorting process is as follows: (1) Sorting feature points on the right side O As the sorting starting point, set its initial three-dimensional coordinate set to (0,0,0); (2) Find the distance from all detected corner points O Point to the three corner points with the smallest Euclidean distance, and then select the corner point with the largest horizontal coordinate as the point B (1,0,0); Next, from the remaining two corner points, select the one with the smallest ordinate and smaller than point O The corner points are C (0,-1,0), and select the point with the largest ordinate that is greater than O The corner points are A (0,1,0); (3) Point C For the new starting point, find the distance point C Then, select the three corner points with the smallest Euclidean distance and the point with the smallest ordinate that is smaller than C The corner points are D (0,-2,0), repeat this process until no corner point that meets the conditions is found, and return the point A ; (4) From point A Start sorting downwards and find the distance point A Then, select the three corner points with the smallest Euclidean distance, and then select the point with the largest ordinate that is greater than A The corner points are E (0,2,0); Repeat this process until no corner point that meets the conditions is found, and return the point B , complete the sorting of the first column of the corner point matrix, and generate an initial set of three-dimensional coordinates corresponding to each one; (5) Excluding the sorted corner points, B As the new starting point, follow the same method to find the distance point B The three corner points with the smallest Euclidean distance, choose the corner point with the largest horizontal coordinate as the point G (2,0,0), and then from the remaining two corner points, select the one with the smallest ordinate and smaller than point B The corner points are H (0,-2,0), and select the point with the largest ordinate that is greater than B The corner points are F (0,2,0); (6) Similarly, a new starting point is set for bidirectional expansion sorting. After each column of corner points is sorted, the next column is sorted. This continues until all corner points are sorted and the corresponding initial set of three-dimensional coordinates is generated. Set point K ( h ,0,0) is the last corner point of the row with the least number of corner points. When executing the corner point sorting algorithm based on bidirectional expansion of feature points, when at point K After the column is sorted, the remaining corner points are sorted. First, the remaining corner points are arranged from small to large according to the ordinate, and then the corner point with the smallest ordinate is selected. L As a starting point, and look for the point K The distance from the corner point in the column L Nearest corner M ( h , l ,0), finally, point L ( , ,0) is inherited from point M , calculate its initial set of three-dimensional coordinates by the following formula: ； Similarly, all remaining corner points on the right side are sorted, and the same method is used to sort the corner points on other sides and generate the corresponding initial set of three-dimensional coordinates.

6. The high-precision rapid camera calibration method based on a large-field-of-view fisheye lens according to claim 1, characterized in that: In step S500, specifically including: are the coordinates of the image distortion point, is the coordinate of the undistorted point of the image, and the six-parameter DM model expression is: ; in, and Distortion points and the undistorted point To the center of distortion P The Euclidean distance of is the radial distortion parameter, and the single-parameter DM model is adopted: ； Set the distortion center P The coordinates of , then the coordinates of the undistorted points in the image are: ； in, is the image distortion point and distortion center The distance satisfies: ； Let the equation of the line ,in k is the slope, b is the intercept, that is, the equation of the undistorted line is , substituting into the above formula, we can get: ； From the above formula, it can be concluded that under the single-parameter DM model, the ideal straight line is distorted into a circular arc curve: ； in, A , B , C The following conditions must be met: ； Solve the parameters by taking three points from the image A , B , C and the distortion center, based on A , B , C The relationship between them is: ； Obtained by solving A, B, C and the distortion center, and obtain the distortion parameters ; The LM algorithm is used to optimize and solve the six-parameter DM model; In the six-parameter model based on LM optimization, As the optimization parameter, the objective function is: ； in, are the coordinates of the distortion point, are the coordinates of the distortion correction points calculated according to the six-parameter division model; The iterative formula of the LM algorithm is: ； in, is the Jacobian matrix; is the identity matrix, is a non-negative number, is regarded as the damping coefficient; when When the optimal solution is missed, the increase Thereby reducing the step size; when When the step length is insufficient, reduce Thereby speeding up the convergence speed; Use a first-order Taylor expansion to construct a local approximation model of the objective function: ； in, The radius is The trust region of the local model is the ratio of the actual error reduction to the local model prediction reduction. As a measure of trust region update, it is shown in the following formula: ； in, represents the initial state of Q; if If it is greater than 1, then decrease ;like If it is less than 1, it will increase ;like If it is close to 1, it indicates that the operation is relatively accurate. Based on the characteristic of large local distortion of the fisheye box, the DM model is solved into six distortion parameters to more accurately describe the distortion degree of the fisheye box image.

7. The high-precision rapid camera calibration method based on a large-field-of-view fisheye lens according to claim 1, characterized in that: In step S600, specifically including: By reversing the model, the distortion correction of the two-dimensional coordinate set of the corner points is achieved, and the undistorted points are obtained. ; Restore the normalized camera coordinates through coordinate system transformation: ； in, is the normalized coordinate in the camera coordinate system, is the inverse matrix of the camera intrinsic parameter matrix; Using the camera's extrinsic matrix ,in, is the rotation matrix and is the translation vector, normalizing the camera coordinates Convert to three-dimensional coordinates in the world coordinate system : ； The Euclidean distance between each point is calculated according to the three-dimensional coordinates of the corner points after three-dimensional reconstruction, and the average of the Euclidean distance is compared with the actual distance to measure the accuracy of the calibration result.

8. The high-precision rapid camera calibration method based on a large-field-of-view fisheye lens according to claim 1, characterized in that: The fisheye box is a rectangular structure with an open top, and each inner surface thereof is painted with black and white square checkerboards in regular n×m rows and columns, and the outer edges of the black and white square checkerboards are spliced ​​with a circle of black and white checkerboard grids surrounding them; blank gaps are set between two adjacent surfaces; two groups of mutually perpendicular line segments are drawn at the center position of the center surface of the fisheye box to form a crosshair; circular marks are drawn on the black and white square checkerboards on the left and right inner surfaces of the fisheye box, respectively, in the middle of a row of checkerboard grids close to the center surface.

9. A high-precision fast camera calibration system based on a large-field-of-view fisheye lens, characterized in that: The system has a program module corresponding to the steps of any one of claims 1 to 8, and executes the steps of the high-precision rapid camera calibration method based on a large-field-of-view fisheye lens when running.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is configured to implement the steps of the high-precision fast camera calibration method based on a large-field-of-view fisheye lens according to any one of claims 1 to 8 when called by a processor.

Images