Vehicle-mounted around-view camera calibration method of unmanned sanitation vehicle

Abstract

The invention provides a vehicle-mounted all-round view camera calibration method for an unmanned sanitation vehicle, and belongs to the technical field of camera calibration, and the method comprises the steps: carrying out the distortion removal processing of the image data of fisheye cameras around the unmanned sanitation vehicle, and carrying out the data enhancement processing through homography transformation; estimating an angle between the vanishing point and the horizon by using a deep learning network, and estimating an external parameter of the fisheye camera according to the angle between the vanishing point and the horizon; and designing an external parameter estimation network based on ResNet-18, and training the external parameter estimation network by taking the estimated external parameters of the fisheye camera as weak supervision, so that the external parameter estimation network performs external parameter calibration of the vehicle-mounted look-around fisheye camera in a natural lane scene, and calibration of the vehicle-mounted look-around fisheye camera is completed. According to the invention, the self-calibration problem of the unmanned sanitation vehicle in a natural scene is solved, so that the unmanned sanitation vehicle does not need to be manually returned to a factory for calibration when external parameters are changed due to the influence of environmental factors.

Description

Technical Field

[0001] The invention belongs to the technical field of camera calibration, and in particular relates to a method for calibrating a vehicle-mounted surround-view fisheye camera of an unmanned sanitation vehicle. Background Art

[0002] The calibration technology of on-board fisheye camera is a key step to ensure that the on-board camera can accurately capture and transmit image information. It includes multiple links such as internal parameter calibration, external parameter calibration, distortion correction and coordinate conversion. Among them, the calibration of internal parameters and distortion parameters is generally implemented by the camera manufacturer and will not change in actual applications. Therefore, the technology here mainly discusses the calibration of external parameters. The external parameters will cause the camera to shift due to vibrations when the vehicle is driving, which will cause errors in the panoramic surround image stitching based on external parameters. Therefore, the calibration of vehicle external parameters in natural scenes is an urgent problem to be solved. At present, for the on-board panoramic system, most of them adopt the calibration workshop calibration method to calibrate the external parameters of the vehicle, that is, the vehicle enters a specially arranged calibration workshop before leaving the factory, and calibrates the external parameters before leaving the factory. The problem brought about by this is that the external parameters are determined after the vehicle leaves the factory, but due to driving problems, the external parameters may change, resulting in failure of the panoramic system of the vehicle, which can only be solved by returning to the factory for repair and calibration.

[0003] In the prior art, a vehicle-mounted fisheye camera is installed on the vehicle, and the internal parameters of the vehicle-mounted fisheye camera are obtained; the driving state of the vehicle is obtained, and when the vehicle is in a driving state, the road condition information is identified. When it is a flat road, the driving speed state of the vehicle is obtained. When the driving speed state of the vehicle is a uniform driving state, the lane line detection result is obtained according to the lane line detection algorithm to determine whether the lane line is a straight line segment. The disadvantages of this method are that the method assumes that the vehicle-mounted fisheye camera has no roll angle; the speed of the vehicle needs to be obtained for auxiliary calculation; the lane line clustering is greatly affected by noise, and the result is unstable; it is impossible to calibrate the vehicle-mounted surround fisheye camera, and only the front and rear fisheye cameras can be calibrated.

[0004] In the prior art, a calibration workshop is used to calibrate the in-vehicle surround-view fisheye camera. First, ensure that the vehicle is on a horizontal plane without obvious slopes. Place four checkerboard calibration plates in the four directions of the front, rear, left, and right of the vehicle in the workshop, ensure uniform lighting, and prevent shadows or specular reflections on the calibration plate from affecting the algorithm's recognition of the calibration plate. The following steps are taken in a horizontal scenario to draw the vehicle body coordinate system on the ground. Find the front and rear symmetric points of the vehicle, draw marks on the ground with a plumb bob, and connect the front and rear points respectively to obtain two lines parallel to the front and rear bumpers of the vehicle. Since the front and rear points are symmetric, the intersection point of the vehicle body's central axis and the front and rear horizontal lines can be obtained through these two points at this time; extend equal distances (such as 1.5 m) from the midpoints of the front and rear horizontal lines to the left and right respectively to obtain symmetric points on both sides. Then, use a horizontal laser level to draw points perpendicular to the front and rear horizontal lines at the extended symmetric points. Through the above operations, the circumscribed rectangle related to the vehicle body is obtained. As long as the calibration plate is parallel to the lines on the ground, it can be ensured that the calibration plate and the vehicle body coordinate system are horizontal. Also, since the calibration plate is guaranteed to be horizontal with the ground through a bracket, the calibration plate can be orthogonal to the vehicle body coordinate system. Finally, take images of the calibration plate in the scenario and select the images that meet the requirements for external parameter calibration. Its disadvantages are that due to the distortion of the fisheye camera and a certain installation angle, when the calibration plate has a certain angle towards the fisheye camera, the checkerboard of the calibration plate will have a large deformation and cannot be recognized by the recognition algorithm. Therefore, it is necessary to ensure that the distortion of the captured calibration plate image is not too large, that is, the angle between the optical axis of the fisheye camera and the normal line of the calibration plate cannot be too large during the shooting process; the range of multiple images of the calibration plate needs to cover the field of view of the fisheye camera as much as possible; the calibration plate needs to be as flat as possible without obvious wrinkles; it relies on the calibration scenario assistance and cannot perform self-calibration.

[0005] Some suppliers also provide an automotive AVM self-calibration method that does not rely on a calibration workshop, that is, the vehicle drives slowly on the road for a certain distance, and uses prior information such as lane lines to calibrate the external parameters of the fisheye camera. According to the perspective projection model of the fisheye camera, parallel lane lines in the three-dimensional fisheye camera coordinate system will intersect at a point on the two-dimensional imaging plane of the fisheye camera, that is, the vanishing point, which corresponds to the infinite point of the parallel lane lines in the three-dimensional fisheye camera coordinate system. First, perform lane line detection. However, due to noise, they will not intersect at a point. Therefore, it is transformed into a problem of solving an overdetermined equation to solve the coordinates of the vanishing point in the image; then, a constraint relationship is constructed through the fisheye camera model, the two-dimensional image coordinates of the vanishing point, and the prior that the Z coordinate of the vanishing point in the three-dimensional coordinates is infinite, and the rotation angles of the fisheye camera on the x, y, and z axes are calculated, and then the attitude of the fisheye camera is calculated. Its disadvantages are that lane line detection is greatly affected by noise, resulting in errors; it can only calibrate the front and rear cameras in the surround-view fisheye camera. Summary of the Invention

[0006] In view of the above deficiencies in the prior art, the present invention provides a method for calibrating an on-vehicle surround-view camera of an unmanned sanitation vehicle, which solves the problem of self-calibration of the unmanned sanitation vehicle in a natural scene and enables it to be calibrated without manual return to the factory when the external parameters change due to environmental factors.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A method for calibrating an on-vehicle surround-view camera of an unmanned sanitation vehicle, comprising the following steps: S1. Perform undistortion processing on the fisheye camera image data around the unmanned sanitation vehicle, and perform data enhancement processing through homography transformation; S2. According to the fisheye camera image data after data enhancement processing, use a deep learning network to estimate the angles between the vanishing points and the horizon of the front and rear fisheye cameras, and estimate the external parameters of the fisheye cameras based on the angles between the vanishing points and the horizon; S3. Design an external parameter estimation network based on ResNet-18, and use the estimated external parameters of the fisheye cameras as weak supervision to train the external parameter estimation network, so that the external parameter estimation network performs on-vehicle surround-view fisheye camera external parameter calibration in a natural lane scene, and completes the calibration of the on-vehicle surround-view fisheye camera of the unmanned sanitation vehicle.

[0008] Further, the step S1 includes the following steps: S101. For a point in the fisheye camera coordinate system around the unmanned sanitation vehicle, use the perspective projection model to obtain the projection point when the fisheye camera image data has not undergone distortion; S102. According to the projection point when there is no distortion, obtain the Cartesian coordinate value by obtaining the polar coordinates of the projection point after distortion of the fisheye camera image data; S103. According to the Cartesian coordinate value, convert the projection point after distortion to the pixel coordinate system through the internal parameters of the fisheye camera, and complete the undistortion processing of the fisheye camera image data; S104. According to the fisheye camera image data after undistortion processing, perform data enhancement processing through homography transformation.

[0009] Still further, the step S2 includes the following steps: S201. According to the fisheye camera image data after data enhancement processing, perform encoding processing using the Tranformer encoding layer; S202. According to the fisheye camera image data after data enhancement processing, perform line detection processing using the line segment detection operator LSD, and input the detection result into the decoding layer together with the encoding result after passing through the convolutional layer; S203. Use the decoding layer to output the angles between the vanishing points and the horizon; S204. Estimate the rotation matrix in the extrinsic parameters according to the angle between the vanishing point and the horizon line, and complete the estimation of the extrinsic parameters of the fisheye camera.

[0010] Furthermore, the expression of the rotation matrix in the extrinsic parameters is as follows:

[0011]

[0012]

[0013]

[0014] Among them, represents the rotation matrix, represents the rotation angle, represents the horizon line angle estimated in step S2, , , represent the three components of the third column of the rotation matrix.

[0015] Furthermore, the expression of the loss function of the extrinsic parameter estimation network based on ResNet-18 is as follows:

[0016]

[0017]

[0018]

[0019]

[0020]

[0021] Among them, represents the loss function of the extrinsic parameter estimation network, represents the current training epoch, represents the total number of training epochs, represents the rotation loss, and respectively represent the coefficients of the orthogonality loss and the rotation loss, represents the orthogonality loss, represents the photometric loss, represents the rotation matrix, represents the fisheye camera i of the pre-estimated extrinsic parameters, represents the index set of adjacent fisheye cameras, represents the selected set of points that meet the requirements, Represents a point in the Bird's Eye View (BEV) coordinate system, Represents at the point The photometric loss value generated, i And j Respectively represent two adjacent fisheye cameras, Represents the fisheye camera The Bird's Eye View (BEV) of, Represents the fisheye camera The Bird's Eye View depth of, Represents the fisheye camera The internal parameters of, Represents the fisheye camera The external parameters of, Represents the internal parameters of the equivalent fisheye camera in the Bird's Eye View (BEV), Represents the scale factor, Represents the fisheye camera The Bird's Eye View (BEV) of, Represents the fisheye camera The Bird's Eye View depth of, Represents the fisheye camera The internal parameters of, Represents the fisheye camera The external parameters of, Represents the region of interest jointly included in the images of adjacent fisheye cameras, And Respectively represent the fisheye camera And The BEV images of, Represents the value of the th row and th column of the rotation matrix of the th fisheye camera.

[0022] The beneficial effects of the present invention are: (1) The present invention solves the problem that the vehicle needs to calibrate the surround-view fisheye cameras in a calibration workshop, and realizes the calibration of the surround-view fisheye cameras in the natural lane scene; at the same time, an algorithm based on deep learning is proposed to avoid the problem of excessive noise in lane line detection; a method for pre-estimating the external parameters based on vanishing point and horizon detection to assist deep learning training is proposed, realizing unsupervised training; improving the environmental perception ability of unmanned sanitation vehicles, promoting the development of in-vehicle intelligence, and driving the application of computer vision and machine learning in the in-vehicle field; (2) For unmanned sanitation vehicles, the external parameter deviation caused by driving bumps can be corrected by the method of the present invention, improving the intelligent perception ability of unmanned sanitation vehicles and reducing the maintenance cost.

[0023] (3) The present invention proposes an online self-calibration method that does not require the use of a calibration object. As long as the road surface in the scene is flat and there is a pair of parallel lines parallel to the vehicle driving direction, these conditions do not need to be manually searched for and judged, and the judgment is completely intelligent. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a flowchart of the method of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0025] The following describes the specific implementation manners of the present invention to facilitate those skilled in the art of the present technology to understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation manners. For those of ordinary skill in the art of the present technology, as long as various changes are within the spirit and scope of the present invention defined and determined by the appended claims, these changes are obvious, and all inventions created using the concept of the present invention are within the scope of protection.

[0026] EXAMPLE As Figure 1 shown, the present invention provides a calibration method for on-vehicle surround cameras of an unmanned sanitation vehicle, and the implementation method is as follows: S1. Perform distortion removal processing on the fisheye camera image data around the unmanned sanitation vehicle, and perform data enhancement processing through homography transformation. The implementation method is as follows: S101. For a point in the fisheye camera coordinate system around the unmanned sanitation vehicle, use the perspective projection model to obtain the projection point when the fisheye camera image data has not undergone distortion; S102. According to the projection point when there is no distortion, obtain the Cartesian coordinate value by obtaining the polar coordinates of the projection point after the fisheye camera image data has undergone distortion; S103. According to the Cartesian coordinate value, convert the projection point after distortion to the pixel coordinate system through the internal parameters of the fisheye camera to complete the distortion removal processing of the fisheye camera image data; S104. Perform data enhancement processing through homography transformation according to the fisheye camera image data after distortion removal processing; S2. According to the fisheye camera image data after data enhancement processing, use a deep learning network to estimate the angles between the vanishing points and the horizon of the front and rear fisheye cameras, and estimate the external parameters of the fisheye cameras according to the angles between the vanishing points and the horizon. The implementation method is as follows: S201. According to the fisheye camera image data after data enhancement processing, perform encoding processing using the Tranformer encoding layer; S202. According to the fisheye camera image data after data enhancement processing, perform line detection processing using the line segment detection operator LSD, and input the detection result into the decoding layer together with the encoding result after passing through the convolutional layer; S203. Output the angle between the vanishing point and the horizon using the decoding layer; S204. Estimate the rotation matrix in the extrinsic parameters based on the angle between the vanishing point and the horizon, and complete the estimation of the extrinsic parameters of the fisheye camera; The expression of the rotation matrix in the extrinsic parameters is as follows:

[0027]

[0028]

[0029]

[0030] Where, represents the rotation matrix, represents the rotation angle, represents the horizon angle estimated in step S2, 、 、 represent the three components of the third column of the rotation matrix.

[0031] S3. Design an extrinsic parameter estimation network based on ResNet-18, and use the estimated extrinsic parameters of the fisheye camera as weak supervision to train the extrinsic parameter estimation network, so that the extrinsic parameter estimation network calibrates the extrinsic parameters of the on-vehicle surround-view fisheye camera in a natural lane scene, and completes the calibration of the on-vehicle surround-view fisheye camera of the unmanned sanitation vehicle.

[0032] In this embodiment, it is assumed that the intrinsic parameters K 、distortion coefficients D of the fisheye camera are known. The equidistant projection model is used, and since the focal length f has nothing to do with the distortion, it is advisable to take the focal length , then there is: Where, represents the projected distance after distortion, represents the projected angle after distortion, 、 、 and respectively represent the four parameters D in the distortion coefficients 。 For a point in the fisheye camera coordinate system, the projected point before distortion can be obtained by the perspective projection model. Let its polar coordinates be , and the projection incident angle be . From the above formula, can be approximately obtained , and , so the projected point The polar coordinates are , and then the Cartesian coordinate values are obtained. Finally, through the internal parameters of the fisheye camera, is transformed into the pixel coordinate system. In the process of distortion removal, a high-order equation about needs to be solved, and the Newton iteration method can be used to solve it.

[0033] In this embodiment, the training data uses an open-source fisheye camera dataset, which is composed of a vehicle photographed in various road scenarios. Since the external parameters of the fisheye cameras equipped on different vehicles in the real scenario usually vary, the collected data should cover different external parameters as much as possible. However, it is time-consuming and costly in manpower to collect data by repeatedly manually adjusting the pose of the fisheye camera and multiple times under different external parameter configurations. Therefore, the external parameters of the fisheye camera remain fixed during the collection process, and the richness of the external parameters of the collected data is improved by applying the homography transformation.

[0034] In this embodiment, it is assumed that the coordinates of a point in space in the fisheye camera coordinate system are , and the coordinates in the pixel coordinate system are , K represents the internal parameters of the fisheye camera, respectively represent the coordinate values, and the following is satisfied: , represents the depth of the point from the projection center. Assume that the coordinates of this point in the fisheye camera coordinate system b are , then , represents the rotation matrix for the coordinate system transformation from b to a, represents the translation vector for the coordinate system transformation from b to a. Transforming to the corresponding pixel coordinate system, we have: . Since there is only a rotation transformation in data augmentation, , and in the homogeneous coordinate system, the coefficient has no influence, so it can be sorted out as: , where represents the coordinates of the point in the b coordinate system, represents b the internal parameters of the fisheye camera, represents the rotation matrix for the coordinate system transformation from b to a, represents the coordinates of the point in the a coordinate system, represents the homography transformation matrix, represents a the internal parameters of the fisheye camera. Since the internal parameters of the fisheye camera do not change, the above , the rotation matrix is calculated from the rotation angle. To simulate the slight change of the external parameters in the vehicle-mounted scenario, the rotation angles on each axis are set within the range of 0.2 rad, and a total of 9 angles from negative to positive are used for data augmentation.

[0035] In this embodiment, a deep learning method is used to estimate the vanishing point and the horizon. The neural network uses the pose estimation network Hourglass network as the backbone to extract feature maps, uses the Tranformer as the basic architecture, and inputs the extracted line features into the decoding layer for detection. Specifically: according to the fisheye camera image data after data augmentation, use the Tranformer encoding layer for encoding processing; according to the fisheye camera image data after data augmentation, use LSD for line detection processing, and input the detection results into the decoding layer together with the encoding results after passing through the convolutional layer; use the decoding layer to output the vanishing point and the horizon angle; according to the vanishing point and the horizon angle, estimate the rotation matrix in the external parameters to complete the estimation of the external parameters of the fisheye camera.

[0036] In this embodiment, the vanishing point coordinates are estimated by the external parameter estimation network and the included angle between the horizon and the horizontal After that, the rotation matrix in the external parameters is estimated by the following derivation. Assume that the internal parameters of the fisheye camera are K , since the vanishing point is the intersection of two parallel lines in the image at infinity, its depth z is infinity. According to the proportional property of the homogeneous coordinate system, assume that the third column of the rotation matrix is , then:

[0037] Among them, represents the vanishing point coordinates, represents the rotation matrix in the external parameters of the fisheye camera, represents the translation vector in the external parameters of the fisheye camera. After transformation, we can get:

[0038] Among them, , and respectively represent the three components of the third column of the rotation matrix. Assume that the rotation angles in each direction are , and the rotation matrices in three directions , , , then , from which we can calculate:

[0039] Among them , substituting the above formula, we can obtain: , , and then the rotation matrix in the external parameter matrix can be calculated, realizing the pre - estimation of the external parameters between the front fisheye camera and the rear fisheye camera.

[0040] In this embodiment, an improved ResNet - 18 is used as the backbone to design a neural network. Four fisheye camera images of 224*224 in the front, rear, left, and right directions are input simultaneously, and the external parameters of the four fisheye cameras are output. The loss function of this network consists of three parts: rotation loss, photometric loss, and orthogonality loss.

[0041] In this embodiment, the rotation loss Loss rot is obtained by calculating the smooth L1 loss between the rotation matrix in the predicted external parameter matrix and the pre - estimated reference rotation matrix. Assume that the pre - estimated external parameters of the fisheye camera i are , and the rotation matrix output by the network is , then: .

[0042] The photometric loss is a self - supervised loss term. Assume that in two adjacent fisheye cameras and fisheye camera , a certain area also contains a point in the world coordinate system. Then, in the ideal state, the gray values of this point in the two fisheye camera images should be equal. Assume that the photometric loss generated by this point is , then:

[0043] Among them, and are respectively and 's BEV images. The above formula can be specifically written as:

[0044] Among them, is a proportionality coefficient introduced to eliminate the influence of exposure differences, which is determined by the exposure time ratio of the two fisheye cameras, that is: . However, since the exposure time is unknown, the ratio of the sum of the gray values in the region of interest can be selected to replace it, that is:

[0045] Among them, is the region of interest jointly included in the adjacent fisheye camera images. For the final photometric loss, it is obtained by summing up each loss term with loss, that is:

[0046] Among them, is the index set of adjacent fish-eye cameras, is the set of selected points that meet the requirements.

[0047] In this embodiment, for the selection of the point set, the gradient of should be large enough to provide good features. Assuming the gradient image is

[0048] where is the mean gradient in the region of interest and is the standard deviation of the gradient. However, for the selection of these points, the object height also has a great influence, such as sidewalks, pedestrians, etc. Such objects may not have matching pixel points. Therefore, it should be ensured that the selected points are as much as possible on the ground. For this, a color-based strategy is adopted here, that is, for a point that meets the requirements, the color difference between it on and should be small. Assuming , are respectively , on the channel above the figure, then the standard deviation can be used to measure the color difference of a point, that is:

[0049] where is the number of color channels, is the mean value of the color ratio in the region, and the color ratio is defined as: , then for the point that meets the requirements, it should satisfy: where is the mean value of the color difference in the region of interest and is the standard deviation.

[0050] In this embodiment, the external parameter matrix is composed of a rotation matrix and a translation vector. Among them, although the rotation matrix has 9 parameters, it has only 3 degrees of freedom, so it is strictly constrained. And it is very difficult to solve the optimization constraint problem. Therefore, a relatively relaxed method is selected, that is, an orthogonal loss is introduced to ensure that the constraint conditions of the rotation matrix are met. For the rotation matrix R, it should satisfy two properties: the column vectors are pairwise orthogonal, that is , ; the column vectors are unit vectors, that is . Therefore, the orthogonal loss is defined as:

[0051] For the training method, we hope to promote the convergence of the network by using the pre-estimated external parameters as the supervision term, and then fine-tune it through photometric loss and orthogonality loss. Therefore, the proportion of the rotation loss is relatively large in the early stage of training and gradually decreases in the later stage, and finally gets rid of the limitation of the pre-estimation. The final loss function is as follows:

[0052] Among them, represents the loss function of the external parameter estimation network, represents the current training epoch, represents the total number of training epochs, represents the rotation loss, and represent the coefficients of the orthogonality loss and the rotation loss respectively, represents the orthogonality loss, represents the photometric loss, represents the rotation matrix, represents the pre-estimated external parameters of the fisheye camera i represents the set of indices of adjacent fisheye cameras, represents the selected set of points that meet the requirements, represents a point in the bird's-eye view BEV coordinates, represents at the point the generated photometric loss value, i and j represent two adjacent fisheye cameras respectively, represents the fisheye camera bird's-eye view BEV, represents the fisheye camera bird's-eye view depth, represents the fisheye camera intrinsic parameters, represents the fisheye camera extrinsic parameters, represents the intrinsic parameters of the equivalent fisheye camera in the bird's-eye view BEV, represents the scale factor, represents the fisheye camera bird's-eye view BEV, represents the fisheye camera bird's-eye view depth, represents the fisheye camera intrinsic parameters, represents the fisheye camera extrinsic parameters, represents the region of interest jointly included in the images of adjacent fisheye cameras, and represent the fisheye camera respectively​ and the BEV image of indicating the value of the th row and th column of the rotation matrix of the th fish-eye camera.

[0053] In this embodiment, after testing the value ranges from 0.1 to 0.2, the value ranges from 0.5 to 1, and the convergence effect is better when the number of training rounds is between 15 and 20.

[0054] In summary, the present invention realizes the external parameter calibration of the unmanned sanitation vehicle in the natural scene, enables the vehicle calibration to be no longer limited by the calibration scene, avoids the additional maintenance cost caused by the change of the external parameters due to driving, and improves the driving safety.

Claims

1. A calibration method for on-vehicle surround cameras of an unmanned sanitation vehicle, characterized in that, It includes the following steps: S1. Perform undistortion processing on the fisheye camera image data around the driverless sanitation vehicle, and perform data augmentation processing through homography transformation; S2. According to the fisheye camera image data after data augmentation processing, use a deep learning network to estimate the angles between the vanishing points and the horizon of the front and rear fisheye cameras, and estimate the external parameters of the fisheye cameras according to the angles between the vanishing points and the horizon; S3. Design an external parameter estimation network based on ResNet-18, and use the estimated external parameters of the fisheye cameras as weak supervision to train the external parameter estimation network, so that the external parameter estimation network calibrates the external parameters of the on-vehicle panoramic fisheye cameras in the natural lane scenario, and completes the calibration of the on-vehicle panoramic fisheye cameras of the driverless sanitation vehicle.

2. The on-vehicle panoramic fisheye camera calibration method for the unmanned sanitation vehicle according to claim 1, wherein, The step S1 includes the following steps: S101. For a point in the fisheye camera coordinate system around the driverless sanitation vehicle, use the perspective projection model to obtain the projection point when the fisheye camera image data is not distorted; S102. According to the projection point when not distorted, obtain the Cartesian coordinate value by obtaining the polar coordinates of the projection point after the fisheye camera image data is distorted; S103. According to the Cartesian coordinate value, convert the distorted projection point to the pixel coordinate system through the internal parameters of the fisheye camera, and complete the undistortion processing of the fisheye camera image data; S104. According to the fisheye camera image data after undistortion processing, perform data augmentation processing through homography transformation.

3. The on-vehicle panoramic fisheye camera calibration method for the unmanned sanitation vehicle according to claim 1, characterized in that, The step S2 includes the following steps: S201. According to the fisheye camera image data after data augmentation processing, perform encoding processing using the Tranformer encoding layer; S202. According to the fisheye camera image data after data augmentation processing, perform line detection processing using the line segment detection operator LSD, and input the detection result and the encoding result into the decoding layer after passing through the convolutional layer; S203. Use the decoding layer to output the angles between the vanishing points and the horizon; S204. According to the angles between the vanishing points and the horizon, estimate the rotation matrix in the external parameters, and complete the estimation of the external parameters of the fisheye cameras.

4. The on-vehicle panoramic fisheye camera calibration method for the unmanned sanitation vehicle according to claim 3, wherein, The expression of the rotation matrix in the external parameters is as follows: Among them, represents a rotation matrix, represents a rotation angle, represents the horizon angle estimated in step S2, , , represent the three components of the third column of the rotation matrix.

5. The on-vehicle panoramic fisheye camera calibration method for the unmanned sanitation vehicle according to claim 1, wherein, The expression of the loss function of the external parameter estimation network based on ResNet-18 is as follows: Among them, represents the loss function of the external parameter estimation network, represents the current training epoch, represents the total number of training epochs, represents the rotation loss, and respectively represent the coefficients of the orthogonal loss and the rotation loss, represents the orthogonal loss, represents the photometric loss, represents the rotation matrix, represents the fisheye camera i of the pre-estimated external parameters, represents the index set of adjacent fisheye cameras, represents the selected set of points that meet the requirements, represents a point in the bird's-eye view BEV coordinates, represents at the point generated photometric loss value, i and j respectively represent two adjacent fisheye cameras, represents the fisheye camera of the bird's-eye view BEV, represents the fisheye camera of the bird's-eye depth, represents the fisheye camera of the internal parameters, represents the fisheye camera of the external parameters, represents the internal parameters of the equivalent fisheye camera in the bird's-eye view BEV, represents the scale factor, represents the fisheye camera of the bird's-eye view BEV, represents the fisheye camera of the bird's-eye depth, represents the fisheye camera of the internal parameters, represents the fisheye camera of the external parameters, represents the region of interest jointly included in the adjacent fisheye camera images, and respectively represent the fisheye camera and of the BEV images, represents the th value of the th row and th column of the rotation matrix of the th fisheye camera.

Images