A laser and multi-camera fusion perception method

Abstract

The present invention discloses a laser and multi-camera fusion perception method. It includes the following steps: S1, obtaining the posture information of the device through IMU, and obtaining the laser radar point cloud through the laser radar, and multiple cameras also synchronously collect images; S2, preprocessing the laser radar point cloud; S3, performing target detection on the preprocessed laser radar point cloud to detect the target object of interest; S4, splicing the images collected by multiple cameras to generate an image with a large field of view; S5, performing target detection on the images collected by multiple cameras; S6, fusing the target detection results of multiple cameras with the target detection results of the laser radar point cloud. The present invention fuses the target detection results of multiple cameras with the point cloud target information of the laser radar, and can obtain more accurate three-dimensional spatial positioning and target recognition effects. This fusion can make up for the limitations of a single sensor and improve the overall detection accuracy and stability of the system.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing and laser point cloud processing, and in particular to a laser and multi-camera fusion perception method. Background Art

[0002] In scenarios involving close-range perception at sea, such as sea navigation, near-port navigation, port entry and exit, etc., it is necessary to accurately identify and locate targets such as identification buoys, navigation buoys, lighthouses, channel beacons, other ships, etc., for navigation obstacle avoidance, route planning, hazard warning, etc. Although ships are usually equipped with sensing equipment such as navigation radar, visible light cameras, and lidar, these devices have obvious deficiencies in handling close-range perception, and the perception system faces a series of challenges.

[0003] Navigation radar has blind spots in close-range perception scenarios, and is ineffective or has poor accuracy for close-range objects, which makes it unable to meet the needs of such scenarios; although lidar can provide accurate spatial information, it has defects in color recognition; and although visible light cameras alone can capture color and visual information, they are limited in their spatial perception capabilities, especially in terms of depth and distance measurement. Conventional lidar and a single visible light camera fusion device can usually only work at a small field of view angle, and the perception range is relatively limited, and cannot support the needs of longer distances and larger fields of view at the same time. Therefore, for close-range perception at sea, a single sensor or a combination of conventional sensors cannot effectively cope with this complex environment.

[0004] However, the image recognition of maritime targets faces high scale transformation requirements, and the ability to recognize small targets directly limits the warning distance. At present, there is a problem of missed detection for the recognition of small ships in the distance. In addition, point cloud target detection is inefficient, point cloud features are limited, and it is difficult to accurately detect small targets. It is also easily affected by water surface reflection and carrier shaking. The current lack of effective laser and multi-camera target fusion methods further exacerbates this challenge.

[0005] Therefore, a more advanced and comprehensive laser and multi-visible light camera fusion method is urgently needed to resolve the contradiction between perception distance and field of view angle. Summary of the invention

[0006] The purpose of the present invention is to overcome the problems existing in the above-mentioned prior art and to provide a laser and multi-camera fusion perception method that can cope with complex perception tasks in different environments.

[0007] In order to achieve the above object, the present invention provides a laser and multi-camera fusion perception method, comprising the following steps:

[0008] S1, obtain the device's attitude information through IMU, and obtain the lidar point cloud through lidar, and multiple cameras also collect images synchronously;

[0009] S2, preprocessing the laser radar point cloud;

[0010] S3, performing target detection on the preprocessed lidar point cloud to detect the target object of interest;

[0011] S4, stitching the images acquired by multiple cameras to generate an image with a large field of view;

[0012] S5, performing target detection on images collected by multiple cameras;

[0013] S6. Fusion the target detection results of multiple cameras with the target detection results of the lidar point cloud.

[0014] Preferably, step S3 specifically includes the following steps:

[0015] S31, correcting the motion error of the lidar point cloud through IMU data;

[0016] S32, projecting the laser radar point cloud into the BEV image;

[0017] S33, clustering objects in the BEV image;

[0018] S34, based on the correspondence between the lidar point cloud and the BEV image, the clustered target box Projection to 3D point cloud plane, thereby filtering out the point cloud belonging to the target box in the lidar point cloud, thereby obtaining the target point cloud of each target.

[0019] Preferably, step S32 specifically includes the following steps:

[0020] S321. Calculate the projection factor, that is, the scaling factor of the LiDAR point cloud on the x-axis and y-axis of the image:

[0021]

[0022] ;

[0023] S322, create a binary image, project the laser radar point cloud into corresponding pixels respectively, and assign values ​​to the binary image.

[0024] Preferably, step S322 specifically includes the following steps:

[0025] S32201. Calculate the row and column indexes of the lidar point cloud in the image:

[0026]

[0027] ;

[0028] S32202, determining whether the index exceeds the image range, and only using the index within the image range;

[0029] S32203. Update the image and pixel list.

[0030] Preferably, step S33 specifically includes the following steps:

[0031] S331, according to the laser radar point cloud projected to the pixel index in the BEV image, the pixels where the point projection exists are expanded, for each input point , with it as the center Traverse within the range to achieve expansion, row range Column Range , get the dilated binary image;

[0032] S332, using a conventional external bounding box method, finding all pixel blocks that are clustered together in the dilated binary image, and obtaining an external bounding box for each block of clustered pixels, wherein all external bounding boxes have no intersection;

[0033] S333: Expand all original external bounding boxes, taking the center point of the bounding box as the reference, and expand the width and height of the box. , so that the bounding boxes that are close to each other form an intersection;

[0034] S334, integrating all the intersecting bounding boxes together through continuous iteration, and re-forming a target box without intersection.

[0035] Preferably, step S334 specifically includes the following steps:

[0036] S33401. Find the largest area among all bounding boxes , with it as the center;

[0037] S33402. Calculate each bounding box and the maximum bounding box The intersection of ; If the width or height of the intersection is greater than zero (i.e. there is an intersection), the current bounding box is merged with the maximum bounding box Perform a union operation and update the maximum bounding box ; If there is no new bounding box to merge in one iteration, it is considered as an independent target box;

[0038] S33403, repeat the above two steps, iterate continuously until no more merging can be done, and obtain multiple expansions There is still no intersection of the target boundaries .

[0039] Preferably, step S34 specifically includes the following steps:

[0040] S341, scaling factors of the laser radar point cloud on the x-axis and y-axis of the image Convert the upper left and lower right pixel coordinates of the target boundary in the BEV image to The conversion method of plane coordinates and points is:

[0041]

[0042] ;

[0043] S342, through The two vertices of the plane construct a point cloud filter box, and the laser radar target point cloud is extracted according to the filter box;

[0044] S343. For the laser radar target point cloud, calculate the target center, center of gravity, length, width, height and other spatial information through statistical methods.

[0045] Preferably, the target detection uses an improved version of the YOLOv5 framework for model training and reasoning, wherein the training part includes the following steps:

[0046] S5101. Collect a dataset of marine scene images, including images of targets such as ships, buoys, and lighthouses, and images of the sea surface background without targets;

[0047] S5102, after importing the image data set into the annotation software, select the target using the box selection annotation method to generate a label file;

[0048] S5103, using the line segment annotation method of the annotation software to connect the clear and continuous parts of the sea surface and the sky to generate a label file;

[0049] S5104, using a script to convert the target box information in the label into a training format txt file of the YOLO framework, where the txt name is consistent with the image file name, and each txt file contains the image information of all targets in the image, including the category number, the upper left x coordinate, the upper left y coordinate, the lower right x coordinate, and the lower right y coordinate of the box;

[0050] S5105, using the line segment to mark box script to convert the line segment mark to the box mark form, and write it into the txt file in step S5104;

[0051] S5106. Train the image and label file according to the YOLOv5 improved version framework process.

[0052] Preferably, the reasoning part of the YOLOv5 improved version framework model includes the following steps:

[0053] S5201, extracting the sea-sky line position information in the image, which is the sea-sky line detection frame, and saving the center point of all sea-sky line target frames in the current image;

[0054] S5202, sending the points in step S5201 to the RANSAC straight line fitting process, calculating the straight line equation of the sea-sky line in the image, and saving the 2D coordinate position of the straight line at the midpoint of the image;

[0055] S5203, using the y-axis coordinate value of the midpoint position in step S5202 as a reference for horizontal cutting of the image, and adjusting the stitching method according to the original image and the input size of the detection model, and sending the stitched image into the model for further detection;

[0056] S5204, restoring the detection box information of the target in the new image to the original image scale;

[0057] S5205, perform IoU screening on all detection boxes, and filter out repeated targets after the fusion of normal scale and small scale by setting an adaptive threshold.

[0058] Preferably, step S4 specifically includes the following steps:

[0059] S41, performing image correction on each image to remove the distortion introduced by the camera lens, and projecting the image on the plane onto the cylindrical surface;

[0060] S42, searching for significant and unique feature points in the image;

[0061] S43, finding the corresponding relationship between them by comparing the feature points in each pair of images;

[0062] S44. According to the corresponding relationship of the feature matching results, the images are fused together.

[0063] Preferably, the process of infield calibration is also included, which specifically includes the following steps:

[0064] A1. Record the camera calibration video on the chessboard, import the calibration software to output the results, and repeat this operation for all cameras;

[0065] A2. Place the April tag calibration plate at the specified location. The camera and lidar observe the calibration plate at the same time. After the data frame rate of the lidar and camera related topics is stable, record the data packets of the related topics. Repeat this step after changing the location of the calibration plate.

[0066] A3. Select the data packet recorded in step A2 on the calibration host computer, select the topic of the lidar and one of the cameras, and import the camera's internal parameters. After the calibration is completed, export the results. Repeat this operation to calibrate the external parameters of all cameras and lidars.

[0067] Compared with the prior art, the present invention has the following beneficial effects:

[0068] 1. The present invention fuses the target detection results of multiple cameras with the point cloud target information of the lidar to obtain more accurate three-dimensional spatial positioning and target recognition effects. This fusion can make up for the limitations of a single sensor, improve the overall detection accuracy and stability of the system, and ensure efficient and accurate target detection and recognition in complex marine environments.

[0069] 2. The present invention aggregates non-continuous but relatively close pixels together. Compared with directly dilating the image, the index can avoid traversing and operating invalid pixels, improve efficiency and reduce overhead. Compared with randomly selecting bounding boxes for merging, the present invention uses the largest bounding box each time to improve merging efficiency and reduce the number of iterations. Finally, multiple relatively independent target point clouds with no intersection are obtained, which reduces the amount of calculation and better solves the recognition of discrete points and targets that are close to each other. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 A flow chart of a laser and multi-camera fusion perception method provided in Example 1;

[0071] Figure 2 A schematic diagram of the structure of a laser and multi-camera fusion sensing device provided in Example 2;

[0072] Figure 3 A schematic diagram of a first method for the specific positions of components in a laser and multi-camera fusion sensing device provided in Example 2;

[0073] Figure 4 A schematic diagram of a second method for the specific positions of components in a laser and multi-camera fusion sensing device provided in Example 2;

[0074] Figure 5 A schematic diagram of a third method for the specific positions of components in a laser and multi-camera fusion sensing device provided in Example 2;

[0075] Figure 6 A fourth schematic diagram of the specific positions of components in a laser and multi-camera fusion sensing device provided in Example 2. DETAILED DESCRIPTION

[0076] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0077] Embodiment 1

[0078] This embodiment provides a laser and multi-camera fusion perception method, such as Figure 1 As shown, the following steps are included:

[0079] Step S1, sensor data acquisition: The device's attitude information is obtained through the IMU (inertial measurement unit), and the point cloud data is obtained through the laser radar. Multiple cameras also obtain the device's attitude information, and the laser radar point cloud is obtained through the laser radar. Multiple cameras also collect images synchronously.

[0080] Step S2, point cloud preprocessing: preprocess the laser radar point cloud, including denoising, filtering and other operations to improve the reliability and accuracy of the data.

[0081] Step S3, point cloud target detection: perform target detection on the preprocessed lidar point cloud to detect the target objects of interest, especially small targets at close range.

[0082] Step S4, image stitching: stitching the images collected by multiple cameras to generate an image with a large field of view, covering a wider perception range.

[0083] Step S5, target detection and small target detection accuracy improvement: target detection is performed on images captured by multiple cameras, and the detection accuracy of small targets is optimized through multiple algorithms to make up for the deficiency of a single sensor in small target detection.

[0084] Step S6, multi-camera and laser point cloud target fusion: fuse the target detection results of multiple cameras with the laser radar point cloud target detection results to obtain more accurate three-dimensional spatial positioning and target recognition effects. This fusion can make up for the limitations of a single sensor and improve the overall detection accuracy and stability of the system.

[0085] The point cloud target detection in step S3 uses laser point cloud distortion correction and BEV-based fast clustering of spatial point cloud targets.

[0086] Laser point cloud distortion correction uses conventional motion compensation, that is, compensating for the error caused by motion through the imu and the timestamp of each point in the point cloud.

[0087] The BEV-based fast clustering of spatial point cloud objects adopts the projection transformation of point cloud-bird's-eye view-point cloud.

[0088] First, the point cloud - bird's-eye view, which is obtained by projecting the laser radar point cloud on the XY coordinate plane after discretization to obtain the XOY plane. The resolution of discretization needs to be manually specified, that is, the size of the rectangular range of the point cloud space. Corresponding to a pixel point of the discretized image. If the point cloud is clustered directly here, the distance between all the points in the point cloud needs to be calculated, which is a huge amount of calculation.

[0089] Then in the bird's-eye view, the image is processed in two dimensions. The main steps are:

[0090] (1) Dilation and erosion of binary images to initially aggregate multiple adjacent but separate pixels;

[0091] (2) Clustering of the initial binary image, converting each clustered pixel block into ;

[0092] (3) All the circumscribed rectangles Iterate clustering, expand each initially formed target frame, and determine whether there is an intersection to aggregate all similar target frames, so as to aggregate discontinuous point clouds collected from the same target together to form a single target frame, and iterate continuously until there is no intersection;

[0093] Finally, the bird's-eye view-point cloud, based on the correspondence between the point cloud and the bird's-eye view, the clustered target box is projected onto the XOY plane of the 3D point cloud, thereby filtering out the point cloud belonging to the target box in the point cloud, and obtaining the target point cloud of each target. For the target point cloud, the target center, center of gravity, length, width, height and other spatial information can be calculated through statistical methods.

[0094] Specifically, the steps of point cloud target detection are as follows:

[0095] (1) Correct the motion error of the lidar point cloud using IMU data.

[0096] (2) Projecting the laser radar point cloud into the BEV image, specifically comprising the following steps:

[0097] a) Calculate the projection factor, that is, the scaling factor of the point cloud on the x-axis and y-axis of the image. Setting different scaling factors and x and y can effectively adapt to different detection ranges and detection accuracies. Meters of detection range use The BEV binary image of the pixel has a detection accuracy of 1 meter and is used BEV binary image of pixels with a detection accuracy of 0.1 meter.

[0098]

[0099]

[0100] b) Create a binary image, project the point cloud data to the corresponding pixels, and assign values ​​to the binary image. The specific steps are as follows:

[0101] i. Calculate the row and column index of the point in the image

[0102]

[0103]

[0104] ii. Determine whether the index exceeds the image range and only use the index within the image range;

[0105] iii. Update the image and pixel list.

[0106] (3) Clustering of objects in the BEV image; the specific steps include:

[0107] a) Dilate the input image with an index to aggregate discontinuous but close pixels together. Compared with directly dilating the image, indexing can avoid traversing and operating invalid pixels, improve efficiency and reduce overhead. Dilate the pixels with point projection according to the pixel index projected to the BEV image by the lidar point cloud. For each input point , with it as the center Traverse within the range to achieve expansion, row range Column Range , get the expanded binary image, directly clustering the original binary image will produce a lot of scattered bounding boxes, and there may be only one point, which is not conducive to the aggregation of scattered points of the same target. The aggregation range can be effectively controlled, making it easier to adapt to point clouds of different resolutions collected by different lidars.

[0108] b) Using a conventional external bounding box method, find all the pixel blocks that are clustered together in the dilated binary image, and obtain the external bounding box of each clustered pixel, and all the external bounding boxes have no intersection.

[0109] c) Expand all original bounding boxes, taking the center point of the bounding box as the reference, and expand the width and height of the box , so that the bounding boxes that are close to each other form an intersection. Merging once or not merging will cause different regions of the same object to be aggregated into separate objects. The parameter can effectively control the merging range of bounding boxes after different aggregations. Adjusting this parameter can make it more suitable for merging objects of different sizes.

[0110] d) After the bounding boxes have an intersection, all the bounding boxes with intersections are integrated together through continuous iterations, and a target box without intersection is re-formed. The specific steps are as follows:

[0111] i. Compared with randomly selecting bounding boxes for merging, using the largest bounding box each time can improve merging efficiency and reduce the number of iterations. Therefore, first find the largest area among all bounding boxes. , with it as the center;

[0112] ii. Calculate each bounding box and the maximum bounding box The intersection of ; If the width or height of the intersection is greater than zero (i.e. there is an intersection), the current bounding box is merged with the maximum bounding box Perform a union operation and update to a new ; If there is no new bounding box to merge in one iteration, it is considered as an independent target box;

[0113] iii. Repeat the above two steps and continue to iterate until no more merging can be done, so that multiple expansions are obtained. There is still no intersection of the target boundaries .

[0114] (4) Based on the correspondence between the lidar point cloud and the BEV image, the clustered target box Projection to 3D point cloud Plane, thereby filtering out the point cloud belonging to the target box in the LiDAR point cloud, and obtaining the target point cloud of each target. The specific steps are as follows:

[0115] a) Scaling factors of the LiDAR point cloud on the x-axis and y-axis of the image Convert the upper left and lower right pixel coordinates of the target boundary in the BEV image to The conversion method of plane coordinates and points is:

[0116]

[0117] ;

[0118] b) Through The two vertices of the plane construct a point cloud filter box, and the laser radar target point cloud is extracted according to the filter box;

[0119] c) For the laser radar target point cloud, the target center, center of gravity, length, width, height and other spatial information are calculated through statistical methods.

[0120] (5) Finally, multiple relatively independent target point clouds with no intersection are obtained, which reduces the amount of calculation and better solves the discrete points and targets that are close to each other.

[0121] The method for improving the accuracy of target detection and small target detection in this embodiment is based on the improvement of the YOLOv5 framework. A large amount of diverse data is collected for complex sea scenes for training to improve the robustness of recognition. The Difussion principle is used to generate a batch of new data for high sea condition scenes or redraw the real collected data to obtain a richer data set. In addition, a multi-target tracking module and a target re-identification and matching function are designed to improve the stability of detection.

[0122] In order to obtain more real-time data, the model structure is modified by reusing the characteristics of the embedded board, and the model is quantized by int8. The zero-copy and multi-threading design on the inference end greatly accelerates the deep learning detection framework.

[0123] The addition of a small target detection head and a small target detection strategy based on sea-sky line detection further improves the accuracy of small target detection.

[0124] In order to improve the target detection accuracy in sea surface scenes, especially to improve the accuracy of small target detection and reduce the missed detection rate of small targets on the sea surface, this embodiment also designs a target detection method based on a multi-scale detection head of sea-sky line detection.

[0125] The target detection part uses the improved version of YOLOv5 framework for model training and reasoning. The training part includes the following steps:

[0126] Step 1: Collect a dataset of marine scene images, including images containing targets such as ships, buoys, and lighthouses, as well as images of the sea surface background without targets;

[0127] Step 2: After importing the image dataset into the annotation software, select the target using the box selection annotation method to generate a label file;

[0128] Step 3: Use the line segment annotation method of the annotation software to connect the clear and continuous parts of the sea surface and the sky (sea-sky line) to generate a label file;

[0129] Step 4. Use the script to convert the target box information in the label into a txt file in the training format of the YOLO framework. The txt file name is consistent with the image file name. Each txt file contains the image information of all targets in the image, including the category number, the upper left x coordinate, upper left y coordinate, lower right x coordinate, and lower right y coordinate of the box.

[0130] Step 5. Use the line segment to annotation box script to convert the line segment annotation into a box annotation and write it into the txt file in step 4.

[0131] The method of converting the line segment to the annotation frame in step 5 includes the following steps:

[0132] 5.1. Read the line segment label file;

[0133] 5.2. Extract line segment information, including the 2D image coordinates of each line segment’s endpoints;

[0134] 5.3. Sample points on the line segment every 32 pixels in the horizontal direction, and generate a 32×32 pixel box centered on the sampling point;

[0135] 5.4. Write all generated boxes into a txt file in the same way as in step 4;

[0136] Step 6: Train the image and label files according to the YOLOv5 framework process.

[0137] The framework network structure includes the backbone network, feature pyramid network, and detection head. The main algorithm flow is: the convolutional network cuts the input image into Each small square is responsible for predicting B candidate boxes and the probability value Pr (Class i |Object), each candidate box prediction value is mainly composed of five parts (x, y, w, h, Pr(Object)), where the (x, y) coordinates represent the distance from the center of the object to the upper left corner of the grid point to which it belongs, and (w, h) is responsible for predicting the width and height ratio of the object relative to the entire image, and Pr(Object) is responsible for predicting the probability value of the object in this candidate box. Finally, all these prediction values ​​can be encoded into a Tensor of .

[0138] Preferably, CSPDarknet53 is used as the feature extraction network, the Ghost convolution module is introduced to reduce the amount of calculation, and the attention mechanism is added to enhance the feature extraction capability; the PANet structure is used for feature fusion, and multi-scale feature map output is used to enhance the detection capability of small targets at a long distance; four detection layers of different scales are newly added to improve small target detection, and the anchor setting is optimized for the size distribution of offshore targets, and the weight ratio of the loss function is optimized.

[0139] In addition to identifying targets at sea, the trained model can also identify the sea-sky line. The steps on the inference side are as follows:

[0140] Step 1: Accurately extract the sea-sky line position information in the image through the sea-sky line detection algorithm. This information is the sea-sky line detection frame, and save the center point of all sea-sky line target frames in the current image;

[0141] Step 2: Send the points in step 1 to the RANSAC straight line fitting process, calculate the straight line equation of the sea-sky line in the image, and save the 2D coordinate position of the straight line at the midpoint of the image;

[0142] The core idea of ​​RANSAC is to randomly select a small number of samples in the data and fit a model based on these samples. Then, by calculating the distance from other data points to the model, data points with a distance less than a certain threshold are classified as inliers, and data points with a distance greater than the threshold are classified as outliers. This process is repeated many times, and the model with the most inliers is selected as the final estimation result.

[0143] Step 3: Use the y-axis coordinate value of the midpoint position in step 2 as a reference for horizontal cutting of the image, and adjust the stitching method according to the original image and the input size of the detection model, and send the stitched image to the detection for further detection;

[0144] Step 3 is further described as:

[0145] If the image used is 1920×1080 pixels in size, the detection model input is 640×640 pixels in size, and the image width 1920 pixels is three times 640, so in the image, two points (640, y) and (1280, y) are equally sampled on the horizontal line where the given y-axis coordinate value is located. The image can be divided into three equal parts horizontally; in order to splice the image vertically into a size of 640×640, only the part with a height of 213 pixels at the y value position is taken from each part of the horizontal three-part image; finally, the three parts of the image are vertically spliced ​​into a size of 640×639 pixels, and the spliced ​​image is sent to the model for detection;

[0146] Step 4: restore the target detection box information in the new image to the original image scale;

[0147] Step 5: Perform IoU (intersection over union) screening on all detection boxes and filter out duplicate targets after the fusion of normal scale and small scale by setting an adaptive threshold.

[0148] Preferably, in order to adapt to the real-time reasoning of the embedded board, the activation function is replaced with SiLU when the model is exported, and the model is quantized by INT8.

[0149] Preferably, in order to achieve continuous and stable perception of the detection target, a target-associated tracking algorithm is added, including:

[0150] Step 1: Calculate the target feature information of the detection result through the re-identification model. The information is encoded as a float type tensor of size 1×2048;

[0151] Step 2: Use Kalman filter algorithm to predict the target detection frame of the previous frame;

[0152] Step 3: Based on the predicted target frame and target feature information, calculate the similarity matrix between the current target and the target in the previous frame, and associate the targets through the Hungarian algorithm to achieve target tracking.

[0153] This embodiment also provides a method for calibrating the external parameters of the camera and lidar based on the calibration plate of the April tag, and a method for calibrating the internal parameters of the camera based on the checkerboard. The preliminary internal and external parameter calibration can be completed on land, and the upper computer software is further developed to fine-tune the parameters during the ship experiment, so that the point cloud and image registration results of the marine target can be visualized and the calibration results can be refined.

[0154] The process of infield calibration has the following steps:

[0155] A1. Record the camera calibration video on the chessboard, import the calibration software to output the results, and repeat this operation for all cameras;

[0156] A2. Place the April tag calibration plate at the specified location. The camera and lidar observe the calibration plate at the same time. After the data frame rate of the lidar and camera related topics is stable, record the data packets of the related topics. Repeat this step after changing the location of the calibration plate.

[0157] A3. Select the data packet recorded in step A2 on the calibration host computer, select the topic of the laser radar and one of the cameras, and import the camera internal parameters. After the calibration is completed, export the results. Repeat this operation to calibrate the external parameters of all cameras and laser radars.

[0158] The external parameter calibration results of the indoor calibration can basically adapt to the outdoor test. In order to obtain more accurate registration results, manual external parameter adjustment can be enabled during the outdoor test, including external parameter yaw angle, pitch angle, roll angle, three-axis offset and adjustment accuracy adjustment. The point cloud can be projected onto the image screen in real time to check whether the external parameters can be aligned.

[0159] In order to ensure the consistency of the transmitted targets in time and space during data communication between multiple cameras and lidars, it is necessary to time align the targets in their messages. Time alignment is mainly based on the Unix timestamp carried by each input and output message. The timestamp alignment method is to obtain a group of messages whose timestamps meet the alignment strategy in the message buffer area of ​​the specified number of frames. Due to the large frequency differences between sensors, some sensors will only output sampling information under specific circumstances, so it is impossible to achieve complete timestamp alignment. The timestamp similarity alignment method can make greater use of the received messages;

[0160] Based on the algorithm of close timestamp alignment, an unsynchronized message buffer is maintained at runtime, and each message received is timestamped and added to the buffer when it is received. When a message is ready to be processed, the strategy looks for other messages in the buffer within a certain time range around the current message timestamp. If there are matching messages, they are delivered at the same time as the current message. If there are no matching messages, the current message is delivered as soon as possible.

[0161] Specifically, assuming that S is the last published set and T is the next set being created, the algorithm steps based on timestamp alignment are as follows:

[0162] 1. When the set S is published, for each type of message, all messages older than those in the set are discarded;

[0163] 2. When a message arrives, it is inserted into the queue of the corresponding type according to the timestamp.

[0164] 3. Once all queues of each kind contain at least one message, find the most recent message at the head of the queue.

[0165] After external parameter calibration and message time alignment, the target of interest is obtained through the target point cloud detector , and then identify the target with the 2D image Matching is performed to obtain the final 2D and 3D corresponding target information, that is, to determine the target object. The main steps are as follows:

[0166] , the information of the target being the object is selected in the laser point cloud, The origin coordinates of the box selection information. is the length, width and height of the selected information, and n is the number of point clouds selected;

[0167] , the information of the target object is selected in the image, [ Vertex pixel coordinates of the box selection information. The width and height of the frame selection information, in pixels;

[0168] 3D target and 2D matching is to identify the result And the test results collection The matching process:

[0169] calculate The center pixel coordinates of , and then calculate its field of view vector,

[0170]

[0171] calculate The field of view vector, the field of view boundary value ,Through the perspective and distance calculation, the three-dimensional information of the field of view boundary and Target_3D is used to obtain the three-dimensional space proportion result

[0172]

[0173]

[0174] calculate and The Euclidean distance between the view vectors

[0175]

[0176] The items with the smallest distance within the preset distance range are selected as mutual matches and passed to result processing;

[0177] If a laser point cloud target is matched to image detection targets of different cameras, the 2D detection result with the highest confidence is selected as the target information release;

[0178] The published targets can be projected onto the panoramic mosaic image for a more intuitive display after turning on the visualization switch.

[0179] The image stitching in this implementation is based on cylindrical stitching, which stitches the images of multiple cameras into a ring view, which can achieve a full-body panoramic observation;

[0180] The stitching parts of the stitched images are further optimized to make the picture transition more natural, and the panoramic images are post-processed and optimized to make the overall brightness, color, etc. more uniform.

[0181] To form a camera image of the entire device, each camera image is stitched together to visualize the device data. The main steps of stitching are as follows:

[0182] (1) Perform image correction on each image to remove the distortion introduced by the camera lens and ensure the accuracy and quality of the image during the stitching process; perform image cylindrical projection to project the image on the plane onto a cylindrical surface to make image stitching easier;

[0183] (2) Image feature detection: finding significant and unique feature points in the image for subsequent image matching and tracking;

[0184] (3) Image feature matching, by comparing the feature points in each pair of images to find the correspondence between them;

[0185] (4) Image fusion: fuse the images together based on the correspondence between feature matching results.

[0186] Embodiment 2

[0187] This embodiment provides a laser and multi-camera fusion sensing device, such as Figure 2As shown in the figure, this embodiment uses laser radar and multiple cameras as data acquisition sensors, and integrates an embedded board to provide integrated intelligent perception equipment for fusion algorithm processing. In addition, an IMU (inertial measurement unit) is integrated for removing laser radar motion distortion in a mobile state. It supports visual perception requirements with a wide range and long distance, and supports spatial three-dimensional point cloud acquisition and processing perception requirements, so as to support multiple types of mobile or stationary platforms.

[0188] The laser and multi-camera fusion sensing device provided in this embodiment has the following characteristics:

[0189] (1) Rigid fixed structure: Multiple cameras, lidars, and IMUs are rigidly fixed on the same structure to ensure that the relative positions of the sensors remain stable, thereby improving the accuracy of data fusion.

[0190] (2) Co-optical center installation: Multiple cameras are installed on the same horizontal plane with a co-optical center design. This enables the cameras to form a common viewing area, optimize the overlapping field of view, and improve the accuracy of target recognition.

[0191] (3) Parallel mounting planes: The mounting plane of the lidar is parallel to the mounting plane of the visible light camera. This configuration can effectively coordinate the data fusion of the two sensors and reduce the measurement error caused by the difference in viewing angles.

[0192] (4) Expanded field of view: The field of view of multiple cameras is larger than that of the lidar, ensuring comprehensive environmental perception over a wider range and improving the ability to detect surrounding objects.

[0193] (5) Flexible data connection: The installation of embedded boards is not restricted by the actual location, which provides flexibility for data connection between sensors and facilitates subsequent system integration and maintenance.

[0194] By designing each module based on the above characteristics, each module can work together to improve the performance of the overall perception system and ensure efficient and accurate target detection and recognition in complex marine environments. Figure 3-Figure 6 As shown, there are four design methods provided in this embodiment.

[0195] Although the present invention is described herein with reference to specific embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the present invention. It should therefore be understood that many modifications may be made to the exemplary embodiments and that other arrangements may be devised without departing from the spirit and scope of the present invention as defined by the appended claims. It should be understood that the various dependent claims and features described herein may be combined in a manner different from that described in the original claims. It should also be understood that features described in conjunction with individual embodiments may be used in other described embodiments.

Claims

1. A laser and multi-camera fusion perception method, characterized in that: The following steps are involved: S1, obtain the device's attitude information through IMU, and obtain the lidar point cloud through lidar, and multiple cameras also collect images synchronously; S2, preprocessing the laser radar point cloud; S3, performing target detection on the preprocessed lidar point cloud to detect the target object of interest; S4, stitching the images acquired by multiple cameras to generate an image with a large field of view; S5, performing target detection on images collected by multiple cameras; S6, fusing the target detection results of multiple cameras with the target detection results of the laser radar point cloud; Step S3 specifically includes the following steps: S31, correcting the motion error of the lidar point cloud through IMU data; S32, projecting the laser radar point cloud into the BEV image; S33, clustering objects in the BEV image; S34, based on the correspondence between the lidar point cloud and the BEV image, the clustered target box Projection to 3D point cloud plane, thereby filtering out the point cloud belonging to the target box in the laser radar point cloud, thereby obtaining the target point cloud of each target; Step S32 specifically includes the following steps: S321. Calculate the projection factor, that is, the scaling factor of the LiDAR point cloud on the x-axis and y-axis of the image: ； ； S322, creating a binary image, projecting the laser radar point cloud into corresponding pixels respectively, and assigning values ​​to the binary image; Step S322 specifically includes the following steps: S32201. Calculate the row and column indexes of the lidar point cloud in the image: ； ； S32202, determining whether the index exceeds the image range, and only using the index within the image range; S32203, update the image and pixel list; Step S33 specifically includes the following steps: S331, according to the laser radar point cloud projection to the pixel index in the BEV image, the pixels where the point projection exists are expanded, for each input point , with it as the center Traverse within the range to achieve expansion, row range Column Range , get the dilated binary image; S332, using a conventional external bounding box method, finding all pixel blocks that are clustered together in the dilated binary image, and obtaining an external bounding box for each block of clustered pixels, wherein all external bounding boxes have no intersection; S333: Expand all original external bounding boxes, taking the center point of the bounding box as the reference, and expand the width and height of the box. , so that the bounding boxes that are close to each other form an intersection; S334, integrating all the bounding boxes with intersections together through continuous iteration, and re-forming a target box without intersections; Step S334 specifically includes the following steps: S33401. Find the largest area among all bounding boxes , with it as the center; S33402. Calculate each bounding box and the maximum bounding box The intersection of ; If the width or height of the intersection is greater than zero, that is, there is an intersection, then the current bounding box is added to the maximum bounding box Perform a union operation and update the maximum bounding box ; If there is no new bounding box to merge in one iteration, it is considered as an independent target box; S33403. Repeat steps S33401 and S33402, and continue to iterate until the merger cannot continue, and obtain multiple expansions. There are still no intersecting target boundaries .

2. The laser and multi-camera fusion perception method according to claim 1 is characterized in that: Step S34 specifically includes the following steps: S341, scaling factors of the laser radar point cloud on the x-axis and y-axis of the image Convert the upper left and lower right pixel coordinates of the target boundary in the BEV image to The conversion method of plane coordinates and points is: ； ； S342, through The two vertices of the plane construct a point cloud filter box, and the laser radar target point cloud is extracted according to the filter box; S343. For the laser radar target point cloud, the target center, center of gravity, length, width and height spatial information are calculated by statistical methods.

3. The laser and multi-camera fusion perception method according to claim 1 is characterized in that: The target detection adopts the improved version of YOLOv5 framework for model training and reasoning, wherein the training part includes the following steps: S5101. Collect a dataset of marine scene images, including images of ships, buoys, and lighthouse targets, and images of the sea surface background without targets; S5102, after importing the image data set into the annotation software, select the target using the box selection annotation method to generate a label file; S5103, using the line segment annotation method of the annotation software to connect the clear and continuous parts of the sea surface and the sky to generate a label file; S5104, using a script to convert the target box information in the label into a training format txt file of the YOLO framework, where the txt name is consistent with the image file name, and each txt file contains the image information of all targets in the image file, including the category number, the upper left x coordinate, the upper left y coordinate, the lower right x coordinate, and the lower right y coordinate of the box; S5105, using the line segment to mark box script to convert the line segment mark to the box mark form, and write it into the txt file in step S5104; S5106. Train the image and label file according to the YOLOv5 improved version framework process.

4. The laser and multi-camera fusion perception method according to claim 3 is characterized in that: The reasoning part of the YOLOv5 improved framework model includes the following steps: S5201, extracting sea-sky-line position information in the image, the sea-sky-line position information is a sea-sky-line detection frame, and saving the center point of all sea-sky-line target frames of the current image; S5202, sending the points in step S5201 to the RANSAC straight line fitting process, calculating the straight line equation of the sea-sky line in the image, and saving the 2D coordinate position of the straight line at the midpoint of the image; S5203, using the y-axis coordinate value of the midpoint position in step S5202 as a reference for horizontal cutting of the image, and adjusting the stitching method according to the original image and the input size of the detection model, and sending the stitched image into the model for further detection; S5204, restoring the detection box information of the target in the new image to the original image scale; S5205, perform IoU screening on all detection boxes, and filter out repeated targets after the fusion of normal scale and small scale by setting an adaptive threshold.

5. The laser and multi-camera fusion perception method according to claim 1, characterized in that: Step S4 specifically includes the following steps: S41, performing image correction on each image to remove the distortion introduced by the camera lens, and projecting the image on the plane onto the cylindrical surface; S42, searching for significant and unique feature points in the image; S43, finding the corresponding relationship between them by comparing the feature points in each pair of images; S44. According to the corresponding relationship of the feature matching results, the images are fused together.

6. The laser and multi-camera fusion perception method according to claim 1, characterized in that: It also includes the process of infield calibration, which includes the following steps: A1. Record the camera calibration video on the chessboard, import the calibration software to output the results, and repeat the steps in A1 for all cameras; A2. Place the April tag calibration plate at the specified location. The camera and lidar observe the calibration plate at the same time. After the data frame rate of the lidar and camera related topics is stable, record the data packets of the related topics. Repeat this step after changing the location of the calibration plate. A3. Select the data packet recorded in step A2 on the calibration host computer, select the topic of the laser radar and one of the cameras, and import the internal parameters of one of the cameras. After the calibration is completed, export the results. Repeat the operation in A3 to calibrate the external parameters of all cameras and laser radars.

Images