Forklift anti-collision system based on radar and vision fusion

Abstract

The present invention relates to the field of forklift anti-collision technology, in particular to a forklift anti-collision system based on radar-vision fusion, including a panoramic picture splicing module, a visual recognition and tracking module, a microwave radar module, a data processing module, and a decision and control module; the panoramic picture splicing module: uses the pictures of four wide-angle cameras in front, behind, left and right to calibrate and splice to form a panoramic picture of the vehicle; the visual recognition and tracking module: performs object detection and target tracking on the panoramic picture in real time, and continuously outputs object coordinates and object types; the microwave radar module: continuously outputs point cloud map data of objects around the vehicle. The present invention can improve the accuracy and reliability of the anti-collision system, reduce the occurrence of false alarms, and enable the anti-collision system to more intelligently and efficiently ensure the safety of forklift operation.

Description

Technical Field

[0001] The present invention relates to the technical field of forklift anti-collision, and in particular to a forklift anti-collision system based on radar and vision fusion. Background Art

[0002] In various industrial production environments, forklifts, as an important material handling equipment, are widely used in daily logistics and warehousing operations. In order to ensure the personal safety of operators and the normal operation of equipment, the application of anti-collision systems on forklifts has become more and more common. This system monitors the environment around the forklift in real time and detects potential obstacles in time, thereby avoiding collision accidents.

[0003] However, most of the anti-collision systems currently on the market rely on radar sensors for obstacle detection. Although radar sensors can effectively identify and warn obstacles to a certain extent, they also have some inherent defects. Specifically, radar sensors often generate false alarms for some non-targets of interest, such as surrounding shelves, stationary objects, etc. These false alarms will cause the anti-collision system to be frequently triggered by mistake, thereby issuing unnecessary alarms and interfering with the normal work of operators. Such frequent false alarms not only affect the normal use of forklifts, but may also paralyze operators to the system's alarms, resulting in slow response when they really encounter danger, increasing the possibility of accidents. Summary of the invention

[0004] The purpose of the present invention is to provide a forklift anti-collision system based on radar and vision fusion, which can improve the accuracy and reliability of the anti-collision system, reduce the occurrence of false alarms, and enable the anti-collision system to more intelligently and efficiently ensure the safety of forklift operation.

[0005] The present invention adopts the following technical solutions to achieve the above purpose:

[0006] The forklift anti-collision system based on radar and vision fusion includes a panoramic image stitching module, a visual recognition and tracking module, a microwave radar module, a data processing module, and a decision-making and control module;

[0007] Panoramic image stitching module: uses the images from four wide-angle cameras in the front, back, left and right to calibrate and stitch them to form a panoramic image of the vehicle;

[0008] Visual recognition and tracking module: performs object detection and target tracking in the panoramic image in real time, and continuously outputs object coordinates and object types;

[0009] Microwave radar module: continuously outputs point cloud data of objects around the vehicle;

[0010] Data processing module: performs real-time matching of the output results of the visual recognition and tracking module and the microwave radar module, marks the successfully matched data as radar-visual fusion targets, marks the unmatched radar targets as other types of tracking targets, and discards the unmatched visual recognition targets;

[0011] Decision and control module: input the matching result list into the collision risk assessment model to output the collision risk level; according to the collision risk level, execute the corresponding decision action;

[0012] The panoramic image stitching module includes a feature point extraction and matching unit, an image registration unit, an image fusion unit, a ghost phenomenon solving unit, and a distortion correction unit; the feature point extraction and matching unit generates a feature descriptor for the image; the image registration unit calculates the homography matrix between the images based on the result of the feature point extraction and matching unit; the image fusion unit decomposes and fuses the image on the basis of the image registration unit, and then reconstructs a seamless panoramic image; the ghost phenomenon solving unit processes the image in blocks, and calculates the homography matrix for each block separately to eliminate the stitching traces; finally, the distortion correction unit performs distortion correction on the image to obtain a normal viewing angle image;

[0013] The visual recognition and tracking module includes a target recognition model framework and a target tracking algorithm framework. The target recognition model framework is used for target recognition, and the target tracking algorithm framework is used for tracking the trajectory of the target.

[0014] Preferably, the panoramic image stitching module is further specifically:

[0015] (1) Feature point extraction and matching unit: Use the ORB algorithm to extract image feature points and generate feature descriptors for subsequent image matching;

[0016] (2) Image registration unit: The random sampling consistency RANSAC algorithm is used to calculate the model parameters by iteratively and randomly selecting sample point pairs, and then the consistency of the model is evaluated to eliminate mismatched points from the feature point matching results; the homography matrix between the images is calculated based on the feature point matching results to determine the geometric transformation relationship between the images;

[0017] Homography matrix calculation steps:

[0018] The homography matrix is ​​a 3x3 matrix with eight degrees of freedom and is normalized to 1; the mathematical scaling is as follows:

[0019] ;

[0020] is the homography matrix, used as a projective transformation, the homography matrix is the matrix internal parameter, where the value range of i and j is [1,2,3]. According to the algebraic formula, we can know that Together they determine the mapping relationship of the x-coordinate. It can be regarded as the linear transformation coefficient of the original coordinates x and y. is a translation term; Determines the mapping relationship of the y coordinate, is the linear transformation coefficient of the original coordinates x and y about y', is the translation term in the y' direction; in the calculation of homogeneous coordinates, Play a normalizing role;

[0021] The matrix form represents the homogeneous coordinates on the 2D image, where It refers to the homogeneous coordinates of the points on the original image plane before the projective transformation. It refers to the homogeneous coordinates of the points on the new image plane after the projective transformation;

[0022] Homography matrix satisfy ;

[0023] (3) Image fusion unit: Based on image registration, the multi-band blending technology is used to perform pyramid decomposition on the image, fuse the images layer by layer, and finally reconstruct a seamless panorama.

[0024] (4) Ghosting problem solving unit: The image is divided into blocks using the APAP algorithm, and the homography matrix is ​​calculated for each block to eliminate the splicing traces;

[0025] (5) Distortion correction unit: For wide-angle images taken with multiple fisheye camera lenses, distortion correction is performed to obtain normal viewing angle images. The correction method adopts a checkerboard correction method.

[0026] Preferably, the image fusion unit is further specifically:

[0027] The calculation steps are as follows:

[0028] 1) Constructing a Laplacian pyramid: First, a Gaussian pyramid is formed by Gaussian blurring and downsampling by halving the size layer by layer. Then, each layer of the Laplacian pyramid is calculated by subtracting the previous layer of Gaussian image that has been upsampled and downsampled from the current layer of Gaussian image. In this process, downsampling is used to reduce the image resolution, and upsampling is used to restore the resolution.

[0029] 2) Fusion weight calculation: For each frequency band, the corresponding weight image is calculated according to the area to be fused; the weight image is used to indicate the proportion of each pixel in the fusion;

[0030] 3) Frequency band fusion: For each frequency band, the images to be fused are weighted fused according to the weight;

[0031] 4) Pyramid reconstruction: reconstruct each fused frequency band and merge them into the final output image;

[0032] 5) Image reconstruction: By iteratively processing each layer of the Gaussian pyramid until the preset maximum number of iterations is reached, the image fusion and detail retention are completed; finally, through the pyramid reconstruction process, starting from the fused top layer image of the Laplacian pyramid, information is gradually added to the next layer to reconstruct the image, retaining the detailed information of the two images, and finally obtaining a fused image.

[0033] Preferably, the ghost phenomenon solving unit is further specifically:

[0034] The calculation steps are as follows:

[0035] 1) Feature point extraction: Use SIFT feature detection algorithm to extract key points and descriptors in two images;

[0036] 2) Feature point matching: Match the extracted feature points to obtain matching point pairs;

[0037] 3) Remove outliers: Use the RANSAC algorithm to remove inaccurate matching points and retain inliers;

[0038] 4) Calculate the global homography matrix: Use the internal points to calculate the global homography matrix for preliminary image alignment;

[0039] 5) Image segmentation: Divide the image into several grid blocks;

[0040] 6) Calculate the local homography matrix: Use the Moving DLT algorithm to calculate the local homography matrix for each grid block; this involves calculating the weight for each grid center point, the weight is determined by the distance between the grid center point and all the internal points, using the weight to update the A matrix in the DLT algorithm, and then decomposing it through SVD to get the local homography matrix;

[0041] 7) Image mapping: Use the local homography matrix of each grid block to map the image onto the panoramic canvas;

[0042] 8) Image fusion: Perform image fusion in overlapping areas to eliminate stitching traces and obtain the final seamless panorama.

[0043] Preferably, the distortion correction unit is further specifically:

[0044] The steps for implementing the chessboard correction method are as follows:

[0045] 1) Prepare a chessboard: Make or obtain a chessboard with black and white squares. The size should be slightly larger than the camera's field of view, and the sides of adjacent squares should be equal.

[0046] 2) Take chessboard images: Place the chessboard in the camera imaging area and take chessboard images from multiple angles and positions; make sure the chessboard fills the entire image area and take enough images to get a good calibration result;

[0047] 3) Corner detection: Use adaptive threshold and non-maximum suppression to detect corners; first convert the image into a grayscale image, then perform binarization, separate the adhered corners through corrosion operations, and finally detect the position of the corners;

[0048] 4) Sub-pixel corner point extraction: The detected corner points are refined with sub-pixel precision to improve the accuracy of the corner point position;

[0049] 5) Calculate the camera's intrinsic parameters and distortion coefficients: Use the detected corner point coordinates to calculate the camera's intrinsic parameter matrix and distortion coefficients through the Zhang Zhengyou calibration method;

[0050] 6) Generate distortion correction mapping: Generate a mapping matrix for distortion correction based on the calculated internal parameters and distortion coefficients;

[0051] 7) Image distortion correction: Use the mapping matrix to correct the distortion of the original image;

[0052] 8) Save correction parameters: Save the calculated intrinsic parameter matrix, distortion coefficient and mapping matrix for subsequent use in distortion correction of other images.

[0053] Preferably, the target recognition model framework includes:

[0054] 1) The Backbone part uses a ResNet variant with optimized depth and width as the Backbone to extract rich feature representations; introduces the Spatial attention mechanism to enhance the model's sensitivity to the target location; integrates the Channel attention mechanism to strengthen effective information channels and reduce the interference of irrelevant information;

[0055] 2) The Neck part adopts the structure of feature pyramid network; the Specific Pathway for Pedestrian module is designed to extract the features of small human bodies under wide-angle lenses. The size of the feature map can be adjusted according to the proportion of the human body in the picture; the optimal number of channels is determined through cross-validation to reduce the amount of calculation and parameters while maintaining the feature expression ability to ensure the real-time performance of the model;

[0056] 3) Adopting a decoupled head structure to separate the classification and detection heads, improving the flexibility and accuracy of the model; following the Anchor-Free design, using CenterNet makes the model more flexible and accurate in processing objects at different scales;

[0057] 4) To overcome the problem of few samples and sample imbalance, Focal Loss is used in the classification part of the loss function, and Distribution Focal Loss is used in the regression part to further improve the performance of the model on unbalanced data sets and the generalization under occlusion;

[0058] 5) In the inference phase, use model pruning, quantization, and knowledge distillation techniques to improve model operation efficiency;

[0059] A) Model quantization: Use quantization-aware training of the deep learning framework Pytorch to achieve model quantization;

[0060] B) Model pruning: Use the weight pruning of the deep learning framework Pytorch, combined with manual structured pruning;

[0061] C) Knowledge Distillation:

[0062] a. The first model trained is a large teacher model with high accuracy;

[0063] b. Design a small student model to learn the knowledge of the teacher model;

[0064] c. Train the student model by using the distillation loss function and taking the output of the teacher model as the target;

[0065] d. Adjust the temperature parameters during the distillation process to balance the softening degree of the soft label;

[0066] e. Deploy the student model for reasoning to verify its performance and reasoning efficiency improvements;

[0067] D) Half-precision reasoning: Depending on the hardware support, choose to use FP16 half-precision or INT8 integer precision for reasoning.

[0068] Preferably, the target tracking algorithm framework includes:

[0069] 1) Initialization: Initialize the tracking list T, create trajectory objects, and store them; in the first frame, all created trajectories will be marked as tracked trajectories;

[0070] 2) Target detection: For each frame in the video sequence V , use the target detector Det to detect and obtain the detection box and score, where k represents the kth frame;

[0071] 3) Detection result classification: According to whether the detection result score is higher than the threshold Th, the detection result is divided into high confidence detection and low confidence detection ;

[0072] 4) Predict tracking position: Use the Kalman filter to predict the position of each track in the current tracking list T;

[0073] 5) First data association: Use the similarity measure matrix to associate the predicted tracking positions with high confidence detections Perform association and obtain preliminary association results;

[0074] 6) Remaining track processing: Get the unmatched tracks from the preliminary association results, denoted as ;

[0075] 7) Second data association: Use the similarity measure matrix to track the remaining With low confidence detection Perform association to obtain the final remaining trace ;

[0076] 8) Delete unmatched traces: Delete unmatched traces from the trace list;

[0077] 9) Initialize new tracking: For the remaining low confidence detections , initialize a new trace and add it to the trace list T;

[0078] 10) Output tracking results: Return the final tracking list.

[0079] Preferably, the microwave radar module is further specified as follows: using a millimeter-wave radar with a detection distance of 20 meters and an operating power of 24 GHz, installing no less than 8 millimeter-wave radars of this type around the vehicle body, and using the ICP algorithm to fuse radar data to construct a point cloud map around the vehicle.

[0080] Preferably, the data processing module is further specifically: converting the visual and radar targets within the field of view into a polar coordinate system, normalizing all coordinates, and obtaining the normalized coordinates of the centroid of the visual object , where i represents the i-th target identified, and the normalized coordinates of the radar object center of mass ,in Between 0 and 1, The value is between 0 and 360; the distance is calculated for all coordinates , if the distance is less than the threshold z, the match is successful; the rest are all thrown into the unmatched target list and use the general warning parameters.

[0081] Preferably, the decision and control module is further specifically:

[0082] Collision risk assessment: Use the trained collision risk assessment model to calculate the collision risk level ,in is the distance between the target and the vehicle, is the direction angle between the target and the front of the forklift, is the target type, is the target relative vehicle speed.

[0083] Beneficial effects of the present invention:

[0084] Improved safety: Through the use of radar and vision fusion technology, the system can more accurately identify and warn of potential collision risks, significantly reducing the false triggering of the anti-collision system due to false alarms, thereby improving the safety of forklift operations.

[0085] Reduce false alarm rate: The system deeply integrates radar data and visual data to accurately determine whether an obstacle poses an actual collision threat, effectively reducing false alarms caused by misjudgment and improving system reliability.

[0086] Enhanced environmental adaptability: The radar sensor's all-weather working capability enables the system to work stably in all weather and lighting conditions, unaffected by changes in the external environment, thus improving the safety of forklifts in complex environments.

[0087] Improve work efficiency: The efficient operation of the system reduces unnecessary alarms caused by false alarms, allowing drivers to focus more on driving tasks, avoiding unnecessary interference and distraction, thereby improving the efficiency of forklift operations.

[0088] Intelligent target recognition: The system uses advanced image recognition technology to identify and classify objects around the forklift, including pedestrians, cargo and other vehicles, providing more accurate target information and enhancing the intelligence of the system.

[0089] Automatic execution of preventive measures: When necessary, the system can not only issue an alarm, but also automatically execute preventive measures, such as slowing down or braking, to avoid collision accidents, enhancing the system's active safety performance.

[0090] Continuous performance optimization: The system has the ability to self-learn and optimize, and can continuously adjust the algorithm according to actual usage to adapt to the changing operating environment and needs, ensuring that the system maintains optimal performance in the long term.

[0091] Reduced driver fatigue: By reducing false alarms and unnecessary alerts, the system reduces driver mental stress and fatigue, and improves driver job comfort and satisfaction.

[0092] Reduce maintenance costs: The high reliability and low false alarm rate of the system reduce equipment loss and maintenance costs caused by false triggering, saving operating costs for enterprises.

[0093] The present invention can improve the accuracy and reliability of the anti-collision system, reduce the occurrence of false alarms, and enable the anti-collision system to more intelligently and efficiently ensure the safety of forklift operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0094] Figure 1 It is the projection relationship diagram between the original image plane and the new image plane.

[0095] Figure 2 It is a schematic diagram of the stitching after image projection transformation. DETAILED DESCRIPTION

[0096] The present invention will be further described below in conjunction with the accompanying drawings.

[0097] The present invention provides a forklift anti-collision system based on radar-vision fusion, including a panoramic image stitching module, a visual recognition and tracking module, a microwave radar module, a data processing module, and a decision and control module;

[0098] Panoramic image stitching module: uses the images from four wide-angle cameras in the front, back, left and right to calibrate and stitch them to form a panoramic image of the vehicle;

[0099] Visual recognition and tracking module: performs object detection and target tracking in the panoramic image in real time, and continuously outputs object coordinates and object types;

[0100] Microwave radar module: continuously outputs point cloud data of objects around the vehicle;

[0101] Data processing module: performs real-time matching of the output results of the visual recognition and tracking module and the microwave radar module, marks the successfully matched data as radar-visual fusion targets, marks the unmatched radar targets as other types of tracking targets, and discards the unmatched visual recognition targets;

[0102] Decision and control module: input the matching result list into the collision risk assessment model to output the collision risk level; according to the collision risk level, execute the corresponding decision action;

[0103] The panoramic image stitching module includes a feature point extraction and matching unit, an image registration unit, an image fusion unit, a ghost phenomenon solving unit, and a distortion correction unit; the feature point extraction and matching unit generates a feature descriptor for the image; the image registration unit calculates the homography matrix between the images based on the result of the feature point extraction and matching unit; the image fusion unit decomposes and fuses the image on the basis of the image registration unit, and then reconstructs a seamless panoramic image; the ghost phenomenon solving unit processes the image in blocks, and calculates the homography matrix for each block separately to eliminate the stitching traces; finally, the distortion correction unit performs distortion correction on the image to obtain a normal viewing angle image;

[0104] The visual recognition and tracking module includes a target recognition model framework and a target tracking algorithm framework. The target recognition model framework is used for target recognition, and the target tracking algorithm framework is used for tracking the trajectory of the target.

[0105] The panoramic image stitching module is further specifically:

[0106] (1) Feature point extraction and matching unit: Use the ORB algorithm to extract image feature points and generate feature descriptors for subsequent image matching;

[0107] (2) Image registration unit: The random sampling consistency RANSAC algorithm is used to calculate the model parameters by iteratively and randomly selecting sample point pairs, and then the consistency of the model is evaluated to eliminate mismatched points from the feature point matching results; the homography matrix between the images is calculated based on the feature point matching results to determine the geometric transformation relationship between the images;

[0108] Homography matrix calculation steps:

[0109] The homography matrix is ​​a 3x3 matrix with eight degrees of freedom and is normalized to 1; the mathematical scaling is as follows:

[0110] ;

[0111] is the homography matrix, used as a projective transformation, the homography matrix is the matrix internal parameter, where the value range of i and j is [1,2,3]. According to the algebraic formula, we can know that Together they determine the mapping relationship of the x-coordinate. It can be regarded as the linear transformation coefficient of the original coordinates x and y. is a translation term; Determines the mapping relationship of the y coordinate, is the linear transformation coefficient of the original coordinates x and y about y', is the translation term in the y' direction; in the calculation of homogeneous coordinates, Play a normalizing role;

[0112] The matrix form represents the homogeneous coordinates on the 2D image, where It refers to the homogeneous coordinates of the points on the original image plane before the projective transformation. It refers to the homogeneous coordinates of the points on the new image plane after the projective transformation;

[0113] Homography matrix satisfy ;

[0114] (3) Image fusion unit: Based on image registration, the multi-band blending technology is used to perform pyramid decomposition on the image, fuse the images layer by layer, and finally reconstruct a seamless panorama.

[0115] (4) Ghosting problem solving unit: The image is divided into blocks using the APAP algorithm, and the homography matrix is ​​calculated for each block to eliminate the splicing traces;

[0116] (5) Distortion correction unit: For wide-angle images taken with multiple fisheye camera lenses, distortion correction is performed to obtain normal viewing angle images. The correction method adopts a checkerboard correction method.

[0117] The image fusion unit is further specifically:

[0118] The calculation steps are as follows:

[0119] 1) Constructing a Laplacian pyramid: First, a Gaussian pyramid is formed by Gaussian blurring and downsampling by halving the size layer by layer. Then, each layer of the Laplacian pyramid is calculated by subtracting the previous layer of Gaussian image that has been upsampled and downsampled from the current layer of Gaussian image. In this process, downsampling is used to reduce the image resolution, and upsampling is used to restore the resolution.

[0120] 2) Fusion weight calculation: For each frequency band, the corresponding weight image is calculated according to the area to be fused; the weight image is used to indicate the proportion of each pixel in the fusion;

[0121] 3) Frequency band fusion: For each frequency band, the images to be fused are weighted fused according to the weight;

[0122] 4) Pyramid reconstruction: reconstruct each fused frequency band and merge them into the final output image;

[0123] 5) Image reconstruction: By iteratively processing each layer of the Gaussian pyramid until the preset maximum number of iterations is reached, the image fusion and detail retention are completed; finally, through the pyramid reconstruction process, starting from the fused top layer image of the Laplacian pyramid, information is gradually added to the next layer to reconstruct the image, retaining the detailed information of the two images, and finally obtaining a fused image.

[0124] The ghost phenomenon solving unit is further specifically:

[0125] The calculation steps are as follows:

[0126] 1) Feature point extraction: Use SIFT feature detection algorithm to extract key points and descriptors in two images;

[0127] 2) Feature point matching: Match the extracted feature points to obtain matching point pairs;

[0128] 3) Remove outliers: Use the RANSAC algorithm to remove inaccurate matching points and retain inliers;

[0129] 4) Calculate the global homography matrix: Use the internal points to calculate the global homography matrix for preliminary image alignment;

[0130] 5) Image segmentation: Divide the image into several grid blocks;

[0131] 6) Calculate the local homography matrix: Use the Moving DLT algorithm to calculate the local homography matrix for each grid block; this involves calculating the weight for each grid center point, the weight is determined by the distance between the grid center point and all the internal points, using the weight to update the A matrix in the DLT algorithm, and then decomposing it through SVD to get the local homography matrix;

[0132] 7) Image mapping: Use the local homography matrix of each grid block to map the image onto the panoramic canvas;

[0133] 8) Image fusion: Perform image fusion in overlapping areas to eliminate stitching traces and obtain the final seamless panorama.

[0134] The distortion correction unit is further specifically:

[0135] The steps for implementing the chessboard correction method are as follows:

[0136] 1) Prepare a chessboard: Make or obtain a chessboard with black and white squares. The size should be slightly larger than the camera's field of view, and the sides of adjacent squares should be equal.

[0137] 2) Take chessboard images: Place the chessboard in the camera imaging area and take chessboard images from multiple angles and positions; make sure the chessboard fills the entire image area and take enough images to get a good calibration result;

[0138] 3) Corner detection: Use adaptive threshold and non-maximum suppression to detect corners; first convert the image into a grayscale image, then perform binarization, separate the adhered corners through corrosion operations, and finally detect the position of the corners;

[0139] 4) Sub-pixel corner point extraction: The detected corner points are refined with sub-pixel precision to improve the accuracy of the corner point position;

[0140] 5) Calculate the camera's intrinsic parameters and distortion coefficients: Use the detected corner point coordinates to calculate the camera's intrinsic parameter matrix and distortion coefficients through the Zhang Zhengyou calibration method;

[0141] 6) Generate distortion correction mapping: Generate a mapping matrix for distortion correction based on the calculated internal parameters and distortion coefficients;

[0142] 7) Image distortion correction: Use the mapping matrix to correct the distortion of the original image;

[0143] 8) Save correction parameters: Save the calculated intrinsic parameter matrix, distortion coefficient and mapping matrix for subsequent use in distortion correction of other images.

[0144] The target recognition model framework includes:

[0145] 1) The Backbone part uses a ResNet variant with optimized depth and width as the Backbone to extract rich feature representations; introduces the Spatial attention mechanism to enhance the model's sensitivity to the target location; integrates the Channel attention mechanism to strengthen effective information channels and reduce the interference of irrelevant information;

[0146] 2) The Neck part adopts the structure of feature pyramid network; the Specific Pathway for Pedestrian module is designed to extract the features of small human bodies under wide-angle lenses. The size of the feature map can be adjusted according to the proportion of the human body in the picture; the optimal number of channels is determined through cross-validation to reduce the amount of calculation and parameters while maintaining the feature expression ability to ensure the real-time performance of the model;

[0147] 3) Adopting a decoupled head structure to separate the classification and detection heads, improving the flexibility and accuracy of the model; following the Anchor-Free design, using CenterNet makes the model more flexible and accurate in processing objects at different scales;

[0148] 4) To overcome the problem of few samples and sample imbalance, Focal Loss is used in the classification part of the loss function, and Distribution Focal Loss is used in the regression part to further improve the performance of the model on unbalanced data sets and the generalization under occlusion;

[0149] 5) In the inference phase, use model pruning, quantization, and knowledge distillation techniques to improve model operation efficiency;

[0150] A) Model quantization: Use quantization-aware training of the deep learning framework Pytorch to achieve model quantization;

[0151] B) Model pruning: Use the weight pruning of the deep learning framework Pytorch, combined with manual structured pruning;

[0152] C) Knowledge Distillation:

[0153] a. The first model trained is a large teacher model with high accuracy;

[0154] b. Design a small student model to learn the knowledge of the teacher model;

[0155] c. Train the student model by using the distillation loss function and taking the output of the teacher model as the target;

[0156] d. Adjust the temperature parameters during the distillation process to balance the softening degree of the soft label;

[0157] e. Deploy the student model for reasoning to verify its performance and reasoning efficiency improvements;

[0158] D) Half-precision reasoning: Depending on the hardware support, choose to use FP16 half-precision or INT8 integer precision for reasoning.

[0159] The target tracking algorithm framework includes:

[0160] 1) Initialization: Initialize the tracking list T, create trajectory objects, and store them; in the first frame, all created trajectories will be marked as tracked trajectories;

[0161] 2) Target detection: For each frame in the video sequence V , use the target detector Det to detect and obtain the detection box and score, where k represents the kth frame;

[0162] 3) Detection result classification: According to whether the detection result score is higher than the threshold Th, the detection result is divided into high confidence detection and low confidence detection ;

[0163] 4) Predict tracking position: Use the Kalman filter to predict the position of each track in the current tracking list T;

[0164] 5) First data association: Use the similarity measure matrix to associate the predicted tracking positions with high confidence detections Perform association and obtain preliminary association results;

[0165] 6) Remaining track processing: Get the unmatched tracks from the preliminary association results, denoted as ;

[0166] 7) Second data association: Use the similarity measure matrix to track the remaining With low confidence detection Perform association to obtain the final remaining trace ;

[0167] 8) Delete unmatched traces: Delete unmatched traces from the trace list;

[0168] 9) Initialize new tracking: For the remaining low confidence detections , initialize a new trace and add it to the trace list T;

[0169] 10) Output tracking results: Return the final tracking list.

[0170] The microwave radar module is further specified as follows: using a millimeter-wave radar with a detection distance of 20 meters and an operating power of 24 GHz, installing no less than 8 millimeter-wave radars of this type around the vehicle body, and using the ICP algorithm to fuse radar data to construct a point cloud map around the vehicle.

[0171] The data processing module is further specifically: converting the visual and radar targets within the field of view into the polar coordinate system, normalizing all the coordinates, and obtaining the normalized coordinates of the centroid of the visual object , where i represents the i-th target identified, and the normalized coordinates of the radar object center of mass ,in Between 0 and 1, The value is between 0 and 360; the distance is calculated for all coordinates , the match is successful if the distance is less than the threshold z; the rest are all thrown into the unmatched target list and use the general warning parameters.

[0172] The decision and control module is further specifically:

[0173] Collision risk assessment: Use the trained collision risk assessment model to calculate the collision risk level ,in is the distance between the target and the vehicle, is the direction angle between the target and the front of the forklift, is the target type, is the target relative vehicle speed.

[0174] The present invention is further described below in conjunction with a specific embodiment:

[0175] Forklift anti-collision system based on radar and vision fusion,

[0176] 1. System composition:

[0177] Triple-proof panoramic camera system: As the front-end sensing device of the system, the camera used in the present invention has the characteristics of waterproof, dustproof and shockproof, ensuring stable operation even in harsh environments. At the same time, the camera uses the images of four wide-angle cameras in the front, back, left and right to calibrate and stitch them to form a panoramic image of the vehicle. The steps of the panoramic image stitching algorithm are as follows:

[0178] Feature point extraction and matching: Use the ORB (Oriented FAST and Rotated BRIEF) algorithm to extract image feature points and generate feature descriptors for subsequent image matching.

[0179] Image registration: The RANSAC algorithm is used to calculate the model parameters by iteratively and randomly selecting sample point pairs. The consistency of the model is then evaluated to remove mismatched points from the feature point matching results. The homography matrix between the images is calculated based on the feature point matching results to determine the geometric transformation relationship between the images.

[0180] Homography matrix calculation steps:

[0181] The homography matrix is ​​a 3x3 matrix with eight degrees of freedom and is normalized to 1; the mathematical scaling is as follows:

[0182] ;

[0183] is the homography matrix, used as a projective transformation, the homography matrix is the matrix internal parameter, where the value range of i and j is [1,2,3]. According to the algebraic formula, we can know that Together they determine the mapping relationship of the x-coordinate. It can be regarded as the linear transformation coefficient of the original coordinates x and y. is a translation term; Determines the mapping relationship of the y coordinate, is the linear transformation coefficient of the original coordinates x and y about y', is the translation term in the y' direction; in the calculation of homogeneous coordinates, Play a normalizing role;

[0184] The matrix form represents the homogeneous coordinates on the 2D image, where It refers to the homogeneous coordinates of the points on the original image plane before the projective transformation. It refers to the homogeneous coordinates of the points on the new image plane after the projective transformation;

[0185] Homography matrix satisfy , where the spatial position point relationship is as follows Figure 1 shown.

[0186] The camera rotates around the projection axis, which is equivalent to considering these points to be on a plane at infinity, such as Figure 2 shown.

[0187] Image fusion: Based on image registration, multi-band blending technology is used to decompose the image into pyramids, fuse them layer by layer, and finally reconstruct a seamless panorama.

[0188] The calculation steps are as follows:

[0189] Pyramid decomposition: Pyramid decomposition is performed on the image to be fused, usually using a Gaussian pyramid or a Laplacian pyramid. The Gaussian pyramid reduces the image resolution by Gaussian filtering and downsampling, while the Laplacian pyramid is based on the Gaussian pyramid. Each layer of the Laplacian pyramid is calculated by subtracting the previous layer of Gaussian image that has been upsampled and downsampled from the current layer of Gaussian image. The purpose of this step is to decompose the image into sub-bands of different frequency ranges.

[0190] Constructing a Laplacian pyramid: First, a Gaussian pyramid is formed by Gaussian blurring and downsampling with half the size layer by layer. Then, each layer of the Laplacian pyramid is calculated by subtracting the Gaussian image of the previous layer that has been upsampled and downsampled from the Gaussian image of the current layer. In this process, downsampling is used to reduce the image resolution, and upsampling is used to restore the resolution.

[0191] Fusion weight calculation: For each frequency band, the corresponding weight image is calculated according to the area that needs to be fused. The weight image is used to indicate the proportion of each pixel in the fusion.

[0192] Frequency band fusion: For each frequency band, the images to be fused are weighted fused according to the weight. This can be achieved using simple weighted average, linear blending or other fusion functions. Different rules are used to fuse images of different layers (images of different frequency bands), with high-frequency parts blending slowly and low-frequency parts blending quickly.

[0193] Pyramid reconstruction: Reconstruct each fused frequency band and merge them into the final output image. Reconstruction is usually performed using methods such as pyramid reconstruction or inverse wavelet transform.

[0194] Image reconstruction: By iteratively processing each layer of the Gaussian pyramid until the preset maximum number of iterations is reached, the image fusion and detail preservation are completed. Finally, through the pyramid reconstruction process, starting from the top image of the fused Laplacian pyramid, information is gradually added to the next layer to reconstruct the image, retaining the detailed information of the two images, and finally obtaining a fused image.

[0195] Pseudocode example:

[0196] function MultiBandBlending(image1, image2)

[0197] / / Step 1: Pyramid Decomposition

[0198] / / Create the Laplacian pyramid of the two images

[0199] pyramid1 = CreateLaplacianPyramid(image1, levels)

[0200] pyramid2 = CreateLaplacianPyramid(image2, levels)

[0201] / / Step 2: Blending Weights Calculation

[0202] / / Calculate the weight image, assuming a simple average value is used as the weight

[0203] weights = CalculateBlendingWeights(pyramid1, pyramid2)

[0204] / / Step 3: Band Blending

[0205] / / Initialize an empty pyramid to store the fused image

[0206] blendedPyramid = []

[0207] / / Start from the lowest frequency layer and merge each layer upwards

[0208] for level from levels-1 down to 0

[0209] / / Get the image and weights of the current layer

[0210] layer1 = pyramid1[level]

[0211] layer2 = pyramid2[level]

[0212] weight = weights[level]

[0213] / / Merge the current layer

[0214] blendedLayer = BlendLayer(layer1, layer2, weight)

[0215] blendedPyramid.insert(0, blendedLayer) / / Insert into the top layer of the pyramid

[0216] / / Step 4: Pyramid Reconstruction

[0217] / / Reconstruct the fused image from the Laplacian pyramid

[0218] blendedImage = ReconstructFromLaplacianPyramid(blendedPyramid)

[0219] return blendedImage

[0220] end function

[0221] function CreateLaplacianPyramid(image, levels)

[0222] / / Initialize the Gaussian pyramid

[0223] gaussianPyramid = []

[0224] currentImage = image

[0225] / / Create a Gaussian pyramid

[0226] for i from 0 to levels-1

[0227] downScaledImage = GaussianDownsample(currentImage)

[0228] gaussianPyramid.push(downScaledImage)

[0229] currentImage = downScaledImage

[0230] / / Create a Laplacian pyramid

[0231] laplacianPyramid = []

[0232] for i from 0 to levels-2

[0233] laplacianImage = SubtractImages(gaussianPyramid[i],GaussianUpsample(gaussianPyramid[i+1]))

[0234] laplacianPyramid.push(laplacianImage)

[0235] / / The last layer is the last layer of the Gaussian pyramid

[0236] laplacianPyramid.push(gaussianPyramid[levels-1])

[0237] return laplacianPyramid

[0238] end function

[0239] function CalculateBlendingWeights(pyramid1, pyramid2)

[0240] / / Calculate the weight, here we use a simple average as an example

[0241] weights = []

[0242] for each layer1, layer2 in zip(pyramid1, pyramid2)

[0243] averageWeight = CalculateAverageWeight(layer1, layer2)

[0244] weights.push(averageWeight)

[0245] return weights

[0246] end function

[0247] function BlendLayer(layer1, layer2, weight)

[0248] / / Use weights to fuse two image layers

[0249] blendedLayer = layer1 * weight + layer2 * (1 - weight)

[0250] return blendedLayer

[0251] end function

[0252] function ReconstructFromLaplacianPyramid(pyramid)

[0253] / / Reconstruct the image from the Laplacian pyramid

[0254] blendedImage = pyramid[0] / / Start from the top layer

[0255] for i from 1 to length(pyramid)-1

[0256] blendedImage = GaussianUpsample(blendedImage) + pyramid[i]

[0257] return blendedImage

[0258] end function

[0259] Solve the ghosting problem: Use the APAP algorithm to process the image in blocks, and calculate the homography matrix for each block separately to eliminate the splicing traces.

[0260] Calculation steps:

[0261] Feature point extraction: Use feature detection algorithms such as SIFT to extract key points and descriptors from two images. SIFT (Scale-Invariant Feature Transform) is a computer vision algorithm used to detect and describe local features in images. It was proposed by David Lowe in 1999. Its main purpose is to effectively extract stable feature points under different image scale, rotation, lighting and other changing conditions. These feature points are very important for applications such as image matching, target recognition, and image stitching.

[0262] Feature point matching: Match the extracted feature points to obtain matching point pairs.

[0263] Remove outliers: Use the RANSAC algorithm to remove inaccurate matching points and retain inliers (correct matching points). RANSAC (Random Sample Consensus) is an iterative algorithm for data fitting. It is mainly used to process data containing a lot of noise or outliers, and is widely used in computer vision, photogrammetry, and robotics. For example, when estimating the homography matrix or base matrix from a set of point pairs containing incorrect matches, the RANSAC algorithm can effectively find the correct model parameters.

[0264] Calculate the global homography matrix: Use the interior points to calculate the global homography matrix for preliminary image alignment.

[0265] Image segmentation: Divide the image into several grid blocks.

[0266] Calculate the local homography matrix: Use the Moving DLT algorithm to calculate the local homography matrix for each grid block. This involves calculating the weight for each grid center point, which is determined by the distance between the grid center point and all internal points, using the weight to update the A matrix in the DLT algorithm, and then decomposing it through SVD to obtain the local homography matrix.

[0267] DLT (Direct Linear Transformation) is a method widely used in computer vision and photogrammetry to estimate the transformation matrix from corresponding point pairs. Moving DLT is an algorithm improvement based on DLT that takes motion scenes into consideration. It is mainly used to deal with situations involving object motion or camera motion, such as target tracking and pose estimation in video sequences, and dynamic scene reconstruction in multi-view geometry.

[0268] SVD (Singular Value Decomposition) is an important matrix decomposition method. It can decompose any matrix into the product of three matrices, one of which is an orthogonal matrix and the other is a diagonal matrix. The elements on the diagonal are called singular values, which is an orthogonal matrix. SVD decomposition has a wide range of applications in many fields, including data compression, signal processing, machine learning, computer vision, etc.

[0269] Image Mapping: The image is mapped onto the panorama canvas using the local homography matrix of each grid block.

[0270] Image fusion: Perform image fusion in overlapping areas to eliminate stitching traces and obtain the final seamless panorama.

[0271] Post-processing: It may be necessary to perform post-processing on the stitched images, such as filtering, blurring, etc. to reduce ghosting.

[0272] Distortion correction: For wide-angle images taken with multiple fisheye camera lenses, distortion correction is performed to obtain normal viewing angle images. The correction method uses the checkerboard correction method, and the implementation steps are as follows:

[0273] Prepare a checkerboard: Make or obtain a checkerboard, usually black and white squares, slightly larger than the camera's field of view, with adjacent squares of equal length.

[0274] Take chessboard images: Place a chessboard in the camera imaging area and take chessboard images from multiple angles and positions. Make sure the chessboard fills the entire image area and take enough images to get a good calibration result.

[0275] Corner detection: Use adaptive threshold and non-maximum suppression to detect corners. First, convert the image into a grayscale image, then perform binarization, separate the adhered corners through corrosion operations, and finally detect the position of the corners.

[0276] Sub-pixel corner extraction: The detected corners are refined with sub-pixel precision to improve the accuracy of the corner positions.

[0277] Calculate the camera's intrinsic parameters and distortion coefficients: Use the detected corner coordinates to calculate the camera's intrinsic parameter matrix and distortion coefficients using Zhang Zhengyou's calibration method

[0278] Generate distortion correction mapping: Generate a mapping matrix for distortion correction based on the calculated intrinsic parameters and distortion coefficients.

[0279] Image distortion correction: Use the mapping matrix to correct the distortion of the original image.

[0280] Save correction parameters: Save the calculated intrinsic parameter matrix, distortion coefficients and mapping matrix for subsequent use in distortion correction of other images.

[0281] Millimeter-wave radar: Use millimeter-wave radar with a detection distance of 20 meters and an operating power of 24GHz. No less than 8 radars of this type are installed around the vehicle body. The ICP (Iterative Closest Point) algorithm is used to fuse radar data to construct a point cloud map around the vehicle.

[0282] Actuators: including sound and light alarms, electronic throttle controllers, etc. More actuators can be customized according to specific implementation requirements. Sound and light alarms and electronic throttle controllers are the most basic units used to implement early warning and control measures.

[0283] Target recognition and tracking: An end-to-end instance segmentation model is designed for target recognition, and a tracking algorithm is used to track the trajectory of the target. The end-to-end target recognition model framework is:

[0284] The Backbone [Backbone refers to the backbone network part, which is mainly responsible for feature extraction] part uses a ResNet [ResNet refers to the residual network structure] variant with optimized depth and width as the Backbone to extract rich feature representations. The Spatial attention mechanism is introduced to enhance the model's sensitivity to the target location. The Channel attention mechanism is integrated to strengthen the effective information channel and reduce the interference of irrelevant information.

[0285] The Neck [Neck usually refers to the network structure part located between the backbone network (Backbone) and the head (Head). It acts like an intermediate processing layer] part adopts the structure of the Feature Pyramid Network (FPN), and realizes multi-scale feature fusion through upsampling and side connections. The Specific Pathway for Pedestrian (SPP) module [SPP refers to a feature extraction module specially designed for pedestrian detection, and the above steps are described as the content of this module] is designed specifically for the feature extraction of small human targets under wide-angle lenses. The size of the feature map can be adjusted according to the proportion of the human body in the picture. The optimal number of channels is determined through cross-validation to reduce the amount of calculation and parameters, while maintaining the feature expression ability and ensuring the real-time performance of the model. Considering that the human target is likely to be a small target in the picture in actual scenarios, the feature map size composition is designed to be [544,272,136,68]

[0286] The decoupled head structure is used to separate the classification and detection heads to improve the flexibility and accuracy of the model. Following the Anchor-Free design [Anchor-Free design means that in the target detection task, the position and size of the target are not predicted without relying on predefined anchor boxes. This method directly learns the representation of the target from the feature map instead of restricting the target box to a predefined set of anchor boxes. It is a design concept noun], CenterNet [CenterNet is an anchor-free target detection algorithm that transforms the target detection problem into a simple regression problem, directly predicting the center point position of the target and the size of the target box. The core idea of ​​this method is to regard the target detection task as a keypoint detection problem, where each target is represented by a set of keypoints (such as the center point or corner point of the target). CenterNet determines the position and size of the target by predicting the positions of these keypoints. ] makes the model more flexible and accurate in processing objects at different scales.

[0287] In order to overcome the problem of few samples and sample imbalance, in addition to using multi-class data enhancement technology. The classification part of the loss function uses Focal Loss [Focal Loss is a loss function used to solve the problem of category imbalance and the problem of imbalance between easy and difficult samples, especially in the target detection task. It was proposed by He Kaiming et al. in the 2017 paper "FocalLoss for Dense Object Detection". The core idea of ​​Focal Loss is to reduce the weight of easy-to-classify samples and increase the weight of difficult-to-classify samples, so that the model pays more attention to samples that are difficult to classify correctly during training. The regression part uses Distribution Focal Loss [Distribution Focal Loss (DFL) is a loss function used in target detection tasks. Its core idea is to transform the bounding box problem in target detection into a distribution estimation problem. In traditional target detection methods, the position of the bounding box is usually represented by fixed coordinate values ​​(such as the coordinates of the upper left corner and the lower right corner). Further improve the performance of the model on unbalanced data sets and the generalization of occlusion.

[0288] In the inference phase, model pruning, quantization, and knowledge distillation techniques are used to improve model operation efficiency. Depending on hardware support, half-precision or INT8 precision can be used for inference.

[0289] Model quantization: Model quantization is implemented using quantization-aware training (QAT) of the deep learning framework Pytorch.

[0290] Model pruning: Use the weight pruning of the deep learning framework Pytorch, combined with manual structured pruning.

[0291] Knowledge Distillation:

[0292] The first model trained is a large teacher model, which has high accuracy.

[0293] Design a small student model to learn the knowledge of the teacher model.

[0294] The student model is trained by taking the output (soft label) of the teacher model as the target through the distillation loss function.

[0295] Adjust the temperature parameter (Temperature) during the distillation process to balance the softening degree of the soft label.

[0296] Deploy the student model for inference to verify its performance and improvement in inference efficiency

[0297] Half-precision reasoning: Depending on the hardware support, you can choose to use FP16 half-precision or INT8 integer precision for reasoning.

[0298] The target tracking algorithm framework is: This algorithm is based on the target tracking algorithm of Bayesian theorem and Kalman filter. Brief technical description:

[0299] Initialization: Initialize the tracking list T, create track objects, and store them. In the first frame, all created tracks will be marked as tracked tracks.

[0300] Object detection: For each frame in the video sequence V [k represents the kth frame], use the object detector Det to detect and obtain the detection box and score. V represents the video sequence, which is a Composed of a collection.

[0301] Detection result classification: According to whether the detection result score is higher than the threshold Th, the detection result is divided into high confidence detection and low confidence detection .

[0302] Predict Track Position: Use a Kalman filter to make a position prediction for each track in the current track list T.

[0303] First data association: Use a similarity measure matrix to associate predicted tracking locations with high confidence detections Perform association and obtain preliminary association results.

[0304] Remaining trace processing: Get the unmatched traces from the preliminary association results, denoted as .

[0305] Second data association: Use the similarity measure matrix to track the remaining With low confidence detection Perform association to obtain the final remaining trace .

[0306] Delete Unmatched Tracks: Delete unmatched tracks from the track list.

[0307] Initialize new tracks: For the remaining low confidence detections , initialize a new trace and add it to the trace list T.

[0308] Output tracking results: Return the final tracking list.

[0309] Data fusion: convert the visual and radar targets within the field of view into polar coordinates, normalize all coordinates, and obtain the normalized coordinates of the center of mass of the visual object. [i represents the i-th target identified], normalized coordinates of the center of mass of the radar object ,in Between 0 and 1, The value is between 0 and 360. Calculate the distance of all coordinates If the distance is less than the threshold [the threshold here is the threshold for spatial distance calculation, which is different from the threshold for tracking calculation and is adjusted according to the vehicle condition], the match is successful. All the remaining ones are thrown into the unmatched target list and use the general warning parameters.

[0310] Collision risk assessment: Use the trained end-to-end collision risk assessment model to calculate the collision risk level ,in is the distance between the target and the vehicle, is the direction angle between the target and the front of the forklift, The target type [including: vehicle, person, cargo, wall, other, etc.], is the target relative vehicle speed. The collision risk assessment model is a simple end-to-end artificial intelligence model of a multi-layer perceptron, which converts the risk assessment output here into a classification problem. The specific internal parameters are determined by the training data and belong to a black box model.

[0311] Decision-making and control: Different early warning measures are implemented according to the risk assessment results. For example, low risk means low-frequency warning lights flashing and normal warning broadcasts. High risk means high-frequency warning lights flashing and tweeter alarms. Emergency collision risk means high-frequency warning lights flashing, tweeter alarms and emergency braking.

[0312] 2. Inter-module collaborative working mechanism:

[0313] The visual recognition and tracking module detects objects and tracks targets in the panoramic image in real time, and continuously outputs object coordinates and object types;

[0314] The microwave radar module also continuously outputs point cloud data of objects around the vehicle;

[0315] The data processing module matches the output results of the visual recognition and tracking module and the microwave radar module in real time, marks the successfully matched data as radar-visual fusion targets, marks the unmatched radar targets as other types of tracking targets, and discards the unmatched visual recognition targets.

[0316] Input the matching result list into the collision risk assessment model to output the collision risk level;

[0317] The decision and control module performs corresponding decision actions according to the risk level output by the collision risk assessment model:

[0318] Low risk: low frequency alarm light flashing and normal warning broadcast;

[0319] High risk: high-frequency alarm lights flashing and loudspeaker alarms;

[0320] Emergency collision risk: high-frequency warning lights flashing, high-pitched horn warning and emergency braking.

[0321] 3. Implementation results:

[0322] This system uses the radar-vision fusion technology to effectively reduce the false triggering of the anti-collision system due to false alarms, and improves the safety and efficiency of forklift operations. Specific implementation effects include:

[0323] Intelligent target recognition alarm: When the forklift is moving parts, it passes through a narrow shelf channel. The radar detects a large number of stationary objects and shelves around it. At the same time, the visual system recognizes and detects these large quantities of goods and shelves around it, and matches the space with the radar vision. According to the user-defined setting, the distance of the goods must be less than 0.5 meters before the alarm is triggered, so in this case, no unnecessary disturbing alarm will be generated. In addition, when the forklift is working, a pedestrian appears 2 meters to the side. At this time, according to the parameters set for the alarm when the person is less than 3 meters away, an audible and visual warning is issued in time to remind the driver that someone is approaching. At the same time, the audible and visual alarm also provides a wake-up effect for pedestrians, informing them that the forklift is working. And according to the pedestrian's subsequent continuous movement trajectory and approach speed, a graded alarm with different frequencies and volumes is issued, thereby effectively avoiding a possible serious collision accident.

[0324] Example of early warning signal classification settings:

[0325] Cargo collision risk level:

[0326] Low risk: the cargo is more than 1 meter away from the vehicle;

[0327] High risk: the distance between the cargo and the vehicle is between 0.5m and 1m;

[0328] Emergency collision risk: cargo is within 0.5 meters from the vehicle;

[0329] Personnel collision risk level:

[0330] Low risk: The person is more than 3 meters away from the vehicle;

[0331] High risk: the person is between 1 meter and 3 meters away from the vehicle;

[0332] Emergency collision risk: Person is within 1 meter of the vehicle;

[0333] Vehicle collision risk level:

[0334] Low risk: vehicles are more than 10 meters apart;

[0335] High risk: vehicles are between 5 and 10 meters apart;

[0336] Emergency collision risk: Vehicles are within 5 meters of each other, or vehicles are within 10 meters of each other and the speed exceeds 15 km / h.

[0337] Generic object [unknown object] collision risk level:

[0338] Low risk: the object is more than 5 meters away from the vehicle;

[0339] High risk: the object is between 1 meter and 5 meters away from the vehicle;

[0340] The object with emergency collision risk is within 1 meter of the vehicle;

[0341] All-weather working capability: Even in adverse weather conditions, such as heavy rain, the images captured by the camera will become blurred due to the coverage of rain and mud, making it difficult to accurately identify people or other smaller objects in the picture. However, the radar fusion system is also equipped with advanced radar sensors that can work normally in various weather and lighting conditions. Even when the camera cannot provide a clear image, the radar sensor can still rely on its basic distance information to make an accurate alarm. Therefore, whether it is day or night, whether it is sunny or rainy, the radar sensor can ensure the stable operation of the system, regardless of any weather and lighting conditions. This all-weather working capability enables the entire monitoring system to maintain efficient and reliable performance in various environments.

[0342] Improve operating efficiency: Through system optimization, false alarms and unnecessary alarms are significantly reduced, allowing drivers to focus more on driving tasks and avoid unnecessary interference and distraction. In this way, the collision system can be truly implemented and accepted by the majority of drivers, avoiding annoyance and actively shutting down the collision system. The efficiency of forklift operations has been significantly improved, the operation process has become smoother, and the overall operation time has been effectively shortened.

[0343] The above description is only a preferred embodiment of the present invention and should not be construed as limiting the present application. All equivalent changes and modifications made within the scope of the patent application of the present invention should fall within the scope of the present invention.

Claims

1. The forklift anti-collision system based on radar and vision fusion is characterized by: It includes panoramic image stitching module, visual recognition and tracking module, microwave radar module, data processing module, decision-making and control module; Panoramic image stitching module: uses the images from four wide-angle cameras in the front, back, left and right to calibrate and stitch them to form a panoramic image of the vehicle; Visual recognition and tracking module: performs object detection and target tracking in the panoramic image in real time, and continuously outputs object coordinates and object types; Microwave radar module: continuously outputs point cloud data of objects around the vehicle; Data processing module: performs real-time matching of the output results of the visual recognition and tracking module and the microwave radar module, marks the successfully matched data as radar-visual fusion targets, marks the unmatched radar targets as other types of tracking targets, and discards the unmatched visual recognition targets; Decision and control module: input the matching result list into the collision risk assessment model to output the collision risk level; According to the collision risk level, execute corresponding decision actions; The panoramic image stitching module includes a feature point extraction and matching unit, an image registration unit, an image fusion unit, a ghost phenomenon solving unit, and a distortion correction unit; The feature point extraction and matching unit generates feature descriptors for the image; the image registration unit calculates the homography matrix between the images based on the results of the feature point extraction and matching unit; the image fusion unit decomposes and fuses the images based on the image registration unit, and then reconstructs a seamless panoramic image; The ghosting problem solving unit processes the image in blocks, and calculates the homography matrix for each block to eliminate the splicing traces. Finally, the distortion correction unit corrects the image distortion to obtain a normal viewing angle image. The visual recognition and tracking module includes a target recognition model framework and a target tracking algorithm framework, wherein the target recognition model framework is used for target recognition, and the target tracking algorithm framework is used for tracking the trajectory of the target; The panoramic image stitching module is further specifically: (1) Feature point extraction and matching unit: Use the ORB algorithm to extract image feature points and generate feature descriptors for subsequent image matching; (2) Image registration unit: The random sampling consistency RANSAC algorithm is used to calculate the model parameters by iteratively and randomly selecting sample point pairs, and then the consistency of the model is evaluated to eliminate mismatched points from the feature point matching results; the homography matrix between the images is calculated based on the feature point matching results to determine the geometric transformation relationship between the images; Homography matrix calculation steps: The homography matrix is ​​a 3x3 matrix with eight degrees of freedom and is normalized to 1; the mathematical scaling is as follows: ； is the homography matrix, used as a projective transformation, the homography matrix is the matrix internal parameter, where the value range of i and j is [1,2,3]; According to the algebraic formula, we can know Together they determine the mapping relationship of the x-coordinate; It can be regarded as the linear transformation coefficient of the original coordinates x and y. is a translation term; Determines the mapping relationship of the y coordinate, is the linear transformation coefficient of the original coordinates x and y about y', is the translation term in the y' direction; in the calculation of homogeneous coordinates, Play a normalizing role; The matrix form represents the homogeneous coordinates on the 2D image, where It refers to the homogeneous coordinates of the points on the original image plane before the projective transformation. It refers to the homogeneous coordinates of the points on the new image plane after the projective transformation; Homography matrix satisfy ; (3) Image fusion unit: Based on image registration, the multi-band blending technology is used to perform pyramid decomposition on the image, fuse the images layer by layer, and finally reconstruct a seamless panorama. (4) Ghosting problem solving unit: The image is divided into blocks using the APAP algorithm, and the homography matrix is ​​calculated for each block to eliminate the splicing traces; (5) Distortion correction unit: For wide-angle images taken with multiple fisheye camera lenses, distortion correction is performed to obtain normal viewing angle images. The correction method adopts a checkerboard correction method.

2. The forklift collision avoidance system based on radar and vision fusion according to claim 1 is characterized in that: The image fusion unit is further specifically: The calculation steps are as follows: 1) Constructing a Laplacian pyramid: First, a Gaussian pyramid is formed by Gaussian blurring and downsampling by halving the size layer by layer. Then, each layer of the Laplacian pyramid is calculated by subtracting the previous layer of Gaussian image that has been upsampled and downsampled from the current layer of Gaussian image. In this process, downsampling is used to reduce the image resolution, and upsampling is used to restore the resolution. 2) Fusion weight calculation: For each frequency band, calculate the corresponding weight image according to the area that needs to be fused; The weight image is used to indicate the weight of each pixel in the fusion; 3) Frequency band fusion: For each frequency band, the images to be fused are weighted fused according to the weight; 4) Pyramid reconstruction: reconstruct each fused frequency band and merge them into the final output image; 5) Image reconstruction: By iteratively processing each layer of the Gaussian pyramid until the preset maximum number of iterations is reached, the image fusion and detail retention are completed; finally, through the pyramid reconstruction process, starting from the fused top layer image of the Laplacian pyramid, information is gradually added to the next layer to reconstruct the image, retaining the detailed information of the two images, and finally obtaining a fused image.

3. The forklift collision avoidance system based on radar and vision fusion according to claim 1 is characterized in that: The ghost phenomenon solving unit is further specifically: The calculation steps are as follows: 1) Feature point extraction: Use SIFT feature detection algorithm to extract key points and descriptors in two images; 2) Feature point matching: Match the extracted feature points to obtain matching point pairs; 3) Remove outliers: Use the RANSAC algorithm to remove inaccurate matching points and retain inliers; 4) Calculate the global homography matrix: Use the internal points to calculate the global homography matrix for preliminary image alignment; 5) Image segmentation: Divide the image into several grid blocks; 6) Calculate the local homography matrix: Use the Moving DLT algorithm to calculate the local homography matrix for each grid block; this involves calculating the weight for each grid center point, the weight is determined by the distance between the grid center point and all the internal points, using the weight to update the A matrix in the DLT algorithm, and then decomposing it through SVD to get the local homography matrix; 7) Image mapping: Use the local homography matrix of each grid block to map the image onto the panoramic canvas; 8) Image fusion: Perform image fusion in overlapping areas to eliminate stitching traces and obtain the final seamless panorama.

4. The forklift collision avoidance system based on radar and vision fusion according to claim 1 is characterized in that: The distortion correction unit is further specifically: The steps for implementing the chessboard correction method are as follows: 1) Prepare a chessboard: Make or obtain a chessboard with black and white squares. The size should be slightly larger than the camera's field of view, and the sides of adjacent squares should be equal. 2) Take chessboard images: Place the chessboard in the camera imaging area and take chessboard images from multiple angles and positions; make sure the chessboard fills the entire image area and take enough images to get a good calibration result; 3) Corner detection: Use adaptive threshold and non-maximum suppression to detect corners; first convert the image into a grayscale image, then perform binarization, separate the adhered corners through corrosion operations, and finally detect the position of the corners; 4) Sub-pixel corner point extraction: The detected corner points are refined with sub-pixel precision to improve the accuracy of the corner point position; 5) Calculate the camera's intrinsic parameters and distortion coefficients: Use the detected corner point coordinates to calculate the camera's intrinsic parameter matrix and distortion coefficients through the Zhang Zhengyou calibration method; 6) Generate distortion correction mapping: Generate a mapping matrix for distortion correction based on the calculated internal parameters and distortion coefficients; 7) Image distortion correction: Use the mapping matrix to correct the distortion of the original image; 8) Save correction parameters: Save the calculated intrinsic parameter matrix, distortion coefficient and mapping matrix for subsequent use in distortion correction of other images.

5. The forklift collision avoidance system based on radar and vision fusion according to claim 1 is characterized in that: The target recognition model framework includes: 1) The Backbone part uses a ResNet variant with optimized depth and width as the Backbone to extract rich feature representations; introduces the Spatial attention mechanism to enhance the model's sensitivity to the target location; integrates the Channel attention mechanism to strengthen effective information channels and reduce the interference of irrelevant information; 2) The Neck part adopts the structure of feature pyramid network; the Specific Pathway for Pedestrian module is designed to extract the features of small human bodies under wide-angle lenses. The size of the feature map can be adjusted according to the proportion of the human body in the picture; the optimal number of channels is determined through cross-validation to reduce the amount of calculation and parameters while maintaining the feature expression ability to ensure the real-time performance of the model; 3) Adopting a decoupled head structure to separate the classification and detection heads, improving the flexibility and accuracy of the model; following the Anchor-Free design, using CenterNet makes the model more flexible and accurate in processing objects at different scales; 4) To overcome the problem of few samples and sample imbalance, Focal Loss is used in the classification part of the loss function, and Distribution Focal Loss is used in the regression part to further improve the performance of the model on unbalanced data sets and the generalization under occlusion; 5) In the inference phase, use model pruning, quantization, and knowledge distillation techniques to improve model operation efficiency; A) Model quantization: Use quantization-aware training of the deep learning framework Pytorch to achieve model quantization; B) Model pruning: Use the weight pruning of the deep learning framework Pytorch, combined with manual structured pruning; C) Knowledge Distillation: a. The first model trained is a large teacher model with high accuracy; b. Design a small student model to learn the knowledge of the teacher model; c. Train the student model by using the distillation loss function and taking the output of the teacher model as the target; d. Adjust the temperature parameters during the distillation process to balance the softening degree of the soft label; e. Deploy the student model for reasoning to verify its performance and reasoning efficiency improvements; D) Half-precision reasoning: Depending on the hardware support, choose to use FP16 half-precision or INT8 integer precision for reasoning.

6. The forklift collision avoidance system based on radar and vision fusion according to claim 1 is characterized in that: The target tracking algorithm framework includes: 1) Initialization: Initialize the tracking list T, create trajectory objects, and store them; in the first frame, all created trajectories will be marked as tracked trajectories; 2) Target detection: For each frame in the video sequence V , use the target detector Det to detect and obtain the detection box and score, where k represents the kth frame; 3) Detection result classification: According to whether the detection result score is higher than the threshold Th, the detection result is divided into high confidence detection and low confidence detection ; 4) Predict tracking position: Use the Kalman filter to predict the position of each track in the current tracking list T; 5) First data association: Use the similarity measure matrix to associate the predicted tracking positions with high confidence detections Perform association and obtain preliminary association results; 6) Remaining track processing: Get the unmatched tracks from the preliminary association results, denoted as ; 7) Second data association: Use the similarity measure matrix to track the remaining With low confidence detection Perform association to obtain the final remaining trace ; 8) Delete unmatched traces: Delete unmatched traces from the trace list; 9) Initialize new tracking: For the remaining low confidence detections , initialize a new trace and add it to the trace list T; 10) Output tracking results: Return the final tracking list.

7. The forklift collision avoidance system based on radar and vision fusion according to claim 1 is characterized in that: The microwave radar module is further specified as follows: using a millimeter-wave radar with a detection distance of 20 meters and an operating power of 24 GHz, installing no less than 8 millimeter-wave radars of this type around the vehicle body, and using the ICP algorithm to fuse radar data to construct a point cloud map around the vehicle.

8. The forklift collision avoidance system based on radar and visual fusion according to claim 1 is characterized in that: The data processing module is further specifically: converting the visual and radar targets within the field of view into the polar coordinate system, normalizing all the coordinates, and obtaining the normalized coordinates of the centroid of the visual object , where i represents the i-th target identified, and the normalized coordinates of the radar object center of mass , in Between 0 and 1, The value is between 0 and 360; the distance is calculated for all coordinates , if the distance is less than the threshold z, the match is successful; the rest are all thrown into the unmatched target list and use the general warning parameters.

9. The forklift collision avoidance system based on radar and vision fusion according to claim 1 is characterized in that: The decision and control module is further specifically: Collision risk assessment: Use the trained collision risk assessment model to calculate the collision risk level ,in is the distance between the target and the vehicle, is the direction angle between the target and the front of the forklift, is the target type, is the target relative vehicle speed.

Images