Forklift collision early warning system and method adopting three-dimensional point cloud information to correct two-dimensional image coordinates

Abstract

The invention discloses a forklift collision early warning system adopting three-dimensional point cloud information to correct two-dimensional image coordinates, and belongs to the field of industrial safety monitoring. Comprising a video data acquisition module, a three-dimensional point cloud data acquisition module, an image target detection module, a three-dimensional point cloud target detection module, a correction model training module, a coordinate correction module, a target tracking module, a target distance calculation module and an alarm module. The invention further discloses a forklift collision early warning method adopting the three-dimensional point cloud information to correct the two-dimensional image coordinates. According to the method, the image target coordinates are corrected by collecting and detecting the image target and the three-dimensional point cloud target and mapping the model between the image target coordinates and the three-dimensional point cloud target coordinates, so that the influence of perspective distortion of the two-dimensional image is effectively reduced, and the accuracy of the forklift collision early warning system is improved while the real-time performance is ensured.

Description

Technical Field

[0001] The present invention relates to the field of industrial safety monitoring, and in particular to a forklift collision warning system and method for correcting two-dimensional image coordinates using three-dimensional point cloud information. Background Art

[0002] With the continuous deepening of industrial automation, forklifts are used more and more frequently in manufacturing workshops. However, due to the complex operating environment, dense personnel, and irregular material stacking in the workshop, forklifts have a high risk of collision during daily operation. Especially when the line of sight is blocked, the lighting conditions are poor, or the operator's perception of the surrounding environment is insufficient, forklift collision accidents occur frequently, which not only causes huge economic losses to the company, but also brings serious safety hazards. With the rapid development of computer vision technology, protection solutions based on technical means are gradually replacing traditional human protection and becoming the focus of workshop safety management.

[0003] At present, most forklift collision warning systems on the market use visual sensors combined with target detection and tracking algorithms to achieve protection warnings. These systems usually use inverse perspective transformation technology to first correct the perspective distortion caused by the camera lens and installation position, and then use the target detection algorithm to identify key objects such as forklifts and pedestrians, and infer the risk of collision by calculating their relative positions. However, the accuracy of this type of technology based on inverse perspective transformation in actual applications is highly dependent on the installation position and calibration accuracy of the camera. If the installation angle of the camera is slightly deviated, or the workshop environment changes and causes the camera position to move slightly, the effect of the inverse perspective transformation will be significantly reduced, resulting in a decrease in detection accuracy.

[0004] In three-dimensional point cloud target detection, the three-dimensional point cloud data directly reflects the object information in the three-dimensional space, and each point contains three-dimensional coordinates, reflection intensity and other information; and the current three-dimensional point cloud target detection technology has a strong pan-scaling ability, which can well solve the problems of the inverse perspective transformation method. However, three-dimensional point clouds require powerful hardware computing resources, which is a huge challenge for tasks with high real-time requirements such as forklift collision warning. In response to the above problems, the present invention provides a forklift collision warning system and method that uses three-dimensional point cloud information to correct two-dimensional image coordinates. While ensuring real-time performance, it can effectively overcome the influence of perspective distortion caused by camera lenses and installation positions, and improve the accuracy of target distance calculation in target detection. Summary of the invention

[0005] The purpose of the present invention is to provide a forklift collision warning system and method that uses three-dimensional point cloud information to correct two-dimensional image coordinates in response to the above-mentioned problems. The system collects video stream data and three-dimensional point cloud data in the forklift working area, and detects targets in the video stream image and point cloud. By training a mapping model between the image target coordinates and the point cloud target coordinates, the image target coordinates are corrected in real time using the mapping model, thereby reducing the influence of perspective distortion caused by the camera lens and the installation position, and improving the accuracy of target distance calculation in target detection. At the same time, the collection and detection of three-dimensional point cloud data and the training of the mapping model are all carried out in the background. While ensuring real-time performance, the mapping model is updated once at a fixed time, which improves the system's ability to adapt to dynamic scene changes.

[0006] The technical solution of the present invention is as follows: The present invention discloses a forklift collision warning system that uses three-dimensional point cloud information to correct two-dimensional image coordinates, including a video data acquisition module, a three-dimensional point cloud data acquisition module, an image target detection module, a three-dimensional point cloud target detection module, a correction model training module, a coordinate correction module, a target tracking module, a target distance calculation module and an alarm module; The video data acquisition module is used to collect video stream data of the forklift working area; The three-dimensional point cloud data acquisition module is used to collect point cloud data of the forklift working area; The image target detection module is used to detect targets in video stream images; The three-dimensional point cloud target detection module is used to detect targets in point cloud data; The correction model training module is used to train a mapping model between image target coordinates and point cloud target coordinates, and to regularly update the model in the coordinate correction module; The coordinate correction module is used to perform regression calculation on the image target coordinates to obtain corrected coordinates; The target tracking module is used to uniquely number each detected target and continuously track the target in subsequent video frames to determine the direction of each target; The target distance calculation module calculates the distance between the forklift and surrounding objects in the forward direction according to the corrected coordinates; The alarm module triggers an alarm when the distance between the forklift and surrounding objects is less than a set threshold, and sends a notification to the workshop management personnel at the same time.

[0007] The present invention also discloses a forklift collision warning method using three-dimensional point cloud information to correct two-dimensional image coordinates, comprising: Step 1: Collect the video stream of the forklift working area in real time and automatically detect the target in the video; Step 2: If a person and a forklift are detected at the same time, the detected two-dimensional coordinates of the target are passed to the coordinate correction module; if the forklift is not detected, return to step 1 and continue the loop monitoring; if a person and a forklift are detected at the same time for k consecutive frames, trigger the collection of 1 frame of 3D point cloud data of the forklift working area, and automatically detect the target in the point cloud to determine whether the point cloud target matches the k-th frame image target; if they match, extract the target's two-dimensional coordinates, image, and three-dimensional coordinates and pass them into the correction model training module; if they do not match, the point cloud data collection waits for the next trigger.

[0008] Step 3: Train the mapping model between the image target coordinates and the point cloud target coordinates through a deep learning algorithm, and regularly update the model in the coordinate correction module; Step 4: Perform regression calculation on the input two-dimensional coordinates according to the mapping model to obtain the corrected coordinates; Step 5: Uniquely number each detected target and continue to track these targets in subsequent video frames; determine the direction of each target by the position change of the target in consecutive frames.

[0009] Step 6: Based on the corrected coordinates and the forward direction of each target, calculate the distance between the forklift and the surrounding objects in its forward direction; when the distance is less than the set threshold, the alarm module triggers an alarm.

[0010] Furthermore, the step 2 also includes: S21, target annotation: extract the target images containing the forklift and the person from the video collected in step 1, annotate these images, generate annotation files containing the target position and category, and construct an image target detection dataset; annotate the forklift and the person for the three-dimensional point cloud data, generate an annotation file containing the target position, direction, and category, and extract the image data collected at the same time to construct a detection dataset containing the image and three-dimensional point cloud targets; S22. Model training: The image target detection part uses the yolov8n algorithm to train the image target detection dataset; the 3D point cloud target detection part uses MVX-Net to train the 3D point cloud dataset, and the image feature extraction part uses ResNet50.

[0011] Furthermore, the two-dimensional coordinates include the center point coordinates (Cx, Cy), width (W) and height (H) of the target detection frame, expressed as (Cx, Cy, W, H).

[0012] Furthermore, the three-dimensional coordinates include the center point coordinates of the target three-dimensional detection frame (Cx point , Cy point , Cz point )、Length(L point )、Width(Wpoint ) and height (H point ), expressed as (Cx point , Cy point , Cz point , L point , W point , H point ).

[0013] Furthermore, in step 6, the distance between the forklift and the surrounding objects :

[0014] in represents the distance between the i-th target and the j-th target, ( , ) represents the coordinates of the center point of the ith target after correction, ( , ) represents the coordinates of the corrected j-th target center point.

[0015] Furthermore, the step three includes: S31, use ResNet50 to extract features from the input image (3×640×640), and output feature vectors of four scales, with shapes of (64×320×320), (256×160×160), (512×80×80), and (1024×40×40); S32, the incoming target two-dimensional coordinates (Cx i-n , Cy i-n , W i-n , H i-n ) are concatenated to obtain an input vector shape of (n×4), where i represents the i-th target, n represents the number of detected targets, and 4 corresponds to the four coordinates of the target; S33. The coordinate input vector is processed by multi-layer perceptron (MLP) in four dimensions, and the shapes of the obtained four feature vectors are (n×320×320), (n×160×160), (n×80×80), and (n×40×40). Then they are concatenated with the four scale features of the image, and the fused feature vectors are ((n+1)×320×320), ((n+1)×160×160), ((n+1)×80×80), and ((n+1)×40×40). S34, upsample all four feature vectors to (320×320) and perform feature fusion, predict the corrected two-dimensional coordinates through convolution and full connection operations, and the shape of the final output result is (n×4); S35, remove the height information in the input three-dimensional coordinates, and only retain the horizontal and vertical coordinates of the center point (Cx point ,Cy point ) and the length of the 3D detection frame (L point )、Width(W point ); Finally, (Cx point-i， Cy point-i， L point-i， W point-i ) as the prediction label for model training; i represents the i-th target, and the loss function uses Smooth L1 loss; the calculation formula of Smooth L1 is:

[0016] in, It represents the difference between the true value and the predicted value. When the difference is small, the loss is expressed as the mean square error, and when the difference is large, it is expressed as the absolute error. , represents the horizontal and vertical coordinates of the three-dimensional coordinate center point of the i-th target (Cx point , Cy point ) and the length of the 3D detection frame (L point )、Width(W point ), , represents the horizontal and vertical coordinates of the three-dimensional coordinate center point of the i-th target (Cx point , Cy point ) and the length of the 3D detection frame (L point )、Width(W point )’s predicted value.

[0017] Furthermore, the step three also includes: when the accumulated images, two-dimensional coordinates, three-dimensional coordinates and other data input into the correction model training module reaches one week's collection amount, the mapping model training between the image target coordinates and the three-dimensional point cloud target coordinates is started, and the training method is to fine-tune the existing model to ensure that the mapping model is updated once a week.

[0018] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are: 1. The present invention solves the problem that the effect of inverse perspective transformation decreases with the change of workshop environment. By collecting and detecting image targets and three-dimensional point cloud targets, and mapping models between image target coordinates and three-dimensional point cloud target coordinates, the image target coordinates are corrected, and the perspective distortion caused by the camera lens and the installation position is corrected, thereby improving the accuracy of target distance calculation in target detection and improving detection accuracy.

[0019] 2. The present invention solves the problem of high hardware requirements in existing three-dimensional point cloud target detection. The two-dimensional image coordinates are corrected by three-dimensional point cloud information, and the three-dimensional point cloud processing is performed in the background, thereby reducing the hardware requirements.

[0020] 3. The present invention improves the system's ability to adapt to dynamic scene changes by updating the mapping model between image target coordinates and three-dimensional point cloud target coordinates once a week.

[0021] 4. The present invention collects data for one week and performs fine-tuning training on the mapping model between the image target coordinates and the three-dimensional point cloud target coordinates, thereby ensuring that the collected information is sufficient to cover the workshop scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The present invention will now be described by way of example with reference to the accompanying drawings, in which: Figure 1 The present invention is a schematic structural diagram of a forklift collision warning system that uses three-dimensional point cloud information to correct two-dimensional image coordinates.

[0023] Figure 2 The present invention is a flow chart of a forklift collision warning method that uses three-dimensional point cloud information to correct two-dimensional image coordinates. DETAILED DESCRIPTION

[0024] All features disclosed in this specification, or steps in all methods or processes disclosed, except mutually exclusive features and / or steps, can be combined in any manner.

[0025] Any feature disclosed in this specification (including any additional claims and abstract), unless otherwise stated, may be replaced by other alternative features that are equivalent or have similar purposes. That is, unless otherwise stated, each feature is only an example of a series of equivalent or similar features.

[0026] The features and performance of the present invention are further described in detail below in conjunction with the embodiments.

[0027] like Figure 1As shown, the present invention discloses a forklift collision warning system that uses three-dimensional point cloud information to correct two-dimensional image coordinates. The system includes a video data acquisition module, a three-dimensional point cloud data acquisition module, an image target detection module, a three-dimensional point cloud target detection module, a correction model training module, a coordinate correction module, a target tracking module, a target distance calculation module and an alarm module; the video data acquisition module is used to collect video streams of the forklift working area; the three-dimensional point cloud data acquisition module is used to collect three-dimensional point cloud data of the forklift working area; the image target detection module is used to detect targets in video stream images; the three-dimensional point cloud target detection module is used to detect targets in three-dimensional point cloud data; the correction model The model training module is used to train the mapping model between the image target coordinates and the three-dimensional point cloud target coordinates, and regularly update the model in the coordinate correction module; the coordinate correction module performs regression calculation on the two-dimensional coordinates of the input image target to obtain the corrected coordinates; the target tracking module is used to uniquely number each detected target, and continuously track the target in subsequent video frames to determine the target's forward direction; the target distance calculation module calculates the distance between the forklift and the surrounding objects in its forward direction based on the corrected coordinates; the alarm module triggers the alarm when the distance between the forklift and the surrounding objects is less than the set threshold, and sends a notification to the workshop management personnel at the same time.

[0028] In the embodiment, the video acquisition module uses a network camera, and the three-dimensional point cloud acquisition module uses a laser radar; the parameter k is obtained by the following formula: Where f image Indicates the video capture frame rate, f point Indicates the frame rate of 3D point cloud acquisition.

[0029] like Figure 2 As shown, the present invention discloses a forklift collision warning method using three-dimensional point cloud information to correct two-dimensional image coordinates, which is as follows: Step 1: The video data acquisition module collects the video stream of the forklift working area in real time and transmits it to the target detection module.

[0030] Step 2: Perform real-time analysis on the input video stream through the target detection module; if a forklift is detected, the detected two-dimensional coordinates of the target are passed to the coordinate correction module; if no forklift is detected, it means that there is no safety hazard in the current area, and return to step 1 to continue the loop monitoring; at the same time, every time k frames of video are detected, the two-dimensional coordinates of the target in the kth frame and the corresponding video frame are extracted and passed to the correction model training module; and the three-dimensional point cloud data acquisition module is triggered to synchronously collect 1 frame of three-dimensional point cloud data, and then the three-dimensional point cloud target detection module performs target detection on the three-dimensional point cloud data. If the detected three-dimensional point cloud target matches the image target, the three-dimensional coordinates of the target are extracted and passed to the correction model training module; if they do not match, the correction model training module abandons the data passed in the current frame, and the three-dimensional point cloud data acquisition module waits for the next trigger.

[0031] In step 2, every time k frames of video are detected, the 3D point cloud data acquisition module is triggered to collect 1 frame of 3D point cloud data. This is because the frame rate of video acquisition is higher than the frame rate of 3D point cloud acquisition, in order to ensure that the 2D coordinates and 3D coordinates in the correction model training module are from the same time.

[0032] In step 2, the detected 3D point cloud target is matched with the image target to ensure data alignment in the correction model training module.

[0033] The two-dimensional coordinates include the center point coordinates (Cx, Cy), width (W) and height (H) of the target detection box, expressed as (Cx, Cy, W, H).

[0034] The three-dimensional coordinates include the center point coordinates of the target three-dimensional detection box (Cx point , Cy point , Cz point )、Length(L point )、Width(W point ) and height (H point ), expressed as (Cx point , Cy point , Cz point , L point , W point , H point ).

[0035] Step 2 also includes: S21, target labeling: extract the target images containing the forklift and the person from the video collected in step 1, and label these images to generate a labeling file containing the target position and category to construct an image target detection dataset; also label the forklift and the person for the three-dimensional point cloud data, generate a labeling file containing the target position, direction, and category, and extract the image data collected at the same time to construct a three-dimensional point cloud target detection dataset containing the image.

[0036] S22. Model training: The image target detection part uses the yolov8n algorithm to train the image target detection dataset. The 3D point cloud target detection part uses MVX-Net to train the 3D point cloud dataset, and the image feature extraction part uses ResNet50.

[0037] Step three: Through the correction model training module, use the deep learning algorithm to train the mapping model between the image target coordinates and the three-dimensional point cloud target coordinates, and regularly update the model in the coordinate correction module.

[0038] In step three, the model in the coordinate correction module is updated regularly to adapt to changes in the scene in the workshop.

[0039] Step three also includes: S31, use ResNet50 to extract features from the input image (3×640×640), and output feature vectors of four scales, with shapes of (64×320×320), (256×160×160), (512×80×80), and (1024×40×40); S32, the incoming target two-dimensional coordinates (Cx i-n , Cy i-n , W i-n , H i-n ) are concatenated to obtain an input vector shape of (n×4), where i represents the i-th target, n represents the number of detected targets, and 4 corresponds to the four coordinates of the target; S33. The coordinate input vector is processed by multi-layer perceptron (MLP) in four dimensions, and the shapes of the obtained four feature vectors are (n×320×320), (n×160×160), (n×80×80), and (n×40×40). Then they are concatenated with the four scale features of the image, and the fused feature vectors are ((n+1)×320×320), ((n+1)×160×160), ((n+1)×80×80), and ((n+1)×40×40). S34, upsample all four feature vectors to (320×320) and perform feature fusion, predict the corrected two-dimensional coordinates through convolution and full connection operations, and the shape of the final output result is (n×4); S35, remove the height information in the input three-dimensional coordinates, and only retain the horizontal and vertical coordinates of the center point (Cx point ,Cy point ) and the length of the 3D detection frame (L point )、Width(W point ); Finally, (Cx point-i， Cy point-i， Lpoint-i， W point-i ) as the prediction label for model training. Where i represents the i-th target, and the loss function uses Smooth L1 loss. The calculation formula of Smooth L1 is:

[0040] in, It represents the difference between the true value and the predicted value. When the difference is small, the loss is expressed as the mean square error, and when the difference is large, it is expressed as the absolute error. Represents the horizontal and vertical coordinates of the three-dimensional coordinate center point of the i-th target (Cx point , Cy point ) and the length of the 3D detection frame (L point )、Width(W point ) is the true value of , Represents the horizontal and vertical coordinates of the three-dimensional coordinate center point of the i-th target (Cx point , Cy point ) and the length of the 3D detection frame (L point )、Width(W point ) is the predicted value of .

[0041] Step three also includes: When the accumulated data of images, 2D coordinates, 3D coordinates, etc. input into the correction model training module reaches a week's collection volume, fine-tuning training of the mapping model between image target coordinates and 3D point cloud target coordinates begins, ensuring that the mapping model is updated once a week. This ensures that the collected information is sufficient to cover changes in workshop scenes and weather reasons, and improves the system's ability to adapt to changes in workshop scenes.

[0042] Step 4: The coordinate correction module performs regression calculation on the input two-dimensional coordinates through the mapping model to obtain the corrected coordinates; then the target tracking module uniquely numbers each detected target, and continues to track these targets in subsequent video frames to determine the direction of each target; at the same time, the target distance calculation module calculates the distance between the forklift and the surrounding objects (people, forklifts) in the forward direction based on the corrected coordinates; when the distance is less than the set threshold, the alarm module triggers the alarm and sends a notification to the workshop manager.

[0043] In step 4, the distance calculation module uses the Euclidean formula to calculate the distance between targets. :

[0044] in represents the distance between the i-th target and the j-th target, ( , ) represents the coordinates of the center point of the ith target after correction, ( , ) represents the coordinates of the corrected j-th target center point.

[0045] Before data collection in step 1, the installation position and angle of the network camera and laser radar are adjusted to ensure that the fields of view of the two overlap as much as possible. At the same time, based on the actual field of view of the laser radar, the video collected by the camera is cropped to remove the part of the field of view that exceeds the field of view of the laser radar, leaving only the overlapping area of ​​the two fields of view.

[0046] The present invention maps a model between image target coordinates and three-dimensional point cloud target coordinates, thereby correcting the image target coordinates while ensuring real-time performance, correcting the perspective distortion caused by the camera lens and the installation position, and improving the accuracy of target distance calculation in target detection; and updating the mapping model once at a fixed time improves the system's ability to adapt to dynamic scene changes.

[0047] The above-mentioned embodiments only express the specific implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the protection scope of the present application. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the technical solution concept of the present application, and these all belong to the protection scope of the present application.

Claims

1. A forklift collision warning system that uses three-dimensional point cloud information to correct two-dimensional image coordinates, characterized in that: It includes video data acquisition module, 3D point cloud data acquisition module, image target detection module, 3D point cloud target detection module, correction model training module, coordinate correction module, target tracking module, target distance calculation module and alarm module; The video data acquisition module is used to collect video stream data of the forklift working area; The three-dimensional point cloud data acquisition module is used to collect point cloud data of the forklift working area; The image target detection module is used to detect targets in video stream images; The three-dimensional point cloud target detection module is used to detect targets in point cloud data; The correction model training module is used to train a mapping model between image target coordinates and point cloud target coordinates, and to regularly update the model in the coordinate correction module; The coordinate correction module is used to perform regression calculation on the image target coordinates to obtain corrected coordinates; The target tracking module is used to uniquely number each detected target and continuously track the target in subsequent video frames to determine the direction of each target; The target distance calculation module calculates the distance between the forklift and surrounding objects in the forward direction according to the corrected coordinates; The alarm module triggers an alarm when the distance between the forklift and surrounding objects is less than a set threshold, and sends a notification to the workshop management personnel at the same time.

2. A forklift collision warning method using three-dimensional point cloud information to correct two-dimensional image coordinates, characterized in that: include: Step 1: Collect the video stream of the forklift working area in real time and automatically detect the target in the video; Step 2: If both a person and a forklift are detected at the same time, the detected two-dimensional coordinates of the target are transmitted to the coordinate correction module; If no forklift is detected, return to step 1 and continue the cycle monitoring; If a person and a forklift are detected simultaneously in k consecutive frames, the acquisition of 1 frame of 3D point cloud data of the forklift working area is triggered, and the target in the point cloud is automatically detected to determine whether the point cloud target matches the target in the k-th frame image; if they match, the 2D coordinates, image, and 3D coordinates of the target are extracted and passed into the correction model training module; If there is no match, point cloud data collection waits for the next trigger; Step 3: Train the mapping model between the image target coordinates and the point cloud target coordinates through a deep learning algorithm, and regularly update the model in the coordinate correction module; Step 4: Perform regression calculation on the input two-dimensional coordinates according to the mapping model to obtain the corrected coordinates; Step 5: Uniquely number each detected target and continue to track these targets in subsequent video frames; determine the direction of each target by the position change of the target in consecutive frames; Step 6: Based on the corrected coordinates and the forward direction of each target, calculate the distance between the forklift and the surrounding objects in its forward direction; when the distance is less than the set threshold, the alarm module triggers an alarm.

3. The forklift collision warning method using three-dimensional point cloud information to correct two-dimensional image coordinates according to claim 2, characterized in that: The step 2 also includes: S21, target annotation: extract the target images containing the forklift and the person from the video collected in step 1, annotate these images, generate annotation files containing the target position and category, and construct an image target detection dataset; annotate the forklift and the person for the three-dimensional point cloud data, generate an annotation file containing the target position, direction, and category, and extract the image data collected at the same time to construct a detection dataset containing the image and three-dimensional point cloud targets; S22. Model training: The image target detection part uses the yolov8n algorithm to train the image target detection dataset; the 3D point cloud target detection part uses MVX-Net to train the 3D point cloud dataset, and the image feature extraction part uses ResNet50.

4. The forklift collision warning method using three-dimensional point cloud information to correct two-dimensional image coordinates according to claim 2, characterized in that: The image target coordinates include the center point coordinates (Cx, Cy), width (W) and height (H) of the target detection frame, expressed as (Cx, Cy, W, H).

5. The forklift collision warning method using three-dimensional point cloud information to correct two-dimensional image coordinates according to claim 2, characterized in that: The point cloud target coordinates include the center point coordinates of the target three-dimensional detection frame (Cx point , Cy point , Cz point ), length (L point )、Width(W point ) and height (H point ), expressed as (Cx point , Cy point , Cz point , L point , W point , H point ).

6. The forklift collision warning method using three-dimensional point cloud information to correct two-dimensional image coordinates according to claim 2, characterized in that: The step three comprises: S31, use ResNet50 to extract features from the input image (3×640×640), and output feature vectors of four scales, with shapes of (64×320×320), (256×160×160), (512×80×80), and (1024×40×40); S32, the incoming target two-dimensional coordinates (Cx i-n , Cy i-n , W i-n , H i-n ) are concatenated to obtain an input vector shape of (n×4), where i represents the i-th target, n represents the number of detected targets, and 4 corresponds to the four coordinates of the target; S33, the coordinate input vector is processed by multi-layer perceptron (MLP) in four dimensions, and the shapes of four feature vectors are obtained respectively: (n×320×320), (n×160×160), (n×80×80), (n×40×40); then they are concatenated with the features of the four scales of the image respectively to obtain the fused feature vectors ((n+1)×320×320), ((n+1)×160×160), ((n+1)×80×80), ((n+1)×40×40); S34, upsample all four feature vectors to (320×320) and perform feature fusion, predict the corrected two-dimensional coordinates through convolution and full connection operations, and the shape of the final output result is (n×4); S35, remove the height information in the input three-dimensional coordinates, and only retain the horizontal and vertical coordinates of the center point (Cx point , Cy point ) and the length of the 3D detection frame (L point )、Width(W point ); Finally, (Cx point-i， Cy point-i， L point-i， W point-i ) as the prediction label for model training; i represents the i-th target, and the loss function uses Smooth L1 loss; the calculation formula of Smooth L1 is: in, It represents the difference between the true value and the predicted value. When the difference is small, the loss is expressed as the mean square error, and when the difference is large, it is expressed as the absolute error. Represents the horizontal and vertical coordinates of the three-dimensional coordinate center point of the i-th target (Cx point ,Cy point ) and the length of the 3D detection frame (L point )、Width(W point ) is the true value of , Represents the horizontal and vertical coordinates of the three-dimensional coordinate center point of the i-th target (Cx point , Cy point ) and the length of the 3D detection frame (L point )、Width(W point ) is the predicted value of .

7. The forklift collision warning method using three-dimensional point cloud information to correct two-dimensional image coordinates according to claim 2, characterized in that: The step three also includes: when the accumulated images, two-dimensional coordinates, three-dimensional coordinates and other data input into the correction model training module reaches a week's collection amount, the mapping model training between the image target coordinates and the three-dimensional point cloud target coordinates is started, and the training method is to fine-tune the existing model to ensure that the mapping model is updated once a week.

8. The forklift collision warning method using three-dimensional point cloud information to correct two-dimensional image coordinates according to claim 2, characterized in that: In step 6, the distance between the forklift and the surrounding objects The calculation formula is: in represents the distance between the i-th target and the j-th target, ( , ) represents the coordinates of the center point of the ith target after correction, ( , ) represents the coordinates of the corrected j-th target center point.

Images