A forklift-oriented target object recognition and distance detection early warning method and system

Abstract

The application discloses a forklift-oriented target object identification and distance detection early warning method and system, comprising the following steps: step 1, constructing a multi-element target object data set; step 2, training and optimizing a multi-stage target object depth identification model; step 3, video frame acquisition and anti-light interference preprocessing; step 4, monocular metric depth estimation and time domain smoothing; step 5, multi-modal obstacle detection and deduplication; step 6, dynamic attitude solution and double-branch adaptive fusion ranging; and step 7, target object three-dimensional space positioning and dynamic risk assessment early warning. Based on the monocular metric depth estimation technology, the forklift pitching bias is compensated in real time through RANSAC ground fitting, the depth estimation precision is improved through the ODEMA, ASG-DCN and DySample serial feature enhancement architecture, the ranging robustness is improved through the physical priori and dense network double-branch ranging architecture, and the collision risk assessment logic is constructed based on the target object three-dimensional position and azimuth angle to realize hierarchical early warning.

Description

Technical Field

[0001] This invention relates to the field of autonomous navigation technology for intelligent forklifts, specifically to a method and system for target object recognition and distance detection and early warning for forklifts. Background Technology

[0002] With the deepening of the "Industry 4.0" and smart logistics concepts, the warehousing and logistics industry is undergoing a profound transformation from traditional manual handling to intelligent and unmanned operations. As a core transport device in modern smart factories and automated warehouses, the efficiency and safety of intelligent forklifts directly determine the overall effectiveness of the logistics system. Environmental perception is the cornerstone of the "intelligence" of intelligent forklifts during autonomous navigation and operation. Compared to expensive LiDAR sensor solutions, computer vision-based perception solutions are gradually becoming an important development direction for intelligent forklift perception systems due to their advantages of low hardware cost, high information density, and convenient installation. Especially in warehousing environments with confined spaces and frequent dynamic objects, how to utilize machine vision technology to accurately reconstruct the 3D of the work scene and measure distances to achieve precise positioning of shelves, goods, and personnel is a key link in ensuring the safe and efficient operation of intelligent forklifts.

[0003] In the field of computer vision, monocular depth estimation is a key technology aimed at inferring pixel-level depth information of a scene from a single 2D RGB image. Unlike binocular stereo matching, which relies on the principle of parallax, monocular depth estimation mainly relies on three types of monocular cues in the image: texture gradients, perspective relationships, and object occlusion. Combined with the powerful feature extraction capabilities of deep neural networks, it reconstructs the 3D structure of the scene. If this technology can be effectively applied to intelligent forklift systems, it will significantly reduce the hardware cost of navigation and obstacle avoidance modules and provide forklifts with richer environmental semantics and geometric information.

[0004] However, while existing monocular depth estimation techniques have made progress in outdoor structured road scenarios for autonomous driving (such as the KITTI dataset), they still have significant limitations and shortcomings in the complex indoor industrial scenarios where intelligent forklifts operate. First, warehouse environments commonly contain numerous low-texture and textureless areas (such as flat cement floors, solid-colored walls, and metal shelf surfaces). This causes self-supervised learning signals based on photometric errors to fail in these areas, making it difficult to extract effective geometric features and resulting in blurred depth estimation results or artifacts. Second, factory lighting conditions are complex, often exhibiting strong reflections (from metal surfaces) or localized darkness, violating the "photometric invariance" assumption in self-supervised learning and severely impacting the robustness of depth inference. Furthermore, most current mainstream algorithms are trained on outdoor datasets, resulting in limited generalization ability. When directly transferred to forklift scenarios with lower viewpoints and unique motion patterns, scale drift or significant distance estimation biases often occur, failing to meet the high-precision depth information requirements of intelligent forklifts during precise operations.

[0005] Therefore, developing a monocular depth estimation method for intelligent forklifts that adapts to strong interference and low-texture industrial environments has significant application value. Summary of the Invention

[0006] The purpose of this invention is to provide a target object identification and distance detection early warning method and system that is low in cost, high in accuracy, and highly adaptable to various scenarios, so as to solve the problems of high hardware cost, large depth estimation deviation, and inaccurate security control in the prior art.

[0007] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0008] A target object recognition and distance detection early warning method includes the following steps:

[0009] Step 1: Construct a multi-target object dataset by collecting data from multiple locations in the warehousing scenario;

[0010] Step 2: Construct a multi-stage target object depth recognition model and train and optimize it;

[0011] Step 3: Acquire video stream in real time using an onboard RGB camera and perform anti-light interference preprocessing;

[0012] Step 4: Perform monocular depth estimation on a single frame image and perform temporal smoothing on consecutive frame images.

[0013] Step 5: Perform multimodal obstacle detection and deduplication on the depth map;

[0014] Step 6: Dynamic attitude calculation and dual-branch adaptive fusion ranging;

[0015] Step 7: 3D spatial positioning of the target object and dynamic risk assessment and early warning.

[0016] Furthermore, the specific steps of step 1 are as follows:

[0017] Step 1.1: Deploy multiple data collection points in the warehouse scenario to collect data, covering different shelf layouts, light intensities, and work postures; use a pre-trained depth measurement model to perform offline inference on the original RGB images to generate an absolute depth map as a pseudo-depth label; limit the depth values ​​to the effective range to remove outliers, and perform median filtering to preserve edge contours;

[0018] Step 1.2: Simultaneously downsample the RGB image and depth pseudo-label to the training target resolution; eliminate unqualified samples by combining grayscale brightness, HSV color features, depth local standard deviation and Laplacian sharpness index; calculate perceptual hash of the filtered images to remove duplicate content, divide the training set and validation set proportionally, and store the depth label in a predetermined format.

[0019] Furthermore, the specific steps of step 2 are as follows:

[0020] Step 2.1: Construct a multi-stage recognition model that includes a target detection module and a target depth calculation module. The target depth calculation module is built based on the DepthAnything model, and the encoder uses DINOv2 to extract multi-scale features.

[0021] Step 2.2: Embed the ODEMA module in the feature fusion stage of the DepthAnything model, and divide the input features into G independent sub-feature groups according to the number of channels, where G is determined by the number of input feature channels C. in Compared with the preset benchmark grouping factor g base The calculation formula is determined together as follows:

[0022]

[0023] Where C in g represents the number of input feature channels. base G represents the preset baseline grouping factor. max Indicates the maximum number of groups;

[0024] For each sub-feature group, adaptive average pooling is performed in both the height and width directions to obtain row and column feature vectors. These vectors are then fused through convolution to generate a row and column attention map. After sigmoid activation, the map is multiplied by the original sub-feature group and normalized to output the enhanced feature. Simultaneously, local detail features are extracted through 3×3 convolution, and association weights are calculated with the channel attention branches to generate the global attention weight W. att The final enhanced feature is output using a zero-initialization damped residual mechanism, as shown in the following formula:

[0025]

[0026] Where X out X represents the output feature. in W represents the input features. att represents the global attention weight, γ represents the learnable scaling factor, and ⊙ represents pixel-wise multiplication;

[0027] Step 2.3: Embed the ASG-DCN module into the feature extraction backbone network, using the structural confidence map S output by ODEMA as the guiding signal, and adaptively constrain the spatial offset of DCNv4 using the structural confidence. The calculation formula for the structural confidence map S is as follows:

[0028]

[0029] Where σ is the Sigmoid activation function, δ is the ReLU activation function, and F EMA This indicates the enhanced characteristics of the ODEMA output;

[0030] Step 2.4: Apply an exponential adaptive soft clamp function to constrain S, as shown in the following formula:

[0031]

[0032] Where ε is the safe minimum value to prevent gradient explosion, and λ is the dynamic clamping strength parameter that the network can learn.

[0033] The final action offset and final mask are calculated using a hybrid deformation strategy based on the momentum factor, as shown in the following formula:

[0034]

[0035] Where Δp represents the final applied offset, and m represents the final mask, Δp raw This represents the original free offset predicted by DCNv4, m raw Let μ represent the original mask for DCNv4 prediction, μ represent the learnable hybrid momentum factor, η represent the guided enhancement coefficient, and S represent the original mask for DCNv4 prediction. clamped This represents the structure confidence map after clamping, and ⊙ indicates pixel-by-pixel multiplication.

[0036] Step 2.5: In the feature upsampling stage, the DySample module is embedded to replace the traditional bilinear interpolation. The sampling scale factor s, the number of groups k and the sampling style are preset. A sampling offset prediction branch is generated through 1×1 convolution. An initial position matrix is ​​introduced. An initial offset with uniform distribution is generated based on the sampling scale. The initial position is corrected by learning the offset. A dynamic sampling grid is constructed based on the generated offset. The input features are sampled in a grid sampling manner to obtain the scale-up feature map.

[0037] Step 2.6: Use absolute depth pseudo-labels as supervision signals to enable the model to learn the physical scale. After training, export the model to ONNX format and simplify it. Generate an optimization engine based on the TensorRT toolkit and deploy it on edge hardware.

[0038] Further, step 3 specifically involves: fixing the RGB camera to the front of the forklift, transmitting the video stream to the edge device in real time, and using a configurable frame interval strategy to reduce the computational load; for strong light and backlight scenes, adaptively enabling CLAHE histogram equalization, converting the RGB image to the LAB color space, and enhancing the brightness channel.

[0039] Furthermore, step 4 consists of the following specific steps:

[0040] Step 4.1: Input the preprocessed single-frame image from Step 3 into the depth calculation engine, such as... Figure 4 As shown, the output is a metric depth matrix aligned with the input image space;

[0041] Step 4.2: Perform EMA exponential moving average smoothing on the depth matrix of consecutive frames pixel by pixel.

[0042] Furthermore, step 5 consists of the following specific steps:

[0043] Step 5.1: Calculate the Sobel gradient of the depth map, identify locations where the gradient magnitude exceeds a preset edge threshold as boundaries of different objects, and remove pixels at these locations from the candidate obstacle mask. The judgment condition is: ,in T represents the gradient magnitude of the depth map at pixel (x,y). edge Indicates the preset edge threshold;

[0044] Step 5.2: Perform black border compensation and remove the segmentation mask corresponding to the filled area of ​​the depth map letterbox; calculate the IoU (Intersection over Union) of the pure depth detection box and the target detection box; for pixels with inconsistent semantic labels in the overlapping area, split them according to the overall median of the depth; output the obstacle bounding box and mask; the IoU deduplication judgment condition is: IoU>τ, where τ represents the IoU deduplication threshold.

[0045] Furthermore, step 6 consists of the following specific steps:

[0046] Step 6.1: The vehicle-mounted RGB camera is calibrated using a checkerboard calibration board to obtain the camera's intrinsic parameters and static installation pitch angle. Pixels in the ground area at the bottom of the image are extracted, and the camera's intrinsic parameters are back-projected into a 3D point set using the initial depth value. The RANSAC algorithm is used to fit the ground plane equation, and the angle between the current frame's real ground plane normal vector and the static calibration normal vector is solved to obtain the dynamic pitch offset. The static calibration pitch angle is then compensated in real time.

[0047] Step 6.2: Construct a dual-branch architecture of physical prior ranging and dense network ranging. The physical prior ranging branch is based on the target object's bounding box pixel height h, camera focal length f, and known true object height H. obj The real-time pitch angle θ obtained in step 6.1 real Calculate the distance D of the target object. branch1 The calculation formula is as follows:

[0048]

[0049] Where h represents the pixel height of the target object's bounding box, f represents the camera's focal length, and H obj Indicates the true height of a known object;

[0050] Step 6.3: The dense network ranging branch inputs the current frame into the pre-trained metric depth network to regress the dense depth map, and takes the median depth of the target area as D. branch2 ; Calculate the gradient variance Var(D) of the depth map; Perform adaptive confidence-weighted fusion of the two branches using Var(D), and when Var(D) is lower than a preset threshold T... var When the confidence level of the physical prior branch is high, Var(D) is higher than T. var The confidence level of dense network branches is relatively high.

[0051] Furthermore, step 7 is detailed below:

[0052] Step 7.1: Based on the camera parameters and the absolute depth data of the target object, convert the pixel coordinates of the target object into three-dimensional coordinates in the camera coordinate system;

[0053] Step 7.2: Based on the camera extrinsic parameters, convert the 3D coordinates in the camera coordinate system to the 3D spatial position coordinates in the forklift operation world coordinate system;

[0054] Step 7.3: Based on the target object's pixel coordinates and camera intrinsic parameters, calculate the horizontal azimuth angle of the target object relative to the forklift's direction of travel;

[0055] Step 7.4: Compare the extracted key feature parameters with the preset safety thresholds to construct the collision risk assessment logic.

[0056] Further, step 7.4 is specifically as follows: if d(t)>d1, or d(t)≤d1 but |α(t)|>α0, it is determined as a safe state; if d2<d(t)≤d1 and |α(t)|≤α0, it is determined as a first-level warning; if d(t)≤d2 and |α(t)|≤α0, it is determined as a second-level warning; where d(t) represents the relative straight-line distance between the target object and the forklift at time t, d1 represents the first-level warning distance threshold, d2 represents the second-level warning distance threshold, and α0 represents the safe azimuth threshold.

[0057] The present invention further provides a target object recognition, distance detection and warning system, comprising a data acquisition module, a model training and optimization module, an edge deployment and inference module, a target object recognition and positioning module, a risk assessment and warning module, and a camera calibration module;

[0058] Data acquisition module: configured to collect multi-element target object datasets and real-time operation scene images, images are collected by an ordinary RGB camera, and depth labels of the datasets are generated offline by a pre-trained metric depth model;

[0059] Model training and optimization module: configured to train and optimize a multi-stage target object depth recognition model, and complete model format conversion and hardware adaptation, wherein the model training unit is configured to complete training and optimization of the multi-stage target object depth recognition model, the depth model optimization unit is configured to embed ASG-DCN and ODEMA to enhance feature expression and integrate DySample to improve upsampling accuracy, and the model conversion and deployment unit is configured to complete ONNX format conversion of the model and hardware adaptation of the TensorRT engine;

[0060] Edge deployment and inference module: constructed based on Jetson Orin NX, equipped with a TensorRT toolkit to implement parallel real-time inference;

[0061] Target object recognition and positioning module: deployed in the edge deployment and inference module, configured to complete target object detection, absolute depth recovery and three-dimensional spatial position positioning through model inference results, and output three-dimensional coordinates of the target object;

[0062] Risk assessment and warning module: in communication with the edge deployment and inference module and the forklift control system, configured to calculate the collision risk coefficient in real time, output graded warning signals and avoidance path reference instructions, and manage and control the driving state of the forklift;

[0063] Camera calibration module: providing Zhang Zhengyou's calibration tool and calibration process, configured to acquire internal and external parameters of the ordinary camera to provide data support for coordinate conversion.

[0064] By adopting the above technical solution, compared with the prior art, the present invention has the following beneficial effects:

[0065] This invention is based on monocular depth estimation technology. It compensates for forklift pitch bias in real time through RANSAC ground fitting, improves depth estimation accuracy through ODEMA, ASG-DCN and DySample serial feature enhancement architecture, enhances ranging robustness through physical prior and dense network dual-branch ranging architecture, and realizes graded early warning by constructing collision risk assessment logic based on the target object's three-dimensional position and azimuth angle. It only requires a regular RGB camera, which greatly reduces hardware costs. Attached Figure Description

[0066] Figure 1 This is a flowchart of the method of the present invention;

[0067] Figure 2 This is a schematic diagram of the network structure of the target object depth recognition model in the method of the present invention;

[0068] Figure 3 This is the original RGB image of the warehouse scene captured by the camera in this embodiment of the invention;

[0069] Figure 4 This is a depth measurement map output by model inference in an embodiment of the present invention;

[0070] Figure 5 This is a schematic diagram of camera calibration in an embodiment of the present invention;

[0071] Figure 6 This is a schematic diagram illustrating the effects of target detection and depth estimation in an embodiment of the present invention. Detailed Implementation

[0072] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0073] The overall flow of the target object identification and distance detection early warning method of the present invention is as follows: Figure 1 As shown, the specific steps include the following:

[0074] Step 1: Construct a multi-object dataset. The specific steps are as follows:

[0075] Step 1.1: Deploy multiple collection points in the warehouse scenario to collect data, covering different shelf layouts, lighting intensities, and work postures. The samples include shelves, goods, pedestrians, work tools, walls, and columns. Use a pre-trained depth measurement model to perform offline inference on the original RGB images to generate an absolute depth map in meters as a pseudo-depth label. Limit the depth values ​​to the valid range to remove outliers and perform median filtering to preserve edge contours.

[0076] Step 1.2: Simultaneously downsample the RGB image and depth pseudo-label to the training target resolution; for issues such as overexposure, reflection, depth jumps, and low texture, eliminate unqualified samples by comprehensively considering grayscale brightness, HSV color features, local depth standard deviation, and Laplacian sharpness index; calculate perceptual hashes to remove duplicate content from the filtered images, and divide the training set and validation set proportionally, storing the depth labels in FP16 format; in this embodiment, the ratio of the training set to the validation set is 7:3;

[0077] Step 2: Train and optimize the multi-stage target depth recognition model. The specific steps are as follows:

[0078] Step 2.1: Construct a multi-stage recognition model that includes a target detection module and a target depth calculation module, such as... Figure 2 As shown, the object depth calculation module is built based on the DepthAnything model, and the encoder uses DINOv2 to extract multi-scale features; the object detection module uses the YOLOv8 detection model, with the model input being the original RGB image from a monocular camera; the model output is the bounding box ROI region.

[0079] Step 2.2: Embed the ODEMA module in the feature fusion stage of the DepthAnything model, and divide the input features into G independent sub-feature groups according to the number of channels, where G is determined by the number of input feature channels C. in Compared with the preset benchmark grouping factor g base The calculation formula is determined by common agreement and is shown in equation (1):

[0080]

[0081] Where C in G represents the number of input feature channels. base G represents the preset baseline grouping factor. max Indicates the maximum number of groups; in this embodiment, g base =32, G max =8, C in =256;

[0082] For each sub-feature group, adaptive average pooling is performed in both the height and width directions to obtain row and column feature vectors. These vectors are then fused through convolution to generate a row and column attention map. After sigmoid activation, the map is multiplied by the original sub-feature group and normalized to output the enhanced feature. Simultaneously, local detail features are extracted through 3×3 convolution, and association weights are calculated with the channel attention branches to generate the global attention weight W. att The final enhanced feature is output using a zero-initialization damping residual mechanism, and the fusion formula is shown in equation (2):

[0083]

[0084] Where X out X represents the output feature. in W represents the input features. att The symbol represents the global attention weight, γ represents the learnable scaling factor, and ⊙ represents pixel-wise multiplication; in this embodiment, γ=10. -6 ;

[0085] Step 2.3: Embed the ASG-DCN module in the feature extraction backbone network, use the structural confidence map S output by ODEMA as the guiding signal, and use the structural confidence to adaptively constrain the spatial offset of DCNv4. The calculation formula of the structural confidence map S is shown in Equation (3):

[0086]

[0087] Where σ is the Sigmoid activation function, δ is the ReLU activation function, and F EMA This indicates the enhanced characteristics of the ODEMA output;

[0088] Step 2.4: Use an exponential adaptive soft clamp function to constrain S, and the calculation formula is shown in equation (4):

[0089]

[0090] Where ε is the safe minimum value to prevent gradient explosion, and λ is the dynamic clamping strength parameter that the network can learn; in this embodiment, ε=10 -6 λ=0.5; The final action offset and final mask are calculated based on the hybrid deformation strategy of momentum factor, and the calculation formula is shown in Equation (5):

[0091]

[0092] Where Δp represents the final applied offset, and m represents the final mask, Δp raw This represents the original free offset predicted by DCNv4, m raw Let μ represent the original mask for DCNv4 prediction, μ represent the learnable hybrid momentum factor, η represent the guided enhancement coefficient, and S represent the original mask for DCNv4 prediction. clamped The diagram represents the confidence level of the clamped structure, and ⊙ represents pixel-by-pixel multiplication; in this embodiment, η=1.5;

[0093] Step 2.5: In the feature upsampling stage, the DySample module is embedded to replace the traditional bilinear interpolation. The sampling scale factor s, the number of groups k, and the sampling style are preset. A sampling offset prediction branch is generated through 1×1 convolution. An initial position matrix is ​​introduced, and a uniformly distributed initial offset is generated based on the sampling scale. The initial position is corrected by learning the offset. A dynamic sampling grid is constructed based on the generated offset. The input features are sampled in a grid sampling manner to obtain the scale-up feature map. In this embodiment, s=2, k=4.

[0094] Step 2.6: Use absolute depth pseudo-labels as supervision signals to enable the model to learn the physical scale. After training, export the model to ONNX format and simplify it. Generate an optimization engine based on the TensorRT toolkit and deploy it on edge hardware.

[0095] Step 3, video frame acquisition and anti-light interference preprocessing, the specific steps are as follows:

[0096] A standard RGB camera is mounted on the front of a forklift, and the video stream is transmitted in real time to an edge device. A single frame of RGB image captured by the camera in a warehouse operation scenario is shown below. Figure 3 As shown, a configurable frame interval strategy is adopted to reduce the computational load; for warehouse strong light and backlight scenes, adaptive CLAHE histogram equalization is enabled to convert RGB images to LAB color space and enhance the brightness channel.

[0097] Step 4, monocular metric depth estimation and temporal smoothing, the specific steps are as follows:

[0098] Step 4.1: Input the preprocessed single-frame image from Step 3 into the depth calculation engine, which outputs a metric depth matrix aligned with the input image space. Then, perform pseudo-color mapping on the metric depth matrix to obtain the following result: Figure 4 The depth map shown;

[0099] Step 4.2: Perform EMA exponential moving average smoothing on the depth matrix of consecutive frames pixel by pixel, and the calculation formula is shown in Equation (6):

[0100]

[0101] Where D smooth D(t) represents the depth matrix after EMA smoothing at time t, D(t) represents the original depth matrix at time t, and β represents the smoothing coefficient; in this embodiment, β=0.85.

[0102] Step 5, multimodal obstacle detection and deduplication, the specific steps are as follows:

[0103] Step 5.1: Calculate the Sobel gradient of the depth map, identify locations where the gradient magnitude exceeds a preset edge threshold as boundaries of different objects, and remove pixels at these locations from the candidate obstacle mask. The judgment condition is: ,in T represents the gradient magnitude of the depth map at pixel (x,y). edge In this embodiment, T represents a preset edge threshold. edge =0.03;

[0104] Step 5.2: Perform black border compensation and remove the segmentation mask corresponding to the filled area of ​​the depth map letterbox to prevent false detection; calculate the IoU (Intersection over Union) of the pure depth detection box and the target detection box; for pixels with inconsistent semantic labels in the overlapping area, split them according to the overall depth median, and output the obstacle bounding box and mask; the IoU deduplication criterion is: IoU > τ, where τ IoU This represents the IoU deduplication threshold; in this embodiment, τ = 0.5.

[0105] Step 6, Dynamic Attitude Calculation and Dual-Branch Adaptive Fusion Ranging, the specific steps are as follows:

[0106] Step 6.1: Calibrate the vehicle-mounted RGB camera using a checkerboard calibration board, such as... Figure 5 As shown, the camera intrinsic parameters and static installation pitch angle are obtained, the pixels of the ground area at the bottom of the image are extracted, and the camera intrinsic parameters and the initial depth value are back-projected into a three-dimensional point set. The RANSAC algorithm is used to fit the ground plane equation, and the angle between the current frame's real ground plane normal vector and the static calibration normal vector is solved to obtain the dynamic pitch offset. The static calibration pitch angle is compensated in real time. The calculation formula is shown in Equation (7):

[0107]

[0108] Where θ real Represents the real-time physical pitch angle, θ static Δθ represents the static calibration pitch angle, and Δθ represents the dynamic pitch offset; in this embodiment, θ static =5°;

[0109] Step 6.2: Construct a dual-branch architecture of physical prior ranging and dense network ranging. The physical prior ranging branch is based on the target object's bounding box pixel height h, camera focal length f, and known true object height H. obj The real-time pitch angle θ obtained in step 6.1 real Calculate the distance D of the target object. branch1 The calculation formula is shown in equation (8):

[0110]

[0111] Where h represents the pixel height of the target object's bounding box, f represents the camera's focal length, and H obj Indicates the true height of a known object;

[0112] Step 6.3: The dense network ranging branch inputs the current frame into the pre-trained metric depth network to regress the dense depth map, and takes the median depth of the target area as D. branch2 ; Calculate the gradient variance Var(D) of the depth map; Perform adaptive confidence-weighted fusion of the two branches using Var(D), and when Var(D) is lower than a preset threshold T... var When the confidence level of the physical prior branch is high, Var(D) is higher than T. var The confidence level of branches in time-dense networks is relatively high, and the calculation formula is shown in equation (9):

[0113]

[0114] Where D final D represents the final fusion distance. branch1 D represents the output distance of the physical prior distance measurement branch. branch2 The distance represents the output distance of the ranging branch in the dense network, and w1 and w2 represent the confidence weights of the two branches, respectively; in this embodiment, a preset threshold T is used. var =0.15, when Var(D)≤0.15, w1=0.8, w2=0.2, when Var(D)>0.15, w1=0.3, w2=0.7;

[0115] Step 7, 3D spatial positioning of the target object and dynamic risk assessment and early warning, the specific steps are as follows:

[0116] Step 7.1: Based on the camera parameters and the absolute depth data of the target object, convert the pixel coordinates of the target object into three-dimensional coordinates in the camera coordinate system. The calculation formula is shown in Equation (10):

[0117]

[0118] Where (u,v) represents the pixel coordinates of the target object in the image, (X... c ,Y c Z c ) represents the three-dimensional coordinates in the camera coordinate system, c x c y f represents the principal point coordinates of the camera. x f y D represents the camera's focal length. abs (u,v) represents the absolute depth value corresponding to pixel (u,v); in this embodiment, f x =f y =800 pixels, principal point coordinates are at the image center;

[0119] Step 7.2: Convert the 3D coordinates in the camera coordinate system to the 3D spatial position coordinates in the forklift operation world coordinate system according to the camera extrinsic parameters, and the calculation formula is as shown in Equation (11):

[0120]

[0121] Where (X w , Y w , Z w ) represents the 3D spatial position coordinates in the world coordinate system, R represents the rotation matrix, H c represents the camera installation height; in this embodiment, H c =2.0 meters, θ static =5°;

[0122] Step 7.3: Based on the pixel coordinates of the target object and the camera intrinsic parameters, calculate the horizontal azimuth angle of the target object relative to the traveling direction of the forklift, and the calculation formula is as shown in Equation (12):

[0123]

[0124] Where α(t) represents the horizontal azimuth angle of the target object at time t relative to the traveling direction of the forklift, u(t) represents the horizontal pixel coordinate of the center of the bounding box of the target object at time t, c x represents the x-coordinate of the camera principal point, f x represents the focal length of the camera in the x direction;

[0125] Step 7.4: Compare the extracted key characteristic parameters with the preset safety threshold to construct a collision risk assessment logic: if d(t)>d1, or d(t)≤d1 but |α(t)|>α0, it is determined as a safe state; if d2<d(t)≤d1 and |α(t)|≤α0, it is determined as a first-level warning; if d(t)≤d2 and |α(t)|≤α0, it is determined as a second-level warning, and the final output result is as Figure 6 shown; where d(t) represents the relative straight-line distance between the target object and the forklift at time t, d1 represents the first-level warning distance threshold, d2 represents the second-level warning distance threshold, and α0 represents the safety azimuth threshold; in this embodiment, d1=3 meters, d2=1.5 meters, α0=15°;

[0126] Taking the public dataset KITTI as an example, the comparison table between the current method and the methods in the field of monocular depth estimation is shown in Table 1:

[0127] Table 1 - Model Performance Comparison Table

[0128] Quantitative comparative tests were conducted on this invention and three classic monocular depth estimation algorithms—Monodepth2, DPT, and Zoedepth—on the KITTI public dataset. The evaluation results show that the absolute relative error (AbsRel) of this invention is 0.046, and the root mean square error (RMSE) is 1.955, both the lowest values ​​among the four methods, representing the percentage of effectively predicted pixels. The index reached 0.982, the highest among all comparison schemes. Compared with Zoedepth, which has the best overall performance, the error indexes of this invention are further reduced, the effective depth prediction pixel ratio is improved, and the advantages over Monodepth2 and DPT algorithms are more prominent. This fully verifies that the improved strategy of this invention through backbone network replacement, feature enhancement, adaptive deformation convolution constraint and dynamic upsampling optimization can effectively reduce the global bias and local relative error of monocular depth prediction. The overall accuracy and reliability of depth estimation are better than the existing mainstream monocular depth estimation methods, and can provide more accurate depth data support for subsequent target 3D coordinate calculation and safety distance warning.

[0129] Corresponding to the above method, the present invention also discloses a target object recognition and distance detection early warning system, comprising:

[0130] Data acquisition module: used to acquire multi-object datasets and real-time operation scene images. It uses a regular RGB camera for image acquisition, and the dataset depth labels are generated offline by a pre-trained depth measurement model.

[0131] Model training and optimization module: used to train and optimize multi-stage target object depth recognition model, complete model format conversion and hardware adaptation, and the deep model optimization unit is specifically used to embed ASG-DCN and ODEMA to enhance feature expression, and integrate DySample to improve upsampling accuracy;

[0132] Edge deployment and inference module: Built on Jetson Orin NX and equipped with the TensorRT toolkit, enabling parallel real-time inference;

[0133] Target object recognition and localization module: Deployed at the edge of the deployment and inference module, it is used to complete target object detection, absolute depth recovery and three-dimensional spatial location through model inference results, and output the three-dimensional coordinates of the target object;

[0134] Risk assessment and early warning module: Communicates with edge deployment and inference module and forklift control system to calculate collision risk coefficient in real time, output graded early warning signals and avoidance path reference instructions, and manage forklift driving status;

[0135] Camera calibration module: Provides Zhang Zhengyou's calibration tools and calibration process to obtain the intrinsic and extrinsic parameters of ordinary cameras and provide data support for coordinate transformation;

[0136] The examples described in this specification are merely enumerations of implementation forms of the inventive concept. The scope of protection of this invention should not be considered as limited to the specific forms stated in the embodiments. The scope of protection of this invention also extends to equivalent technical means that can be conceived by those skilled in the art based on the inventive concept.

Claims

1. A method for target object recognition and distance detection and early warning for forklifts, characterized in that, Includes the following steps: Step 1: Construct a multi-target object dataset by collecting data from multiple locations in the warehousing scenario; Step 2: Construct a multi-stage target object depth recognition model and train and optimize it; Step 3: Acquire video stream in real time using an onboard RGB camera and perform anti-light interference preprocessing; Step 4: Perform monocular depth estimation on the single-frame image after preprocessing in Step 3, and perform temporal smoothing on the continuous frame images. Step 5: Perform multimodal obstacle detection and deduplication on the depth map obtained in Step 4; Step 6: Perform dynamic attitude calculation and dual-branch adaptive fusion ranging on the depth map processed in Step 5. Step 7: Based on the calculation results of Step 6, perform three-dimensional spatial positioning and dynamic risk assessment and early warning of the target object.

2. The target object recognition and distance detection early warning method for forklifts according to claim 1, characterized in that, Step 1: The specific steps are as follows: Step 1.1: Deploy multiple data collection points in the warehouse scenario to collect data, covering different shelf layouts, light intensities, and work postures; use a pre-trained depth measurement model to perform offline inference on the original RGB images to generate an absolute depth map as a pseudo-depth label; Outliers are eliminated by limiting the depth values ​​to the valid range, and median filtering is performed to preserve the edge contours. Step 1.2: Simultaneously downsample the RGB image and depth pseudo-label to the training target resolution; eliminate unqualified samples by combining grayscale brightness, HSV color features, depth local standard deviation and Laplacian sharpness index; calculate perceptual hash of the filtered images to remove duplicate content, divide the training set and validation set proportionally, and store the depth label in a predetermined format.

3. The target object recognition and distance detection early warning method for forklifts according to claim 1, characterized in that, Step 2 is detailed below: Step 2.1: Construct a multi-stage recognition model that includes a target detection module and a target depth calculation module. The target depth calculation module is built based on the DepthAnything model, and the encoder uses DINOv2 to extract multi-scale features. Step 2.2: Embed the ODEMA module in the feature fusion stage of the DepthAnything model, and divide the input features into G independent sub-feature groups according to the number of channels, where G is determined by the number of input feature channels C. in Compared with the preset benchmark grouping factor g base The calculation formula is determined together as follows: ； Where C in G represents the number of input feature channels. base G represents the preset baseline grouping factor. max Indicates the maximum number of groups; For each sub-feature group, adaptive average pooling is performed in the height and width directions to obtain row and column feature vectors. These vectors are then fused by convolution to generate row and column attention maps. After Sigmoid activation, these maps are multiplied by the original sub-feature groups and normalized to output enhanced features. Simultaneously, local detail features are extracted through 3×3 convolution, and associated weights are calculated with the channel attention branches to generate the global attention weight W. att The final enhanced feature is output using a zero-initialization damped residual mechanism, as shown in the following formula: ； Where X out X represents the output feature. in W represents the input features. att represents the global attention weight, γ represents the learnable scaling factor, and ⊙ represents pixel-wise multiplication; Step 2.3: Embed the ASG-DCN module into the feature extraction backbone network, using the structural confidence map S output by ODEMA as the guiding signal, and adaptively constrain the spatial offset of DCNv4 using the structural confidence. The calculation formula for the structural confidence map S is as follows: ； Where σ is the Sigmoid activation function, δ is the ReLU activation function, and F EMA This indicates the enhanced characteristics of the ODEMA output; Step 2.4: Apply an exponential adaptive soft clamp function to constrain S, as shown in the following formula: ； Where ε is the safe minimum value to prevent gradient explosion, and λ is the dynamic clamping strength parameter that the network can learn. The final action offset and final mask are calculated using a hybrid deformation strategy based on the momentum factor, as shown in the following formula: ； Where Δp represents the final applied offset, m represents the final mask, and Δp raw This represents the original free offset predicted by DCNv4, m raw This represents the original mask for DCNv4 prediction. S represents the learnable hybrid momentum factor, η represents the guided enhancement coefficient, and S clamped This represents the structure confidence map after clamping, and ⊙ indicates pixel-by-pixel multiplication. Step 2.5: In the feature upsampling stage, the DySample module is embedded to replace the traditional bilinear interpolation. The sampling scale factor s, the number of groups k and the sampling style are preset. A sampling offset prediction branch is generated through 1×1 convolution. An initial position matrix is ​​introduced. An initial offset with uniform distribution is generated based on the sampling scale. The initial position is corrected by learning the offset. A dynamic sampling grid is constructed based on the generated offset. The input features are sampled in a grid sampling manner to obtain the scale-up feature map. Step 2.6: Use absolute depth pseudo-labels as supervision signals to enable the model to learn the physical scale. After training, export the model to ONNX format and simplify it. Generate an optimization engine based on the TensorRT toolkit and deploy it on edge hardware.

4. The target object recognition and distance detection early warning method for forklifts according to claim 1, characterized in that, Step 3 specifically involves: fixing the RGB camera to the front of the forklift, transmitting the video stream to the edge device in real time, and using a configurable frame interval strategy to reduce the computational load; for strong light and backlight scenes, adaptively enabling CLAHE histogram equalization, converting the RGB image to the LAB color space, and enhancing the brightness channel.

5. The target object recognition and distance detection early warning method for forklifts according to claim 1, characterized in that, Step 4 is detailed below: Step 4.1: Input the preprocessed single-frame image from Step 3 into the depth calculation engine, and output a metric depth matrix that is spatially aligned with the input image. Step 4.2: Perform EMA exponential moving average smoothing on the depth matrix of consecutive frames pixel by pixel.

6. The target object recognition and distance detection early warning method for forklifts according to claim 1, characterized in that, Step 5 is detailed below: Step 5.1: Calculate the Sobel gradient of the depth map, identify locations where the gradient magnitude exceeds a preset edge threshold as boundaries of different objects, and remove pixels at these locations from the candidate obstacle mask. The judgment condition is: ,in T represents the gradient magnitude of the depth map at pixel (x,y). edge Indicates the preset edge threshold; Step 5.2: Perform black border compensation and clear the segmentation mask corresponding to the filled area of ​​the depth map letterbox; calculate the IoU (Intersection over Union) of the pure depth detection box and the target detection box; for pixels with inconsistent semantic labels in the overlapping area, split them according to the overall median of the depth; and output the obstacle bounding box and mask. The condition for determining duplicate IoU is: IoU > τ, where τ represents the IoU deduplication threshold.

7. The target object recognition and distance detection early warning method for forklifts according to claim 1, characterized in that, Step 6 is detailed below: Step 6.1: Extract the pixels of the ground area at the bottom of the image, combine them with the initial depth values ​​of the camera to back-project into a 3D point set, use the RANSAC algorithm to fit the ground plane equation, solve the angle between the real ground plane normal vector and the static calibration normal vector in the current frame, obtain the dynamic pitch offset, and perform real-time compensation for the static calibration pitch angle. Step 6.2: Construct a dual-branch architecture of physical prior ranging and dense network ranging. The physical prior ranging branch is based on the target object's bounding box pixel height h, camera focal length f, and known true object height H. obj The real-time pitch angle θ obtained in step 6.1 real Calculate the distance D of the target object. branch1 The calculation formula is as follows: ； Where h represents the pixel height of the target object's bounding box, f represents the camera's focal length, and H obj Indicates the true height of a known object; Step 6.3: The dense network ranging branch inputs the current frame into the pre-trained metric depth network to regress the dense depth map, and takes the median depth of the target area as D. branch2 ; Calculate the gradient variance Var(D) of the depth map; Perform adaptive confidence-weighted fusion of the two branches using Var(D), and when Var(D) is lower than a preset threshold T... var When the confidence level of the physical prior branch is high, Var(D) is higher than T. var The confidence level of dense network branches is relatively high.

8. The target object recognition and distance detection early warning method for forklifts according to claim 1, characterized in that, Step 7 is detailed below: Step 7.1: Based on the camera parameters and the absolute depth data of the target object, convert the pixel coordinates of the target object into three-dimensional coordinates in the camera coordinate system; Step 7.2: Based on the camera extrinsic parameters, convert the 3D coordinates in the camera coordinate system to the 3D spatial position coordinates in the forklift operation world coordinate system; Step 7.3: Based on the target object's pixel coordinates and camera intrinsic parameters, calculate the horizontal azimuth angle of the target object relative to the forklift's direction of travel; Step 7.4: Compare the extracted key feature parameters with the preset safety thresholds to construct the collision risk assessment logic.

9. A target object recognition and distance detection early warning method for forklifts according to claim 8, characterized in that, Step 7.4 is specifically as follows: if d(t) > d1, or d(t) ≤ d1 but |α(t)| > α0, it is determined as a safe state; if d2 < d(t) ≤ d1 and |α(t)| ≤ α0, it is determined as a first-level warning; if d(t) ≤ d2 and |α(t)| ≤ α0, it is determined as a second-level warning; where d(t) represents the relative straight-line distance between the target object and the forklift at time t, d1 represents the first-level warning distance threshold, d2 represents the second-level warning distance threshold, and α0 represents the safe azimuth threshold.

10. A target object recognition and distance detection early warning system for forklifts, characterized in that, includes: Data acquisition module: configured to acquire a multi-element target object dataset and real-time operation scene images; Model training and optimization module: configured to train and optimize a multi-stage deep target object recognition model, and complete model format conversion and hardware adaptation; Edge deployment and inference module: constructed based on Jetson Orin NX, equipped with a TensorRT toolkit to implement parallel real-time inference; Target object recognition and positioning module: deployed in the edge deployment and inference module, configured to complete target object detection, absolute depth restoration and three-dimensional spatial position positioning through model inference results, and output three-dimensional coordinates of the target object; Risk assessment and warning module: in communication with the edge deployment and inference module and the forklift control system, configured to calculate the collision risk coefficient in real time, output graded warning signals and avoidance path reference instructions, and manage and control the driving state of the forklift; Camera calibration module: provided with Zhang Zhengyou's calibration tool and calibration process, configured to acquire internal and external parameters of the camera to provide data support for coordinate conversion.

Images