Unmanned intelligent sensing forklift system and control method

Abstract

This invention discloses an unmanned intelligent sensing forklift system and control method. The system includes a multimodal intelligent sensing module, an edge computing hub module, a precision motion control module, and an onboard interactive communication module. The control method uses a simplified warehouse base map as a reference, synchronously acquiring and preprocessing multimodal data, and combining the MMDetection3D framework to achieve high-precision target identification and tracking. Based on the Cartographer framework, it completes real-time SLAM mapping and local dynamic updates, and achieves precise forklift positioning through IMU dead reckoning and LiDAR feature matching. The SAC-Auto deep reinforcement learning algorithm is used for dynamic path planning and obstacle avoidance. Precise motion execution is achieved through closed-loop feedback control. This invention is suitable for dynamic warehousing scenarios, has low positioning error, high operating efficiency, and can realize automated cargo handling, possessing significant industrial practicality and promotional value.

Description

Technical Field

[0001] This invention belongs to the field of intelligent forklift technology, and particularly relates to an unmanned intelligent sensing forklift system and control method. Background Technology

[0002] In the field of intelligent warehousing and logistics, unmanned forklifts, as core equipment for automated handling, directly determine operational efficiency, safety, and the level of intelligence through their perception, positioning, planning, and control capabilities.

[0003] Existing unmanned forklift technology suffers from the following drawbacks: First, it relies heavily on pre-built high-precision static warehouse maps. When the warehouse environment undergoes dynamic changes such as cargo movement, shelf adjustments, or the establishment of temporary work areas, the static map is prone to failure, leading to positioning errors, unreasonable path planning, and extremely high costs for re-mapping and updating. Second, environmental perception often uses single sensors, resulting in low accuracy in dynamic target recognition, numerous blind spots, and weak anti-interference capabilities. In scenarios where pedestrians, AGVs, and other dynamic obstacles coexist, it is prone to missed or false detections and slow obstacle avoidance response. Third, the coordination between positioning and mapping is poor. Although some technologies use SLAM mapping, it is not synchronized with the forklift operation process and lacks a high-precision positioning strategy based on multi-sensor fusion, which easily leads to accumulated errors. Fourth, path planning is mostly pre-planned and cannot be optimized in real time according to dynamic environmental changes, making it difficult to achieve precise micro-motion control in complex scenarios such as narrow passages. Fifth, the closed-loop feedback between motion control and path planning is insufficient, resulting in large actual execution deviations and failing to meet the stringent requirements for operational accuracy in warehousing scenarios. Summary of the Invention

[0004] The purpose of this invention is to provide an unmanned intelligent sensing forklift system and control method to solve the above-mentioned problems in the prior art.

[0005] To solve the above-mentioned technical problems, the present invention adopts the following solution:

[0006] An unmanned intelligent sensing forklift system includes a multimodal intelligent sensing module, an edge computing hub module, a precision motion control module, and an on-board interactive communication module installed on the forklift body. Each module achieves bidirectional data interaction through industrial bus or wireless communication. The system can seamlessly connect with a warehouse management system (WMS) without the need to build a high-precision static map of the warehouse in advance, only requiring a simple basic base map of the warehouse.

[0007] The multimodal intelligent sensing module includes a lidar, a vision camera, a depth camera, a millimeter-wave radar, and an inertial measurement unit (IMU). Each sensor is time-synchronized and spatially calibrated to synchronously collect point cloud data, visual image data, depth image data, obstacle distance / velocity data, and the forklift's own pose, velocity, and acceleration data of the surrounding environment.

[0008] The edge computing hub module is an embedded industrial control computer running the Ubuntu operating system. It is compatible with the OpenMMLab, Cartographer, and PyTorch open-source frameworks. It has built-in multimodal fusion perception algorithm, SLAM real-time mapping and dynamic update algorithm, and SAC-Auto deep reinforcement learning dynamic path planning algorithm. It is used for multi-source data preprocessing, feature fusion, target recognition and localization. Based on a simple warehouse base map, it realizes SLAM real-time mapping and local dynamic updates during the operation process, completes the real-time localization of the forklift itself and environmental targets, and realizes dynamic path planning and optimization.

[0009] The precision motion control module includes a servo controller, a travel motor controller, and a fork actuator controller, which are electrically connected to the forklift's steering mechanism, travel mechanism, and fork actuator, respectively. It receives control commands to achieve precise control of the forklift's speed, steering, and fork movements. Specifically, the servo controller uses a PID position closed-loop drive algorithm; the travel motor controller uses an FOC field-oriented control algorithm; and the fork actuator controller uses a stepper motor microstepping drive algorithm. This module receives standardized control commands from the edge computing hub module, drives the forklift's actuators to complete precise travel, steering, and fork movements, and collects the actual motion state data of the actuators, feeding it back to the edge computing hub module.

[0010] The in-vehicle interactive communication module includes an industrial touchscreen and a 5G / wireless communication unit. It is used to visualize perception, mapping, positioning, and planning data, and to communicate bidirectionally with the warehouse management system. It receives a simplified warehouse base map and operational tasks, and uploads forklift operation status and mapping data. The 5G / wireless communication unit uses an industrial-grade communication protocol to achieve bidirectional communication with the warehouse management system. It receives the simplified warehouse base map and operational tasks (pickup / placement points, task priority, and timing requirements) from the warehouse management system, and synchronously uploads the forklift's real-time operational status, positioning information, SLAM mapping data, and anomaly information to the warehouse management system, enabling global data interoperability and unified scheduling within the warehouse.

[0011] Further optimization involves aligning the data acquisition timestamps of each sensor in the multimodal intelligent sensing module using the PTP time synchronization protocol, and establishing a spatial coordinate system using Zhang's calibration method. A LiDAR is mounted on the front top of the forklift, with a vision / depth camera positioned adjacent to the LiDAR. Millimeter-wave radars are located at the front and rear ends of the forklift, and an inertial measurement unit is installed at the center of the forklift body, enabling multi-source data acquisition without blind spots. This module synchronously collects point cloud data, visual image data, depth image data, obstacle distance / velocity / azimuth data, and the forklift's own pose, velocity, acceleration, angular velocity, and other physical state data, providing a raw data foundation for subsequent perception, localization, and mapping.

[0012] Further optimization involves an embedded industrial control computer running Ubuntu 22.04 or later, equipped with an embedded GPU, and compatible with open-source frameworks such as OpenMMLab, Cartographer, PyTorch, and ROSNoetic, featuring lightweight design and high computing power. This module serves as the core control hub of the system, incorporating multimodal fusion environmental perception and target localization algorithms, SLAM real-time mapping and dynamic update algorithms, SAC-Auto deep reinforcement learning dynamic path planning algorithms, and precise motion control and closed-loop feedback correction algorithms. It preprocesses, fuses features, identifies and localizes targets from data collected by the multimodal intelligent perception module, and uses a simplified warehouse base map as a reference to achieve real-time SLAM mapping and local dynamic updates during operations. This enables real-time precise localization of the forklift and its surroundings, generates and optimizes the optimal path based on dynamic environmental information, calculates motion control deviations, and dynamically adjusts control commands, achieving full-process algorithm computation and command scheduling.

[0013] A method for controlling an unmanned intelligent sensing forklift based on the above system includes the following steps:

[0014] Step S1, System Initialization and Basic Information Acquisition: Start the system, the vehicle-mounted interactive communication module receives the simplified basic base map of the warehouse and the operation tasks issued by the warehouse management system, and the edge computing hub module loads the base map and completes the parameter initialization of each module.

[0015] Step S2, Multimodal Data Synchronous Acquisition and Preprocessing: The multimodal intelligent sensing module synchronously acquires environmental multi-source data and forklift body status data, and the edge computing hub module performs noise reduction, filtering and coordinate system preprocessing on the raw data;

[0016] Step S3, Multimodal Fusion Environment Perception and Target Localization: Input the preprocessed data into the 3D target detection model based on the MMDetection3D framework, fuse multimodal features through feature-level fusion algorithm, and combine millimeter-wave radar data to achieve high-precision identification and three-dimensional localization of environmental targets, continuously track dynamic targets and update their status in real time;

[0017] Step S4, SLAM real-time mapping and dynamic updates to achieve precise positioning of the forklift: Using a simplified warehouse base map as the global coordinate reference, SLAM real-time mapping is completed synchronously during forklift operation based on the Cartographer framework, integrating LiDAR, IMU, and target positioning data. When environmental changes are detected, only the changed areas are dynamically updated locally in the SLAM map; precise positioning and real-time position updates of the forklift are achieved through IMU dead reckoning and LiDAR map feature matching.

[0018] Step S5: Dynamic path planning and tracking optimization based on deep reinforcement learning: Input the SLAM real-time map, forklift pose, target positioning information, and task into the SAC-Auto deep reinforcement learning dynamic path planning algorithm to construct a state space, action space, and reward function specific to the warehouse scenario, generate the optimal path and evaluate its feasibility in real time, realize dynamic obstacle avoidance and real-time path optimization, and simultaneously perform tracking optimization on the planned path.

[0019] Step S6, Precise Motion Control and Closed-Loop Feedback Adjustment: The edge computing hub module converts the path planning results into control commands and sends them to the precision motion control module to drive the forklift to perform the operation; the multimodal intelligent sensing module collects the actual motion state of the forklift and provides feedback, and the edge computing hub module calculates the deviation and dynamically adjusts the control commands to achieve closed-loop precise control;

[0020] Step S7, Task Iteration and Data Synchronization: After completing one task objective, receive the next task and repeat steps S3-6 to achieve continuous iteration of multi-task operations. At the same time, the forklift operation status, positioning and mapping data are synchronously uploaded to the warehouse management system.

[0021] Further optimization involves the following simplified warehouse base map described in step S1: overall warehouse boundary, approximate coordinates of shelf layout, narrow passage range, restricted / operational area division, and warehouse location coordinates.

[0022] Further optimization, step S3 specifically includes:

[0023] Step S3.1, Model Pre-deployment and Adaptation, specifically includes:

[0024] Step S3.1.1 Build the MMDetection3D framework runtime environment for the edge computing hub of the unmanned intelligent sensing forklift;

[0025] Step S3.1.2 Construct a 3D object detection model based on the MMDetection3D framework, which includes multi-branch feature extraction and attention fusion. The model comprises:

[0026] LiDAR branch: The PointPillars network is used as the backbone to convert the point cloud into 0.2m×0.2m Pillar features, and the SECONDFPNNeck network outputs a 128-dimensional point cloud feature map.

[0027] Visual / Depth Camera Branch: Using a pre-trained ResNet50 as the backbone network (with the weights of the first few layers frozen), 128-dimensional texture semantic features of visual images and 64-dimensional spatial distance features of depth images are extracted and fused through the FPNNeck network to output a 256-dimensional feature map.

[0028] Millimeter-wave radar branch: It adopts a 3-layer fully connected MLP network, inputting 4-dimensional features of range, velocity, azimuth angle and acceleration, and outputting a 128-dimensional dynamic feature tensor.

[0029] Feature-level fusion module: Construct a channel-space dual attention fusion network. The channel attention module assigns dynamic weights to each modality according to the information entropy. The spatial attention module increases the feature weights of narrow channels and shelf areas by 1.2 to 1.5 times. After fusion through 3 layers of 3×3 convolutional layers, a 512-dimensional fused feature map is output.

[0030] 3D Inspection Head: The Anchor3DHead inspection head is used, and the anchor sizes of five types of targets are set: goods, pedestrians, AGVs, fixed facilities, and unknown obstacles. It includes a Softmax classifier and a 3D bounding box regression branch.

[0031] Step S3.2, Preprocessing Data Standardization Input: Standardize the multimodal data after spatiotemporal synchronization, denoising filtering, and coordinate unification: convert the LiDAR point cloud into a 0.02m×0.02m resolution BEV pseudo-image tensor and remove invalid point clouds; scale the visual / depth images to 800×600 pixels and normalize them, binding the pixel coordinates to the warehouse global coordinate system mapping relationship; convert the millimeter-wave radar data into a 4D dynamic feature tensor. Then, encapsulate all the processed data into a unified batch containing timestamps, sensor IDs, and coordinate mapping relationships.

[0032] Step S3.3, Multimodal Feature Extraction and Fusion, specifically:

[0033] Step S3.3.1: Input the standardized data into the corresponding branch network to extract the geometric contour features of the LiDAR, the visual / depth texture space features, and the dynamic features of the millimeter-wave radar respectively;

[0034] Step S3.3.2: Execute the deep convolutional fusion algorithm with enhanced attention mechanism: Dynamic weights are assigned through the channel attention module, and the spatial attention module strengthens the features of key regions. After the weighted features are concatenated by channels and fused by 3 layers of 3×3 convolution, they are optimized by BN batch normalization and ReLU activation function to output a 512-dimensional fused feature map.

[0035] Step S3.4, D-target recognition and 3D global coordinate localization, specifically includes:

[0036] Step S3.4.1: Based on the fused feature map, the Softmax classifier (threshold 0.85) is used to classify the five target classes. The regression branch is used to predict the 3D bounding box parameters in the global coordinate system of the warehouse, including the center coordinates, length, width and height, and orientation angle.

[0037] Step S3.4.2: Fusion of millimeter-wave radar dynamic features corrects the 3D bounding box parameters, converts the bounding box center coordinates into three-dimensional coordinates (X, Y, Z) in the warehouse global coordinate system, and outputs the target size, orientation angle, and category label. The specific implementation is as follows:

[0038] 1) Using the center, size, and orientation angle of the 3D bounding box output by the MMDetection3D detection head as the initial observation values, and the target range, azimuth angle, and relative velocity output by the millimeter-wave radar as dynamic constraints, a weighted correction model is constructed.

[0039] 2) Based on the high ranging accuracy and fast dynamic response of millimeter-wave radar, the center coordinates of the bounding box are corrected: the target polar coordinates measured by the radar are converted into local Cartesian coordinates, and then weighted and fused with the laser-vision fusion detection results. The correction formula is as follows:

[0040] u cor =λ w u det +(1-λ w )u radar ;r cor =λ w r det +(1-λ w )r radar ;

[0041] Where, λ w λ is the confidence weighting coefficient, used when the detection confidence is ≥ 0.85. w When the value is 0.7, and the detection confidence level is 0.5~0.85, λ w When the confidence level is 0.5, λ is less than 0.5. w The value is 0.3, achieving complementarity between static geometric features and dynamic measurement features; u cor r cor The corrected coordinates of the target center; u detr det The coordinates of the target center output by laser, vision / depth detection; u radar r radar The target center coordinates after measurement and conversion by millimeter-wave radar;

[0042] 3) Utilize the target motion direction output by millimeter-wave radar to correct the orientation angle of the 3D bounding box, eliminating angle jitter caused by pure point cloud / visual detection. The correction formula is:

[0043] θ cor =λ w θ det +(1-λ w )θ radar ;

[0044] Where, θ cor The corrected target 3D bounding box orientation angle; θ det The target orientation angle output by MMDetection3D; θ radar The target motion direction angle measured by millimeter-wave radar;

[0045] 4) By using the pre-calibrated sensor extrinsic parameter matrix and global coordinate mapping relationship, the corrected target local coordinates are converted into three-dimensional coordinates (X,Y,Z) in the warehouse global coordinate system.

[0046] 5) The final output includes the target category label, 3D global coordinates, dimensions (length / width / height), orientation angle, and confidence level.

[0047] Step S3.5, Continuous tracking and status update of dynamic targets, specifically:

[0048] Step S3.5.1 Assign a unique ID to the dynamic target and establish a feature library, and predict the target state using Kalman filtering with a period of 0.1s;

[0049] Step S3.5.2 uses the Hungarian matching algorithm (IOU threshold 0.7) to complete the inter-frame target ID association, and combines the dynamic characteristics of millimeter-wave radar to correct the Kalman filter prediction value. The specific implementation is as follows:

[0050] Let the set of tracked targets in the previous frame be... The target set detected in the current frame is First, Kalman filtering is performed on the tracked target to predict the predicted state vector; then, a fusion matching cost matrix C is constructed, with matrix elements C... ij The matching cost between the i-th tracked target and the j-th detected target is represented by the following formula: ;

[0051] Among them, IOU 3DThe intersection-union ratio is 3D, where dist is the Euclidean distance from the target center, and D is the intersection-union ratio. max For maximum sensing distance, Feat sim For feature similarity, v radar The target velocity is measured by millimeter-wave radar, and α, β, γ, and δ are normalized weighting coefficients. This represents the current frame prediction state of the i-th tracked target after Kalman filtering. With minimizing the total matching cost as the optimization objective, the Hungarian algorithm is used to solve for the optimal matching in the bipartite graph, obtaining a one-to-one correspondence between tracked and detected targets. For successfully matched targets, the range, velocity, and azimuth observations output by the millimeter-wave radar are introduced into the Kalman filter update stage to correct the predicted state vector and suppress state estimation drift. Unmatched tracked targets are marked as pending cleanup, and unmatched detected targets are initialized as new tracked targets and assigned a globally unique ID.

[0052] Step S3.5.3: Output dynamic target status information (ID, 3D coordinates, motion speed, orientation angle, occlusion label) every 0.1s; targets that are not matched for 5 consecutive frames or whose confidence is less than 0.5 for 5 consecutive frames are judged as disappeared and removed.

[0053] Step S3.6, Result Output and Anomaly Handling: Encapsulate the target perception results into a standardized output format. Static targets include category, 3D coordinates, size, orientation angle, and confidence level. Dynamic targets include ID, motion parameters, tracking status, and update timestamp. If the target detection confidence level is less than 0.8 for 5 consecutive frames or the dynamic target tracking loss rate is ≥30%, trigger sensor self-check and output an anomaly prompt.

[0054] Further optimization, step S4 specifically includes:

[0055] Step S4.1, Cartographer framework adaptation and initialization, specifically:

[0056] Step S4.1.1: Deploy the lightweight CartographerSLAM framework for the edge computing center of the unmanned intelligent sensing forklift and enable GPU-accelerated point cloud processing;

[0057] Step S4.1.2: Read the simplified basic base map of the warehouse, and map the geographical coordinates of the base map to the global Cartesian coordinate system for forklift operations using a coordinate transformation algorithm. This coordinate system will be used as the global reference for SLAM mapping. The coordinate transformation is achieved using a translation-rotation-scale unified affine transformation, as shown in the following formula:

[0058] ;

[0059] Where: X g Y g For global Cartesian coordinates; Xm Y m For simplified base map coordinates; This is the scaling factor; T is the rotation angle; x T y This represents the translation of the origin.

[0060] Step S4.1.3: Configure the sensor parameters of the lidar and IMU to complete the spatiotemporal synchronization of the lidar and IMU.

[0061] Step S4.2, SLAM real-time mapping, specifically:

[0062] Step S4.2.1: During forklift operation, the Cartographer framework receives the pre-processed LiDAR point cloud, IMU body number, and static target 3D coordinates output by the 3D target detection module in real time, and removes dynamic target point cloud.

[0063] Step S4.2.2: Adopt a layered mapping strategy: Construct a 2D navigation layer (marking channels and restricted areas) and a 3D feature layer (extracting features of fixed facilities) based on the LiDAR point cloud. Generate a submap in units of 10 LiDAR frames and complete the submap stitching through the Ceres-solver optimizer.

[0064] Step S4.3, Local dynamic update of the SLAM map, specifically:

[0065] Step S4.3.1: Execute the environmental change detection algorithm: Extract the key features of the current frame point cloud and perform ICP matching with the corresponding area features of the existing map. Set the matching error threshold to 0.05m. If the matching error is greater than 0.05m for 3 consecutive frames, it is determined to be an area of ​​environmental change. Output the global coordinate range of the changed area.

[0066] ICP is used for registration of the current frame point cloud with local map features, and the implementation steps are as follows:

[0067] 1) Extract local map point cloud feature sets in the corresponding areas of the existing SLAM map. ;

[0068] 2) Extract the real-time point cloud feature set of the region corresponding to the current lidar frame. ;

[0069] 3) Find the nearest point in the map point cloud for each real-time point and establish an initial matching relationship;

[0070] 4) Construct the ICP optimization objective function: Where l is the index of the matching point pair; N is the total number of valid matching point pairs participating in the optimization; q l p represents the laser feature points in the current frame.l R represents the corresponding feature points on the map; R is the rotation matrix; and T is the translation vector.

[0071] 5) Iteratively solve for the rotation matrix R and translation vector T until the objective function converges;

[0072] 6) Calculate the average matching error after convergence: .

[0073] The logic for determining environmental changes is as follows:

[0074] Calculate the average error E of ICP matching frame by frame icp Set the judgment threshold d th =0.1m;

[0075] Judgment rule: If three consecutive frames satisfy E icp >d th If the environment changes, it is considered a change in environment; otherwise, if the environment has not changed, the map update will not be triggered.

[0076] Output of the changed region: Based on the global coordinates of the matched anomaly point set, calculate the minimum bounding rectangle and output the global coordinate range of the changed region [X]. min ,Y min ,X max ,Y max ]. [X min ,Y min ,X max ,Y max [ ] represents the smallest bounding rectangle coordinate range of the environmental change area in the global Cartesian coordinate system of the warehouse. Xmin and Xmax are the minimum and maximum values ​​of the change area on the X-axis in the global coordinate system, respectively; Ymin and Ymax are the minimum and maximum values ​​of the change area on the Y-axis in the global coordinate system, respectively.

[0077] Step S4.3.2: Perform local update for the changed area: lock the corresponding submap, collect 10 frames of point cloud to regenerate the local submap, and stitch it into the global map after calibrating the pose with the surrounding submaps.

[0078] Step S4.4, precise positioning of the forklift itself, specifically:

[0079] Step S4.4.1: Execute dead reckoning algorithm based on IMU: Using the initial global pose of the forklift as a reference, calculate the real-time heading angle by integrating the angular velocity and the real-time displacement by integrating the acceleration, and output the coarse positioning result.

[0080] Specifically:

[0081] Input: Initial global pose of the forklift: X0, Y0, θ0;

[0082] IMU real-time output: acceleration, angular velocity ω (z-axis);

[0083] Heading angle calculation: Current heading angle = Initial heading angle + Integral of angular velocity over time, i.e. ;

[0084] Displacement calculation: First, integrate the acceleration to obtain the velocity, then integrate again to obtain the displacement.

[0085] ;

[0086] Output coarse localization result: (X t ,Y t ,θ t ).

[0087] Step S4.4.2: Extract key feature points (≥50) from the current frame of the LiDAR every 0.1s, and compare them with SLAM data. Figure 3 ICP matching is performed on the D feature layer to calculate the pose deviation and correct the IMU dead reckoning results. The calibrated global pose is output with a positioning error ≤0.03m and a heading angle error ≤0.1°.

[0088] Specifically, it triggers every 0.1 seconds to extract key feature points, including corner points and edge points, from the current LiDAR frame. This is done within the SLAM (Simultaneous Localization and Mapping) framework. Figure 3 Finding matches in the D feature layer: Use the ICP algorithm to register real-time features with map features and solve for rotation deviation Δθ and translation deviation ΔX and ΔY.

[0089] Corrected IMU dead reckoning results:

[0090] ;

[0091] Output calibrated global accurate pose (X) final ,Y final ,θ final The positioning error is ≤0.05m.

[0092] Step S4.4.3: Output the forklift's global pose, heading angle, and position confidence every 0.1s. When the confidence is <0.9 for 5 consecutive frames, trigger emergency positioning compensation and output an abnormal prompt.

[0093] Step S4.5, Mapping-Location Closed-Loop Iteration: Feed back the real-time positioning results of the forklift to the SLAM mapping module to optimize the point cloud coordinate mapping accuracy. When the positioning deviation is greater than 0.03m for multiple consecutive frames, trigger local secondary calibration of the map.

[0094] Further optimization, step S5 specifically includes:

[0095] Step S5.1, SAC-Auto algorithm framework adaptation and initialization, specifically:

[0096] Step S5.1.1: Deploy the lightweight SAC-Auto framework for the forklift edge computing hub, and use quantization-aware training to compress model weights; initialize the actor network, critic network, and target critic network, and set the discount factor γ. d =0.95, entropy temperature coefficient 0.2.

[0097] The actor network is a policy network that takes a 210-dimensional state space S as input and outputs forklift control commands corresponding to the continuous action space A, namely linear velocity increments and angular velocity increments. The critic network and the target critic network are value networks that take state-action pairs as input and output state value estimates. The target critic network has the same structure as the critic network, but its parameter update rate lags behind that of the critic network. It is used to calculate the target value to stabilize the training process of the actor network.

[0098] Step S5.1.2: Standardize the four types of data: SLAM real-time map, forklift pose, target localization, and task data. Convert the 2D grid map into a 100×100 tensor, normalize the forklift pose / target localization / task data to [-1,1], and stitch them together into a 1×210 dimensional state vector.

[0099] Step S5.2, Construction of a dedicated decision-making space for the warehousing scenario, specifically:

[0100] Step S5.2.1: Define the state space The dimension is 210, where M flatten The map flattening tensor has 194 dimensions.

[0101] P forklift The 5D forklift pose vector [X,Y,θ] f ,v f ,ω f X and Y are global coordinates, θ f Let v be the heading angle. f Let ω be the linear velocity. f Angular velocity;

[0102] T target 8-dimensional target vector ;

[0103] J task The task vector is 3D, [priority, timeleft, typetask], representing the task attributes.

[0104] Step S5.2.2: Define the continuous motion space Adapting to forklift physical constraints; among which ∈[-0.5,0.5]m / s, representing the linear velocity increment; ∈[-5,5]° / s, representing the increment of angular velocity;

[0105] Step S5.2.3: Construct a multi-dimensional reward function ;

[0106] Where: w1, w2, w3, and w4 are the weighting coefficients for each sub-reward, used to adjust the decision priority of target approach, obstacle avoidance, path efficiency, and safety compliance; combined with the dynamic operation scenario of the warehouse, the values ​​are w1=1.0, w2=2.0, w3=1.0, and w4=2.0.

[0107] Target proximity reward d dis Let d be the Euclidean distance from the current forklift to the target. prev The distance corresponding to the previous frame is given, and k is the distance decay coefficient, which rewards the forklift's behavior of moving towards the target. The value range is [-1, 1].

[0108] Obstacle Avoidance Penalty R obstacle The value is -10 when the distance between the forklift and the obstacle is less than 0.5m; -1 when the distance between the forklift and the obstacle is greater than or equal to 0.5m and less than 1m; and 0 when the distance between the forklift and the obstacle is greater than or equal to 1m.

[0109] Path efficiency reward ; where L current L represents the current planned path length. shortest The shortest path length when there are no obstacles, and the decision that rewards the shorter path, with a value range of [-1, 1].

[0110] Safety Compliance Rewards R safety The value is 0.5 when the speed / angular velocity is compliant and -2 when it is non-compliant.

[0111] Step S5.3, the execution flow of the SAC-Auto algorithm, is as follows:

[0112] Step S5.3.1 Offline pre-training: Build a warehouse scene simulator, collect 1 million training samples and store them in the experience replay pool, train the actor network (3 layers of fully connected, 210-dimensional input, 2-dimensional action output) and the dual critic network, soft update the target critic network, and train until the average reward on the validation set is ≥1.0;

[0113] Step S5.3.2 Online Real-time Planning: Input the state vector at a frequency of 10Hz, the actor network outputs the optimal action, and a global path is generated by fitting a Bézier curve; evaluate the feasibility of the path through collision detection and dynamic constraint checks, and resample the action if it is not feasible.

[0114] Step S5.3.3 Dynamic obstacle avoidance and optimization: Update dynamic obstacle data every 50ms. When an obstacle enters the path within 1m, the forklift deviates from the target direction, or the task priority changes, trigger local incremental optimization of the path and stitch the old and new paths together by interpolation.

[0115] Step S5.4, Path Tracking Optimization, specifically:

[0116] Step S5.4.1: Calculate the lateral deviation e in real time. y Heading angle deviation Speed ​​deviation e v ;

[0117] Step S5.4.2: Employ the Model Predictive Control (MPC) algorithm to predict the pose of the forklift in the next 5 steps based on the forklift's Ackerman steering model, and minimize the weighted sum of tracking errors. Output corrected motion commands to ensure lateral deviation ≤ 0.05m and heading angle deviation ≤ 0.5°;

[0118] Step S5.4.3 When the lateral deviation is greater than 0.1m or the heading angle deviation is greater than 1°, it is judged as a tracking anomaly. For a slight anomaly, the speed is reduced and the correction weight is increased. For a severe anomaly, tracking is suspended and local replanning is triggered. At the same time, the error data is fed back to the SAC-Auto algorithm to optimize the planning strategy.

[0119] Further optimization, step S6 specifically includes:

[0120] Step S6.1, control command parsing and standardization conversion, specifically includes:

[0121] Step S6.1.1 The edge computing hub module receives the optimal path and motion instruction set from the path planning module, and converts the global Cartesian coordinates of the planned path into forklift body coordinates through a spatial transformation matrix to eliminate coordinate deviation.

[0122] The transformation formula from the global coordinate system to the vehicle body coordinate system is:

[0123] ; ;

[0124] Where X b Y b For the vehicle body coordinate system, X g Y g X represents the coordinates of the global path point. fg Y fg θ f Real-time global pose and heading angle of the forklift;

[0125] Step S6.1.2 Analyze and extract the desired linear velocity v d Desired angular velocity ωd Combined with the physical constraints of the forklift, the amplitude is limited, and the fork operation requirements are parsed and converted into pulse number instructions;

[0126] Step S6.1.3 Encapsulate the steering, traveling, and forklift action commands into standardized data frames according to the CAN bus communication protocol, including command type, value, execution timing, and check code.

[0127] Step S6.2: The precision motion control module drives the actuator, specifically as follows:

[0128] Step S6.2.1: The steering mechanism adopts a PID position closed-loop drive algorithm, and the servo controller drives the steering gear at the desired angular velocity ω. d The target value is to collect the actual angle feedback value of the steering servo in real time, calculate the angle deviation and output the PWM drive signal to control the steering motor.

[0129] The walking mechanism adopts the FOC field orientation control algorithm, and the walking motor controller controls the desired linear velocity v. d Using the target value and combining the actual speed feedback from the motor encoder, closed-loop precise speed control is achieved;

[0130] Step S6.2.2: The fork actuator adopts a stepper motor microstepping drive algorithm to convert the pulse number command into microstepping pulses (microstepping multiple 1000) to drive the fork lifting / tilting / reverse stepper motor to move. At the same time, the actual action value is collected by the displacement sensor / angle sensor to realize position closed-loop control and ensure that the deviation between the fork action and the command value is ≤0.005m / 0.05°.

[0131] Step S6.2.3: Based on the operation sequence, realize the coordinated linkage of turning, walking and fork movements, with an action interval of ≤0.1s.

[0132] Step S6.3: The multimodal sensing module acquires the actual motion state:

[0133] Step S6.3.1: Acquire the actual linear velocity v of the forklift using an IMU. fa Actual angular velocity ω fa Actual heading angle θ a The lidar acquires the actual global pose (Xga, Yga);

[0134] Step S6.3.2: Collect the actual motion values ​​of each actuator through the encoder and displacement / angle sensor, with a sampling frequency of 20Hz;

[0135] Step S6.3.3: Filter and denoise the collected data (IMU complementary filtering, sensor median filtering), align by timestamp to ensure the spatiotemporal consistency of the feedback data, with a time synchronization error ≤1ms, and upload it to the edge computing hub module at a frequency of 20Hz.

[0136] Step S6.3.4, Status Data Upload: The preprocessed actual status data is uploaded to the edge computing hub module via the industrial bus at a fixed frequency. The data format includes the actual value, collection timestamp, and data validity identifier to ensure that the edge computing hub can monitor the forklift's execution status in real time.

[0137] Step S6.4, Deviation Calculation and Multi-Dimensional Dynamic Correction, specifically includes:

[0138] Step S6.4.1 Calculate the walking speed deviation Angular velocity / heading angle deviation , Fork movement deviation and lifting height Forward / backward tilt angle deviation Path tracking overall deviation ;

[0139] Among them, v fd Let v be the target linear velocity. fa ω represents the actual linear velocity of travel. fd Let ω be the target turning angular velocity. fa θ is the actual steering angular velocity. d Let θ be the target heading angle. a h is the actual heading angle. d h is the target elevation height. a θ represents the actual lifting height. fd θ is the angle of forward / backward tilt of the target. fa This refers to the actual forward / backward tilt angle; X gd Y gd Let X be the global coordinates of the target point. ga Y ga These are the global coordinates of the actual point.

[0140] Step S6.4.2 For different types of deviation, an appropriate closed-loop control algorithm is used to generate correction instructions to ensure that the deviation converges quickly to the allowable range. Specifically, the travel / steering mechanism adopts an incremental PID correction algorithm, the fork actuator adopts a position PID and feedforward compensation correction algorithm, and the path tracking adopts a fuzzy PID correction algorithm. The correction amount is calculated through the algorithm and correction instructions are generated.

[0141] Step S6.4.3 When operating in a narrow passage, trigger the low-speed, high-precision correction mode, limit the linear speed to 0.01~0.1m / s, and the single correction amount for turning is ≤0.5° to ensure that the deviations in each dimension converge to the allowable range.

[0142] Step S6.5, closed-loop control iteration and exception handling, specifically includes:

[0143] Step S6.5.1: Complete the closed-loop iteration of "command issuance - status acquisition - deviation calculation - command correction" at a frequency of 20Hz to ensure rapid convergence of deviations; set allowable thresholds for deviations in each dimension, where the allowable threshold for walking speed deviation is 0.01m / s, the allowable threshold for turning angle deviation is 0.1°, the allowable threshold for fork movement deviation is 0.005m, and the allowable threshold for overall path deviation is 0.03m.

[0144] Step S6.5.2: In case of a minor abnormality, increase the PID proportional coefficient and reduce the driving speed; in case of a severe abnormality, immediately issue an emergency braking command and output an alarm message; after the abnormality is resolved, reset the forklift and restore control.

[0145] Step S6.5.3: Store the closed-loop correction data into the database, optimize the PID coefficients and fuzzy rule base through offline learning, and realize the self-calibration of control parameters.

[0146] A computer-readable electronic storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described unmanned intelligent sensing forklift control method.

[0147] Compared with the prior art, the present invention has the following beneficial effects:

[0148] 1. The invention can achieve operation based on a simple base map of the warehouse management system. There is no need to invest manpower and resources in building a high-precision static map of the warehouse in advance. The operation map is built in real time by the forklift through SLAM technology during the operation and is dynamically updated locally in response to environmental changes. This solves the problems of traditional static maps being prone to failure and having high update costs in dynamic warehousing scenarios. It can adapt to environmental changes such as goods movement, shelf adjustment, and dynamic obstacle movement in real time.

[0149] 2. The invention achieves multi-source data acquisition from laser, vision, depth, millimeter-wave radar and IMU through a multimodal intelligent sensing module. Combined with a feature-level fusion algorithm enhanced by attention mechanism and a 3D target detection model of the MMDetection3D framework, it breaks through the perception limitations of a single sensor and achieves high-precision identification and three-dimensional positioning of five types of targets in warehouse scenarios. It has small static target positioning error, high dynamic target tracking accuracy, and continuous dynamic target tracking capability, with no perception blind spots, missed detections and misjudgments.

[0150] 3. This invention enables the synchronous execution of SLAM real-time mapping and forklift operation. It adopts a layered mapping strategy to construct a high-precision 2D navigation layer and a 3D feature layer, with a mapping resolution of less than 0.02m. Combining IMU dead reckoning and LiDAR ICP feature matching in a two-stage positioning strategy, it achieves centimeter-level accurate positioning of the forklift itself. Furthermore, it continuously improves accuracy through a closed-loop iteration of "mapping-positioning," thus solving the problem of accumulated error in traditional positioning technologies.

[0151] 4. This invention constructs a state space, action space, and reward function specific to the warehousing scenario based on the SAC-Auto deep reinforcement learning algorithm, realizing high-frequency online real-time path planning; it adopts local incremental path optimization for dynamic obstacles instead of global replanning, which greatly reduces computing power consumption, and achieves smooth path tracking through Bézier curve fitting and MPC model predictive control, with a lateral deviation of ≤0.05m, and can still achieve accurate path planning in complex scenarios such as narrow passages.

[0152] 5. This invention constructs a closed-loop motion control system of "instruction parsing - execution drive - state feedback - deviation correction", and adopts a variety of high-precision control algorithms such as PID, FOC, fuzzy control, and model predictive control. Combined with a special optimization mode for narrow passages, it realizes precise control of forklift travel, steering, and fork movements.

[0153] 6. The system hardware of this invention adopts a standardized modular design, compatible with mainstream sensors and controllers. The edge computing hub module is compatible with open source frameworks such as OpenMMLab, Cartographer, and PyTorch. The algorithm adopts a lightweight design to adapt to the computing power characteristics of the forklift edge. At the same time, it supports offline pre-training and online incremental optimization of the algorithm. The algorithm parameters can be continuously optimized iteratively by collecting actual operation data to adapt to the operation requirements of different warehousing scenarios. It has high engineering feasibility and is easy to scale up and apply. Attached Figure Description

[0154] Figure 1 This is a schematic diagram of the unmanned intelligent sensing forklift control method of the present invention;

[0155] Figure 2 This is a schematic diagram of the multimodal fusion environmental perception and target localization process of the present invention;

[0156] Figure 3 This is a schematic diagram illustrating the process of real-time SLAM mapping and precise positioning of the forklift itself in this invention.

[0157] Figure 4 This is a flowchart illustrating the SAC-Auto deep reinforcement learning dynamic path planning and tracking optimization process of the present invention.

[0158] Figure 5 This is a schematic diagram of the precise motion control and closed-loop feedback adjustment process of the present invention. Detailed Implementation

[0159] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0160] This embodiment applies the unmanned intelligent sensing forklift system and control method of the present invention to an automated handling operation scenario in a large intelligent warehouse. The warehouse contains multiple rows of shelves, and the narrow aisle width is 1.2 times the width of the forklift. The operation scenario contains dynamic obstacles such as pedestrians and AGVs. The warehouse management system only has a simple basic map containing the approximate layout of the shelves, the aisle range, and the location number, without a high-precision static map. The operation requirement is to achieve unmanned continuous operation of automated picking, handling, and placement of goods. The operation efficiency requirement is ≥50% higher than that of traditional manual forklifts, and the operation accuracy requirement is positioning error ≤0.05m and path deviation ≤0.03m.

[0161] Example 1: Unmanned intelligent sensing forklift system, including:

[0162] Multimodal intelligent sensing module: Utilizing a 16-line LiDAR, a 2-megapixel industrial vision camera, a binocular depth camera, a 77GHz millimeter-wave radar, and a six-axis inertial measurement unit. Each sensor achieves timestamp alignment via the PTP time synchronization protocol, and the spatial coordinate system is established using the Zhang calibration method. Installed at designated locations on the forklift, it enables multi-source data acquisition without blind spots.

[0163] Edge computing hub module: It uses an embedded industrial computer with a Core i7 processor, RTX 3050 graphics card, and 32GB of memory, running the Ubuntu 22.04 operating system and ROSNoetic framework. It deploys open-source frameworks such as MMDetection3Dv1.4.0, Cartographer, and PyTorch 2.0, and compiles and optimizes each core algorithm.

[0164] Precision motion control module: High-precision servo servo controller, AC servo travel motor controller, and stepper motor fork actuator controller are selected and connected to the forklift's steering servo, travel motor, and fork lifting / tilting / reverse actuator respectively. The controller parameters are calibrated to ensure steering control accuracy of ±0.1°, travel speed accuracy of ±0.01m / s, and fork movement deviation of ≤0.005m / 0.05°.

[0165] Vehicle-mounted interactive communication module: An industrial touch screen is selected to develop a visual operation interface that displays information such as perception, mapping, positioning, planning, and operation status; an industrial-grade 5G wireless communication unit is selected to realize two-way communication with the warehouse management system.

[0166] The above modules are connected via an industrial bus to complete the overall system debugging and joint testing, ensuring normal data interaction between modules, stable algorithm operation, and precise response of the actuator.

[0167] Example 2: Figure 1 As shown, the control method for an unmanned intelligent sensing forklift specifically includes the following steps:

[0168] Step S1: System Initialization and Basic Information Acquisition: Start the system and complete the self-check and parameter initialization of each module. The precision motion control module completes zero-position calibration. The vehicle-mounted interactive communication module receives the simplified basic map and the first operation task (picking up goods from shelf A-1 and placing them on shelf B-2) issued by the warehouse management system. The edge computing hub module converts the simplified basic map into a global Cartesian coordinate system to determine the operation boundary and constraints.

[0169] Step S2: Synchronous acquisition and preprocessing of multimodal data: The multimodal intelligent sensing module synchronously acquires multi-source data of the environment and the subject. The edge computing hub module performs preprocessing on the data, such as noise reduction, filtering, and coordinate unification, to eliminate spatiotemporal deviations.

[0170] Step S3, Multimodal Fusion for Environmental Perception and Target Localization: The preprocessed data is input into the 3D target detection model to complete multimodal feature extraction and fusion, achieving high-precision identification and 3D localization of goods on shelf A-1, surrounding AGVs, and pedestrians. For example... Figure 2 As shown.

[0171] Step S4, SLAM Real-time Mapping and Precise Self-Location: The Cartographer framework is launched to achieve real-time SLAM mapping, with a mapping resolution ≤0.02m and a global coordinate deviation of 0.03m. When AGV movement is detected, the map is dynamically updated locally with an update delay of 0.08s. Forklift self-localization is achieved through IMU dead reckoning and LiDAR feature matching, with a positioning error of 0.03m, a heading angle error of 0.08°, and a positioning update frequency of 10Hz. During local updates, 10 frames of point cloud data are collected to regenerate the local submap. Figure 3 As shown.

[0172] Step S5, SAC-Auto Deep Reinforcement Learning Dynamic Path Planning and Tracking Optimization: Standardized multi-source data is input into the SAC-Auto algorithm to generate the optimal path from the forklift stop to shelf A-1. During travel, if a pedestrian is detected entering within 1m of the planned path, local path optimization is triggered, taking 15ms, with an obstacle avoidance response time of 0.15s. Path tracking is achieved through MPC model predictive control, with a lateral deviation of 0.04m and a heading angle deviation of 0.4°. Figure 4 As shown.

[0173] Step S6, Precision Motion Control and Closed-Loop Feedback Adjustment: The planned path instructions are converted into standardized control instructions and sent to the precision motion control module to drive the actions of each actuator of the forklift; the actual motion status is collected in real time and fed back, and after calculating the deviation, it is dynamically corrected through algorithms such as incremental PID and fuzzy PID, with a travel speed deviation of 0.008m / s, a steering angle deviation of 0.08°, and a comprehensive path deviation of 0.025m; when operating in narrow passages, a low-speed, high-precision correction mode is triggered to achieve precise passage without scraping. Figure 5 As shown.

[0174] Step S7, Task Iteration and Full-Process Data Synchronization: After picking up goods from shelf A-1, the system determines that the task is completed and uploads a signal, then receives the next task (placing goods on shelf B-2); repeat steps 3 to 6 to achieve continuous operation of goods handling and placement, with a task switching response time of 0.8s; throughout the process, the forklift operation status, positioning, and mapping data are synchronously uploaded to the warehouse management system to achieve global data interoperability.

[0175] This embodiment completed 72 hours of continuous real-world verification in the aforementioned intelligent warehousing center, where forklifts achieved unmanned continuous operation of picking, handling, and placing goods without any safety accidents occurring.

[0176] Example 3:

[0177] A computer-readable electronic storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described unmanned intelligent sensing forklift control method.

[0178] Based on the above-described preferred embodiments of the present invention, and through the foregoing description, those skilled in the art can make various changes and modifications without departing from the inventive concept. The technical scope of this invention is not limited to the contents of the specification, but must be determined according to the scope of the claims.

Claims

1. An unmanned intelligent sensing forklift system, characterized in that, This includes setting up a multimodal intelligent sensing module, an edge computing hub module, a precision motion control module, and an in-vehicle interactive communication module on the forklift itself; The multimodal intelligent sensing module includes a lidar, a vision camera, a depth camera, a millimeter-wave radar, and an inertial measurement unit. Each sensor is time-synchronized and spatially calibrated to synchronously collect point cloud data, visual image data, depth image data, obstacle distance / velocity data, and the forklift's own pose, velocity, and acceleration data of the surrounding environment. The edge computing hub module is an embedded industrial control computer running the Ubuntu operating system. It is compatible with the OpenMMLab, Cartographer, and PyTorch open-source frameworks. It has built-in multimodal fusion perception algorithm, SLAM real-time mapping and dynamic update algorithm, and SAC-Auto deep reinforcement learning dynamic path planning algorithm. It is used for multi-source data preprocessing, feature fusion, target recognition and localization. Based on the simple warehouse base map, it realizes SLAM real-time mapping and local dynamic updates during the operation process, completes the real-time localization of the forklift itself and environmental targets, and realizes dynamic path planning and optimization. The precision motion control module includes a servo controller, a travel motor controller, and a fork actuator controller, which are electrically connected to the steering mechanism, travel mechanism, and fork actuator of the forklift, respectively, and are used to receive control commands to achieve precise control of the forklift's speed, steering, and fork movements. The vehicle-mounted interactive communication module includes an industrial touch screen and a 5G / wireless communication unit. The industrial touch screen is used to visualize and display perception, mapping, positioning and planning data. The 5G / wireless communication unit communicates bidirectionally with the warehouse management system, receives the simplified basic map of the warehouse and the operation tasks, and uploads the forklift operation status and mapping data.

2. The unmanned intelligent sensing forklift system according to claim 1, characterized in that, The sensors of the multimodal intelligent sensing module achieve data acquisition timestamp alignment through the PTP time synchronization protocol and complete the spatial coordinate system one through Zhang's calibration method; among them, the lidar is installed at the front of the top of the forklift, the vision camera and depth camera are arranged next to the lidar, the millimeter-wave radar is set at the front and rear of the forklift, and the inertial measurement unit is installed at the center of the forklift body.

3. The unmanned intelligent sensing forklift system according to claim 1, characterized in that, The SLAM real-time mapping and dynamic update algorithm of the edge computing hub module is developed based on the Cartographer framework, and only performs local dynamic updates for areas where the warehouse environment changes; the SAC-Auto deep reinforcement learning dynamic path planning algorithm uses the gymnasium framework to realize vectorized environmental simulation, and supports multi-source observation fusion and PER priority experience playback.

4. A method for controlling an unmanned intelligent sensing forklift based on the system described in any one of claims 1-3, characterized in that, Includes the following steps: Step S1, System Initialization and Basic Information Acquisition: Start the system, the vehicle-mounted interactive communication module receives the simplified basic base map of the warehouse and the operation tasks issued by the warehouse management system, and the edge computing hub module loads the base map and completes the parameter initialization of each module. Step S2, Multimodal Data Synchronous Acquisition and Preprocessing: The multimodal intelligent sensing module synchronously acquires environmental multi-source data and forklift body status data, and the edge computing hub module performs noise reduction, filtering and coordinate system preprocessing on the raw data; Step S3, Multimodal Fusion Environment Perception and Target Localization: Input the preprocessed data into the 3D target detection model based on the MMDetection3D framework, fuse multimodal features through feature-level fusion algorithm, and combine millimeter-wave radar data to achieve high-precision identification and three-dimensional localization of environmental targets, continuously track dynamic targets and update their status in real time; Step S4, SLAM real-time mapping and dynamic updates to achieve precise positioning of the forklift: Using a simplified warehouse base map as the global coordinate reference, SLAM real-time mapping is completed synchronously during forklift operation based on the Cartographer framework, integrating LiDAR, IMU, and target positioning data. When environmental changes are detected, only the changed areas are dynamically updated locally in the SLAM map; precise positioning and real-time position updates of the forklift are achieved through IMU dead reckoning and LiDAR map feature matching. Step S5: Dynamic path planning and tracking optimization based on deep reinforcement learning: Input the SLAM real-time map, forklift pose, target positioning information, and task into the SAC-Auto deep reinforcement learning dynamic path planning algorithm to construct a state space, action space, and reward function specific to the warehouse scenario, generate the optimal path and evaluate its feasibility in real time, realize dynamic obstacle avoidance and real-time path optimization, and simultaneously perform tracking optimization on the planned path. Step S6, Precise Motion Control and Closed-Loop Feedback Adjustment: The edge computing hub module converts the path planning results into control commands and sends them to the precision motion control module to drive the forklift to perform the operation; the multimodal intelligent sensing module collects the actual motion state of the forklift and provides feedback, and the edge computing hub module calculates the deviation and dynamically adjusts the control commands to achieve closed-loop precise control; Step S7, Task Iteration and Data Synchronization: After completing one task objective, receive the next task and repeat steps S3-6 to achieve continuous iteration of multi-task operations. At the same time, the forklift operation status, positioning and mapping data are synchronously uploaded to the warehouse management system.

5. The unmanned intelligent sensing forklift control method according to claim 4, characterized in that, The simplified basic warehouse map mentioned in step S1 includes the overall warehouse boundary, approximate coordinates of the shelving layout, narrow passage range, restricted / operational area division, and warehouse location coordinates.

6. The unmanned intelligent sensing forklift control method according to claim 5, characterized in that, Step S3 specifically includes: Step S3.1, Model Pre-deployment and Adaptation, specifically includes: Step S3.1.1: Build the MMDetection3D framework runtime environment for the edge computing hub of the unmanned intelligent sensing forklift; Step S3.1.2: Construct a 3D object detection model based on the MMDetection3D framework, which includes multi-branch feature extraction and attention fusion. The object detection model comprises: LiDAR branch: The Point Pillars network is used as the backbone to convert point clouds into Pillar features, and the SECONDFPN Neck network is used to output a 128-dimensional point cloud feature map. Visual / Depth Camera Branch: Using a pre-trained ResNet50 as the backbone network, 128-dimensional texture semantic features of visual images and 64-dimensional spatial distance features of depth images are extracted and fused through the FPN Neck network to output a 256-dimensional feature map. Millimeter-wave radar branch: It adopts a 3-layer fully connected MLP network, inputting 4-dimensional features of range, velocity, azimuth angle and acceleration, and outputting a 128-dimensional dynamic feature tensor. Feature-level fusion module: Construct a channel-space dual attention fusion network. The channel attention module assigns dynamic weights to each modality according to information entropy, and the spatial attention module enhances the feature weights of narrow channels and shelf areas. After fusion through 3 layers of 3×3 convolutional layers, a 512-dimensional fused feature map is output. 3D Inspection Head: The Anchor3DHead inspection head is used, and the anchor size of five types of targets, namely goods, pedestrians, AGVs, fixed facilities, and unknown obstacles, is set. It includes a Softmax classifier and a 3D bounding box regression branch. Step S3.2, Preprocessing data standardization input: Standardize the multimodal data after spatiotemporal synchronization, noise reduction filtering, and coordinate unification, and then encapsulate all the processed data into a unified batch containing timestamps, sensor IDs, and coordinate mapping relationships; Step S3.3, Multimodal Feature Extraction and Fusion, specifically: Step S3.3.1: Input the standardized data into the corresponding branch network to extract the geometric contour features of the LiDAR, the visual / depth texture space features, and the dynamic features of the millimeter-wave radar respectively; Step S3.3.2: Execute the deep convolutional fusion algorithm with enhanced attention mechanism: Dynamic weights are assigned through the channel attention module, and the spatial attention module strengthens the features of key regions. After the weighted features are concatenated by channels and fused by 3 layers of 3×3 convolution, they are optimized by BN batch normalization and ReLU activation function to output a 512-dimensional fused feature map. Step S3.4, 3D target recognition and 3D global coordinate localization, specifically includes: Step S3.4.1: Based on the fused feature map, the Softmax classifier is used to classify the five types of targets, and the regression branch is used to predict the 3D bounding box parameters in the global coordinate system of the warehouse. Step S3.4.2: Fuse millimeter-wave radar dynamic features to correct 3D bounding box parameters, convert the bounding box center coordinates into three-dimensional coordinates in the warehouse global coordinate system, and output target size, orientation angle, and category label; Step S3.5, Continuous tracking and status update of dynamic targets, specifically: Step S3.5.1: Assign a unique ID to the dynamic target and establish a feature library. Predict the target state using Kalman filtering with a period of 0.1s. Step S3.5.2: Use the Hungarian matching algorithm to complete the inter-frame target ID association, and combine the dynamic characteristics of millimeter-wave radar to correct the Kalman filter prediction value; Step S3.5.3: Output dynamic target status information every 0.1s. Targets that are not matched for 5 consecutive frames or whose confidence is less than 0.5 for 5 consecutive frames are judged as disappeared and removed. Step S3.6, Result Output and Anomaly Handling: Encapsulate the target perception results into a standardized output format. Static targets include category, 3D coordinates, size, orientation angle, and confidence level. Dynamic targets include ID, motion parameters, tracking status, and update timestamp. If the target detection confidence level is less than 0.8 for 5 consecutive frames or the dynamic target tracking loss rate is ≥30%, trigger sensor self-check and output an anomaly prompt.

7. The unmanned intelligent sensing forklift control method according to claim 6, characterized in that, Step S4 specifically includes: Step S4.1, Cartographer framework adaptation and initialization, specifically: Step S4.1.1: Deploy the lightweight CartographerSLAM framework for the edge computing center of the unmanned intelligent sensing forklift and enable GPU-accelerated point cloud processing; Step S4.1.2: Read the simplified basic base map of the warehouse, and use a coordinate transformation algorithm to map the geographical coordinates of the base map to the global Cartesian coordinate system for forklift operations, and use this coordinate system as the global reference for SLAM mapping; Step S4.1.3: Configure the sensor parameters of the lidar and IMU to complete the spatiotemporal synchronization of the lidar and IMU; Step S4.2, SLAM real-time mapping, specifically: Step S4.2.1: During forklift operation, the Cartographer framework receives the pre-processed LiDAR point cloud, IMU body number, and static target 3D coordinates output by the 3D target detection module in real time, and removes dynamic target point cloud. Step S4.2.2: Adopt a layered mapping strategy, construct a 2D navigation layer and a 3D feature layer based on the LiDAR point cloud, generate a submap in units of 10 LiDAR frames, and complete the submap stitching through the Ceres-solver optimizer. Step S4.3, Local dynamic update of the SLAM map, specifically: Step S4.3.1: Execute the environmental change detection algorithm, extract the key features of the current frame point cloud and perform ICP matching with the corresponding area features of the existing map. The matching error threshold is set to 0.05m. When the matching error is greater than 0.05m for 3 consecutive frames, it is determined to be an environmental change area, and the global coordinate range of the change area is output. Step S4.3.2: Perform local update for the changed area: lock the corresponding submap, collect 10 frames of point cloud to regenerate the local submap, and stitch it into the global map after calibrating the pose with the surrounding submaps. Step S4.4, precise positioning of the forklift itself, specifically: Step S4.4.1: Execute the dead reckoning algorithm based on IMU. Using the initial global pose of the forklift as a reference, calculate the real-time heading angle by integrating the angular velocity and calculate the real-time displacement by integrating the acceleration, and output the coarse positioning result. Step S4.4.2: Extract key features of the current frame of the LiDAR every 0.1s, perform ICP matching with the 3D feature layer of the SLAM map, calculate the pose deviation and correct the IMU dead reckoning results, and output the calibrated global pose. Step S4.4.3: Output the forklift's global pose, heading angle, and position confidence every 0.1s. When the confidence is less than 0.9 for 5 consecutive frames, trigger emergency positioning compensation and output an abnormal prompt. Step S4.5, Mapping-Location Closed-Loop Iteration: Feed back the real-time positioning results of the forklift to the SLAM mapping module to optimize the point cloud coordinate mapping accuracy. When the positioning deviation is greater than 0.03m for multiple consecutive frames, trigger local secondary calibration of the map.

8. The unmanned intelligent sensing forklift control method according to claim 7, characterized in that, Step S5 specifically includes: Step S5.1, SAC-Auto algorithm framework adaptation and initialization, specifically: Step S5.1.1: Deploy the lightweight SAC-Auto framework for the forklift edge computing hub, and use quantization-aware training to compress model weights; initialize the actor network, critic network, and target critic network; Step S5.1.2: Standardize the four types of data: SLAM real-time map, forklift pose, target localization, and work task. Step S5.2, Construction of a dedicated decision-making space for the warehousing scenario, specifically: Step S5.2.1: Define the state space The total dimensions are 210, of which The map flattening tensor has 194 dimensions. The 5D forklift pose vector [X,Y,θ] f ,v f ,ω f X and Y are the coordinates of the forklift in the warehouse global coordinate system, and θ is the coordinate of the forklift. f v is the forklift heading angle. f Let ω be the current linear velocity of the forklift. f This represents the current angular velocity of the forklift. For 8-dimensional target and dynamic obstacle state orientation ;in, , The global coordinates of the task target; Encode the target type; The global coordinates of the nearest dynamic obstacle, number 1; The linear velocity of dynamic obstacle number 1; The global coordinates of the second nearest dynamic obstacle, number 2; 3D task vector , represents the attributes of the task; priority indicates the task priority, the larger the value, the higher the priority, and is used for weight allocation when scheduling multiple tasks; timeleft indicates the remaining time or time window of the task, and is used to prioritize urgent tasks; typetask indicates the task type code, and is used to adapt the planning strategy to different task scenarios. Step S5.2.2: Define the continuous motion space Adapting to forklift physical constraints; among which ∈[-0.5,0.5]m / s, representing the linear velocity increment; ∈[-5,5]° / s, representing the increment of angular velocity; Step S5.2.3: Construct a multi-dimensional reward function ; Where: w1, w2, w3, and w4 are the weighting coefficients for each sub-reward, used to adjust the decision priority of target approach, obstacle avoidance, path efficiency, and safety compliance; combined with the dynamic operation scenario of the warehouse, the values ​​are w1=1.0, w2=2.0, w3=1.0, and w4=2.

0. Target proximity reward d dis Let d be the Euclidean distance from the current forklift to the target. prev The distance corresponding to the previous frame is given, and k is the distance decay coefficient, which rewards the forklift's behavior of moving towards the target. The value range is [-1, 1]. Obstacle Avoidance Penalty R obstacle The value is -10 when the distance between the forklift and the obstacle is less than 0.5m; -1 when the distance between the forklift and the obstacle is greater than or equal to 0.5m and less than 1m; and 0 when the distance between the forklift and the obstacle is greater than or equal to 1m. Path efficiency reward ; where L current L represents the current planned path length. shortest The shortest path length when there are no obstacles, and the decision that rewards the shorter path, with a value range of [-1, 1]. Safety Compliance Rewards R safety The value is 0.5 when the speed / angular velocity is compliant and -2 when it is non-compliant; Step S5.3, the execution flow of the SAC-Auto algorithm, is as follows: Step S5.3.1, Offline pre-training: Build a warehouse scene simulator, collect 1 million training samples and store them in the experience replay pool, train the actor network and the dual Critic network, soft update the target critic network, and train until the average reward of the validation set is ≥1.0; Step S5.3.2, Online Real-time Planning: Input the state vector at a frequency of 10Hz, the actor network outputs the optimal action, and a global path is generated by fitting a Bézier curve; evaluate the feasibility of the path through collision detection and dynamic constraint checks, and resample the action if it is not feasible. Step S5.3.3, Dynamic Obstacle Avoidance and Optimization: Update dynamic obstacle data every 50ms. When an obstacle enters the path within 1m, the forklift deviates from the target direction, or the task priority changes, trigger local incremental optimization of the path and stitch the old and new paths together by interpolation. Step S5.4, Path Tracking Optimization, specifically: Step S5.4.1: Calculate the lateral deviation e in real time. y Heading angle deviation Speed ​​deviation e v ; Step S5.4.2: Employ the Model Predictive Control (MPC) algorithm to predict the pose of the forklift in the next 5 steps based on the forklift's Ackerman steering model, and minimize the weighted sum of tracking errors. Output the corrected motion command; Step S5.4.3: When the lateral deviation is greater than 0.1m or the heading angle deviation is greater than 1°, it is judged as a tracking anomaly. For a slight anomaly, the speed is reduced and the correction weight is increased. For a severe anomaly, tracking is suspended and local replanning is triggered. At the same time, the error data is fed back to the SAC-Auto algorithm to optimize the planning strategy.

9. The unmanned intelligent sensing forklift control method according to claim 8, characterized in that, Step S6 specifically includes: Step S6.1, control command parsing and standardization conversion, specifically includes: Step S6.1.1: The edge computing hub module receives the optimal path and motion instruction set from the path planning module, and converts the global Cartesian coordinates of the planned path into the forklift body coordinates through a spatial transformation matrix to eliminate coordinate deviation. The transformation formula from the global coordinate system to the vehicle body coordinate system is: ； ； Where X b Y b For the vehicle body coordinate system, X g Y g X represents the coordinates of the global path point. fg Y fg θ f Real-time global pose and heading angle of the forklift; Step S6.1.2: Analyze and extract the desired linear velocity v d Desired angular velocity ω d Combined with the physical constraints of the forklift, the amplitude is limited, and the fork operation requirements are parsed and converted into pulse number instructions; Step S6.1.3: Encapsulate the steering, traveling, and fork action commands into standardized data frames according to the CAN bus communication protocol, including command type, value, execution sequence, and check code; Step S6.2: The precision motion control module drives the actuator, specifically as follows: Step S6.2.1: The steering mechanism adopts a PID position closed-loop drive algorithm, and the servo controller drives the steering gear at the desired angular velocity ω. d The target value is to collect the actual angle feedback value of the steering servo in real time, calculate the angle deviation and output a PWM drive signal to control the steering motor action; the travel mechanism adopts the FOC field orientation control algorithm, and the travel motor controller drives the travel motor at the desired linear velocity v. d Using the target value and combining the actual speed feedback from the motor encoder, closed-loop precise speed control is achieved; Step S6.2.2: The fork actuator adopts a stepper motor microstepping drive algorithm to convert the pulse number command into microstepping pulses, drive the fork lifting / tilting / reverse stepper motor to move, and collect the actual action value through displacement sensor / angle sensor to realize position closed-loop control. Step S6.2.3: Implement coordinated steering, walking, and fork movements during operation; Step S6.3: The multimodal sensing module collects the actual motion state, specifically as follows: Step S6.3.1: Acquire the actual linear velocity v of the forklift using an IMU. fa Actual angular velocity ω fa Actual heading angle θ a The lidar acquires the actual global pose (Xga, Yga); Step S6.3.2: Collect the actual motion values ​​of each actuator through the encoder and displacement / angle sensor; Step S6.3.3: Filter and denoise the collected data, align it according to the timestamp to ensure the spatiotemporal consistency of the feedback data, and upload it to the edge computing hub module at a frequency of 20Hz; Step S6.3.4, Status Data Upload: The preprocessed actual status data is uploaded to the edge computing hub module via the industrial bus at a fixed frequency. The data format includes the actual value, collection timestamp, and data validity identifier to ensure that the edge computing hub can monitor the forklift's execution status in real time. Step S6.4, Deviation Calculation and Multi-Dimensional Dynamic Correction, specifically includes: Step S6.4.1 Calculate the walking speed deviation Angular velocity / heading angle deviation , Fork movement deviation and lifting height Forward / backward tilt angle deviation Path tracking overall deviation ; Among them, v fd Let v be the target linear velocity. fa ω represents the actual linear velocity of travel. fd Let ω be the target turning angular velocity. fa θ is the actual steering angular velocity. d Let θ be the target heading angle. a h is the actual heading angle. d h is the target elevation height. a θ represents the actual lifting height. fd θ is the angle of forward / backward tilt of the target. fa This refers to the actual forward / backward tilt angle; X gd Y gd Let X be the global coordinates of the target point. ga Y ga The global coordinates of the actual point; Step S6.4.2: For different types of deviation, an appropriate closed-loop control algorithm is used to generate correction instructions to ensure that the deviation converges quickly to the allowable range. Specifically, the travel / steering mechanism adopts an incremental PID correction algorithm, the fork actuator adopts a position PID and feedforward compensation correction algorithm, and the path tracking adopts a fuzzy PID correction algorithm. The correction amount is calculated through the algorithm and correction instructions are generated. Step S6.4.3: When operating in a narrow passage, trigger the low-speed high-precision correction mode, limit the linear speed to 0.01~0.1m / s, and the single correction amount for turning is ≤0.5° to ensure that the deviations in each dimension converge to the allowable range; Step S6.5, closed-loop control iteration and exception handling, specifically includes: Step S6.5.1: Complete the closed-loop iteration of "command issuance - status acquisition - deviation calculation - command correction" at a frequency of 20Hz to ensure rapid convergence of deviation; Step S6.5.2: In case of a minor abnormality, increase the PID proportional coefficient and reduce the driving speed; in case of a severe abnormality, immediately issue an emergency braking command and output an alarm message; after the abnormality is resolved, reset the forklift and restore control. Step S6.5.3: Store the closed-loop correction data into the database, optimize the PID coefficients and fuzzy rule base through offline learning, and realize the self-calibration of control parameters.

10. A computer-readable electronic storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the unmanned intelligent sensing forklift control method as described in any one of claims 4 to 9.

Images