External visual system for forklift operation

Abstract

The invention discloses an external visual system for forklift operation, and relates to the technical field of forklifts. Comprising an acquisition module, a processing module and an output module. The acquisition module stores a warehouse information data set, a cargo information data set, a forklift information data set, an equipment information data set and a problem information data set. According to the invention, through multi-factor decision fusion of a plurality of data sets stored in the acquisition module, a dynamic environment map of the processing module and an analysis model, collaborative optimization of cargo transportation safety and path planning robustness in a storage scene is realized; according to the system, historical accident data, dynamic obstacle contours and warehouse static structures can be fused in real time, high-risk areas are actively avoided, collision accidents and cargo damage risks are remarkably reduced, the problem of path failure caused by dynamic environment changes is effectively solved, and full-process intelligent safety guarantee is provided for high-density storage scenes.

Description

Technical Field

[0001] This invention relates to the field of forklift technology, specifically to an external vision system for forklift operation. Background Technology

[0002] In warehousing and logistics systems, forklifts are core handling equipment, and their operational safety and efficiency are highly dependent on environmental perception capabilities. Forklifts have significant limitations in highly dynamic warehouse environments: metal racks cause multipath interference, goods stacking and obstruction, and sudden mixing of people and vehicles, making it difficult for existing systems to build real-time and accurate environmental maps, which in turn leads to safety hazards such as path planning failure and a sharp increase in collision risk.

[0003] A search revealed a forklift control system and method based on laser positioning and vision guidance, patent publication number CN117891186A. The main control module processes data collected in real time from the laser positioning module, vision detection module, and laser obstacle avoidance module, and controls the forklift's operation according to received remote commands. The laser positioning module collects the forklift's positioning information throughout the work area; the vision detection module collects visible light visual information required for the forklift to visually locate goods; the voice prompt module broadcasts prompts during forklift operation; and the laser obstacle avoidance module collects laser information about obstacles around the forklift. This system offers the advantage of being able to operate within a large factory area, and also features flexible driving routes, diverse remote command reception methods, comprehensive safety guarantees, and user-friendly voice prompts.

[0004] While the aforementioned system enables forklifts to move flexibly within a large factory area, trucks are easily obstructed during cargo handling, laser positioning drifts due to multipath interference in metal racking areas, and in highly dynamic and dense warehouse scenarios, interference from multiple obstacles can easily lead to the failure to receive remote commands, making it unusable. Summary of the Invention

[0005] The purpose of this invention is to provide an external vision system for forklift operation to solve the problems mentioned in the background art.

[0006] To achieve the above objectives, the present invention provides the following technical solution: an external vision system for forklift operation, comprising a data acquisition module, a processing module, and an output module:

[0007] The acquisition module stores warehouse information datasets, cargo information datasets, forklift information datasets, equipment information datasets, and problem information datasets. Based on the warehouse information dataset and equipment information dataset, the acquisition module installs first vision devices at key locations in the warehouse to perform real-time monitoring and acquire warehouse video information sets. Based on the forklift information dataset and equipment information dataset, the acquisition module sets second vision devices at appropriate locations on the forklifts to perform real-time monitoring and acquire work-related video information sets.

[0008] The processing module is equipped with an analysis model. It scans goods to obtain order information using a second vision device. Based on the order information, it generates the optimal path using a path generation method. Based on real-time positioning data from the forklift information dataset, it integrates warehouse video information and work video information to construct and update a dynamic environment map in real time. Based on the dynamic environment map and the problem information dataset, it obtains a set of alert areas. The analysis model monitors and analyzes the optimal path in real time based on the work video information, alert area set, and dynamic environment map to obtain a freight information set. When path blockage or high-risk events are detected, it triggers real-time replanning of the optimal path. The analysis model also analyzes the status of goods during handling in real time using a second vision device to detect tilting, displacement, or damage risks and incorporates these risks into the freight information set. Based on the risk level of the freight information set, the processing module issues graded early warning instructions.

[0009] The output module provides visual output based on a dynamic environment map and issues alarms based on warning commands;

[0010] The method for creating the dynamic environment map includes:

[0011] S1: Establish a connection. Based on the forklift information dataset, set up a multimodal sensing unit and a unique forklift encoder on the forklift to obtain the operation information set. Establish a connection between the warehouse first vision device and the forklift sensing unit through a private local area network. Build a clock synchronization network based on a precise time protocol to align the timestamps of the operation information set to an error of ≤10ms.

[0012] S2: Establish a coordinate system, obtain the real-time location and order information of the forklift based on the forklift encoder ID binding, establish a global coordinate system with the pre-embedded reference point at the warehouse entrance as the origin, obtain the first transformation matrix M1 from the first vision device to the global coordinate system through dual-target plate calculation, and obtain the second transformation matrix M2 based on the fusion calculation of the millimeter-wave radar point cloud and UWB coordinates of the preset calibration point.

[0013] S3: Data fusion. Based on the warehouse video information set, the shelf point cloud model is extracted. The static structure layer in the global coordinate system is obtained through M1 mapping. Based on the work video information set and the millimeter-wave radar point cloud, the dynamic obstacle 3D contour is reconstructed through M2 and DBSCAN clustering algorithms, and its minimum directed bounding box is calculated. Based on the historical accident coordinates in the problem information dataset, the risk probability heat map is generated through Gaussian kernel density estimation to obtain the risk semantic layer. A unified raster spatial index is established through the global coordinate system. The OBB bounding boxes in the dynamic obstacle layer are associated and mapped to the corresponding raster in the risk semantic layer. A fused map data volume containing the static structure layer, dynamic obstacle layer, risk semantic layer and spatial index is generated and published to the analysis model.

[0014] S4: Incremental update condition. When the high-risk event flag of the freight information set is true, the risk semantic mark of the dynamic obstacle layer is dynamically updated. When the displacement of the forklift in the global coordinate system is detected to be >0.5m, the forklift coordinate data in the dynamic obstacle layer is refreshed in real time. When path replanning is triggered, the passage status of the risk semantic layer is updated based on the 10m range along the new path.

[0015] Furthermore, the method for obtaining order information includes:

[0016] A1: Based on the forklift information dataset, a multispectral imaging unit is set on the forklift to obtain light-adaptive raw recognition data. Based on the raw recognition data, the goods are scanned by setting an OCR-barcode dual-mode recognition engine to obtain a set of goods parameters, which includes goods ID, batch number and specification parameters.

[0017] A2: Obtain the storage information set of the corresponding goods by matching the goods parameter set and the goods information dataset. The storage information set includes the warehouse storage location and the warehouse storage quantity.

[0018] A3: Generate order information based on the association and matching of the cargo parameter set and the stored information set.

[0019] Furthermore, the method for creating the analytical model includes:

[0020] B1: Based on warehouse video information set, work video information set, forklift location information set and problem information dataset, and based on the spatial structure of dynamic environment map, a random occlusion generator is set to obtain simulated cargo damage samples.

[0021] B2: Based on the heat map of the alert area, the outline of the dynamic obstacle layer, and the path coordinates, a path risk analysis sub-model is obtained by setting a spatiotemporal graph convolutional network. Based on cargo image data, a cargo mask is obtained by setting a segmentation network, and a tilt angle calculation model is obtained based on the geometric features of the mask. Based on the path blocking probability and the cargo tilt angle, a multi-factor decision fusion sub-model is obtained by setting a fuzzy Petri net inference engine. The multi-factor decision fusion sub-model outputs a cargo information set, which includes cargo tilt angle, path risk level, and high-risk event flag.

[0022] B3: The analysis model is obtained by training and optimizing the multi-factor decision fusion sub-model, and the predictive analysis model is updated regularly.

[0023] Furthermore, the training and optimization methods include:

[0024] C1: Based on the pre-trained weights of the public scene dataset, the model parameter initialization scheme is obtained by setting the transfer learning strategy, and the optimized model parameters are obtained by setting the Bayesian optimizer based on the multi-objective loss function.

[0025] C2: Based on edge computing units, a real-time deployment scheme is obtained by setting the TensorRT inference engine. Based on the dynamic environment map change threshold, a model update strategy is obtained by setting an incremental learning mechanism.

[0026] Furthermore, the method for obtaining the alert area information set includes:

[0027] D1: The risk semantic layer based on the dynamic environment map obtains the risk probability value of each grid by setting a heat map grid parser, and obtains the set of historical accident points related to the current path by setting an accident point matching engine based on the historical accident coordinates of the problem information dataset.

[0028] D2: Based on the coordinate sequence of the optimal path, the path influence area is obtained by setting a buffer generator, and based on the path influence area and the risk semantic layer, a high-risk raster set is obtained by setting a raster filter.

[0029] D3: Based on the latest 3D contours of the dynamic obstacle layer and its OBB bounding box, the dynamic obstacle occupancy grid in the risk semantic layer is obtained by setting a contour projection converter. The high-risk grid set and the dynamic obstacle occupancy grid are merged and the alert area set is obtained by setting a calculator.

[0030] Furthermore, the path generation method includes:

[0031] E1: Path initialization, based on the warehouse storage location in the order information, the initial node path sequence is obtained by setting the A* algorithm path planner, and the safe passage area boundary is obtained by setting the grid expansion processor based on the static structure layer of the dynamic environment map.

[0032] E2: Dynamic path optimization, based on the alert area set and the initial node path sequence, generates the best path and its risk distribution map by setting a path risk mapper;

[0033] E3: Real-time replanning trigger. When the analysis model detects that the path blockage or high-risk event flag is true and the path risk level is ≥0.8, based on the latest dynamic obstacle layer, the optimal path is dynamically corrected by setting a dynamic window method to generate a local detour path. When the freight information set contains the risk of cargo tilting, the speed reduction command is obtained by setting a speed ratio controller based on the risk level.

[0034] Furthermore, the method for generating the tiered early warning instructions includes:

[0035] F1: Based on the risk level of the freight information aggregation, physical early warning signals are obtained by setting up a multi-mode alarm with sound, light and vibration.

[0036] F2: When the cargo tilt angle is greater than 10°, a forced deceleration command is obtained by setting the forklift controller's CAN bus interception module;

[0037] F3: Risk semantic layer based on dynamic environment map, which generates red flashing effect of high-risk area in the visualization interface of the output module by setting AR projection overlay unit.

[0038] Furthermore, the method for determining the key location:

[0039] G1: Based on the shelving layout diagram in the warehouse information dataset, the coordinate set of all shelving aisle intersection points is obtained by setting an aisle intersection detection algorithm;

[0040] G2: Based on historical accident records in the problem information dataset, a set of coordinates for high-incidence areas with an accident frequency > 0.5 times / month is obtained by setting an accident frequency statistician;

[0041] G3: Based on the coordinate set of the intersection points of shelf aisles and the coordinate set of accident-prone areas, a candidate key point set is obtained by setting an Euclidean distance clustering engine;

[0042] G4: Based on the field of view parameters of the first vision device in the candidate keypoint set and device information dataset, the minimum installation position and angle of the first vision device are obtained by setting the field of view coverage optimizer.

[0043] Furthermore, the first vision device is a global monitoring industrial camera, the second vision device is a forklift multispectral anti-shake gimbal camera, the multimodal sensing unit includes a millimeter-wave radar, a UWB positioning module and an inertial measurement unit, and the AR projection overlay unit is a holographic waveguide head-up display.

[0044] Compared with the prior art, the beneficial effects of the present invention are:

[0045] An external vision system for forklift operation achieves coordinated optimization of cargo transportation safety and path planning robustness in warehousing scenarios by integrating multiple datasets stored in the acquisition module, dynamic environmental maps in the processing module, and multi-factor decision-making in the analysis model. Compared with traditional systems that rely on preset paths or single sensor monitoring, this system can integrate historical accident data, dynamic obstacle outlines, and static warehouse structures in real time, actively avoid high-risk areas, significantly reduce collision accidents and cargo damage risks, effectively solve the path failure problem caused by dynamic environmental changes, and provide full-process intelligent safety assurance for high-density warehousing scenarios.

[0046] Meanwhile, the system ensures the real-time and accuracy of environmental perception through an incremental update mechanism of dynamic environmental maps and a vigilant approach to generating regional sets. Through hierarchical early warning commands linked by the analysis model, including audible and visual vibration alarms, forced speed reduction on the CAN bus, and red light flashing on the AR interface, the system synchronously responds to cargo tilting / displacement risks and path blockage events, triggering speed adjustments or path replanning. Compared with existing technologies, this system is the first to achieve three-dimensional collaborative decision-making based on cargo status, path risks, and environmental maps, breaking through the limitations of traditional forklift systems in terms of lagging cargo monitoring and rigid path planning, and significantly improving the continuity and safety of warehousing operations. Attached Figure Description

[0047] Figure 1 This is a schematic diagram of the overall architecture of the present invention;

[0048] Figure 2 This is a schematic diagram of the dynamic environment map architecture of the present invention;

[0049] Figure 3 This is a schematic diagram of the dynamic obstacle reconstruction process of the present invention;

[0050] Figure 4 This is a schematic diagram of the warehouse interior according to the present invention. Detailed Implementation

[0051] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0052] Example 1: As Figures 1-4 As shown, the present invention provides a technical solution: an external vision system for forklift operation, comprising a data acquisition module, a processing module, and an output module.

[0053] The acquisition module stores warehouse information datasets, cargo information datasets, forklift information datasets, equipment information datasets, and problem information datasets. Based on the warehouse information dataset and equipment information dataset, the acquisition module installs first vision devices at key locations in the warehouse to perform real-time monitoring and acquire warehouse video information sets. Based on the forklift information dataset and equipment information dataset, the acquisition module sets second vision devices at appropriate locations on the forklifts to perform real-time monitoring and acquire work-related video information sets.

[0054] It is important to note that the warehouse information dataset includes the warehouse layout map, the precise coordinates of shelves, aisles, and columns, and the lighting distribution; the cargo information dataset includes cargo IDs, physical parameters, and stacking limits; the forklift information dataset includes forklift IDs and load data; the equipment information dataset includes camera intrinsic parameter matrices, radar calibration parameters, field of view range, and spectral response curves; and the problem information dataset includes historical accident coordinates, accident type codes, high-frequency risk areas, and environmental interference records. The appropriate location is the driver's blind spot.

[0055] The processing module is equipped with an analysis model. It scans goods to obtain order information using a second vision device. Based on the order information, it generates the optimal path using a path generation method. Based on real-time positioning data from the forklift information dataset, it integrates warehouse video information and work-related video information to construct and update a dynamic environment map in real time. Based on the dynamic environment map, it matches the problem information dataset to obtain a set of alert areas. The analysis model monitors and analyzes the optimal path in real time based on the work-related video information set, the alert area set, and the dynamic environment map to obtain a freight information set. When path blockage or high-risk events are detected, it triggers real-time replanning of the optimal path. The analysis model also analyzes the status of goods during handling in real time using a second vision device to detect tilting, displacement, or damage risks and incorporates these risks into the freight information set. Based on the risk level of the freight information set, the processing module issues graded early warning instructions.

[0056] The output module provides visual output based on a dynamic environment map and issues alerts based on warning commands;

[0057] The method for creating a dynamic environment map includes: S1: Establishing connections. Based on the forklift information dataset, multimodal sensing units and a unique forklift encoder are set on the forklift to obtain the operational information set. A connection is established between the warehouse's first vision device and the forklift sensing unit via a private local area network. A clock synchronization network is constructed based on a precise time protocol to align the timestamps of the operational information set to an error ≤10ms. S2: Establishing a coordinate system. Based on the forklift encoder ID binding, the real-time location and order information of the forklift are obtained. A global coordinate system is established with the pre-embedded reference point at the warehouse entrance as the origin. The first transformation matrix M1 from the first vision device to the global coordinate system is obtained through dual-target calibration. The second transformation matrix M2 is obtained by fusing the millimeter-wave radar point cloud and UWB coordinates based on the preset calibration points. S3: Data fusion. Based on the warehouse video information set, the shelf point cloud model is extracted. The static structure layer under the global coordinate system is obtained through mapping M1. Based on the work stop video information set and... Millimeter-wave radar point cloud fusion is used to reconstruct the 3D contours of dynamic obstacles through M2 and DBSCAN clustering algorithms, and calculate their minimum directed bounding boxes. Based on historical accident coordinates in the problem information dataset, a risk probability heatmap is generated through Gaussian kernel density estimation to obtain the risk semantic layer. A unified raster spatial index is established through a global coordinate system, and the OBB bounding boxes in the dynamic obstacle layer are associated and mapped to the corresponding raster in the risk semantic layer. A fused map data volume containing the static structure layer, dynamic obstacle layer, risk semantic layer, and spatial index is generated and published to the analysis model. S4: Incremental update condition. When the high-risk event flag in the freight information set is true, the risk semantic markers of the dynamic obstacle layer are dynamically updated. When the displacement of the forklift in the global coordinate system is detected to be >0.5m, the forklift coordinate data in the dynamic obstacle layer is refreshed in real time. When path replanning is triggered, the passage status of the risk semantic layer is updated based on a 10m range along the new path.

[0058] It is important to note that the IEEE 1588v2 protocol is used, with the master clock deployed on an edge server (model: NIPXIe-6674T). A fiber optic switch connects the warehouse industrial camera and the forklift sensing unit. The sensor uses a hardware timestamp chip (DP83640), with a synchronization error ≤8ms. UWB anchor points are pre-embedded, and a 10×10 checkerboard calibration board is placed at the warehouse entrance. The camera extrinsic parameter matrix is ​​solved using OpenCV's solvePnP function. At five pre-set calibration points, millimeter-wave radar point clouds and UWB coordinates are fused, and the least squares method is used to fit the transformation matrix, with an error <3cm. The pre-set calibration point spacing is 2m, and the angular resolution of the millimeter-wave radar point cloud is 0.5°. The static structure layer is achieved through warehouse video... The information set extracts the shelf point cloud model, which is then mapped to the global coordinate system via ICP registration. The dynamic obstacle layer reconstructs the 3D contours of dynamic obstacles by fusing the outgoing video information set and millimeter-wave radar point cloud data and using the DBSCAN clustering algorithm (parameters ε = 0.5m, minPts = 4). The risk semantic layer processes historical accident coordinates through Gaussian kernel density estimation (bandwidth σ = 1.2m) to generate a risk probability heatmap. The spatial index layer establishes a 0.5m × 0.5m rasterized index structure through the global coordinate system. Incremental updates refresh the forklift coordinate data in real time when a forklift displacement > 0.5m is detected. During path replanning, the risk semantic layer data within ±10m of the new path centerline is updated.

[0059] The order information acquisition methods include: A1: Based on the forklift information dataset, a multispectral imaging unit is set on the forklift to obtain the original recognition data with adaptive illumination. Based on the original recognition data, the goods are scanned by setting an OCR-barcode dual-mode recognition engine to obtain the goods parameter set, which includes the goods ID, batch number and specification parameters. A2: Based on the goods parameter set and the goods information dataset, the corresponding goods storage information set is obtained by matching. The storage information set includes the warehouse storage location and warehouse storage quantity. A3: Based on the association and matching of the goods parameter set and the storage information set, order information is generated.

[0060] It should be noted that the cargo parameter set is obtained by scanning the cargo using an OCR-barcode dual-mode recognition engine. The OCR engine uses a CRNN model to recognize the text on the cargo, and the barcode recognition uses the ZBar library to decode the Code128 / EAN-13 format. Order information is generated by associating the cargo parameter set with the stored information set, and the storage location is mapped to global coordinate system coordinates, for example, "A-03-2" → (x:12.5m, y:7.2m).

[0061] The method for creating the analysis model includes: B1: Based on warehouse video information sets, work video information sets, forklift location information sets, and problem information datasets, a standard dataset with a time synchronization deviation ≤10ms is obtained by setting a timestamp alignment module. Based on the spatial structure of the dynamic environment map, a random occlusion generator is used to obtain simulated cargo damage samples. B2: Based on the heat map of the alert area, the outline of the dynamic obstacle layer, and the path coordinates, a path risk analysis sub-model is obtained by setting a spatiotemporal graph convolutional network. Based on cargo image data, a cargo mask is obtained by setting a segmentation network, and a tilt angle calculation model is obtained based on the geometric features of the mask. Based on the path blocking probability and cargo tilt angle, a multi-factor decision fusion sub-model is obtained by setting a fuzzy Petri net inference engine. The multi-factor decision fusion sub-model outputs a cargo information set, which includes cargo tilt angle, path risk level, and high-risk event flags. B3: The analysis model is obtained by training and optimizing the multi-factor decision fusion sub-model, and the predictive analysis model is updated regularly.

[0062] It is important to note that the cargo damage simulation uses a random occlusion generator to add 4-8 polygonal occlusions to simulate fork-shaped occlusion scenarios. The path risk analysis sub-model takes as input: a heatmap of alert areas + a path coordinate sequence. The heatmap of alert areas is a 64×64 grid, and the path coordinate sequence has a length of 50. The network architecture is a 3-layer spatiotemporal graph convolution with 64-128-256 channels. The output is the path blocking probability P. b ∈[0,1], the tilt angle calculation model is based on the cargo mask output by YOLOv7-Seg, and calculates the major axis angle of the minimum bounding rectangle: α: (For second-order central moments), fuzzy Petri net rule base:

[0063]

[0064] The training and optimization methods include: C1: Pre-training weights based on public scene datasets, obtaining model parameter initialization schemes by setting transfer learning strategies, and obtaining optimized model parameters by setting a Bayesian optimizer based on multi-objective loss functions; C2: Obtaining real-time deployment schemes by setting TensorRT inference engines based on edge computing units, and obtaining model update strategies by setting incremental learning mechanisms based on dynamic environment map change thresholds.

[0065] It should be noted that incremental learning (learning rate 0.001) is triggered when the dynamic environment map change rate is greater than 20% (raster change ratio).

[0066] The methods for obtaining the alert area information set include: D1: Based on the risk semantic layer of the dynamic environment map, the risk probability value of each grid is obtained by setting a heatmap grid parser. Based on the historical accident coordinates of the problem information dataset, the set of historical accident points related to the current path is obtained by setting an accident point matching engine. D2: Based on the coordinate sequence of the optimal path, the path influence area is obtained by setting a buffer generator. Based on the path influence area and the risk semantic layer, the high-risk grid set is obtained by setting a grid filter. D3: Based on the latest 3D contour of the dynamic obstacle layer and its OBB bounding box, the grid occupied by dynamic obstacles in the risk semantic layer is obtained by setting a contour projection converter. The high-risk grid set and the dynamic obstacle occupancy grid are merged and the alert area set is obtained by setting a calculator.

[0067] It is important to note the risk probability value. Where x i For historical accident points, σ ​​= 1.2m, the path influence area = path centerline ± 1.5m, where 1.5m is the forklift width + safety margin, and the logical rule is: if a grid simultaneously meets the requirements and is occupied by dynamic obstacles, it is marked as a warning area.

[0068] The path generation method includes: E1: Path initialization, which obtains the initial node path sequence based on the warehouse storage location in the order information by setting the A* algorithm path planner, and obtains the safe passage area boundary based on the static structure layer of the dynamic environment map by setting the grid expansion processor; E2: Dynamic path optimization, which generates the best path and its risk distribution map based on the alert area set and the initial node path sequence by setting the path risk mapper; E3: Real-time replanning trigger, which generates a local detour path by dynamically correcting the best path based on the latest dynamic obstacle layer and setting the dynamic window method when the analysis model detects that the path blockage or high-risk event flag is true and the path risk level is ≥0.8, and obtains the deceleration command based on the risk level by setting the speed ratio controller when the freight information set contains the risk of cargo tilting.

[0069] It is important to note that the A* algorithm's heuristic function is f(n) = g(n) + 1.2 × h(n), where h(n) is the Manhattan distance. The safety boundary expansion radius equals the forklift width plus 0.3m. The dynamic window method parameters are: maximum linear velocity 1.5m / s, maximum angular velocity 40° / s, safety distance threshold 0.8m, and prediction time 3s. The deceleration command formula is: v new =v orig ×(1-0.1×risk level ).

[0070] The generation methods for graded early warning instructions include: F1: Based on the risk level in the freight information set, physical early warning signals are obtained by setting up a multi-mode alarm with sound, light and vibration; F2: When the tilt angle of the goods is >10°, a forced speed reduction instruction is obtained by setting up the CAN bus interception module of the forklift controller; F3: Based on the risk semantic layer of the dynamic environment map, a red light flashing effect in the high-risk area is generated in the visualization interface of the output module by setting up an AR projection overlay unit.

[0071] It is important to note that the audible, visual, and vibration alarms—including a buzzer, flashing red LEDs, and a seat vibration motor for forced speed reduction—are implemented by sending SPN 6142 (target speed value) via the J1939 protocol, overriding the forklift throttle signal.

[0072] AR projection: Holographic waveguide display renders RGBA, with a flicker frequency of 2Hz.

[0073] Methods for determining key locations: G1: Based on the shelving layout map in the warehouse information dataset, obtain the coordinate set of all shelving aisle intersection points by setting an aisle intersection detection algorithm; G2: Based on historical accident records in the problem information dataset, obtain the coordinate set of high-incidence areas with an accident frequency > 0.5 times / month by setting an accident frequency statistician; G3: Based on the shelving aisle intersection point coordinate set and the high-incidence area coordinate set, obtain the candidate key point set by setting an Euclidean distance clustering engine; G4: Based on the candidate key point set and the field of view angle parameters of the first vision device in the equipment information dataset, obtain the minimum installation position and angle of the first vision device by setting a field of view coverage optimizer.

[0074] It is important to note that the aisle intersection detection is based on the equation of the shelf line segments to find the intersection points, filtering out neighboring points with a distance of less than 2m, using an integer linear programming model: min∑x i st∑a ij x i ≥1, where a ij =1 When point j is within the field of view of i, the angle of view is ≤45°.

[0075] The first vision device is a global monitoring industrial camera; the second vision device is a forklift multispectral anti-shake gimbal camera; the multimodal sensing unit includes millimeter-wave radar, UWB positioning module and inertial measurement unit; and the AR projection overlay unit is a holographic waveguide head-up display.

[0076] Example 2: A workflow scenario of an external vision system for forklift operation: When the forklift encounters a dynamic obstacle during transport:

[0077] Data acquisition: Millimeter-wave radar detects moving obstacles 1.8m ahead (point cloud cluster size 0.6×0.4m), and multispectral camera monitors cargo tilt angle = 12°;

[0078] Dynamic map update: Forklift displacement 0.6m → refresh dynamic obstacle layer, match historical accident point (coordinates (12.3, 7.1), accident frequency 0.8 times / month);

[0079] Decision and Output: Analysis model output: Path risk level 0.9 + tilt angle 12° triggers F2 level warning; Execution: Reduce speed to 3km / h + red light flashing on AR interface + generate detour using dynamic window method. Path.

[0080] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended embodiments and their equivalents.

Claims

1. An external vision system for forklift operation, comprising a data acquisition module, a processing module, and an output module, characterized in that: The acquisition module stores warehouse information datasets, cargo information datasets, forklift information datasets, equipment information datasets, and problem information datasets. Based on the warehouse information dataset and equipment information dataset, the acquisition module installs first vision devices at key locations in the warehouse to perform real-time monitoring and acquire warehouse video information sets. Based on the forklift information dataset and equipment information dataset, the acquisition module sets second vision devices at appropriate locations on the forklifts to perform real-time monitoring and acquire work-related video information sets. The processing module is equipped with an analysis model. It scans goods to obtain order information using a second vision device. Based on the order information, it generates the optimal path using a path generation method. Based on real-time positioning data from the forklift information dataset, it integrates warehouse video information and work video information to construct and update a dynamic environment map in real time. Based on the dynamic environment map and the problem information dataset, it obtains a set of alert areas. The analysis model monitors and analyzes the optimal path in real time based on the work video information, alert area set, and dynamic environment map to obtain a freight information set. When path blockage or high-risk events are detected, it triggers real-time replanning of the optimal path. The analysis model also analyzes the status of goods during handling in real time using a second vision device to detect tilting, displacement, or damage risks and incorporates these risks into the freight information set. Based on the risk level of the freight information set, the processing module issues graded early warning instructions. The output module provides visual output based on a dynamic environment map and issues alarms based on warning commands; The method for creating the dynamic environment map includes: S1: Establish a connection. Based on the forklift information dataset, set up a multimodal sensing unit and a unique forklift encoder on the forklift to obtain the operation information set. Establish a connection between the warehouse first vision device and the forklift sensing unit through a private local area network. Build a clock synchronization network based on a precise time protocol to align the timestamps of the operation information set to an error of ≤10ms. S2: Establish a coordinate system, obtain the real-time location and order information of the forklift based on the forklift encoder ID binding, establish a global coordinate system with the pre-embedded reference point at the warehouse entrance as the origin, obtain the first transformation matrix M1 from the first vision device to the global coordinate system through dual-target plate calculation, and obtain the second transformation matrix M2 based on the fusion calculation of the millimeter-wave radar point cloud and UWB coordinates of the preset calibration point. S3: Data fusion. Based on the warehouse video information set, the shelf point cloud model is extracted. The static structure layer in the global coordinate system is obtained through M1 mapping. Based on the work video information set and the millimeter-wave radar point cloud, the dynamic obstacle 3D contour is reconstructed through M2 and DBSCAN clustering algorithms, and its minimum directed bounding box is calculated. Based on the historical accident coordinates in the problem information dataset, the risk probability heat map is generated through Gaussian kernel density estimation to obtain the risk semantic layer. A unified raster spatial index is established through the global coordinate system. The OBB bounding boxes in the dynamic obstacle layer are associated and mapped to the corresponding raster in the risk semantic layer. A fused map data volume containing the static structure layer, dynamic obstacle layer, risk semantic layer and spatial index is generated and published to the analysis model. S4: Incremental update condition. When the high-risk event flag of the freight information set is true, the risk semantic mark of the dynamic obstacle layer is dynamically updated. When the displacement of the forklift in the global coordinate system is detected to be >0.5m, the forklift coordinate data in the dynamic obstacle layer is refreshed in real time. When path replanning is triggered, the passage status of the risk semantic layer is updated based on the 10m range along the new path.

2. The external vision system for forklift operation according to claim 1, characterized in that: The method for obtaining order information includes: A1: Based on the forklift information dataset, a multispectral imaging unit is set on the forklift to obtain light-adaptive raw recognition data. Based on the raw recognition data, the goods are scanned by setting an OCR-barcode dual-mode recognition engine to obtain a set of goods parameters, which includes goods ID, batch number and specification parameters. A2: Obtain the storage information set of the corresponding goods by matching the goods parameter set and the goods information dataset. The storage information set includes the warehouse storage location and the warehouse storage quantity. A3: Generate order information based on the association and matching of the cargo parameter set and the stored information set.

3. The external vision system for forklift operation according to claim 1, characterized in that: The method for creating the analysis model includes: B1: Based on warehouse video information set, work video information set, forklift location information set and problem information dataset, a standard dataset with a time synchronization deviation ≤10ms is obtained by setting a timestamp alignment module. Based on the spatial structure of the dynamic environment map, a simulated sample of cargo damage is obtained by setting a random occlusion generator. B2: Based on the heat map of the alert area, the outline of the dynamic obstacle layer, and the path coordinates, a path risk analysis sub-model is obtained by setting a spatiotemporal graph convolutional network. Based on cargo image data, a cargo mask is obtained by setting a segmentation network, and a tilt angle calculation model is obtained based on the geometric features of the mask. Based on the path blocking probability and the cargo tilt angle, a multi-factor decision fusion sub-model is obtained by setting a fuzzy Petri net inference engine. The multi-factor decision fusion sub-model outputs a cargo information set, which includes cargo tilt angle, path risk level, and high-risk event flag. B3: The analysis model is obtained by training and optimizing the multi-factor decision fusion sub-model, and the predictive analysis model is updated regularly.

4. The external vision system for forklift operation according to claim 3, characterized in that: The training and optimization methods include: C1: Based on the pre-trained weights of the public scene dataset, the model parameter initialization scheme is obtained by setting the transfer learning strategy, and the optimized model parameters are obtained by setting the Bayesian optimizer based on the multi-objective loss function. C2: Based on edge computing units, a real-time deployment scheme is obtained by setting the TensorRT inference engine. Based on the dynamic environment map change threshold, a model update strategy is obtained by setting an incremental learning mechanism.

5. The external vision system for forklift operation according to claim 1, characterized in that: The method for obtaining the alert zone information set includes: D1: The risk semantic layer based on the dynamic environment map obtains the risk probability value of each grid by setting a heat map grid parser, and obtains the set of historical accident points related to the current path by setting an accident point matching engine based on the historical accident coordinates of the problem information dataset. D2: Based on the coordinate sequence of the optimal path, the path influence area is obtained by setting a buffer generator, and based on the path influence area and the risk semantic layer, a high-risk raster set is obtained by setting a raster filter. D3: Based on the latest 3D contours of the dynamic obstacle layer and its OBB bounding box, the dynamic obstacle occupancy grid in the risk semantic layer is obtained by setting a contour projection converter. The high-risk grid set and the dynamic obstacle occupancy grid are merged and the alert area set is obtained by setting a calculator.

6. The external vision system for forklift operation according to claim 1, characterized in that: The path generation method includes: E1: Path initialization, based on the warehouse storage location in the order information, the initial node path sequence is obtained by setting the A* algorithm path planner, and the safe passage area boundary is obtained by setting the grid expansion processor based on the static structure layer of the dynamic environment map. E2: Dynamic path optimization, based on the alert area set and the initial node path sequence, generates the best path and its risk distribution map by setting a path risk mapper; E3: Real-time replanning trigger. When the analysis model detects that the path blockage or high-risk event flag is true and the path risk level is ≥0.8, based on the latest dynamic obstacle layer, the optimal path is dynamically corrected by setting a dynamic window method to generate a local detour path. When the freight information set contains the risk of cargo tilting, the speed reduction command is obtained by setting a speed ratio controller based on the risk level.

7. The external vision system for forklift operation according to claim 1, characterized in that: The method for generating the tiered early warning instructions includes: F1: Based on the risk level of the freight information aggregation, physical early warning signals are obtained by setting up a multi-mode alarm with sound, light and vibration. F2: When the cargo tilt angle is greater than 10°, a forced deceleration command is obtained by setting the forklift controller's CAN bus interception module; F3: Risk semantic layer based on dynamic environment map, which generates red flashing effect of high-risk area in the visualization interface of the output module by setting AR projection overlay unit.

8. The external vision system for forklift operation according to claim 1, characterized in that: The method for determining the key locations: G1: Based on the shelving layout diagram in the warehouse information dataset, the coordinate set of all shelving aisle intersection points is obtained by setting an aisle intersection detection algorithm; G2: Based on historical accident records in the problem information dataset, a set of coordinates for high-incidence areas with an accident frequency > 0.5 times / month is obtained by setting an accident frequency statistician; G3: Based on the coordinate set of the intersection points of shelf aisles and the coordinate set of accident-prone areas, a candidate key point set is obtained by setting an Euclidean distance clustering engine; G4: Based on the field of view parameters of the first vision device in the candidate keypoint set and device information dataset, the minimum installation position and angle of the first vision device are obtained by setting the field of view coverage optimizer.

9. The external vision system for forklift operation according to claim 1, characterized in that: The first vision device is a global monitoring industrial camera, the second vision device is a forklift multispectral anti-shake gimbal camera, the multimodal sensing unit includes a millimeter-wave radar, a UWB positioning module and an inertial measurement unit, and the AR projection overlay unit is a holographic waveguide head-up display.

Images