Loading and unloading positioning method, device and equipment based on eagle eye system and medium

Abstract

The embodiment of the invention discloses a loading and unloading positioning method, device and equipment based on an eagle-eye system and a medium, applied to the eagle-eye system, and the eagle-eye system comprises an input module, an identification module and an output module; comprising the steps that an input module collects point cloud data and image data through sensors in multiple directions; the recognition module recognizes detection targets of loading and unloading through a deep learning network model based on the image data, and the detection targets comprise trays, vehicles and obstacles; the identification module identifies the position information of the detection target based on the point cloud data; and the output module sends the position information of the detection target to the cloud, so that the cloud sends a corresponding loading and unloading instruction based on the position information of the detection target, and automatically completes a corresponding loading and unloading task based on the loading and unloading instruction.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and in particular to a method, device, equipment, and medium for positioning loading and unloading cargo based on an Hawkeye system. Background Art

[0002] With the rapid development of global trade and e-commerce, improving logistics efficiency has become a critical issue for major companies, ports, warehousing, and logistics centers. Traditional truck loading and unloading operations often rely on manual labor, which not only leads to low efficiency and waste of resources, but also easily leads to safety accidents due to improper operation. Therefore, how to use technology to improve loading and unloading efficiency, ensure safety, and achieve intelligent scheduling has become a key issue in the development of the logistics industry. Summary of the Invention

[0003] One or more embodiments of this specification provide a method, device, equipment and medium for positioning loading and unloading based on an Hawkeye system, which is used to solve the technical problems raised by the background technology.

[0004] One or more embodiments of this specification adopt the following technical solutions:

[0005] One or more embodiments of this specification provide a method for positioning cargo loading and unloading based on an Hawkeye system. The method is applied to the Hawkeye system, which includes an input module, a recognition module, and an output module; including:

[0006] The input module collects point cloud data and image data respectively through sensors in multiple directions;

[0007] The recognition module identifies detection targets for loading and unloading based on the image data through a deep learning network model, wherein the detection targets include pallets, vehicles, and obstacles;

[0008] The recognition module identifies the position information of the detection target based on the point cloud data;

[0009] The output module sends the location information of the detection target to the cloud, so that the cloud sends corresponding loading and unloading instructions based on the location information of the detection target and automatically completes the corresponding loading and unloading tasks based on the loading and unloading instructions.

[0010] It should be noted that the embodiments of this specification have the following beneficial effects through the above content:

[0011] Multimodal environmental perception improves operational safety: Multi-directional sensors simultaneously collect point cloud and image data to build a three-dimensional environmental perception network. Point cloud data accurately reconstructs three-dimensional spatial structures, while image data provides high-resolution visual information. The fusion of these two eliminates the blind spots of a single sensor. This technology enables real-time identification of pallets, vehicles, and dynamic obstacles (such as mobile equipment and personnel entering the area), effectively preventing collisions caused by blind spots during loading and unloading, and providing a safe operating boundary for automated equipment.

[0012] Highly adaptable object recognition driven by deep learning: This system utilizes a deep learning network model to handle complex scenarios, overcoming the environmental limitations of traditional algorithms. Trained with massive amounts of logistics scenario data, the system is capable of recognizing multiple pallet sizes (including damaged and deformed ones), different vehicle types (such as vans and flatbed trucks), and non-standard obstacles. This robust generalization allows the system to adapt to diverse storage environments, cargo stacking configurations, and changing lighting conditions, ensuring stable operation in complex working conditions.

[0013] Precise spatial positioning enables precise operation: Spatial registration technology using point cloud data and visual recognition enables centimeter-level precision positioning of target objects. This feature enables automated loading and unloading equipment (such as robotic arms and AGVs) to accurately calculate grasping points and motion trajectories, resolving the issues of cargo drops and equipment damage caused by positioning errors in traditional manual operations. This is particularly effective for handling unusually shaped cargo or densely stacked items, eliminating the efficiency losses caused by repeated adjustments.

[0014] Cloud-based collaborative scheduling optimizes resource utilization: After real-time location information is uploaded to the cloud, the system dynamically plans the task sequence and routing for loading and unloading equipment. By intelligently matching loading and unloading needs with equipment status, this reduces the idle waiting time of equipment under traditional models and enables coordinated scheduling of loading and unloading resources across platforms and regions. This global optimization capability is particularly useful in scenarios where multiple vehicles are operating in parallel, significantly improving turnover efficiency at loading and unloading ports.

[0015] Furthermore, the position information of the detection target is the coordinates of the pallet, and the recognition module identifies the position information of the detection target based on the point cloud data, including:

[0016] The recognition module performs a clustering operation on the point cloud data based on the type of the tray hole and the size of the tray as fitting rules to identify the coordinates of the tray.

[0017] It should be noted that the embodiments of this specification have the following beneficial effects through the above content:

[0018] By presetting the hole types (such as the number and arrangement of square and circular holes) and standard sizes as key recognition parameters, the point cloud clustering algorithm is equipped with physical feature verification capabilities. This design incorporates prior knowledge of pallet structure into data processing, effectively distinguishing between real pallets and similarly sized interference objects (such as stacked containers and equipment components), avoiding misjudgments due to similar point cloud shapes and ensuring that automated equipment only operates on compliant pallets.

[0019] Furthermore, the position information of the detection target is the coordinates of the vehicle;

[0020] The recognition module identifies the location information of the detection target based on the point cloud data, including:

[0021] The recognition module performs a clustering operation on the point cloud data based on the front size and body size of the vehicle as a fitting rule to identify the coordinates of the vehicle.

[0022] It should be noted that the embodiments of this specification have the following beneficial effects through the above content:

[0023] Using vehicle front dimensions (such as cab height and front bumper length) and standard body dimensions (such as wheelbase and cargo box length) as core recognition parameters, digital verification rules for vehicle physical characteristics are constructed. This mechanism accurately distinguishes target vehicles from other large obstacles (such as forklifts and temporary cargo piles) within a point cloud, avoiding false triggering due to similar volumes and ensuring that loading and unloading equipment only operates on compliant vehicles.

[0024] Furthermore, the position information of the detection target is the coordinates of the obstacle;

[0025] The recognition module identifies the location information of the detection target based on the point cloud data, including:

[0026] The recognition module performs a clustering operation on the point cloud data based on the type of the obstacle and the size of the obstacle as a fitting rule to identify the coordinates of the obstacle.

[0027] It should be noted that the embodiments of this specification have the following beneficial effects through the above content:

[0028] By simultaneously analyzing obstacle types (such as people, mobile equipment, and loose cargo) and physical dimensions (height and volume), a dynamic risk classification mechanism is established. The system can distinguish between low-risk static obstacles (such as fixed fences) and high-risk dynamic obstacles (such as shuttle trucks), providing differentiated obstacle avoidance strategies for automated equipment and avoiding the efficiency losses caused by the "one-size-fits-all" emergency stop approach used in traditional solutions.

[0029] At the same time, the dual rules of type semantics and size constraints can effectively identify unstructured obstacles (such as tilted and collapsed cargo boxes and temporary stacking tools). This technology breaks through the limitations of traditional geometric shape matching and achieves reliable detection of unconventional obstacles such as damaged shelves and irregular-shaped equipment through feature combination judgment, improving the system's adaptability in complex working conditions.

[0030] Furthermore, 3D dimensional analysis based on point cloud data can accurately calculate the spatial extent of obstacles in the loading and unloading area (such as the safe height below the overhead conveyor belt and the rotation radius of the mobile robot arm). Compared to two-dimensional planar obstacle avoidance solutions, this can prevent hidden collision risks in three-dimensional space (such as interference between high-level stackers and lifting forks), ensuring safety in multi-layer operation scenarios.

[0031] Furthermore, by clustering and updating obstacle coordinates in real time, the system can capture sudden obstacle displacements (such as personnel mistakenly entering the work area or AGVs temporarily changing their routes). This capability enables loading and unloading equipment to dynamically adjust their motion trajectory without interrupting the work process, resolving the system lock-up problem caused by sudden environmental changes in traditional solutions and maintaining operational continuity.

[0032] Furthermore, the Hawkeye system further includes a tracking module, and the method further includes:

[0033] The tracking module performs tracking smoothing processing on the position information of the detection target, so as to perform weighted smoothing processing on the position information of the detection target in multiple frames to obtain position information with stable time sequence.

[0034] It should be noted that the embodiments of this specification have the following beneficial effects through the above content:

[0035] Anti-transient interference ensures continuous operation: Multi-frame data fusion eliminates transient interference such as sensor noise and brief occlusion during single-frame detection, preventing sudden changes in target coordinates that could cause sudden equipment stops or path oscillations. This feature ensures continuous and stable operation of equipment such as loading and unloading robots and AGVs in harsh working conditions such as dust, rain, and fog.

[0036] Dynamic trajectory prediction improves control accuracy: Time series smoothing captures the motion trends of target objects (such as a moving forklift or goods on a conveyor belt) and uses a weighted algorithm to predict the appropriate position range at the next moment. This capability enables automated equipment to calculate grasping timing and motion compensation in advance, resolving the problem of inaccurate grasping caused by processing delays in traditional solutions.

[0037] Multi-target association enhances scene understanding: Based on trajectory correlation analysis of historical location information, it can intelligently distinguish different targets with similar appearances (such as pallets of the same model side by side). Through motion continuity verification, it effectively prevents loading and unloading command confusion caused by target ID jumps, ensuring the orderly operation of multiple targets in parallel.

[0038] Motion status visualization aids decision-making: Smoothed time-series position sequences generate target motion heat maps, visually reflecting hotspots of equipment activity and unusual movement patterns (such as a broken-down vehicle that remains for an extended period) within the loading and unloading area. This data layer provides a dynamic behavioral analysis basis for optimizing operational processes and adjusting equipment layouts.

[0039] Adaptive weight adjustment enhances robustness: Dynamically adjusts the smoothing algorithm's weight parameters based on target type (e.g., increasing the weight of recent frames for fast-moving forklifts and prioritizing historical frame stability for static pallets), enabling differentiated tracking strategies for moving and static targets. This mechanism balances tracking response speed with position stability, adapting to the dynamic changes of complex loading and unloading scenarios.

[0040] Furthermore, the Hawkeye system further includes a coordinate conversion module, and the method further includes:

[0041] The coordinate conversion module associates the position information of the detection target with a global map, where the global map is a point cloud map pre-established for a global operating area.

[0042] It should be noted that the embodiments of this specification have the following beneficial effects through the above content:

[0043] Unified spatial reference eliminates positioning ambiguity: By binding real-time detection coordinates to the global map, a unified spatiotemporal coordinate system is established for the entire work area. This mechanism eliminates coordinate system conflicts caused by multi-sensor perspective deviations (such as differences in local coordinate systems at different loading and unloading platforms), ensuring that positioning instructions received by actuators such as AGVs and robotic arms are globally consistent, and avoiding the risk of cross-border collisions caused by coordinate mismatches.

[0044] Dynamic environmental self-calibration enhances robustness: The global map serves as a spatial reference system, automatically compensating for coordinate drift caused by slow environmental changes such as equipment vibration and ground subsidence. Even if the work area undergoes local modifications (such as shelf relocation or platform expansion), the system can still achieve real-time coordinate correction by matching key landmarks, maintaining long-term operational stability.

[0045] Improved cross-regional collaborative scheduling capabilities: Global coordinate association enables equipment across loading and unloading ports and storage zones to share unified spatial semantics. The cloud-based scheduling system can coordinate cross-regional transfer routes for multiple AGVs based on global location information, eliminating blind spots in traditional segmented scheduling and achieving optimal resource allocation across the entire operation network.

[0046] Heterogeneous data fusion builds digital twins: Real-time coordinates are continuously linked to the global map, forming a dynamic mapping between physical space and digital models. This capability supports the spatial integration of multi-dimensional data such as loading and unloading equipment status, cargo location, and environmental risks, providing a high-fidelity data foundation for intelligent decision-making (such as congestion prediction and hotspot optimization) through virtual-physical integration.

[0047] Scalable architecture adapts to business evolution: The modular global map supports flexible, iterative updates. Adjustments to the work area layout (such as adding an automated warehouse) require only expanding the map coverage without reconfiguring the coordinate parsing algorithm. This feature enables the system to quickly respond to business expansion or process reconfiguration needs at logistics nodes.

[0048] Furthermore, the position information of the detection target is the coordinates of the pallet and the coordinates of the vehicle, and the recognition module identifies the position information of the detection target based on the point cloud data, including:

[0049] The recognition module performs a clustering operation on the point cloud data based on the front size and body size of the vehicle as a fitting rule to identify the coordinates of the vehicle;

[0050] The recognition module obtains the positional relationship between the vehicle and the pallet based on the current loading and unloading task;

[0051] The recognition module determines the coordinates of the pallet based on the positional relationship and the coordinates of the vehicle, and marks a designated area of ​​the pallet in the point cloud data based on the coordinates of the pallet;

[0052] The recognition module detects the number of point clouds in the designated area, and if the number of point clouds is lower than a preset threshold, re-determines the coordinates of the pallet based on the positional relationship and the coordinates of the vehicle.

[0053] It should be noted that the embodiments of this specification have the following beneficial effects through the above content:

[0054] Dynamic collaborative positioning improves recognition efficiency: By analyzing the relationship between vehicle coordinates and pallet positions, a "vehicle-to-pallet" positioning paradigm is established. Using the vehicle as a spatial reference point significantly reduces the pallet scanning range, avoiding the computational redundancy of traditional full-area point cloud traversal. This mechanism is particularly suitable for loading and unloading sites with densely packed vehicles, quickly locking onto the target pallet and reducing misidentification interference from adjacent vehicles' cargo piles.

[0055] Spatial relationship constraints enhance logical verification: Physical constraints such as the standard distance between the vehicle and the pallet, as well as the azimuth angle, are incorporated into coordinate calculations, creating a dual verification mechanism. Even if the pallet is tilted and the point cloud morphology is abnormal, logical correction can still be performed based on the relative position of the vehicle. This addresses the pain point of single-visual recognition being susceptible to cargo deformation and ensures the rationality of the automated equipment's grasping posture.

[0056] Incremental focused scanning optimizes resource allocation: After determining the theoretical coordinates of the pallet, it specifically marks designated areas for point cloud density detection. This strategy concentrates system computing resources on critical areas, ensuring positioning accuracy while reducing the overall data processing load. This intelligently balances detailed scanning of high-value areas with rapid filtering of background areas.

[0057] A self-correction mechanism addresses partial occlusions: When the point cloud of a designated area is insufficient (e.g., a pallet is partially obscured by a loading and unloading robot), the system automatically triggers a coordinate recalculation based on spatial relationships. This dynamic compensation capability overcomes the limitations of traditional rigid matching algorithms, ensuring continuous operation under non-ideal conditions such as misaligned cargo stacking and temporary occlusion.

[0058] Enhanced semantic understanding of loading and unloading scenarios: By correlating the positioning of vehicles and pallets, the system not only outputs discrete coordinates but also constructs the spatial topological relationship between vehicle and cargo. This capability provides the intelligent scheduling system with business-level semantic information such as loading and unloading progress and vehicle loading status, supporting the coordinated rhythm control of loading and unloading equipment and transport vehicles.

[0059] One or more embodiments of this specification provide a loading and unloading positioning device based on the Hawkeye system. The device is applied to the Hawkeye system. The Hawkeye system includes an input module, a recognition module, and an output module, including:

[0060] The acquisition unit, the input module respectively acquires point cloud data and image data through sensors in multiple directions;

[0061] A detection target recognition unit, wherein the recognition module recognizes detection targets for loading and unloading cargo based on the image data through a deep learning network model, wherein the detection targets include pallets, vehicles, and obstacles;

[0062] A position information recognition unit, wherein the recognition module recognizes the position information of the detection target based on the point cloud data;

[0063] The position information sending unit, the output module sends the position information of the detection target to the cloud, so that the cloud sends corresponding loading and unloading instructions based on the position information of the detection target, and automatically completes the corresponding loading and unloading tasks based on the loading and unloading instructions.

[0064] One or more embodiments of this specification provide a loading and unloading positioning device based on the Hawkeye system, which is applied to the Hawkeye system. The Hawkeye system includes an input module, a recognition module, and an output module, including:

[0065] at least one processor; and,

[0066] a memory communicatively connected to the at least one processor; wherein,

[0067] The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to:

[0068] The input module collects point cloud data and image data respectively through sensors in multiple directions;

[0069] The recognition module identifies detection targets for loading and unloading based on the image data through a deep learning network model, wherein the detection targets include pallets, vehicles, and obstacles;

[0070] The recognition module identifies the position information of the detection target based on the point cloud data;

[0071] The output module sends the location information of the detection target to the cloud, so that the cloud sends corresponding loading and unloading instructions based on the location information of the detection target and automatically completes the corresponding loading and unloading tasks based on the loading and unloading instructions.

[0072] One or more embodiments of this specification provide a non-volatile computer storage medium, which is applied to a Hawkeye system. The Hawkeye system includes an input module, a recognition module, and an output module, and stores computer-executable instructions. When executed by a computer, the computer-executable instructions can achieve the following:

[0073] The input module collects point cloud data and image data respectively through sensors in multiple directions;

[0074] The recognition module identifies detection targets for loading and unloading based on the image data through a deep learning network model, wherein the detection targets include pallets, vehicles, and obstacles;

[0075] The recognition module identifies the position information of the detection target based on the point cloud data;

[0076] The output module sends the location information of the detection target to the cloud, so that the cloud sends corresponding loading and unloading instructions based on the location information of the detection target and automatically completes the corresponding loading and unloading tasks based on the loading and unloading instructions.

[0077] At least one of the above technical solutions adopted in the embodiments of this specification can achieve the following beneficial effects:

[0078] Multimodal environmental perception improves operational safety: Multi-directional sensors simultaneously collect point cloud and image data to build a three-dimensional environmental perception network. Point cloud data accurately reconstructs three-dimensional spatial structures, while image data provides high-resolution visual information. The fusion of these two eliminates the blind spots of a single sensor. This technology enables real-time identification of pallets, vehicles, and dynamic obstacles (such as mobile equipment and personnel entering the area), effectively preventing collisions caused by blind spots during loading and unloading, and providing a safe operating boundary for automated equipment.

[0079] Highly adaptable object recognition driven by deep learning: This system utilizes a deep learning network model to handle complex scenarios, overcoming the environmental limitations of traditional algorithms. Trained with massive amounts of logistics scenario data, the system is capable of recognizing multiple pallet sizes (including damaged and deformed ones), different vehicle types (such as vans and flatbed trucks), and non-standard obstacles. This robust generalization allows the system to adapt to diverse storage environments, cargo stacking configurations, and changing lighting conditions, ensuring stable operation in complex working conditions.

[0080] Precise spatial positioning enables precise operation: Spatial registration technology using point cloud data and visual recognition enables centimeter-level precision positioning of target objects. This feature enables automated loading and unloading equipment (such as robotic arms and AGVs) to accurately calculate grasping points and motion trajectories, resolving the issues of cargo drops and equipment damage caused by positioning errors in traditional manual operations. This is particularly effective for handling unusually shaped cargo or densely stacked items, eliminating the efficiency losses caused by repeated adjustments.

[0081] Cloud-based collaborative scheduling optimizes resource utilization: After real-time location information is uploaded to the cloud, the system dynamically plans the task sequence and routing for loading and unloading equipment. By intelligently matching loading and unloading needs with equipment status, this reduces the idle waiting time of equipment under traditional models and enables coordinated scheduling of loading and unloading resources across platforms and regions. This global optimization capability is particularly useful in scenarios where multiple vehicles are operating in parallel, significantly improving turnover efficiency at loading and unloading ports. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the following briefly introduces the drawings required for the embodiments or the description of the prior art. Obviously, the drawings described below are only some of the embodiments described in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without inventive work. In the drawings:

[0083] Figure 1 A schematic flow chart of a method for positioning cargo loading and unloading based on the Hawkeye system provided in one or more embodiments of this specification;

[0084] Figure 2A functional structure diagram of the Hawkeye system provided for one or more embodiments of this specification;

[0085] Figure 3 A diagram of the Hawkeye vision function provided for one or more embodiments of this specification;

[0086] Figure 4 A schematic diagram of truck parking spots provided for one or more embodiments of this specification;

[0087] Figure 5 A schematic structural diagram of a loading and unloading positioning device based on the Hawkeye system provided in one or more embodiments of this specification;

[0088] Figure 6 A schematic structural diagram of a loading and unloading positioning device based on the Hawkeye system provided in one or more embodiments of this specification. DETAILED DESCRIPTION

[0089] The embodiments of this specification provide a method, device, equipment and medium for positioning loading and unloading based on the Hawkeye system.

[0090] To help those skilled in the art better understand the technical solutions in this specification, the following will provide a clear and complete description of the technical solutions in the embodiments of this specification, in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this specification without creative work should fall within the scope of protection of this specification.

[0091] Figure 1 This is a schematic diagram of a method for locating cargo for loading and unloading using the HawkEye system, provided in one or more embodiments of this specification. This process can be executed by the HawkEye system, which includes an input module, a recognition module, and an output module. Certain input parameters or intermediate results in the process can be manually adjusted to help improve accuracy.

[0092] The method steps of the embodiment of this specification are as follows:

[0093] S101, the input module collects point cloud data and image data respectively through sensors in multiple directions.

[0094] In the embodiments of this specification, regarding S101, the following specific implementation scheme can be adopted:

[0095] Sensor network deployment: LiDAR and high-definition cameras are deployed in multiple directions in the loading and unloading area to form a sensor matrix with multi-perspective coverage.

[0096] Synchronous trigger acquisition: The hardware synchronization signal is used to trigger each sensor to simultaneously collect point cloud data and RGB images, ensuring the alignment of spatiotemporal data.

[0097] Data preprocessing: Perform noise reduction on the original point cloud (such as removing rain and snow noise), perform lighting equalization on the image, and improve the quality of subsequent processing.

[0098] S102, the recognition module identifies detection targets for loading and unloading based on the image data through a deep learning network model, and the detection targets include pallets, vehicles and obstacles.

[0099] In the embodiments of this specification, regarding S102, the following specific implementation scheme can be adopted:

[0100] Model loading and inference: Load a pre-trained multi-target detection neural network (such as the improved YOLO architecture) and cut the image into a grid for parallel processing.

[0101] Target classification and filtering: The classification branch outputs the category labels and confidence levels of pallets, vehicles, and obstacles, and filters low-confidence targets (such as blurred objects in the distance).

[0102] Abnormal state identification: For the pallet detection branch, abnormal state identification such as damage and tilt is output synchronously to provide prior knowledge for subsequent point cloud processing.

[0103] S103: The recognition module recognizes the position information of the detection target based on the point cloud data.

[0104] In the embodiments of this specification, regarding S103, the following specific implementation scheme can be adopted:

[0105] Data association and registration: Map the S102 visual detection frame to the point cloud coordinate system, extract the point cloud subset of the corresponding area, and reduce the invalid calculation range.

[0106] Hierarchical feature clustering:

[0107] Vehicle positioning: Initial clustering is performed based on the vehicle's front dimensions (such as the windshield inclination angle), followed by secondary verification based on the vehicle's body length, and the vehicle's center coordinates and heading angle are output.

[0108] Pallet positioning: Based on the pallet size, the pallet center coordinates are output through clustering.

[0109] Obstacle monitoring: Density clustering is performed on non-vehicle / pallet point cloud areas, and size thresholds and motion continuity analysis are combined to distinguish fixed obstacles from dynamic risk targets.

[0110] S104, the output module sends the location information of the detection target to the cloud, so that the cloud sends corresponding loading and unloading instructions based on the location information of the detection target, and automatically completes the corresponding loading and unloading tasks based on the loading and unloading instructions.

[0111] In the embodiments of this specification, regarding S104, the following specific implementation scheme can be adopted:

[0112] Data encapsulation and transmission: The target coordinates, type, and timestamp are encapsulated into a standardized JSON message and uploaded to the cloud control center via the 5G private network with encryption.

[0113] Dynamic task allocation: The cloud calculates the optimal loading and unloading equipment (such as the nearest idle AGV) and path based on real-time coordinates, and generates an instruction set including grasping posture and movement trajectory.

[0114] Equipment linkage control: After receiving the command, the loading and unloading robot starts the safety verification process (such as obstacle distance recalculation). After confirming that it is correct, it executes actions such as pallet grabbing and cargo stacking, and sends back the execution status in real time.

[0115] Abnormal fuse mechanism: When the deviation between the device feedback coordinates and the actual point cloud exceeds the limit, the operation is immediately suspended and a system-level alarm is triggered, and environmental perception and path planning are reinitiated by the cloud.

[0116] It should be noted that the embodiments of this specification have the following beneficial effects through the above content:

[0117] Multimodal environmental perception improves operational safety: Multi-directional sensors simultaneously collect point cloud and image data to build a three-dimensional environmental perception network. Point cloud data accurately reconstructs three-dimensional spatial structures, while image data provides high-resolution visual information. The fusion of these two eliminates the blind spots of a single sensor. This technology enables real-time identification of pallets, vehicles, and dynamic obstacles (such as mobile equipment and personnel entering the area), effectively preventing collisions caused by blind spots during loading and unloading, and providing a safe operating boundary for automated equipment.

[0118] Highly adaptable object recognition driven by deep learning: This system utilizes a deep learning network model to handle complex scenarios, overcoming the environmental limitations of traditional algorithms. Trained with massive amounts of logistics scenario data, the system is capable of recognizing multiple pallet sizes (including damaged and deformed ones), different vehicle types (such as vans and flatbed trucks), and non-standard obstacles. This robust generalization allows the system to adapt to diverse storage environments, cargo stacking configurations, and changing lighting conditions, ensuring stable operation in complex working conditions.

[0119] Precise spatial positioning enables precise operation: Spatial registration technology using point cloud data and visual recognition enables centimeter-level precision positioning of target objects. This feature enables automated loading and unloading equipment (such as robotic arms and AGVs) to accurately calculate grasping points and motion trajectories, resolving the issues of cargo drops and equipment damage caused by positioning errors in traditional manual operations. This is particularly effective for handling unusually shaped cargo or densely stacked items, eliminating the efficiency losses caused by repeated adjustments.

[0120] Cloud-based collaborative scheduling optimizes resource utilization: After real-time location information is uploaded to the cloud, the system dynamically plans the task sequence and routing for loading and unloading equipment. By intelligently matching loading and unloading needs with equipment status, this reduces the idle waiting time of equipment under traditional models and enables coordinated scheduling of loading and unloading resources across platforms and regions. This global optimization capability is particularly useful in scenarios where multiple vehicles are operating in parallel, significantly improving turnover efficiency at loading and unloading ports.

[0121] Furthermore, the position information of the detection target is the coordinates of the pallet. When the recognition module identifies the position information of the detection target based on the point cloud data, the recognition module can perform clustering operations on the point cloud data based on the type of the pallet hole and the size of the pallet as fitting rules to identify the coordinates of the pallet.

[0122] In the examples of this specification, the above content can be implemented by the following specific implementation schemes:

[0123] Pallet feature rule library construction: A pre-built ISO pallet standard parameter library includes feature templates for hole spacing, hole diameter, and arrangement patterns for different pallet hole types (e.g., European standard 6-hole, American standard 9-hole, etc.). A dimensional rule tree is established to define the standard ranges and tolerance thresholds for pallet length, width, and height, compatible with the deformation tolerance of wooden and plastic pallets.

[0124] Point cloud data preprocessing: Voxel filtering is performed on the original point cloud to reduce noise and remove flying points (such as dust and rain noise). Through direct filtering, the point cloud is intercepted in the height range of 0.1-1.5 meters above the ground to filter out ground and high-altitude interference objects.

[0125] Hole Feature Matching: Euclidean clustering is performed on the denoised point cloud to isolate candidate areas suspected of being pallets. Plane fitting is performed on each candidate area, and the upper surface point cloud is extracted and projected onto a 2D plane. Circular and square holes are detected using a Hough transform, and matched against pre-set hole templates to verify that the number and spacing of holes conform to the rule base.

[0126] Size rule verification: For candidate areas that pass the hole verification, calculate their 3D bounding box dimensions (length × width × height). Compare them with the standard size range in the rule library and eliminate out-of-tolerance candidate areas (such as misjudged cargo boxes).

[0127] Spatial coordinate calculation: For a pallet point cloud cluster that has passed dual verification, its geometric center point is calculated as the initial coordinate. Corner point detection is performed based on the pallet hole distribution pattern (e.g., four-corner symmetrical holes), and the center coordinate is corrected to the actual pallet gripping position.

[0128] Dynamic compensation mechanism: If a pallet tilt is detected (the point cloud plane normal vector deviates from the vertical direction), the coordinates of the actual pallet support points are inferred based on the position of the support holes. If the point cloud density is insufficient, the coordinates of the missing areas are interpolated based on historical loading and unloading data.

[0129] Logical closed-loop verification: Map the output coordinates back to the original image data to verify the spatial consistency between the visual inspection box and the point cloud coordinates. If the discrepancy exceeds the limit, a partial rescan of the point cloud and fine-tuning of the rule base parameters are triggered.

[0130] It should be noted that the embodiments of this specification have the following beneficial effects through the above content:

[0131] By presetting the hole types (such as the number and arrangement of square and circular holes) and standard sizes as key recognition parameters, the point cloud clustering algorithm is equipped with physical feature verification capabilities. This design incorporates prior knowledge of pallet structure into data processing, effectively distinguishing between real pallets and similarly sized interference objects (such as stacked containers and equipment components), avoiding misjudgments due to similar point cloud shapes and ensuring that automated equipment only operates on compliant pallets.

[0132] Furthermore, the position information of the detection target is the coordinates of the vehicle; when the recognition module identifies the position information of the detection target based on the point cloud data, the recognition module can perform a clustering operation on the point cloud data based on the front size and body size of the vehicle as fitting rules to identify the coordinates of the vehicle.

[0133] In the examples of this specification, the above content can be implemented by the following specific implementation schemes:

[0134] Vehicle feature rule library construction: A pre-built parameter library for mainstream vehicle models includes key vehicle front dimensions (such as windshield height and bumper width) and standard body parameters (such as wheelbase and cargo box length), compatible with vehicle standards in different countries (such as European trucks and American trailers). Spatial association rules are established to define relative orientation constraints between the vehicle front and body (such as the front-to-back distance threshold between the cab and cargo box).

[0135] Point cloud data preprocessing: Statistical filtering is performed on the original point cloud to remove discrete noise points (such as flying birds and floating objects). Through-through filtering, point clouds are retained within the 0.5-4 meter height range above the ground, filtering out ground reflections and interference from overhead hoisting equipment.

[0136] Vehicle head feature matching: Euclidean clustering is performed on the preprocessed point cloud to isolate candidate regions. An oriented bounding box (OBB) is fitted to each candidate region to extract its length, width, and height parameters. The candidate clusters that match the windshield tilt angle (e.g., 50°-70°) and vehicle head height (2-3.5 meters) are selected and marked as the initial vehicle head location.

[0137] Body Correlation Verification: Using the vehicle's head coordinates as a reference, the scanning area is expanded along the vehicle's direction of travel. The point cloud continuity of the extended area is checked, and compliance with cargo box / trailer standards (e.g., 6-16 meters) is verified by length and width. Isolated vehicle heads (e.g., tractor heads without cargo boxes) are eliminated to ensure vehicle-body integrity.

[0138] Spatial coordinate calculation: For the verified vehicle point cloud cluster, calculate its geometric center as the basic coordinate.

[0139] Combining the vehicle head orientation angle (calculated by the windshield plane normal vector) with the cargo box geometric center, the vehicle parking azimuth and loading and unloading port coordinates are output.

[0140] Dynamic Compensation Mechanism: When a part of the vehicle body point cloud is missing (e.g., the side of the cargo box is obscured), the vehicle body coordinates are interpolated based on the vehicle head dimensions and historical vehicle model data. If multiple vehicles are detected parked side by side, adjacent vehicles are distinguished based on headway spacing rules (e.g., >1.5 meters) to avoid cluster confusion.

[0141] It should be noted that the embodiments of this specification have the following beneficial effects through the above content:

[0142] Using vehicle front dimensions (such as cab height and front bumper length) and standard body dimensions (such as wheelbase and cargo box length) as core recognition parameters, digital verification rules for vehicle physical characteristics are constructed. This mechanism accurately distinguishes target vehicles from other large obstacles (such as forklifts and temporary cargo piles) within a point cloud, avoiding false triggering due to similar volumes and ensuring that loading and unloading equipment only operates on compliant vehicles.

[0143] Furthermore, the position information of the detection target is the coordinates of the obstacle; when the recognition module identifies the position information of the detection target based on the point cloud data, the recognition module performs a clustering operation on the point cloud data based on the type of the obstacle and the size of the obstacle as fitting rules to identify the coordinates of the obstacle.

[0144] In the examples of this specification, the above content can be implemented by the following specific implementation schemes:

[0145] Obstacle feature rule library construction: A pre-built parameter library of typical obstacles in logistics scenarios is included, including type labels, typical size ranges (e.g., human height 0.5-2 meters, forklift width 1-1.5 meters), and shape characteristics (e.g., cylindrical columns, rectangular cargo boxes) for both static obstacles (e.g., columns, shelves) and dynamic obstacles (e.g., people, forklifts). Motion feature rules for dynamic obstacles (e.g., forklift speed range, continuity of people's movement trajectories) are defined to distinguish instantaneous noise from real obstacles.

[0146] Point cloud data preprocessing: Ground segmentation is performed on the original point cloud, ground reflection point clouds are removed, and the area above 0.2 meters above the ground is retained. Radius filtering is applied to eliminate isolated noise points (such as flying insects and floating debris) and retain continuous point cloud clusters.

[0147] Initial screening of dynamic / static obstacles: Perform Euclidean clustering on non-vehicle / pallet point cloud areas to separate independent point cloud clusters. Based on the motion trajectory analysis of point cloud clusters (compared with historical frame data), mark dynamic / static attributes:

[0148] Static obstacles: Match the bounding box size with the rule library (e.g., column diameter > 0.3 meters);

[0149] Dynamic obstacles: through speed estimation and motion continuity verification (such as personnel movement speed <2m / s).

[0150] Type matching and logic verification: Shape feature extraction: perform normal vector analysis and curvature calculation on point cloud clusters to identify geometric features (such as planes and cylinders).

[0151] Column: Verify radial symmetry by cylindrical fitting;

[0152] Cargo box: Verify the right-angle edge features through cuboid fitting;

[0153] People: Identified by point cloud height stratification (spherical features of head and shoulders + columnar features of legs).

[0154] Size rule filtering: Eliminate false positive clusters that exceed the preset type size threshold (such as misclassifying a large cargo box as a small package).

[0155] Space coordinate solution:

[0156] Static obstacles: Calculate the geometric center coordinates of the point cloud cluster, superimpose the bounding box size, and output the safety boundary.

[0157] Dynamic obstacles: Output real-time position and motion direction vector through multi-frame point cloud trajectory prediction.

[0158] Special scene processing: For partially occluded obstacles, coordinates are completed based on the point cloud interpolation of the visible area; for dense small obstacles (such as scattered goods), high-density clustering mode is enabled to segment independent individuals.

[0159] Dynamic safety boundary generation: Based on obstacle types, preset safety distance rules are matched (e.g., a 1-meter safety zone for personnel, a 2-meter operating area for forklifts). Real-time coordinates are superimposed to generate a 3D safety heat map, marking dangerous areas where equipment is prohibited from entering.

[0160] It should be noted that the embodiments of this specification have the following beneficial effects through the above content:

[0161] By simultaneously analyzing obstacle types (such as people, mobile equipment, and loose cargo) and physical dimensions (height and volume), a dynamic risk classification mechanism is established. The system can distinguish between low-risk static obstacles (such as fixed fences) and high-risk dynamic obstacles (such as shuttle trucks), providing differentiated obstacle avoidance strategies for automated equipment and avoiding the efficiency losses caused by the "one-size-fits-all" emergency stop approach used in traditional solutions.

[0162] At the same time, the dual rules of type semantics and size constraints can effectively identify unstructured obstacles (such as tilted and collapsed cargo boxes and temporary stacking tools). This technology breaks through the limitations of traditional geometric shape matching and achieves reliable detection of unconventional obstacles such as damaged shelves and irregular-shaped equipment through feature combination judgment, improving the system's adaptability in complex working conditions.

[0163] Furthermore, 3D dimensional analysis based on point cloud data can accurately calculate the spatial extent of obstacles in the loading and unloading area (such as the safe height below the overhead conveyor belt and the rotation radius of the mobile robot arm). Compared to two-dimensional planar obstacle avoidance solutions, this can prevent hidden collision risks in three-dimensional space (such as interference between high-level stackers and lifting forks), ensuring safety in multi-layer operation scenarios.

[0164] Furthermore, by clustering and updating obstacle coordinates in real time, the system can capture sudden obstacle displacements (such as personnel mistakenly entering the work area or AGVs temporarily changing their routes). This capability enables loading and unloading equipment to dynamically adjust their motion trajectory without interrupting the work process, resolving the system lock-up problem caused by sudden environmental changes in traditional solutions and maintaining operational continuity.

[0165] Furthermore, the Hawkeye system also includes a tracking module, which performs tracking and smoothing processing on the position information of the detection target, so as to perform weighted smoothing processing on the position information of the detection target in multiple frames to obtain time-stable position information.

[0166] In the examples of this specification, the above content can be implemented by the following specific implementation schemes:

[0167] Multi-target tracker initialization: When a detection target (pallet / vehicle / obstacle) is first identified, an independent tracking ID is generated based on its type and initial coordinates, and a dedicated status queue is created.

[0168] 1. Bind dynamic weight parameters to the target

[0169] Static targets (such as pillars): Use low decay weights to enhance the persistence of historical location data;

[0170] Dynamic targets (such as forklifts): Use high sensitivity weights to prioritize responding to new coordinate changes.

[0171] Sliding window weighted smoothing: Construct a time series sliding window (such as the latest 5 frames of data) and arrange the target coordinate sequence in reverse order by timestamp.

[0172] 2. Apply exponential decay weight distribution

[0173] The current frame has the highest weight, which decreases frame by frame (e.g., weight coefficient 0.5^t), and the weighted average coordinates are calculated.

[0174] Superimpose velocity prediction compensation for dynamic targets: Based on the displacement vector of the previous frame, predict the theoretical position of the current frame and merge it with the detected coordinates.

[0175] 3. Motion continuity check

[0176] Static target: Verify the offset between the smoothed coordinates and the historical reference position. If the offset exceeds the limit (e.g., >10cm), trigger environmental re-sensing to eliminate interference from building structure changes or sensor drift.

[0177] Dynamic target: Through speed-acceleration continuity analysis (such as the instantaneous speed of the forklift cannot exceed 5m / s 2 ) to eliminate abnormal jitter coordinates (such as false detection points caused by flying insects).

[0178] 4. Multi-target trajectory association

[0179] When an object reappears after a brief occlusion, the new and old IDs are matched based on motion trend consistency (such as the rate of change of angular direction) and spatial proximity to avoid duplicate tracker creation. For densely packed object clusters (such as side-by-side pallets), independent tracking is maintained through the spatial topology (adjacent / inclusive) of the bounding box to prevent ID jumps.

[0180] 5. Adaptive parameter adjustment

[0181] Dynamically adjusts smoothing strength based on target type:

[0182] For targets requiring high precision (such as pallet grasping points): reduce the sliding window (3 frames) and reduce the smoothing force to preserve subtle pose changes;

[0183] For low-sensitivity targets (such as far-field obstacles), increase the window size (10 frames) and enhance noise suppression. When the handling equipment reports execution errors (such as grasping offset), reversely optimize the smoothing parameters (such as the weight decay coefficient).

[0184] 6. Tracking lifecycle management

[0185] Tracking continuation condition: If the target is not detected for three consecutive frames and there is no associated logic (for example, the vehicle has not left the loading and unloading area), tracking is terminated after a delay of three frames and resources are released.

[0186] Cross-region tracking relay: When the target moves out of the current sensor coverage area, its coordinates and motion vector are uploaded to the cloud, triggering the tracking takeover of the adjacent Hawkeye system.

[0187] It should be noted that the embodiments of this specification have the following beneficial effects through the above content:

[0188] Anti-transient interference ensures continuous operation: Multi-frame data fusion eliminates transient interference such as sensor noise and brief occlusion during single-frame detection, preventing sudden changes in target coordinates that could cause sudden equipment stops or path oscillations. This feature ensures continuous and stable operation of equipment such as loading and unloading robots and AGVs in harsh working conditions such as dust, rain, and fog.

[0189] Dynamic trajectory prediction improves control accuracy: Time series smoothing captures the motion trends of target objects (such as a moving forklift or goods on a conveyor belt) and uses a weighted algorithm to predict the appropriate position range at the next moment. This capability enables automated equipment to calculate grasping timing and motion compensation in advance, resolving the problem of inaccurate grasping caused by processing delays in traditional solutions.

[0190] Multi-target association enhances scene understanding: Based on trajectory correlation analysis of historical location information, it can intelligently distinguish different targets with similar appearances (such as pallets of the same model side by side). Through motion continuity verification, it effectively prevents loading and unloading command confusion caused by target ID jumps, ensuring the orderly operation of multiple targets in parallel.

[0191] Motion status visualization aids decision-making: Smoothed time-series position sequences generate target motion heat maps, visually reflecting hotspots of equipment activity and unusual movement patterns (such as a broken-down vehicle that remains for an extended period) within the loading and unloading area. This data layer provides a dynamic behavioral analysis basis for optimizing operational processes and adjusting equipment layouts.

[0192] Adaptive weight adjustment enhances robustness: Dynamically adjusts the smoothing algorithm's weight parameters based on target type (e.g., increasing the weight of recent frames for fast-moving forklifts and prioritizing historical frame stability for static pallets), enabling differentiated tracking strategies for moving and static targets. This mechanism balances tracking response speed with position stability, adapting to the dynamic changes of complex loading and unloading scenarios.

[0193] Furthermore, the Hawkeye system also includes a coordinate conversion module, which associates the location information of the detection target with a global map, where the global map is a point cloud map pre-established for the global operating area.

[0194] In the examples of this specification, the above content can be implemented by the following specific implementation schemes:

[0195] 1. Global map construction and calibration

[0196] Mobile laser scanning equipment is used to collect multi-perspective point clouds across the entire loading and unloading area, covering key nodes such as platforms, aisles, and shelf areas.

[0197] SLAM technology is used to perform spatiotemporal registration of discrete point cloud frames to generate a three-dimensional reference map with an absolute coordinate system (including ground elevation, column coordinates, loading and unloading port posture, etc.).

[0198] The benchmark map is subjected to noise reduction and semantic annotation, and logical areas such as loading and unloading areas, passage areas, and restricted areas are divided to form a digital operation sandbox.

[0199] 2. Multi-source coordinate system alignment

[0200] The sensors (lidar, camera) of the Hawkeye system are jointly calibrated to determine their installation offset and attitude angle relative to the origin of the global map.

[0201] Establish a transformation matrix from the sensor's local coordinate system to the global coordinate system to ensure that the real-time detection coordinates can be mapped to a unified spatial reference system.

[0202] 3. Real-time point cloud matching and pose calculation

[0203] Perform ICP (Iterative Closest Point) matching on the current frame point cloud and the reference map to calculate the real-time pose of the Hawkeye system itself (because the device may be installed on a mobile lifting mechanism).

[0204] Based on the pose correction, the position information of the detection target (pallet / vehicle / obstacle coordinates) is converted from the sensor coordinate system to the global coordinate system.

[0205] 4. Dynamic map incremental update

[0206] Compare the real-time point cloud with the baseline map, detect new fixed obstacles (such as temporarily stacked cargo boxes), and add their geometric features to the dynamic layer.

[0207] The real-time coordinates of movable targets (such as AGVs and forklifts) are retained without modifying the baseline map, maintaining the stability of the static layer of the map.

[0208] 5. Business logic space association

[0209] Bind the target's global coordinates to the loading and unloading task logic: pallet coordinates are associated with preset storage location numbers, triggering "cargo-storage location" binding; vehicle coordinates are mapped to platform partitions, assigning corresponding loading and unloading equipment service domains; obstacle coordinates are superimposed on safety electronic fences to generate equipment obstacle avoidance paths.

[0210] 6. Cross-system coordinate synchronization

[0211] The target position in the global coordinate system is converted into a data format that can be recognized by heterogeneous systems such as the AGV navigation coordinate system and the robotic arm joint coordinate system.

[0212] The timestamp alignment mechanism ensures that the coordinate information received by multiple devices has temporal consistency.

[0213] It should be noted that the embodiments of this specification have the following beneficial effects through the above content:

[0214] Unified spatial reference eliminates positioning ambiguity: By binding real-time detection coordinates to the global map, a unified spatiotemporal coordinate system is established for the entire work area. This mechanism eliminates coordinate system conflicts caused by multi-sensor perspective deviations (such as differences in local coordinate systems at different loading and unloading platforms), ensuring that positioning instructions received by actuators such as AGVs and robotic arms are globally consistent, and avoiding the risk of cross-border collisions caused by coordinate mismatches.

[0215] Dynamic environmental self-calibration enhances robustness: The global map serves as a spatial reference system, automatically compensating for coordinate drift caused by slow environmental changes such as equipment vibration and ground subsidence. Even if the work area undergoes local modifications (such as shelf relocation or platform expansion), the system can still achieve real-time coordinate correction by matching key landmarks, maintaining long-term operational stability.

[0216] Improved cross-regional collaborative scheduling capabilities: Global coordinate association enables equipment across loading and unloading ports and storage zones to share unified spatial semantics. The cloud-based scheduling system can coordinate cross-regional transfer routes for multiple AGVs based on global location information, eliminating blind spots in traditional segmented scheduling and achieving optimal resource allocation across the entire operation network.

[0217] Heterogeneous data fusion builds digital twins: Real-time coordinates are continuously linked to the global map, forming a dynamic mapping between physical space and digital models. This capability supports the spatial integration of multi-dimensional data such as loading and unloading equipment status, cargo location, and environmental risks, providing a high-fidelity data foundation for intelligent decision-making (such as congestion prediction and hotspot optimization) through virtual-physical integration.

[0218] Scalable architecture adapts to business evolution: The modular global map supports flexible, iterative updates. Adjustments to the work area layout (such as adding an automated warehouse) require only expanding the map coverage without reconfiguring the coordinate parsing algorithm. This feature enables the system to quickly respond to business expansion or process reconfiguration needs at logistics nodes.

[0219] Furthermore, the position information of the detection target is the coordinates of the pallet and the coordinates of the vehicle. When the recognition module identifies the position information of the detection target based on the point cloud data, the recognition module performs a clustering operation on the point cloud data based on the front size and body size of the vehicle as fitting rules to identify the coordinates of the vehicle; the recognition module obtains the positional relationship between the vehicle and the pallet based on the current loading and unloading task; the recognition module determines the coordinates of the pallet based on the positional relationship and the coordinates of the vehicle, and marks the designated area of ​​the pallet in the point cloud data based on the coordinates of the pallet; the recognition module detects the number of point clouds in the designated area, and if the number of point clouds is lower than a preset threshold, the coordinates of the pallet are re-determined based on the positional relationship and the coordinates of the vehicle.

[0220] In the examples of this specification, the above content can be implemented by the following specific implementation schemes:

[0221] 1. Construction of vehicle feature rule base and loading and unloading logic presetting

[0222] Establish a vehicle parameter library to define the key dimensions of the front of the vehicle (windshield height, bumper width) and the body length range (including trailer / van differences).

[0223] Pre-set loading and unloading direction rule library, such as side loading and unloading (the pallet is located within 1 meter on the left side of the vehicle body), tail loading and unloading (the pallet is located directly behind the rear of the vehicle), and other scenario-related logic.

[0224] 2. Point cloud preprocessing and vehicle positioning

[0225] Perform statistical filtering on the original point cloud to remove flying point noise.

[0226] Vehicle head recognition: Candidate areas are separated through Euclidean clustering, target clusters are filtered using vehicle head size rules (such as windshield inclination angle of 50°-70°), and the center coordinates and heading angle of the vehicle head are output.

[0227] Body association: Expand the scanning area along the vehicle's heading angle, match continuous point cloud clusters based on the vehicle's length rule (e.g., 6-16 meters), and calculate the vehicle's geometric center coordinates.

[0228] 3. Dynamic calculation of pallet area

[0229] Retrieve the preset position relationship based on the current loading and unloading task type:

[0230] Side loading and unloading tasks: Based on the center of the vehicle, the pallet scanning area is defined by extending the preset offset (e.g. 1.5 meters) to the left / right.

[0231] Tail loading and unloading task: Define a rectangular scanning area extending 2-3 meters along the rear of the vehicle.

[0232] Perform high-density point cloud clustering within the estimated area to screen candidate clusters that meet the pallet height (0.1-1.2 meters).

[0233] 4. Precise positioning and calibration of pallets

[0234] Perform hole feature matching on the candidate clusters (such as the four-corner circular point cloud distribution) and output the coordinates of the pallet center.

[0235] Point cloud density detection: A 0.5m×0.5m cubic area is defined around the pallet coordinates and the number of valid points is counted.

[0236] If the number of points meets the requirement: mark it as a valid pallet and output the coordinates.

[0237] If the number of points is insufficient: it is determined to be blocked or offset, and the estimated area is expanded according to the loading and unloading direction rules (such as expanding the side loading and unloading area to a range of 2 meters) and rescanning.

[0238] 5. Dynamic compensation mechanism

[0239] If recalculation still fails to locate the vehicle, historical data review is triggered: the relative positions of the pallets in the same vehicle's historical loading and unloading tasks are compared, and the current heading angle of the vehicle is superimposed to fine-tune the estimated area.

[0240] If multiple pallets are detected side by side, separate the independent individuals using bounding box spacing rules (e.g., >0.3 meters) to avoid cluster adhesion.

[0241] 6.Logical closed loop verification

[0242] Reverse-map the pallet point cloud coordinates to the visual inspection frame to verify spatial consistency (e.g., whether the pallet is within the vehicle's associated area).

[0243] When the vehicle-pallet logical relationship conflicts (e.g., the pallet coordinates exceed the preset range of the loading and unloading direction), local point cloud re-collection and rule base parameter self-checking are triggered.

[0244] It should be noted that the embodiments of this specification have the following beneficial effects through the above content:

[0245] Dynamic collaborative positioning improves recognition efficiency: By analyzing the relationship between vehicle coordinates and pallet positions, a "vehicle-to-pallet" positioning paradigm is established. Using the vehicle as a spatial reference point significantly reduces the pallet scanning range, avoiding the computational redundancy of traditional full-area point cloud traversal. This mechanism is particularly suitable for loading and unloading sites with densely packed vehicles, quickly locking onto the target pallet and reducing misidentification interference from adjacent vehicles' cargo piles.

[0246] Spatial relationship constraints enhance logical verification: Physical constraints such as the standard distance between the vehicle and the pallet, as well as the azimuth angle, are incorporated into coordinate calculations, creating a dual verification mechanism. Even if the pallet is tilted and the point cloud morphology is abnormal, logical correction can still be performed based on the relative position of the vehicle. This addresses the pain point of single-visual recognition being susceptible to cargo deformation and ensures the rationality of the automated equipment's grasping posture.

[0247] Incremental focused scanning optimizes resource allocation: After determining the theoretical coordinates of the pallet, it specifically marks designated areas for point cloud density detection. This strategy concentrates system computing resources on critical areas, ensuring positioning accuracy while reducing the overall data processing load. This intelligently balances detailed scanning of high-value areas with rapid filtering of background areas.

[0248] A self-correction mechanism addresses partial occlusions: When the point cloud of a designated area is insufficient (e.g., a pallet is partially obscured by a loading and unloading robot), the system automatically triggers a coordinate recalculation based on spatial relationships. This dynamic compensation capability overcomes the limitations of traditional rigid matching algorithms, ensuring continuous operation under non-ideal conditions such as misaligned cargo stacking and temporary occlusion.

[0249] Enhanced semantic understanding of loading and unloading scenarios: By correlating the positioning of vehicles and pallets, the system not only outputs discrete coordinates but also constructs the spatial topological relationship between vehicle and cargo. This capability provides the intelligent scheduling system with business-level semantic information such as loading and unloading progress and vehicle loading status, supporting the coordinated rhythm control of loading and unloading equipment and transport vehicles.

[0250] It's important to note that with the rapid development of global trade and e-commerce, improving logistics efficiency has become a critical issue for major companies, ports, warehousing, and logistics centers. Traditional truck loading and unloading operations often rely on manual labor, which not only leads to low efficiency and waste of resources, but also easily leads to safety accidents due to improper operation. Therefore, how to use technology to improve loading and unloading efficiency, ensure safety, and achieve intelligent scheduling has become a key issue in the development of the logistics industry.

[0251] Against this backdrop, with the advancement of the Internet of Things, big data, artificial intelligence, and automation, many companies are exploring the use of these technologies to transform truck loading and unloading operations into intelligent systems. Technologies such as unmanned forklifts, robotic automation systems, and intelligent dispatching systems are gradually being applied to the logistics industry, driving the automation, digitization, and intelligentization of loading and unloading operations.

[0252] In addition, with the development of technologies such as intelligent transportation systems (ITS), cloud computing, and edge computing, more and more logistics companies and industries are accelerating their development towards intelligence and automation, aiming to reduce costs, improve efficiency, and enhance operational safety.

[0253] With the continuous advancement of industrialization, truck loading and unloading plays a vital role in material transportation and loading and unloading operations. In traditional truck loading and unloading operations, there are several problems:

[0254] (1) Low recognition accuracy: In the existing technology, traditional material recognition systems usually rely on traditional image processing algorithms or manual methods to identify materials. The recognition accuracy is not high, and the adaptability to materials of different shapes and sizes in automated processing is weak.

[0255] (2) Low loading and unloading efficiency: Existing loading and unloading operations usually rely on manual forklift operations, with a long operation cycle. Especially when the quantity of goods is large and the operations are complicated, the efficiency is low, which easily leads to waste of resources and shortage of personnel.

[0256] (3) High safety risk: During manual operation, workers are easily injured by collisions or improper operation during cargo handling, and improper operation of mechanical equipment also increases safety hazards.

[0257] (4) Low level of intelligence: In the existing system, the scheduling of most truck loading and unloading tasks relies on manual arrangements. Although some warehouse management systems have introduced robots and automation technologies, they lack intelligent optimization scheduling and resource management, cannot fully improve operational efficiency, and are prone to process errors.

[0258] (5) Insufficient real-time monitoring: Under the current operation mode, monitoring mostly relies on manual intervention, making it difficult to grasp the operation progress and cargo status in real time, resulting in information lag and affecting timely decision-making.

[0259] Therefore, there is an urgent need for a system that can monitor in real time, perform intelligent analysis, and provide optimization suggestions to improve the safety and efficiency of loading and unloading.

[0260] 1. Efficient and accurate material identification system: By combining deep learning algorithms with visual recognition technology and laser point cloud post-fusion processing technology, the present invention can automatically identify the type, size and position of materials and their pallets, providing high-precision 3D target recognition information, effectively reducing manual intervention.

[0261] 2. Improve loading and unloading efficiency: By automating the loading and unloading process, manual intervention is reduced, and the efficiency of truck loading and unloading operations is significantly improved. Especially in scenarios with large quantities of goods, it can effectively reduce operation time and labor intensity.

[0262] 3. Improve safety: Automatically transport goods through unmanned forklifts and robots to avoid injuries that may occur during manual operation, reduce operational risks, and ensure the safety of operators.

[0263] 4. Achieve intelligent scheduling and optimization: Leveraging real-time monitoring, data analysis, and cloud platform scheduling, automatically optimize job resource allocation, ensure accurate scheduling of job progress, improve resource utilization, and reduce human errors.

[0264] 5. Real-time monitoring and feedback: Through the integration of road-side equipment, sensors, and cameras, relevant data on trucks and cargo are collected in real time, providing accurate and timely information to the control center, supporting decision optimization and exception handling, and realizing intelligent collaborative work between unmanned forklifts, Eagle Eye, and the cloud (vehicle-road-cloud).

[0265] This paper proposes an Eagle Eye system for industrial truck loading and unloading based on roadside equipment and its application. By deploying a laser radar and high-precision cameras on the roadside, combined with advanced image recognition algorithms, laser clustering algorithms, smooth tracking algorithms, and data analysis technologies, the system enables real-time monitoring of the loading and unloading process, truck position identification, pallet type identification, pallet location identification, anomaly detection, and dynamic cloud-based vehicle scheduling, thus achieving vehicle-road-cloud collaborative control.

[0266] The present invention introduces an Eagle Eye system for truck loading and unloading in industrial scenarios based on road-side equipment and its application from a software perspective.

[0267] The Hawkeye system consists of five parts in terms of software: input module, recognition module, tracking module, coordinate conversion module, and output module. Figure 2 The figure shows the functional structure of the Hawkeye system, in which the recognition module includes three parts: pallet recognition module, truck recognition module, and obstacle recognition module. The tracking module, coordinate conversion module, and output module correspond to the recognition module nodes respectively.

[0268] The Eagle Eye input module includes point cloud data from multiple sensors and their synchronously uploaded image data. The laser and point cloud are synchronized using timestamps, forming a data set of 1 frame of laser data + 1 frame of image data. The laser data timestamp is used as a reference, and the nearest neighbor timestamp and a set time threshold (usually set to less than 50ms between the laser and image timestamps) are used. The camera needs to be calibrated for internal parameters, distortion parameters, and external parameters according to the calibration plate method. The laser needs to be calibrated for external parameters, and the camera and laser need to be jointly calibrated for external parameters.

[0269] The Eagle Eye recognition module uses a vision + laser post-fusion algorithm.

[0270] First, target detection is performed on all images. The process can be found in Figure 3The Eagle Eye vision function diagram shown in the figure detects pallets, trucks, and obstacles. Obstacles include two types of dynamic targets: non-truck vehicles and pedestrians. All detected targets are output as 2D bounding boxes (2dbbox). The method used is a deep learning network model. Commonly used network models include classic network models such as the RCNN series, Yolo series, SSD, RetinaNet, EfficientDet, and CenterNet.

[0271] Then, based on the camera's internal and external parameters and the relative external parameters of the laser calibration, the point cloud is calculated for each target in the image. Since the image 2dBbox is two-dimensional information, the same 2dBbox will retain point cloud data at different depths. Therefore, the point cloud within each 2dBbox needs to be clustered to obtain accurate 3D target information (3DInfo). Common clustering algorithms include template matching, Euclidean clustering, DBSCAN, PointNet++, PointCNN, post-processing clustering, point cloud segmentation, and other classic methods.

[0272] For pallet recognition, since the pallet target is stationary and has strong regularity when recognizing the pallet, the pallet's support holes and pallet size information can usually be used as fitting rules. Pallet post-processing clustering operations can be implemented on the image point cloud benchmark. You can also use PointNet++, DBSCAN, Euclidean distance and other clustering methods to achieve high-precision pallet 3D target recognition. The recognized content includes pallet type (type), pallet coordinates L (x, y, z, roll, pitch, yaw), pallet size (length, width, height), etc. Among them, the pallet types include plastic Sichuan pallet (yellow), plastic Sichuan pallet (blue), plastic Sichuan pallet (black), wooden Sichuan pallet, etc.

[0273] For truck identification, when Eagle Eye is loading or unloading, the truck will stop at the designated location, see Figure 4The diagram below shows a schematic diagram of truck parking spots. The truck is divided into a front and a body, and the body position is what the Hawkeye system needs to accurately identify. Therefore, when a truck is recognized in the image, a truck point cloud clustering algorithm is used to identify the truck in the point cloud. The clustering algorithm can use various clustering methods such as template matching, PointNet++, and DBSCAN, combined with the image recognition 2dbbox. After the two are realized and fused, a 3D truck bounding box (3dbbox) is obtained. Then, based on the 3dbbox, the parking spot of the truck body head is identified according to the characteristics of the front and body. Depending on the Hawkeye layout, the relative position of the parking spot and the body to be calculated is also different. For example, the double-sided Hawkeye needs to identify the center point of the body head as the parking spot, while the single-sided Hawkeye needs to identify the intersection point on one side of the body head as the parking spot. The recognition output includes the parking spot of the truck body, that is, the truck position information L (x, y, z, roll, pitch, yaw).

[0274] The method used for obstacle recognition is similar to that for truck recognition. Both methods use a clustering algorithm to identify fixed targets. The deep learning model PointNet++ can be used to simultaneously identify truck point clouds and various obstacle point clouds. The obstacle point clouds are then fused with the 2dbbox image recognition method to obtain detailed information about each obstacle, including the obstacle category (class_id) and obstacle location information L (x, y, z, roll, pitch, yaw).

[0275] The tracking module performs tracking and smoothing on the positional information output by pallets, trucks, and obstacles. It typically uses Kalman filtering to perform weighted smoothing on multiple frames of data, achieving stable target output and ensuring the uniqueness of each target ID output, effectively supporting downstream pallet loading and unloading tasks. In particular, after the tracking module passes through the pallet information, the pallet ID becomes more stable and can be sorted based on the current pallet ID to form the sort_id operation sequence for unloading.

[0276] The coordinate conversion module, for the position information L (x, y, z, roll, pitch, yaw) output by pallets, trucks, obstacles, etc., must be associated with the global map. The global map is a SLAM point cloud map established in advance for the global operating area, and the coordinates recognized by the Hawkeye system are all based on the Hawkeye radar itself as the zero point. Therefore, all targets need to be coordinate converted before output. At the beginning of equipment installation, it is necessary to calibrate the Hawkeye point cloud and the global map to obtain the rotation matrix R and translation vector T of the two spaces. Finally, all recognized target information L (x, y, z, roll, pitch, yaw) is calculated according to the RT formula to obtain the global coordinate position information G (x, y, z, roll, pitch, yaw).

[0277] Output module: Pallet output information includes pallet ID (0 to n, n represents the number of pallets - 1), pallet type (type), pallet dimensions (length, width, height), and pallet global coordinates G (x, y, z, roll, pitch, yaw). Truck output information includes truck ID (default is 0) and truck head global coordinates G (x, y, z, roll, pitch, yaw). Obstacle output information includes obstacle ID (0 to n, n represents the number of obstacles - 1), obstacle category ID (class_id), and obstacle global coordinates G (x, y, z, roll, pitch, yaw).

[0278] After the Hawkeye system goes through the five parts of input module, recognition module, tracking module, coordinate conversion module and output module, it can provide high-precision target information during loading and unloading to the cloud. The cloud can realize the automatic loading and unloading scheduling function based on this content. In this invention, the Hawkeye system focuses on providing high-precision target information and does not involve automatic loading and unloading scheduling, so this part will not be introduced in detail.

[0279] 1. The Hawkeye system in this invention adopts a vision + laser fusion recognition solution, which not only includes a visual deep learning model, but also a laser deep learning model. At the same time, a tracking module is added to enhance timing, ensuring that it can efficiently and accurately identify various targets, providing an effective basis for realizing intelligent scheduling.

[0280] 2. This invention can provide accurate location information for unmanned forklifts and robots when loading and unloading cargo, significantly improving the efficiency of truck loading and unloading operations. This can effectively reduce operation time and labor intensity, especially in scenarios involving large quantities of cargo.

[0281] 3. This invention provides a set of effective Hawkeye design solutions. Through the Hawkeye equipment, all conditions in the unloading area can be observed in all directions, providing an effective basis for intelligent scheduling.

[0282] 4. The present invention can observe obstacles in the loading and unloading area in real time, and combined with the obstacle information, it can improve the safety of unmanned forklifts and robots in automated loading and unloading of goods.

[0283] Figure 5 A structural schematic diagram of a loading and unloading positioning device based on the Hawkeye system provided for one or more embodiments of this specification, wherein the device is applied to the Hawkeye system, and the Hawkeye system includes an input module, an identification module and an output module, including: an acquisition unit 501, a detection target identification unit 502, a position information identification unit 503 and a position information sending unit 504.

[0284] The acquisition unit 501, the input module respectively acquires point cloud data and image data through sensors in multiple directions;

[0285] Detection target recognition unit 502, the recognition module identifies the detection targets of loading and unloading based on the image data through a deep learning network model, the detection targets include pallets, vehicles and obstacles;

[0286] A position information recognition unit 503, wherein the recognition module recognizes the position information of the detection target based on the point cloud data;

[0287] The position information sending unit 504, the output module sends the position information of the detection target to the cloud, so that the cloud sends corresponding loading and unloading instructions based on the position information of the detection target, and automatically completes the corresponding loading and unloading tasks based on the loading and unloading instructions.

[0288] Figure 6 This is a schematic diagram of the structure of a loading and unloading positioning device based on the Hawkeye system provided in one or more embodiments of this specification, which is applied to the Hawkeye system. The Hawkeye system includes an input module, a recognition module, and an output module, including:

[0289] at least one processor; and,

[0290] a memory communicatively connected to the at least one processor; wherein,

[0291] The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to:

[0292] The input module collects point cloud data and image data respectively through sensors in multiple directions;

[0293] The recognition module identifies detection targets for loading and unloading based on the image data through a deep learning network model, wherein the detection targets include pallets, vehicles, and obstacles;

[0294] The recognition module identifies the position information of the detection target based on the point cloud data;

[0295] The output module sends the location information of the detection target to the cloud, so that the cloud sends corresponding loading and unloading instructions based on the location information of the detection target and automatically completes the corresponding loading and unloading tasks based on the loading and unloading instructions.

[0296] One or more embodiments of this specification provide a non-volatile computer storage medium, which is applied to a Hawkeye system. The Hawkeye system includes an input module, a recognition module, and an output module, and stores computer-executable instructions. When executed by a computer, the computer-executable instructions can achieve the following:

[0297] The input module collects point cloud data and image data respectively through sensors in multiple directions;

[0298] The recognition module identifies detection targets for loading and unloading based on the image data through a deep learning network model, wherein the detection targets include pallets, vehicles, and obstacles;

[0299] The recognition module identifies the position information of the detection target based on the point cloud data;

[0300] The output module sends the location information of the detection target to the cloud, so that the cloud sends corresponding loading and unloading instructions based on the location information of the detection target and automatically completes the corresponding loading and unloading tasks based on the loading and unloading instructions.

[0301] The various embodiments in this specification are described in a progressive manner. Similar portions between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from the other embodiments. In particular, the device, apparatus, and non-volatile computer storage medium embodiments are generally similar to the method embodiments, so their descriptions are relatively simplified. For relevant details, refer to the descriptions of the method embodiments.

[0302] The various embodiments in this specification are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences from other embodiments. In particular, the device embodiments are generally similar to the method embodiments, so the description is relatively simple. For relevant parts, refer to the description of the method embodiments.

[0303] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0304] In the embodiments provided in this application, it should be understood that the disclosed devices / network equipment and methods can be implemented in other ways. For example, the device / network equipment embodiments described above are merely illustrative. For example, the division of the modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0305] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0306] In addition, the functional units in the various embodiments of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above units may be implemented in the form of hardware or software.

[0307] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by the processor, it can implement the steps of the above-mentioned various method embodiments. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0308] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A method for positioning loading and unloading cargo based on the Hawkeye system, characterized in that: The method is applied to the Hawkeye system, which includes an input module, a recognition module, and an output module; including: The input module collects point cloud data and image data respectively through sensors in multiple directions; The recognition module identifies detection targets for loading and unloading based on the image data through a deep learning network model, wherein the detection targets include pallets, vehicles, and obstacles; The recognition module identifies the position information of the detection target based on the point cloud data; The output module sends the location information of the detection target to the cloud, so that the cloud sends corresponding loading and unloading instructions based on the location information of the detection target and automatically completes the corresponding loading and unloading tasks based on the loading and unloading instructions.

2. The method according to claim 1, characterized in that The position information of the detection target is the coordinates of the pallet, and the recognition module identifies the position information of the detection target based on the point cloud data, including: The recognition module performs a clustering operation on the point cloud data based on the type of the tray hole and the size of the tray as fitting rules to identify the coordinates of the tray.

3. The method according to claim 1, characterized in that The position information of the detection target is the coordinates of the vehicle; The recognition module identifies the location information of the detection target based on the point cloud data, including: The recognition module performs a clustering operation on the point cloud data based on the front size and body size of the vehicle as a fitting rule to identify the coordinates of the vehicle.

4. The method according to claim 1, wherein The position information of the detection target is the coordinates of the obstacle; The recognition module identifies the location information of the detection target based on the point cloud data, including: The recognition module performs a clustering operation on the point cloud data based on the type of the obstacle and the size of the obstacle as a fitting rule to identify the coordinates of the obstacle.

5. The method according to claim 1, characterized in that The Hawkeye system further includes a tracking module, and the method further includes: The tracking module performs tracking smoothing processing on the position information of the detection target, so as to perform weighted smoothing processing on the position information of the detection target in multiple frames to obtain position information with stable time sequence.

6. The method according to claim 1, characterized in that The Hawkeye system further includes a coordinate conversion module, and the method further includes: The coordinate conversion module associates the position information of the detection target with a global map, where the global map is a point cloud map pre-established for a global operating area.

7. The method according to claim 1, characterized in that The position information of the detection target is the coordinates of the pallet and the coordinates of the vehicle. The recognition module identifies the position information of the detection target based on the point cloud data, including: The recognition module performs a clustering operation on the point cloud data based on the front size and body size of the vehicle as a fitting rule to identify the coordinates of the vehicle; The recognition module obtains the positional relationship between the vehicle and the pallet based on the current loading and unloading task; The recognition module determines the coordinates of the pallet based on the positional relationship and the coordinates of the vehicle, and marks a designated area of ​​the pallet in the point cloud data based on the coordinates of the pallet; The recognition module detects the number of point clouds in the designated area, and if the number of point clouds is lower than a preset threshold, re-determines the coordinates of the pallet based on the positional relationship and the coordinates of the vehicle.

8. A loading and unloading positioning device based on the Hawkeye system, characterized in that: The device is applied to the Hawkeye system, which includes an input module, a recognition module and an output module, including: The acquisition unit, the input module respectively acquires point cloud data and image data through sensors in multiple directions; A detection target recognition unit, wherein the recognition module recognizes detection targets for loading and unloading cargo based on the image data through a deep learning network model, wherein the detection targets include pallets, vehicles, and obstacles; A position information recognition unit, wherein the recognition module recognizes the position information of the detection target based on the point cloud data; The position information sending unit, the output module sends the position information of the detection target to the cloud, so that the cloud sends corresponding loading and unloading instructions based on the position information of the detection target, and automatically completes the corresponding loading and unloading tasks based on the loading and unloading instructions.

9. A loading and unloading positioning device based on the Hawkeye system, characterized in that: Applied to the Hawkeye system, the Hawkeye system includes an input module, a recognition module and an output module, including: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to: The input module collects point cloud data and image data respectively through sensors in multiple directions; The recognition module identifies detection targets for loading and unloading based on the image data through a deep learning network model, wherein the detection targets include pallets, vehicles, and obstacles; The recognition module identifies the position information of the detection target based on the point cloud data; The output module sends the location information of the detection target to the cloud, so that the cloud sends corresponding loading and unloading instructions based on the location information of the detection target and automatically completes the corresponding loading and unloading tasks based on the loading and unloading instructions.

10. A non-volatile computer storage medium, characterized in that Applied to the Hawkeye system, the Hawkeye system includes an input module, a recognition module, and an output module, and stores computer-executable instructions. When the computer executes the computer, the computer-executable instructions can achieve: The input module collects point cloud data and image data respectively through sensors in multiple directions; The recognition module identifies detection targets for loading and unloading based on the image data through a deep learning network model, wherein the detection targets include pallets, vehicles, and obstacles; The recognition module identifies the position information of the detection target based on the point cloud data; The output module sends the location information of the detection target to the cloud, so that the cloud sends corresponding loading and unloading instructions based on the location information of the detection target and automatically completes the corresponding loading and unloading tasks based on the loading and unloading instructions.

Images