AGV control method based on multimodal warehouse and transportation integration, eagle eye system and program product

By linking the multimodal Eagle Eye system and the WMS system, a multimodal Eagle Eye AI model is constructed, which solves the problem of low AGV control efficiency caused by the independent operation of warehousing and transportation, and realizes intelligent operation and integrated management of AGVs in the warehouse.

CN121477896BActive Publication Date: 2026-07-07HEFEI GEN SONG AUTOMATION TECH
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202511819850.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2025-12-04
Publication Date
2026-07-07
Estimated Expiration
2045-12-04

AI Technical Summary

Technical Problem

In existing technologies, warehousing and transportation are independent of each other, which makes it impossible for AGV control to be intelligently adjusted based on the specific conditions of warehousing and transportation, thus reducing transportation efficiency and management efficiency.

Method used

A multimodal eagle-eye system is adopted to acquire image data and point cloud data in the warehouse, construct a multimodal eagle-eye AI model, and combine it with the WMS system and scheduling system to realize intelligent scheduling and operation optimization of AGVs.

Benefits of technology

It has achieved global optimization and control of AGV operation within the warehouse, improved transportation efficiency and the level of intelligence in warehouse management, and realized intelligent management of integrated warehousing and transportation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN121477896B_ABST
    Figure CN121477896B_ABST
Patent Text Reader

Abstract

The present application relates to the technical field of intelligent warehousing and transportation, in particular to an AGV regulation method based on multimodal warehousing and transportation integration, an eagle eye system and a program product. The AGV regulation method processes the image data and point cloud data collected in each area of the warehouse through the construction of the eagle eye system, and outputs the target information of the corresponding task according to the task type. The WMS system obtains the business type executed by the current warehouse, generates different carrying tasks according to the target information output by the business processing module, and sends them to the scheduling system. The scheduling system generates different AGV scheduling strategies according to the obtained carrying tasks, and then schedules the operation of the AGV, so as to realize the closed loop from the perception result to the regulation of the AGV operation in the whole warehouse, thereby realizing the optimization and regulation of the AGV operation from the global perspective, and improving the regulation effect of the AGV operation.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of intelligent warehousing and transportation technology, specifically to an AGV control method, Eagle Eye system, and program product based on multimodal warehousing and transportation integration. Background Technology

[0002] With the rapid development of e-commerce and logistics, warehousing and transportation, as two indispensable core links in the modern logistics system, together constitute the complete chain of commodity circulation. Among them, the core function of warehousing is to achieve centralized storage and effective allocation of materials through scientific management, while flexibly controlling AGVs in the existing logistics and transportation system can improve production efficiency and reduce costs.

[0003] In the current management system, warehousing and transportation are managed separately. For warehousing, monitoring and management typically involves installing cameras in storage locations / areas and connecting them to a Warehouse Management System (WMS). However, this approach focuses primarily on loading docks, specific storage areas, or shelves, lacking a unified understanding of multiple areas within the warehouse. Furthermore, the data obtained through this method remains at the statistical and visualization level, unable to be integrated with subsequent AGV driving and scheduling. Similarly, in the transportation process, loading, unloading, and safety management are also independent, resulting in data silos between these stages. This hinders effective AGV scheduling and reduces transportation efficiency. Moreover, the lack of data integration between warehousing and transportation in current technologies prevents existing methods from simultaneously applying the specific circumstances of both stages to intelligently control AGV operation. Summary of the Invention

[0004] To address the technical problem that existing technologies cannot simultaneously and intelligently control the operation of AGVs based on the specific circumstances of both the warehousing and transportation stages, this invention provides an AGV control method, Eagle Eye system, and program product based on multimodal warehousing and transportation integration.

[0005] This invention employs the following technical solution: an AGV control method based on multimodal warehousing and transportation integration, comprising: acquiring image data and point cloud data of all areas within the warehouse; labeling and preprocessing targets on the image data and point cloud data respectively. Labeled targets include: personnel, operating equipment, storage locations, cargo types, cargo-carrying tools, and transportation vehicles. An eagle-eye system is constructed, comprising a multimodal eagle-eye AI model and a business processing module; the multimodal eagle-eye AI model receives both the preprocessed image data and point cloud data as inputs and outputs target detection results and depth estimation results respectively; wherein, the target detection results include the category of all targets within the warehouse and the pose of each target; the depth estimation results include a monocular depth map; the business processing module generates target information corresponding to the business based on the output of the multimodal eagle-eye AI model and in conjunction with the specific business type.

[0006] The WMS system obtains the current business type being executed in the warehouse and generates different handling tasks based on the target information output by the business processing module, which are then sent to the scheduling system. The scheduling system generates different AGV scheduling strategies based on the obtained handling tasks, thereby scheduling the operation of the AGVs.

[0007] As a further improvement of this invention, the multimodal eagle-eye AI model includes a Backbone module, a Neck module, and a Heads module. The Backbone module acquires preprocessed image data and point cloud data, and extracts image features from the image data and point cloud features from the point cloud data, respectively. The Neck module transforms the image features and point cloud features extracted by the Backbone module into the same spatial coordinate system for feature alignment and fusion to obtain a fused multimodal feature map. The Heads module includes a detection head and a depth estimation head. The detection head outputs the target category and pose of each target based on the multimodal feature map; the depth estimation head outputs a monocular depth map of the same size as the multimodal feature map.

[0008] As a further improvement of this invention, the multimodal Eagle Eye AI model is trained using a hybrid supervised and unsupervised training method, with the total loss function during the training process... The expression is:

[0009] ;

[0010] In the formula: All are coefficients of the weighted terms; The loss function for the detection head; The loss function for the depth estimation head; The loss function is for depth smoothing constraints.

[0011] As a further improvement of this invention, the detection head is trained using supervised learning, and the loss function of the detection head is:

[0012] ;

[0013] In the formula: Cross-entropy loss; For bounding box regression loss; For GIoU loss; These are the coefficients for the weighting terms.

[0014] As a further improvement to this invention, the loss function of the depth estimation head adopts different expressions depending on the camera mounting method:

[0015] With the camera fixed in place, the loss function of the depth estimation head is: or ;

[0016] in, ;

[0017] In the formula: A set of pixels with depth annotations. The size of the set of pixels labeled with depth; d ( p (This refers to the depth map predicted by the multimodal Eagle Eye AI model.) This is the true depth map obtained using a calibration board, lidar scanning, and structured light.

[0018]

[0019] In the formula: The pseudo-label depth generated by the teacher model or simulation; N is the total number of pixels involved in the calculation;

[0020] When the camera is mounted on a mobile device, the expression for the loss function of the depth estimation head is:

[0021] ;

[0022] In the formula: I t ( p () represents the pixel value of the current frame; The pixel value is obtained by reprojecting adjacent frames onto the current frame using the predicted depth and camera pose; δ is the weighting coefficient.

[0023] As a further improvement of this invention, the expression for the loss function of the depth smoothing constraint is:

[0024] ;

[0025] In the formula: and These represent the predicted depth maps. d At pixel p Place, extension x direction and y Gradient of direction; and These represent the original input images respectively. I At pixel p Place, extension x and y Gradient of direction.

[0026] As a further improvement of the present invention, the types of business performed by the warehouse include: warehouse area management business, personnel management business, automated loading business, and automated unloading business.

[0027] As a further improvement of this invention, the AGV operation control strategy during warehouse management is as follows: The WMS system extracts the warehouse location status information from the Eagle Eye system: empty warehouse location, full warehouse location, outbound, inbound, and the number of stacked layers of goods within the warehouse location. Upon inbound, the system determines the warehouse location based on the obtained warehouse location status information, the type and quantity of inbound goods, and generates an inbound handling task. The scheduling system then controls the AGV based on the inbound handling task, ensuring that the AGV moves the inbound goods to the corresponding warehouse location. Similarly, upon outbound, the WMS system determines the outbound warehouse location based on the obtained warehouse location status information, the type and quantity of outbound goods, and generates an outbound handling task. The scheduling system then controls the AGV based on the outbound handling task, ensuring that the AGV moves the goods out of the corresponding outbound warehouse location.

[0028] As a further improvement to this invention, the AGV operation control strategy during personnel management is as follows: The WMS system extracts the personnel's pose information from the eagle-eye system and performs spatial matching with the preset control zone range in the WMS system, then makes the following judgment: if the personnel's pose information is within the preset control zone range in the WMS system, it indicates that personnel have entered the control zone; otherwise, it indicates that no personnel have entered the control zone. Furthermore, when personnel enter the control zone, the scheduling system triggers the AGV to decelerate or even stop, and promptly updates the AGV's running path and speed.

[0029] As a further improvement of this invention, the AGV operation control strategy during automated loading is as follows: The WMS system extracts the location information of the loading truck, the type and quantity of the loaded goods, and the status information of the storage location from the eagle-eye system. The WMS system matches storage locations based on the type and quantity of the loaded goods and obtains the initial pose information of the goods in the successfully matched storage locations, thereby generating an automated loading and handling task. The scheduling system obtains the automated loading task and generates the AGV's running path and speed.

[0030] As a further improvement of this invention, the AGV operation control strategy during automated unloading is as follows: The WMS system extracts the type and quantity of unloaded goods, the pose information of the unloaded goods, and the status information of the storage location from the eagle-eye system. The WMS system matches storage locations based on the type and quantity of unloaded goods and generates the pose information of the unloaded goods based on the status information of the successfully matched storage locations, thereby generating an automated unloading and handling task. The scheduling system obtains the automated unloading and handling task and generates the AGV's running path and speed.

[0031] As a further improvement to this invention, the multimodal Eagle Eye AI model outputs the truck's location information in the following manner: Raw point cloud data of the truck is acquired using LiDAR, and ground points are separated from the raw point cloud using the RANSAC plane fitting algorithm. Then, the point cloud clusters of the truck's main body are obtained from the non-ground point cloud using DBSCAN clustering. Finally, a three-dimensional bounding box is fitted to the truck's point cloud clusters using the minimum bounding rectangle algorithm, thus obtaining the three-dimensional geometric model of the truck's cargo compartment.

[0032] This invention also includes a multi-modal warehousing and transportation integrated AGV control "eagle eye" system, which adopts the aforementioned multi-modal warehousing and transportation integrated AGV control method. The AGV control "eagle eye" system includes a data acquisition module, a data processing module, and an execution module. The data acquisition module is used to acquire the status and location information of storage locations, personnel location information, operating equipment location information, cargo type, location information of the tools carrying the cargo, location information of the transport vehicles, cargo pose information, and the type of business currently being performed in the warehouse. The data processing module is used to process the data acquired by the data acquisition module and generate AGV control strategies. The execution module is used to acquire the control strategies generated by the data processing module to control the operation of the AGV.

[0033] The present invention also includes a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the above-described control method based on multimodal warehousing and transportation integration, thereby enabling the selection of corresponding control strategies to regulate the running path and speed of the AGV according to different business types performed by the warehouse.

[0034] The technical solution provided by this invention has the following beneficial effects:

[0035] (1) The AGV control method based on multimodal warehousing and transportation integration provided by the present invention transmits the output of the Eagle Eye system as input to the WMS system, thereby realizing the linkage between the Eagle Eye system and the WMS system. The WMS system also generates different handling tasks based on the output of the Eagle Eye system and the current business type being executed in the warehouse and sends them to the scheduling system. The scheduling system generates different AGV scheduling strategies based on the acquired handling tasks, thereby realizing the scheduling of AGV operation. This achieves a closed loop from the perception result to the control of AGV operation in the entire warehouse, thereby realizing the optimization and control of AGV operation from a global perspective, thus improving the control effect of AGV operation.

[0036] (2) The AGV control method based on multimodal warehousing and transportation integration provided by the present invention can realize the unification of the entire chain of warehouse monitoring, safety protection, loading and unloading guidance WMS and scheduling, realize the interconnection of diversified information data across scenarios in the warehouse, and thus realize intelligent management of the warehouse based on multimodal warehousing and transportation integration; and control the operation of AGV based on the acquired information such as warehouse status information and safety protection, so as to realize more intelligent warehouse operation management.

[0037] (3) The AGV control method based on multimodal warehousing and transportation integration provided by the present invention connects the Eagle Eye system and the WMS system in real time, so that the storage location can be automatically updated under the synergistic effect of the Eagle Eye system and the WMS system, reducing manual intervention; it can also drive the AGV to move goods according to the actual storage location status information and through the control system, so as to realize timely adjustment of the storage of goods, making the storage of goods more intelligent, and achieving the construction of an AGV control method based on warehousing and transportation integration, making the operation of AGV in the warehouse more intelligent, thereby improving the operating efficiency of the warehouse. Attached Figure Description

[0038] Figure 1 This is a flowchart of the steps of the AGV control method based on multimodal warehousing and transportation integration provided in Embodiment 1 of the present invention.

[0039] Figure 2 This is an architecture diagram of the multimodal Eagle Eye AI model in Embodiment 1 of the present invention.

[0040] Figure 3 This is a flowchart illustrating the operation of controlling the AGV during the warehousing process in Embodiment 1 of the present invention.

[0041] Figure 4 This is a flowchart illustrating the operation of controlling the AGV during outbound operations in Embodiment 1 of the present invention.

[0042] Figure 5This is a flowchart illustrating the operation of controlling the AGV during personnel management in Embodiment 1 of the present invention.

[0043] Figure 6 This is a flowchart illustrating the operation of controlling the AGV during the execution of automated loading services in Embodiment 1 of the present invention.

[0044] Figure 7 This is a flowchart illustrating the operation of controlling the AGV during the execution of automatic unloading services in Embodiment 1 of the present invention. Detailed Implementation

[0045] The present invention will now be further described in conjunction with specific embodiments. It should be noted that, without conflict, the various embodiments or technical features described below can be arbitrarily combined to form new embodiments.

[0046] In the description of this invention, it should be noted that directional terms such as "center," "lateral," "longitudinal," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," and "counterclockwise" indicate the orientation and positional relationship based on the orientation or positional relationship shown in the accompanying drawings. These terms are used only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. They should not be construed as limiting the specific scope of protection of this invention. The terms "first," "second," etc., in the specification and claims of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. The terms "comprising" and "having," and any variations thereof, in the specification and claims of this invention, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or devices.

[0047] Example 1

[0048] This embodiment provides an AGV control method based on multimodal warehousing and transportation integration. Please refer to [link / reference]. Figure 1 It includes the following steps:

[0049] (a) Data acquisition and preprocessing

[0050] Acquire image and point cloud data for all areas within the warehouse, and annotate and preprocess the targets on both the image and point cloud data. Data sources can include various types of sensors installed at key locations within the warehouse, such as storage locations, loading / unloading areas, and aisles. These sensors may include industrial cameras, panoramic cameras, 2D LiDAR, and 3D LiDAR. Data types acquired include image data (RGB or grayscale) and point cloud data (XYZI). The acquired image and point cloud data need to be annotated and preprocessed separately. The annotated objects mainly cover typical elements in warehousing and transportation scenarios, including personnel, operating equipment, storage locations, cargo types, cargo-carrying tools, and transportation vehicles. Personnel include: workers, visitors, and inspection personnel; operating equipment includes: forklifts, AGVs, and small handling robots; cargo types include: standard containers, drums, bags, and bulk tonnes; cargo-carrying tools include: pallets, crates, turnover boxes, and cage trucks; and transportation vehicles include: various trucks (such as flatbed trucks, high-sided trucks, and container trucks). Standard strategies can employ traditional manual annotation, semi-automatic annotation, or active learning annotation. Traditional manual annotation requires frame-by-frame bounding box selection and classification using an annotation platform. Semi-automatic annotation requires automatic generation of candidate boxes based on a pre-trained model, followed by manual correction. Active learning annotation requires the model to select uncertain samples for focused annotation during iteration, thereby reducing overall annotation costs. It is understandable that, in terms of spatial information acquisition, for point cloud data, in addition to labeling the target category, keypoints, bounding boxes, pose parameters, etc., can be annotated to enable subsequent models to perform more complex recognition and localization tasks. For image data, distortion correction, color normalization, and scale adjustment preprocessing are required. For point cloud data, filtering and coordinate system alignment preprocessing are required. Since the preprocessing methods for image data and point cloud data are existing, the specific preprocessing procedures will not be described in detail in this application.

[0051] (II) Constructing the Eagle Eye System

[0052] The Eagle Eye system comprises a multimodal Eagle Eye AI model and a business processing module. The multimodal Eagle Eye AI model receives pre-processed image and point cloud data as input, and outputs target detection and depth estimation results, respectively. The target detection results include the category and pose of all targets within the warehouse; the depth estimation results include a monocular depth map. The business processing module uses the output of the multimodal Eagle Eye AI model and combines it with specific business types to generate target information corresponding to the business.

[0053] To construct a multimodal Eagle Eye AI model based on a typical multi-task learning framework, please refer to [link / reference]. Figure 2 The multimodal Eagle Eye AI model includes the Backbone module, Neck module, and Heads module.

[0054] The Backbone module acquires preprocessed image and point cloud data, and extracts image features from the image data and point cloud features from the point cloud data, respectively. The Backbone module can use network structures such as ResNet, EfficientNet, ConvNet, Swin Transformer, PointNet++, or VoxelNet to extract multi-scale features from the input data. It is understood that combining camera and LiDAR data in this solution can improve the spatial localization and pose estimation accuracy of the constructed multimodal eagle-eye AI model.

[0055] The Neck module is used to transform the image features and point cloud features extracted by the Backbone module into the same spatial coordinate system for feature alignment and fusion to obtain a fused multimodal feature map. The Neck module can adopt an FPN (Feature Pyramid Network) or BiFPN (Bi-directional FPN) network structure, which can fuse features of different scales in a top-down and bottom-up manner to balance detection accuracy and speed.

[0056] The Heads module includes a Detection Head and a Depth Estimation Head. The Detection Head can adopt a YOLO or FCOS-like architecture. Based on multimodal feature maps, it outputs the target category and pose of each target; the Depth Estimation Head outputs a monocular depth map of the same size as the multimodal feature maps. Individual depth maps can be scaled and mapped to actual spatial units (meters / millimeters). Specifically, the Detection Head identifies all visible and distinguishable objects in the acquired data and outputs the target category, location (2D bounding box or 3D bounding box), and confidence score. These results can be directly used for warehouse monitoring (e.g., determining if a storage location is empty or stacked), area security management (whether personnel have entered a restricted area), and loading / unloading operations (identifying truck or cargo location). The Depth Estimation Head outputs a pixel-by-pixel depth map for the image data, converting a two-dimensional image into approximately three-dimensional spatial information. The results from the Depth Estimation Head allow for the determination of cargo stacking levels and shelf occupancy status, obtaining this information without relying on expensive 3D cameras. The multimodal Eagle Eye AI model provided in this solution possesses both detection and monocular depth estimation capabilities, enabling the processing of multimodal data with a single model, thereby reducing the deployment costs of multiple models. Furthermore, the multimodal Eagle Eye AI model constructed in this solution uses a depth estimation head to replace expensive 3D cameras for 3D monitoring of storage locations, thus reducing the monitoring costs of storage locations.

[0057] The outline and location of the transport vehicle can be obtained not only by point cloud scanning with lidar, but also by visual inspection to identify the outline and location of the transport vehicle (such as a truck); it can also be obtained by ultra-wideband (UWB), RFID, or infrared beam sensors; and its docking and loading coordinates can be directly obtained through vehicle-to-vehicle communication (V2X) between the vehicle and the warehouse control platform.

[0058] In this embodiment, the multimodal Eagle Eye AI model is trained using a hybrid supervised and unsupervised training method. Since the training method of the multimodal Eagle Eye AI model in this solution is the same as existing training methods, it will not be described in detail here. The total loss function of the multimodal Eagle Eye AI model during training is... The expression is:

[0059] ;

[0060] In the formula: All are coefficients of the weighted terms; The loss function for the detection head; The loss function for the depth estimation head; The loss function is for depth smoothing constraints.

[0061] The loss function for the detection head is trained using supervised learning. The loss of the detection head includes classification loss, bounding box regression loss, and IoU loss. The expression for the loss function of the detection head is as follows:

[0062] ;

[0063] In the formula: Cross-entropy loss; For bounding box regression loss; For GIoU loss; These are the coefficients for the weighting terms.

[0064] Specifically, the cross-entropy loss function (also known as the focus loss function) is expressed as follows:

[0065] ;

[0066] In the formula: y i Indicates the first i The true category of each target Indicates the first i The predicted category of each target, D This represents the total number of targets in the original data.

[0067] Bounding box regression loss function The expression is as follows:

[0068] ;

[0069] In the formula: b i Indicates the first i The true bounding box of each target; Indicates the first i Predicted bounding boxes for each target; D This represents the total number of targets in the original data.

[0070] The IoU loss function can be GIoU loss function, DIoU loss function or CIoU loss function, which is used to improve the accuracy of positioning.

[0071] The depth estimation head supports both fixed-mount and mobile-mounted cameras or LiDARs, improving its adaptability to different scenarios. Furthermore, the depth estimation head can be trained using unsupervised or weakly supervised learning methods depending on the camera or LiDAR's mounting mode. Therefore, the depth estimation head in this solution can switch training strategies based on the camera or LiDAR's mounting mode during training, thereby improving the accuracy of the trained multimodal Eagle Eye AI model. Simultaneously, the loss function of the depth estimation head can use different expressions depending on the camera's mounting method.

[0072] (1) The camera is fixed in place.

[0073] Since the camera is fixed in its mounting configuration, it cannot use the adjacent frame reprojection constraint. Therefore, when the camera is fixed in its mounting orientation, one of the following two methods can be used to describe the loss function of the depth estimation head.

[0074] 1.1 A sparse depth weak supervision approach is used to obtain the loss function of the depth estimation head. In this approach, sparse depth points can be obtained using calibration boards, LiDAR scanning, structured light, etc. The loss function of the depth estimation head is then... ;

[0075] in, ;

[0076] In the formula: A set of pixels with depth annotations. The size of the set of pixels labeled with depth; d ( p (This refers to the depth map predicted by the multimodal Eagle Eye AI model.) This is the true depth map obtained using a calibration board, lidar scanning, and structured light.

[0077] 1.2 The loss function of the depth estimation head is obtained by using synthetic data / model distillation. The loss function of the depth estimation head at this point is: .

[0078] in, ; N This represents the total number of pixels involved in the calculation.

[0079] In the formula: The depth of pseudo-labels generated for teacher models or simulations.

[0080] (2) The camera is mounted on a mobile device.

[0081] When the camera is mounted on a mobile device, the loss function of the depth estimation head can be constructed using a photometric consistency loss constraint. The expression for the loss function of the depth estimation head in this case is:

[0082] ;

[0083] In the formula: I t ( p () represents the pixel value of the current frame; δ represents the pixel value obtained by reprojecting adjacent frames onto the current frame using predicted depth and camera pose; δ is the weighting coefficient. In this scheme, δ can be between 0 and 1 and is used to balance the importance of the subsequent two losses (e.g., 0.85). SSIM is the structural similarity index, which is a more advanced image quality assessment metric than simple absolute value. It compares two image patches from three dimensions: luminance, contrast, and structure. Its output value is between [-1, 1], and when it is 1, it indicates that the two image patches are completely identical.

[0084] In this scheme, a loss function with depth smoothing constraints is constructed to maintain the local smoothness of depth prediction while preserving edge details. The expression for the loss function with depth smoothing constraints is as follows:

[0085] ;

[0086] In the formula: and These represent the predicted depth maps. d At pixel p Place, extension x direction and y The gradient in a direction measures the magnitude of change in the depth map in a particular direction. If the depth map changes drastically in a certain region (large gradient value), it indicates that the area may not be smooth, and this term will incur a large penalty (loss value). and These represent the original input images respectively. I At pixel p Place, extension x and y The gradient in the direction. Wherein, and This is the essence of the entire formula; it introduces an edge-aware mechanism. and These represent the original input images respectively. I At pixel p Place, extension x and y The gradient of direction. Areas with large gradients in an image usually correspond to drastic changes in color, texture, or brightness, and these areas are often the edges of objects. It is an exponential decay function. When the image gradient... When it is very large (i.e. at the edge of the object). This will become a very small weight; conversely, in flat regions of the image (small gradients), this weight is close to 1. Multiplying these two terms together: This results in an edge-aware smoothing loss. Within the object's surface, the image gradient... Very small It approaches 1. At this point, if the depth gradient... The larger the gradient, the larger the loss. This penalizes the unevenness of the depth map inside the object's surface, forcing the model to predict smooth depth changes in these areas. At the object's edges, the image gradient... Very large It approaches 0. At this point, even the depth gradient... The value is very large, and the entire product will approach 0. This means that at the edges of the image, the loss function allows for drastic and discontinuous changes in depth. This perfectly matches physical reality: a box placed on the floor, at the edge of the box, the depth value is the depth from the depth of the box to the depth of the floor.

[0087] By constructing the loss function with the aforementioned depth smoothing constraint, the smoothing penalty can be reduced at locations with large changes in image gradient (i.e., object edges), thereby improving the accuracy of the results after model processing.

[0088] (iii) Regulating warehouse operations

[0089] The WMS system obtains the current business type being executed in the warehouse and generates different handling tasks based on the target information output by the business processing module, which are then sent to the scheduling system. The scheduling system generates different AGV scheduling strategies based on the obtained handling tasks, thereby scheduling the operation of the AGVs.

[0090] The current business operations performed in the warehouse include: warehouse area management, personnel management, automated loading, and automated unloading. It is understandable that the results obtained by the Eagle Eye system can be organically transmitted to the WMS system as input in this step, thereby enabling linkage between the Eagle Eye system and the WMS system. The WMS system, based on the output of the Eagle Eye system and the current business operations performed in the warehouse, generates different handling tasks and sends them to the scheduling system. The scheduling system generates different AGV scheduling strategies based on the acquired handling tasks, thus scheduling the operation of the AGVs. This achieves a closed loop from sensing results to controlling AGV operation within the entire warehouse, thereby optimizing and controlling AGV operation from a global perspective and improving the effectiveness of AGV operation control.

[0091] As described above, the control method provided by this solution can achieve unified control of the entire warehouse chain, including warehouse area monitoring, safety protection, loading and unloading guidance, WMS, and scheduling. It enables the interconnection of diversified information data across different scenarios within the warehouse, thereby achieving intelligent warehouse management based on multimodal warehousing and transportation integration. Furthermore, it controls the operation of AGVs based on acquired information such as warehouse status information and safety protection, achieving more intelligent control of AGVs.

[0092] The following section describes in detail the different control strategies adopted by the warehouse for AGV operation when performing different business types.

[0093] (3.1) Warehouse performs warehouse area management business

[0094] The AGV operation control strategy is as follows: The WMS system automatically extracts the status information of each storage location in the Eagle Eye system and generates corresponding handling tasks based on the required task information (such as inbound or outbound operations). The scheduling system generates different AGV scheduling strategies based on the acquired handling tasks, thereby scheduling the operation of the AGVs. Please refer to [the relevant documentation / reference]. Figure 3 When performing inbound operations, the WMS system extracts the location status information from the Eagle Eye system: empty location, full location, in the process of outbound, in the process of inbound, and the number of stacking layers of goods within the location. Based on the obtained location status information, the type and quantity of inbound goods, the WMS system determines the appropriate storage location. It then generates an inbound handling task based on the goods' positions before and after inbound. The scheduling system generates an AGV control strategy based on this task, determining the AGV's path and speed. This strategy also considers information from the Eagle Eye system, such as personnel positions and the tools used to carry the goods, to determine the optimal path and speed for AGV handling. Finally, the scheduling system controls the AGV to move the inbound goods to the corresponding storage location along this path and at this speed. For outbound operations, please refer to [the relevant documentation / reference]. Figure 4The WMS system extracts the storage location status information from the Eagle Eye system: empty storage location, full storage location, outbound, inbound, and the number of stacked layers of goods within the storage location. The WMS system also determines the storage location for outbound goods based on the acquired storage location status information, the type and quantity of outbound goods, and generates outbound handling tasks based on the positions of the goods before and after outbound. The scheduling system acquires the outbound handling tasks and generates AGV control strategies. Based on these strategies, it determines the AGV's operating path and speed, taking into account information such as the location of personnel and the tools used to carry the goods from the Eagle Eye system, thus determining the optimal path and speed for AGV handling. Finally, the control system regulates the AGV to move the goods from the corresponding outbound storage location. In this solution, by integrating the Eagle Eye system and the WMS system in real time, the collaborative operation of the two systems enables automatic updates of storage locations, reducing manual intervention. The WMS system can also generate different handling tasks based on the actual warehouse location status information. The scheduling system obtains the handling tasks and generates corresponding AGV control strategies. Based on the generated AGV control strategies, it drives the AGVs to handle the goods, so as to realize timely adjustment of the storage of goods, making the storage of goods more intelligent. This achieves the construction of an AGV control method based on the integration of warehousing and transportation, making the operation of AGVs in the warehouse more intelligent, thereby improving the operating efficiency of AGVs in the warehouse.

[0095] (3.2) Warehouse personnel management business

[0096] The control strategy for AGV operation is as follows: When performing personnel management tasks, please refer to... Figure 5 The Eagle Eye system identifies and tracks personnel in real time based on continuous frame target detection results, outputting their pose information. When performing personnel management tasks, the dispatch system extracts the pose information from the Eagle Eye system and spatially matches it with the preset control zone range in the WMS system, making the following judgment: if the personnel's pose information is within the preset control zone range in the WMS system, it indicates that personnel have entered the control zone; otherwise, it indicates that no personnel have entered the control zone. Furthermore, when personnel enter the control zone, the dispatch system triggers AGVs currently operating in that zone or about to pass through it to slow down or even stop, and promptly updates the AGV's running path and speed. This ensures the safety of AGV operation.

[0097] (3.3) The warehouse performs automated loading operations.

[0098] The control strategy for AGV operation is as follows: Please refer to... Figure 6The WMS system extracts the precise coordinates of the transport vehicle's compartment, the type and quantity of loaded goods, and the status information of the storage location from the Eagle Eye system. Then, based on the type and quantity of loaded goods, it matches storage locations and obtains the initial pose information of the goods within the successfully matched locations. The WMS system also generates automated unloading and handling tasks based on the truck's location information and the initial pose information of the goods. The scheduling system obtains the automated unloading and handling tasks and generates corresponding AGC control strategies, which then determine the AGV's operating path and speed. Simultaneously, when generating the AGV's operating path and speed, it considers information such as the personnel's location and the tools carrying the goods obtained from the Eagle Eye system, thus determining the optimal path and speed for AGV handling. Finally, by controlling the AGV, it moves the goods from the corresponding outbound storage location to the truck compartment, thereby completing the automated loading operation.

[0099] The multimodal Eagle Eye AI model can obtain the precise coordinates of the loading truck's cargo compartment in the following way: First, the multimodal Eagle Eye AI model uses the target detection results obtained by the detection head to quickly determine the approximate location and type of the loading truck, thereby filtering out irrelevant interference in the environment. Based on this, the multimodal Eagle Eye AI model uses a point cloud algorithm to accurately model and locate the cargo compartment and its ancillary structures (such as doors, side panels, support rods, etc.), thus obtaining the three-dimensional geometric boundary of the cargo compartment. This boundary is then transformed into a coordinate system to obtain the precise coordinates of the cargo compartment. The obtained precise coordinates of the cargo compartment can provide a basis for AGV loading path planning and motion guidance, thereby achieving precise loading operations without human intervention. One specific method for obtaining the precise coordinates of the cargo compartment is as follows: The original point cloud data of the truck is obtained through LiDAR, and the RANSAC plane fitting algorithm is used to separate ground points from the original point cloud. Next, point cloud clusters representing the main body of the truck body are obtained from the non-terrestrial point cloud using DBSCAN clustering. Finally, a 3D bounding box is fitted to the point cloud clusters of the truck body using the minimum bounding rectangle algorithm, thus obtaining the 3D geometric model of the truck body. In this method, geometric feature detection (such as line fitting and edge detection) can be used to separate structures such as doors, side panels, and support rods in the point cloud. During the separation process, Huogh's line transform can be used to detect long straight edges in the projection plane, and local vector consistency can be used to distinguish between planar and rod-like structures.

[0100] (3.4) The warehouse performs automated unloading operations.

[0101] The control strategy for AGV operation is as follows: Please refer to... Figure 7The WMS system extracts the type and quantity of unloaded goods, their pose information, and the status information of the storage location from the Eagle Eye system. Based on the type and quantity of unloaded goods, it matches storage locations and generates post-unload pose information based on the status information of the matched locations. The WMS system also generates automated unloading and handling tasks based on the pose information of the unloaded goods and their post-unload pose information. The scheduling system generates AGV control strategies by acquiring these tasks, and then determines the AGV's path and speed based on these strategies. Simultaneously, when generating the AGV's path and speed, it considers information such as the location of personnel and the tools used to carry the goods obtained from the multimodal Eagle Eye AI model, thus determining the optimal path and speed for AGV handling. Finally, it controls the AGV to move the goods from the truck bed to the corresponding outbound storage location, completing the automated unloading operation.

[0102] As described above, the output of the Eagle Eye system provided by this solution can be used as input for various business types executed in the warehouse. This effectively avoids the need to design separate perception modules when different business types are executed in the warehouse, thereby improving the versatility and scalability of the Eagle Eye system in the entire warehouse management process.

[0103] The Eagle Eye system obtains the pose information of cargo to be unloaded in the following way: First, it collects point cloud data of the cargo using LiDAR and removes the background using the vehicle boundary to obtain point cloud clusters of candidate cargo. Then, it separates individual cargo point cloud clusters from the candidate cargo point clouds using DBSCAN clustering and outlier removal. Next, it uses 6DOF pose estimation technology to obtain the specific pose information of the cargo. The specific operation steps of 6DOF pose estimation technology are as follows: First, it obtains the centroid / geometric center point of the cargo and uses this geometric center as the translation vector t in the pose. Then, it uses PCA principal component analysis to estimate the main orientation of the cargo, thereby determining the rotational attitude R of the cargo. Finally, it combines the translation vector t and the rotation matrix R into a rigid transformation matrix, which transforms a point from the local coordinate system of the cargo to the world coordinate system. Then, it uses the ICP (Iterative Closest Point) algorithm to align the detected cargo point cloud with the standard model to optimize the pose information of the cargo, thereby obtaining more accurate cargo pose information. The positional information of the goods obtained by the above method can be directly used to guide AGVs or other handling machinery to grasp and transport the goods, ensuring stable unloading even under complex stacking and irregular posture conditions.

[0104] During the process of AGVs moving goods within or to the carriage, they primarily rely on the parking position and posture information provided by the Eagle Eye system for precise docking. When achieving precise docking between the AGV and the carriage, any of the following commonly used existing docking methods can be employed: Method 1: The AGV's onboard sensors (such as LiDAR, depth cameras, etc.) directly identify the carriage boundaries, thus achieving docking. Method 2: The AGV uses guiding devices (such as mechanical limit blocks, embedded guide grooves) installed in the carriage to assist in positioning, completing the docking. Method 3: Magnetic navigation, landmark recognition, or other methods are used to achieve docking between the AGV and the carriage.

[0105] Example 2

[0106] This embodiment provides an AGV control "eagle eye" system based on multimodal warehousing and transportation integration, building upon Embodiment 1. It includes a data acquisition module, a data processing model, and an execution module. The data processing module acquires the status and location information of storage locations, personnel location information, operating equipment location information, cargo type, location information of the tools carrying the cargo, location information of the transportation vehicles, cargo pose information, and the current business type being performed by the warehouse. In this embodiment, the status and location information of storage locations, personnel location information, operating equipment location information, cargo type, location information of the tools carrying the cargo, location information of the transportation vehicles, and cargo pose information can all be acquired using the "eagle eye" system described in Embodiment 1. The current business type being performed by the warehouse can be acquired using a WMS system in the prior art. The data processing module processes the data acquired by the data acquisition module and generates an AGV control strategy. It is understood that during data processing, this embodiment can first obtain the current business type being performed by the warehouse through a WMS system, and then the WMS system extracts the pose information of targets related to the specific business type being performed from the "eagle eye" system, thereby generating the AGV control strategy. The execution module is used to obtain the control strategy generated by the data processing module to control the operation of the AGV.

[0107] In this example, the camera and radar in the AGV control eagle eye system can be fixedly installed on the top of a warehouse or platform, or they can be installed mobilely. Mobile installation can be achieved in any of the following ways: 1. Mount the AGV control eagle eye system on a movable bracket that can move along a ground track. 2. Mount it on an automatic lifting column or a telescopic robotic arm, and adjust its shooting position as needed. 3. Combine it with a drone or AGV, allowing it to fly or move above the truck parking point for identification when needed.

[0108] The AGV control eagle eye system in this embodiment can operate as an independent host computer system, or it can be integrated into the AGV scheduling system for unified backend processing; it can also be distributed on various camera nodes to enable independent processing of data collected by each camera and uploading the processing results to the backend.

[0109] Example 3

[0110] This embodiment provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the AGV control method based on multimodal warehousing and transportation integration as described in Embodiment 1, thereby enabling the selection of corresponding control strategies to regulate the AGV's running path and speed according to different business types performed by the warehouse.

[0111] The AGV control method based on multimodal warehousing and transportation integration in Example 1 can be applied in the form of software, such as by designing a program that can run independently on a computer-readable storage medium, which can be a USB flash drive. The program is designed to start the entire method through an external trigger.

[0112] The computer program product provided in this embodiment is essentially a computer device used to implement the AGV control method in Embodiment 1. This computer device can be an embedded module, or it can be a smart terminal, tablet computer, laptop computer, desktop computer, rack server, blade server, tower server, or cabinet server (including independent servers or server clusters composed of multiple servers), etc., capable of executing programs.

[0113] The computer device mentioned in this embodiment includes, but is not limited to, a memory and a processor that can be interconnected via a system bus.

[0114] In this embodiment, the memory (i.e., the readable storage medium) includes flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory can be an internal storage unit of a computer device, such as the hard disk or RAM of the computer device. In other embodiments, the memory can also be an external storage device of the computer device, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the computer device. Of course, the memory can also include both internal storage units and external storage devices of the computer device. In this embodiment, the memory is typically used to store the operating system and various application software installed on the computer device. In addition, the memory can also be used to temporarily store various types of data that have been output or will be output.

[0115] In some embodiments, the processor may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor is typically used to control the overall operation of a computer device. In this embodiment, the processor is used to run program code stored in memory or process data to implement the steps of the AGV control method based on multimodal warehousing and transportation integration in Embodiment 1.

[0116] The basic principles, main features, and advantages of this invention have been described above. Those skilled in the art should understand that this invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely principles of the invention. Various changes and modifications can be made without departing from the spirit and scope of the invention, and all such changes and modifications fall within the scope of the invention as claimed. The scope of protection claimed by this invention is defined by the appended claims and their equivalents.

Claims

1. A method for controlling AGVs based on multimodal warehousing and transportation integration, characterized in that, It includes: Acquire image data and point cloud data for all areas within the warehouse, and label the targets on the image data and point cloud data respectively: personnel, operating equipment, storage locations, goods types, tools carrying goods, and transportation vehicles; and preprocess the labeled image data and point cloud data. An Eagle Eye system is constructed, comprising a multimodal Eagle Eye AI model and a business processing module. The multimodal Eagle Eye AI model receives pre-processed image data and point cloud data as inputs, and outputs target detection results and depth estimation results respectively. The target detection results include the category of all targets within the warehouse and the pose of each target; the depth estimation results include a monocular depth map. The business processing module generates target information corresponding to the specific business type based on the output of the multimodal Eagle Eye AI model. The WMS system obtains the current business type being executed in the warehouse and generates different handling tasks based on the target information output by the business processing module, which are then sent to the scheduling system. The scheduling system generates different AGV scheduling strategies based on the obtained handling tasks, thereby scheduling the operation of the AGVs. The multimodal Eagle Eye AI model is trained using a hybrid supervised and unsupervised training method, with the total loss function during training being... The expression is: ; In the formula: All are coefficients of the weighted terms; The loss function for the detection head; The loss function for the depth estimation head; The loss function is for depth smoothing constraints; The loss function of the depth estimation head uses different expressions depending on the camera mounting method: With the camera fixed in place, the loss function of the depth estimation head is: or ; in, ; In the formula: A set of pixels with depth annotations. The size of the set of pixels labeled with depth; d ( p (This refers to the depth map predicted by the multimodal Eagle Eye AI model.) d gt ( p (This refers to the true depth map obtained using a calibration board, lidar scanning, and structured light.) In the formula: The depth of pseudo-labels generated for teacher models or simulations; N The total number of pixels involved in the calculation; When the camera is mounted on a mobile device, the expression for the loss function of the depth estimation head is: ; In the formula: I t ( p () represents the pixel value of the current frame; The pixel value is obtained by reprojecting adjacent frames onto the current frame using the predicted depth and camera pose; δ is the weighting coefficient.

2. The AGV control method based on multimodal warehousing and transportation integration as described in claim 1, characterized in that, The multimodal Eagle Eye AI model includes: The Backbone module is used to acquire preprocessed image data and point cloud data, and extract image features from the image data and point cloud features from the point cloud data, respectively. The Neck module is used to transform the image features and point cloud features extracted by the Backbone module into the same spatial coordinate system for feature alignment and fusion to obtain a fused multimodal feature map; The Heads module includes a detection head and a depth estimation head. The detection head outputs the target category and pose of each target based on the multimodal feature map. The depth estimation head outputs a monocular depth map of the same size as the multimodal feature map based on the multimodal feature map.

3. The AGV control method based on multimodal warehousing and transportation integration as described in claim 1, characterized in that, The detection head is trained using supervised learning, and its loss function is: ; In the formula: Cross-entropy loss; For bounding box regression loss; For GIoU loss; They are respectively and Weighting coefficients.

4. The AGV control method based on multimodal warehousing and transportation integration as described in claim 1, characterized in that, The The expression for the loss function with depth smoothing constraints is: ; In the formula: and These represent the predicted depth maps. d At pixel p Place, extension x direction and y Gradient of direction; and These represent the original input images respectively. I At pixel p Place, extension x and y Gradient of direction.

5. The AGV control method based on multimodal warehousing and transportation integration as described in claim 1, characterized in that, The warehouse performs the following business operations: warehouse area management, personnel management, automated loading, and automated unloading. When performing warehouse area management, the AGV operation control strategy is as follows: The WMS system extracts the warehouse location status information from the Eagle Eye system based on the warehouse area management operations: empty location, full location, outbound, inbound, and the number of stacked layers of goods within the location. Upon inbound, the system determines the warehouse location based on the obtained warehouse location status information, the type and quantity of inbound goods, and generates an inbound handling task. The scheduling system controls the AGV to move the inbound goods to the corresponding warehouse location based on the inbound handling task. Similarly, upon outbound, the system determines the outbound location based on the obtained warehouse location status information, the type and quantity of outbound goods, and generates an outbound handling task. The scheduling system controls the AGV to move the goods out of the corresponding outbound location based on the outbound handling task. When performing personnel management tasks, the AGV operation control strategy is as follows: The WMS system extracts the personnel's pose information from the Eagle Eye system and performs spatial matching with the preset control area range in the WMS system to make the following judgment: If the personnel's pose information is within the preset control area range in the WMS system, it means that a person has entered the control area; otherwise, it means that no person has entered the control area; and when a person enters the control area, the scheduling system triggers the AGV to decelerate or even stop, and updates the AGV's running path and speed in a timely manner. When performing automated loading operations, the AGV operation control strategy is as follows: The WMS system extracts the location information of the loading truck, the type and quantity of the loaded goods, and the status information of the storage location from the Eagle Eye system; the storage location is matched according to the type and quantity of the loaded goods, and the initial pose information of the goods in the successfully matched storage location is obtained, thereby generating an automated loading and handling task; the scheduling system obtains the path and speed of the AGV operation generated by the automated loading and handling task. When performing automated unloading operations, the AGV's operation control strategy is as follows: The WMS system extracts the type and quantity of unloaded goods, the pose information of unloaded goods, and the status information of the storage location from the Eagle Eye system; the storage location is matched according to the type and quantity of unloaded goods, and the pose information of the goods after unloading is generated based on the status information of the successfully matched storage location; the WMS system generates an automated unloading and handling task based on the pose information of the unloaded goods and the pose information of the goods after unloading, and the scheduling system obtains the automated unloading and handling task and generates the AGV's running path and speed.

6. The AGV control method based on multimodal warehousing and transportation integration as described in claim 1, characterized in that, The multimodal Eagle Eye AI model outputs the truck's location information in the following way: it acquires the truck's original point cloud data through LiDAR and uses the RANSAC plane fitting algorithm to separate ground points from the original point cloud; then it uses DBSCAN clustering to obtain the point cloud cluster of the truck's main body from the non-ground point cloud; finally, it uses the minimum bounding rectangle algorithm to fit a three-dimensional bounding box to the truck's point cloud cluster, thus obtaining the three-dimensional geometric model of the truck's cargo compartment.

7. An AGV control system based on multimodal warehousing and transportation integration, which adopts the AGV control method based on multimodal warehousing and transportation integration as described in any one of claims 1-6; The AGV control eagle eye system includes: The data acquisition module is used to acquire the status and location information of the storage location, the location information of personnel, the location information of the operating equipment, the type of goods, the location information of the tools carrying the goods, the location information of the transportation tools, the position information of the goods, and the type of business currently being performed by the warehouse. The data processing module is used to process the data acquired by the data acquisition module and generate the control strategy for the AGV; The execution module is used to obtain the control strategy generated by the data processing module to control the operation of the AGV.

8. A computer program product comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the control method based on multimodal warehousing and transportation integration as described in any one of claims 1-6, thereby enabling the selection of corresponding control strategies to regulate the running path and speed of the AGV according to different business types performed by the warehouse.

Citation Information

Patent Citations

  • Logistics storage tool identification method and device, equipment and storage medium

    CN117409365A

  • Loading and unloading positioning method, device and equipment based on eagle eye system and medium

    CN120539738A

  • Method and device for controlling transfer robot

    WO2022193762A1