Method for determining a path of an unmanned loader
By identifying the status of the silo gate through real-time images and point cloud data and dynamically updating the grid map, the problem of low passage efficiency of unmanned loaders in narrow construction projects is solved, and flexible path planning and safe and efficient material transfer are realized.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SINOHYDRO BUREAU 5
- Filing Date
- 2026-04-30
- Publication Date
- 2026-06-02
AI Technical Summary
In narrow and dynamically changing construction projects, the fixed trajectory of unmanned loaders cannot adapt to real-time road conditions, resulting in low traffic efficiency. Furthermore, the status of gates that rely solely on visual recognition are easily affected by light, shadows, and dirt, resulting in low robustness.
By acquiring real-time operational images and point cloud data of the unmanned loader, the status of the hopper gate is identified, the grid map is dynamically updated, the optimal path is planned using a search algorithm, and the path is fine-tuned by combining sensor monitoring of dynamic obstacles.
It improves the mobility and path flexibility of unmanned loaders in complex environments, reduces deployment and maintenance costs, ensures on-site operation safety, and adapts to dynamically changing construction scenarios.
Smart Images
Figure CN122130091A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of electronic technology, and more specifically to a path determination method for unmanned loaders. Background Technology
[0002] In construction projects, the main roads of mixing plants are usually designed to be narrow, and the silo gates open and close frequently. Traditional unmanned loaders often rely on preset fixed tracks to operate. In narrow and dynamically changing environments, fixed tracks cannot adapt to real-time road conditions, which can easily lead to the following problems: when the gates open, although the space available for passage is temporarily increased, the unmanned loader still travels along the original track, unable to make flexible use of the space, resulting in low traffic efficiency. Summary of the Invention
[0003] The purpose of this invention is to provide a path determination method for unmanned loaders to solve the technical problems existing in related technologies.
[0004] To achieve the above objectives, the present invention provides a path determination method for an unmanned loader, comprising:
[0005] Real-time operation images and real-time point cloud data of an unmanned loader operating in a work scenario are acquired, wherein the real-time operation images and the real-time point cloud data are acquired at the same time.
[0006] The gate status of the hopper in the real-time operation image is determined based on the real-time operation image and real-time point cloud data.
[0007] The hoppers with the gate in the open state are identified as temporary access areas, and the grid map is updated through the temporary access areas to obtain the target grid map, where the grid map is a grid map of the work scene;
[0008] Obtain the current location of the unmanned loader, and determine the optimal path for the unmanned loader to perform its work in the target grid map based on the current location using a search algorithm.
[0009] Optionally, determining the gate status of the silo in the real-time operation image based on the real-time operation image and real-time point cloud data includes:
[0010] Identify the gate bounding box for each hopper in the real-time operation image;
[0011] For the gate bounding box of each hopper, the pixel coordinates of the gate bounding box are back-projected to the lidar coordinate system where the real-time point cloud data is located according to the calibration matrix, and the corresponding three-dimensional point cloud is extracted to obtain the three-dimensional point cloud cluster of the gate bounding box. The calibration matrix is used to characterize the relationship between the camera coordinate system and the lidar coordinate system.
[0012] Based on the 3D point cloud clusters of the gate bounding box, the first center pose of the left gate and the second center pose of the right gate are calculated using a clustering algorithm.
[0013] Calculate the distance between the first center pose and the second center pose, and determine the gate state of the hopper based on the distance and a preset distance threshold.
[0014] Optionally, determining the gate status of the hopper based on the spacing and a preset distance threshold includes:
[0015] When the distance is greater than the preset distance threshold, the gate status of the hopper is determined to be open.
[0016] When the spacing is less than or equal to the preset distance threshold, the gate status of the hopper is determined to be closed.
[0017] Optionally, identifying the gate bounding box of each hopper in the real-time operation image includes:
[0018] The real-time operation image is input into the target detection model to obtain the gate bounding box of each hopper, wherein the target detection model is the Mask R-CNN model.
[0019] Optionally, the step of defining the silos with the gate in the open state as temporary access areas and updating the grid map using these temporary access areas to obtain the target grid map includes:
[0020] When the gate of the hopper is in the open state, determine the size information of the hopper;
[0021] The area corresponding to the size information of the silo is designated as a temporary passage area;
[0022] Determine the first state of the silo in the grid map, and update the first state of the silo in the grid map to the second state according to the temporary passage area to obtain the target grid map, where the first state is an impassable state and the second state is a passable state.
[0023] Optionally, the method further includes:
[0024] When the unmanned loader operates in the work scenario according to the optimal path, the sensors installed on the unmanned loader monitor in real time whether there are dynamic obstacles in the optimal path. When there are dynamic obstacles in the optimal path, the optimal path is fine-tuned by the local window method or the time elastic band method to obtain the target optimal path.
[0025] Optionally, acquiring real-time operational images and real-time point cloud data of the unmanned loader operating in the work scenario includes:
[0026] Real-time operation images are acquired by an image acquisition module installed on the unmanned loader, and real-time point cloud data is acquired by a lidar installed on the unmanned loader.
[0027] Through the above technical solution, during the operation of the unmanned loader, the gate status of each hopper in the operation scene is determined based on real-time operation images and real-time point cloud data. A temporary passage area is then determined based on the gate status of each hopper, and the grid map is updated based on the temporary passage area to obtain the target grid map. Finally, the optimal path for the unmanned loader in the operation scene can be determined using a search algorithm based on the target grid map. This achieves optimal path planning based on the opening status of the hoppers. Compared with related technologies that rely on preset trajectories and pure visual detection, this improves the flexibility of the optimal path and the passage efficiency of the unmanned loader.
[0028] Other features and advantages of the present invention will be described in detail in the following detailed description section. Attached Figure Description
[0029] Figure 1 This is a schematic diagram illustrating a path determination method for an unmanned loader according to an exemplary embodiment of the present invention.
[0030] Figure 2 This is a flowchart illustrating a path determination method for an unmanned loader according to an exemplary embodiment of the present invention.
[0031] Figure 3 This is a schematic diagram illustrating a grid map update according to an exemplary embodiment of the present invention.
[0032] Figure 4 This is a schematic diagram of path planning according to an exemplary embodiment of the present invention. Detailed Implementation
[0033] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, so as to provide a better understanding of the concept of the present invention, the technical problem solved, the technical features constituting the technical solution, and the technical effects brought about.
[0034] The inventors discovered that when determining the operating path of an unmanned loader using related technologies, the following problems exist: the unmanned loader needs to operate according to a pre-deployed precise trajectory, which cannot adapt to dynamic changes in the environment and has poor flexibility; and the gate status is easily affected by light, shadow, and dirt when identified purely by vision, resulting in low robustness of identification.
[0035] In view of this, the present invention provides a path determination method for unmanned loaders to solve the technical problems existing in the above-mentioned related technologies.
[0036] like Figure 1 As shown, Figure 1 This is a schematic diagram illustrating a path determination method for an unmanned loader according to an exemplary embodiment of the present invention, with reference to... Figure 1 The method includes;
[0037] S101: Acquire real-time operation images and real-time point cloud data of the unmanned loader operating in the work scene, wherein the real-time operation images and the real-time point cloud data are acquired at the same time;
[0038] S102: Determine the gate status of the hopper in the real-time operation image based on the real-time operation image and real-time point cloud data;
[0039] S103: The hopper with the gate in the open state is identified as a temporary passage area, and the grid map is updated through the temporary passage area to obtain the target grid map, where the grid map is the grid map of the work scene;
[0040] S104: Obtain the current position of the unmanned loader, and determine the optimal path for the unmanned loader to perform operations in the work scenario in the target grid map based on the current position using a search algorithm.
[0041] Through the above technical solution, during the operation of the unmanned loader, the gate status of each hopper in the operation scene is determined based on real-time operation images and real-time point cloud data. A temporary passage area is then determined based on the gate status of each hopper, and the grid map is updated based on the temporary passage area to obtain the target grid map. Finally, the optimal path for the unmanned loader in the operation scene can be determined using a search algorithm based on the target grid map. This achieves optimal path planning based on the opening status of the hoppers, which, compared to related technologies that operate according to preset trajectories, improves the flexibility of the optimal path and increases the passage efficiency of the unmanned loader.
[0042] To enable those skilled in the art to better understand the path determination method for unmanned loaders provided by this invention, the above steps are illustrated in detail below.
[0043] For example, the operational scenario could be an unmanned loader operating at a concrete mixing plant. This scenario includes multiple hoppers, each used for storing materials. In this embodiment, an image acquisition module and a LiDAR can be installed on the unmanned loader. The image acquisition module acquires real-time operational images, and the LiDAR acquires real-time point cloud data. The image acquisition module can be a camera, and the LiDAR can be a multi-line LiDAR, with 16 or 32 lines. Both the input terminals of the image acquisition module and the LiDAR can be connected to a control module. When acquiring real-time operational images and point cloud data, the control module sends signals for both the acquired images and point cloud data simultaneously to both the image acquisition module and the LiDAR, ensuring that the real-time operational images acquired by the image acquisition module and the real-time point cloud data acquired by the LiDAR are at the same time.
[0044] Specifically, when the work scenario is a mixing plant, the unmanned loader starts at the entrance of the main road of the mixing plant. The image acquisition module can acquire real-time work images at a resolution of 1920x1080 at a frequency of 15Hz, and the lidar can acquire real-time point cloud data of the surrounding environment at a frequency of 10Hz.
[0045] For example, after determining the real-time operation image and real-time point cloud data, the gate status of all silos in the real-time operation image at the current moment can be determined based on the real-time operation image and real-time point cloud data.
[0046] In one possible manner, determining the gate status of the silo in the real-time operation image based on the real-time operation image and real-time point cloud data includes:
[0047] Identify the gate bounding box for each hopper in the real-time operation image;
[0048] For the gate bounding box of each hopper, the pixel coordinates of the gate bounding box are back-projected to the lidar coordinate system where the real-time point cloud data is located according to the calibration matrix, and the corresponding three-dimensional point cloud is extracted to obtain the three-dimensional point cloud cluster of the gate bounding box. The calibration matrix is used to characterize the relationship between the camera coordinate system and the lidar coordinate system.
[0049] Based on the 3D point cloud clusters of the gate bounding box, the first center pose of the left gate and the second center pose of the right gate are calculated using a clustering algorithm.
[0050] Calculate the distance between the first center pose and the second center pose, and determine the gate state of the hopper based on the distance and a preset distance threshold.
[0051] It should be understood that a gate can be a door to a hopper. When an unmanned loader needs to operate in a work environment, it needs to determine whether the gate of the hopper is open. If the gate is open, the loader can enter the corresponding hopper to work; otherwise, it cannot operate. Therefore, after obtaining a real-time work image, the gate bounding box in the real-time work image can be identified. Specifically, this can be done through a model or through a vector machine learning method. This embodiment of the invention does not specifically limit the method. The identified gate bounding box can be represented by the following coordinates: [x_center, y_center, width, height, confidence, class], where (x_center, y_center) are the normalized coordinates of the bounding box center in the image, width is the width information, height is the height information, confidence is the confidence level, and class is the class.
[0052] After recognizing the gate bounding box in the real-time operation image, such as Figure 2 As shown, for each gate boundary frame, its pixel coordinates in the pixel coordinate system can be determined. Then, based on the calibration matrix, the pixel coordinates of the gate boundary frame are projected back onto the lidar coordinate system where the real-time point cloud data is located. The corresponding 3D point cloud data is then extracted from the real-time point cloud data to obtain the 3D point cloud cluster of the gate boundary frame. The calibration matrix can be used to characterize the relationship between the camera coordinate system and the lidar coordinate system. Specifically, before the unmanned loader begins operation, the control module can control the image acquisition module and lidar to trigger synchronous signals, acquiring images and point cloud data at the same time. The images and point cloud data are then transformed using a joint calibration method from related technologies to obtain the calibration matrix between the image acquisition module and the lidar. A 3D point cloud cluster can be a set of point cloud data that has spatial correlation and belongs to the same physical target.
[0053] After obtaining the 3D point cloud cluster, the first center pose of the left gate and the second center pose of the right gate can be calculated using a clustering algorithm. The clustering algorithm can be DBSCAN (Density-Based Spatial Clustering of Applications with Noise) or Euclidean clustering. The first center pose can be the center pose of the left gate, and the second center pose can be the center pose of the right gate. After calculating the first and second center poses, the distance between them can be calculated. This distance can be a distance perpendicular to the gate's orientation direction, and this distance is used as the opening width W_open. The gate state of the hopper can then be determined based on the distance and a preset distance threshold, improving the robustness of the calculation.
[0054] In one possible manner, determining the gate status of the hopper based on the spacing and a preset distance threshold includes:
[0055] When the distance is greater than the preset distance threshold, the gate status of the hopper is determined to be open.
[0056] When the spacing is less than or equal to the preset distance threshold, the gate status of the hopper is determined to be closed.
[0057] It should be understood that the preset distance threshold can be used to characterize the critical value between the open and closed states of the gate. When the distance is greater than the preset distance threshold, the gate state of the hopper is determined to be open; when the distance is less than or equal to the preset distance threshold, the gate state of the hopper is determined to be closed. This allows the determination of the gate state of all hoppers. In specific implementation, the preset distance threshold is set to 0.15 meters. When W_open = 2.1 meters, which is much greater than the threshold W_threshold = 0.15 meters, the gate state of the hopper is determined to be open.
[0058] In one possible manner, identifying the gate bounding box of each hopper in the real-time operation image includes:
[0059] The real-time operation image is input into the target detection model to obtain the gate bounding box of each hopper, wherein the target detection model is the Mask R-CNN model.
[0060] It should be understood that the object detection model can be used to detect the gate bounding boxes in real-time operation images. After acquiring real-time operation images, these images can be input into the object detection model to obtain the gate bounding boxes of the silo. The object detection model can annotate the gate bounding boxes in historical operation images and train the model using these annotated historical operation images. This object detection model can be a Mask R-CNN model (Mask Region-Based Convolutional Neural Network).
[0061] For example, a grid map can be a map that discretizes the real-world environment into fixed-size grids, with each grid cell marked as having three states: idle, occupied, and unknown. It can be used for environmental representation methods in robot localization, path planning, and obstacle avoidance. In this embodiment of the invention, after determining the work scenario, a corresponding grid map can be obtained based on that scenario. When the gate of a silo in the real-time work image is detected to be in an open state, the corresponding gate area is determined as a temporary passage area. Simultaneously, the grid map corresponding to the work scenario is updated based on the temporary passage area to obtain the target grid map. Specifically, if the gate of silo A is in an occupied state, when the gate of silo A is detected to be in an open state, the occupied state of silo A in the grid map is updated to an idle state to obtain the target grid map. When the gate of silo A changes to a closed state, the idle state of silo A in the target grid map is changed back to an occupied state.
[0062] In one possible approach, the step of identifying the hopper with the gate in the open state as a temporary access area and updating the grid map using the temporary access area to obtain the target grid map includes:
[0063] When the gate of the hopper is in the open state, determine the size information of the hopper;
[0064] The area corresponding to the size information of the silo is designated as a temporary passage area;
[0065] Determine the first state of the silo in the grid map, and update the first state of the silo in the grid map to the second state according to the temporary passage area to obtain the target grid map, where the first state is an impassable state and the second state is a passable state.
[0066] It should be understood that the first state can be an occupied state, and the second state can be an idle state; however, this embodiment of the invention does not specifically limit this. In the grid map, a grid can be determined based on the size of each hopper, and the state of the grid can be determined based on the state of the gate corresponding to the hopper. For example, when the hopper gate is in the closed state, the state in the grid can be determined as the first state; when the hopper gate is in the open state, the state in the grid can be determined as the second state. When the hopper gate state changes from closed to open, the grid area corresponding to the hopper can be determined as a temporary passage area. Then, in the grid map, the first state corresponding to the hopper is updated to the second state to facilitate path planning for the unmanned loader during subsequent material retrieval.
[0067] For example, after obtaining the target grid map, the current position of the unmanned loader in the target grid map can be determined. Then, based on the current position, a search algorithm can be used to determine the optimal path for the unmanned loader to perform operations in the target grid map. The cost function of the search function can be f(n) = g(n) + h(n), where g(n) is the actual cost from the starting point to node n, and h(n) is the estimated cost from node n to the target point. This cost can be the Euclidean distance.
[0068] In some possible ways, the method further includes:
[0069] When the unmanned loader operates in the work scenario according to the optimal path, the sensors installed on the unmanned loader monitor in real time whether there are dynamic obstacles in the optimal path. When there are dynamic obstacles in the optimal path, the optimal path is fine-tuned by the local window method or the time elastic band method to obtain the target optimal path.
[0070] It should be understood that when an unmanned loader is operating according to the optimal path in a work environment, there may be situations where it is misaligned with other loaders. Therefore, a sensor is installed on the top of the unmanned loader. This sensor can monitor the road conditions ahead of the loader in real time during operation and determine whether there are dynamic obstacles based on the monitored images. If dynamic obstacles are present, the optimal path can be fine-tuned using either the local window method or the time-flexible band method to obtain the target optimal path, and the loader will then travel according to this target optimal path. The local window method is a real-time, fast, short-line-of-sight obstacle avoidance method that optimizes speed; the time-flexible band method is a local planning method that treats the global path as a time-constrained flexible band.
[0071] Specifically, when the current position of the unmanned loader is P1, the optimal path requires the unmanned loader to move from the current position P1 to the hopper position P2. During the operation of the unmanned loader, if the main road in its planned optimal path is narrow and there are oncoming vehicles, the gate of hopper A in the operation scenario is in the open state. According to the target grid map, a path can be planned so that the unmanned loader can briefly drive into the empty area in front of hopper A to avoid oncoming vehicles, and then return to the main road in the optimal path to continue driving. The route of this driving is the target optimal path.
[0072] In actual processing, such as Figure 3 and Figure 4 As shown, in a mixing plant operation scenario, the main road is narrow, with multiple silos distributed on both sides. Initially, all silo gates are closed, and the corresponding areas in the grid map are marked as occupied. An unmanned loader starts from the starting point and needs to proceed to silo B for loading operations. According to the grid map, the path planning result requires bypassing the silo B area. When the unmanned loader reaches the middle of the main road, the gate of silo B opens. The image acquisition module and LiDAR simultaneously acquire real-time data. After target detection and point cloud projection analysis, it is determined that the gate opening width is 2.1 meters, which is greater than the threshold of 0.15 meters, indicating an open status. The system then determines that the area of silo B is a temporary passage area and updates the corresponding grid in the grid map from occupied to free, obtaining the target grid map. The updated target grid map is used for real-time path replanning. The search algorithm calculates the optimal path for a straight line into silo B, and the unmanned loader directly enters silo B to complete the loading operation. Afterwards, the gate closes, the corresponding area on the map reverts to occupied, and the system automatically adjusts the subsequent path. This dynamic update mechanism eliminates detours and waiting for the loader, significantly improving operational efficiency. The aforementioned technical solution dynamically updates passable areas by real-time identification of gate opening and closing status, enabling unmanned loaders to flexibly plan paths even in narrow passages, improving traffic efficiency. Combining camera and LiDAR data, deep learning is used to identify gates and extract their point cloud positions, determining opening and closing status through patterns, significantly improving detection stability in complex environments. Without the need for pre-deployed fixed trajectories, the system autonomously plans the optimal path based on real-time environmental information, making it particularly suitable for industrial scenarios with dynamically changing structures. Dynamic lane borrowing reduces waiting and detours for the unmanned loader in narrow main roads, improving material handling efficiency; it adapts to dynamically changing operating environments without relying on fixed trajectories, reducing deployment and maintenance costs; reliable environmental perception and real-time map updates avoid collisions and congestion, ensuring on-site operational safety; it can be applied to autonomous vehicles in various narrow and dynamic scenarios such as factories, docks, and warehouses, showing broad application prospects; it achieves fully automated operation, reducing the need for manual intervention and scheduling.
[0073] The above technical solutions improve the robustness of detection. By fusing images and point clouds to detect gate status, deep learning is used to identify gate bounding boxes, and point cloud geometry is combined to calculate the opening width. This effectively avoids the shortcomings of pure visual methods, which are susceptible to interference from lighting, shadows, and dirt, significantly improving the stability and accuracy of gate status detection in complex environments. By dynamically updating the grid map, open gate areas are temporarily marked as passable, allowing unmanned loaders to utilize temporary channels in real time, avoiding detours and waiting, and greatly improving material transfer efficiency in narrow main roads. Through real-time path replanning, the system can autonomously adjust the optimal path according to dynamic environmental changes without the need for preset fixed trajectories, adapting to dynamic industrial scenarios such as mixing plants, factories, and docks, reducing deployment and maintenance costs. Reliable environmental perception and real-time map updates avoid collisions and congestion, ensuring on-site operational safety. It can be applied to autonomous vehicles in various narrow and dynamic scenarios to achieve fully automated operation, reducing the need for manual intervention and scheduling.
[0074] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A path determination method for an unmanned loader, characterized in that, include: Real-time operation images and real-time point cloud data of an unmanned loader operating in a work scenario are acquired, wherein the real-time operation images and the real-time point cloud data are acquired at the same time. The gate status of the hopper in the real-time operation image is determined based on the real-time operation image and real-time point cloud data. The hoppers with the gate in the open state are identified as temporary access areas, and the grid map is updated through the temporary access areas to obtain the target grid map, where the grid map is a grid map of the work scene; Obtain the current location of the unmanned loader, and determine the optimal path for the unmanned loader to perform its work in the target grid map based on the current location using a search algorithm.
2. The path determination method for an unmanned loader according to claim 1, characterized in that, The step of determining the gate status of the silo in the real-time operation image based on the real-time operation image and real-time point cloud data includes: Identify the gate bounding box for each hopper in the real-time operation image; For the gate bounding box of each hopper, the pixel coordinates of the gate bounding box are back-projected to the lidar coordinate system where the real-time point cloud data is located according to the calibration matrix, and the corresponding three-dimensional point cloud is extracted to obtain the three-dimensional point cloud cluster of the gate bounding box. The calibration matrix is used to characterize the relationship between the camera coordinate system and the lidar coordinate system. Based on the 3D point cloud clusters of the gate bounding box, the first center pose of the left gate and the second center pose of the right gate are calculated using a clustering algorithm. Calculate the distance between the first center pose and the second center pose, and determine the gate state of the hopper based on the distance and a preset distance threshold.
3. The path determination method for an unmanned loader according to claim 2, characterized in that, The process of determining the gate status of the hopper based on the spacing and a preset distance threshold includes: When the distance is greater than the preset distance threshold, the gate status of the hopper is determined to be open. When the spacing is less than or equal to the preset distance threshold, the gate status of the hopper is determined to be closed.
4. The path determination method for an unmanned loader according to claim 2, characterized in that, The identification of the gate boundary box for each hopper in the real-time operation image includes: The real-time operation image is input into the target detection model to obtain the gate bounding box of each hopper, wherein the target detection model is the Mask R-CNN model.
5. The path determination method for an unmanned loader according to claim 1, characterized in that, The process of identifying silos with open gates as temporary access areas and updating the grid map using these temporary access areas to obtain the target grid map includes: When the gate of the hopper is in the open state, determine the size information of the hopper; The area corresponding to the size information of the silo is designated as a temporary passage area; Determine the first state of the silo in the grid map, and update the first state of the silo in the grid map to the second state according to the temporary passage area to obtain the target grid map, where the first state is an impassable state and the second state is a passable state.
6. The path determination method for an unmanned loader according to claim 1, characterized in that, The method further includes: When the unmanned loader operates in the work scenario according to the optimal path, the sensors installed on the unmanned loader monitor in real time whether there are dynamic obstacles in the optimal path. When there are dynamic obstacles in the optimal path, the optimal path is fine-tuned by the local window method or the time elastic band method to obtain the target optimal path.
7. The path determination method for an unmanned loader according to any one of claims 1-6, characterized in that, The acquisition of real-time operational images and real-time point cloud data of the unmanned loader operating in the work scenario includes: Real-time operation images are acquired by an image acquisition module installed on the unmanned loader, and real-time point cloud data is acquired by a lidar installed on the unmanned loader.