Granary automatic operation robot system based on multi-sensor fusion and operation method

Abstract

The invention discloses a granary automatic operation robot system based on multi-sensor fusion and an operation method.The system comprises an unmanned aerial vehicle, a grain suction robot, a grain conveying robot, a sensor module, a communication module and a cloud server, and the sensor module and the communication module are arranged on the unmanned aerial vehicle, the grain suction robot and the grain conveying robot; the unmanned aerial vehicle, the grain suction robot and the grain conveying robot are connected with the cloud server through the communication module. The unmanned aerial vehicle collects environmental data in the granary through the sensor module, constructs a granary map, performs remaining grain estimation, and uploads information to the cloud server; the grain suction robot and the grain conveying robot download the granary map from the cloud server and then perform collaborative operation; and the cloud server carries out coordinated scheduling distribution on tasks of the unmanned aerial vehicle, the grain suction robot and the grain conveying robot and monitors related numerical values. Manual dependence in traditional granary operation is reduced, the labor intensity is reduced, the safety risk is reduced, and the operation efficiency is improved.

Description

Technical Field

[0001] This invention relates to an automated grain storage robot system and operation method based on multi-sensor fusion, belonging to the field of smart agriculture technology. Background Technology

[0002] In recent years, with the rapid development of artificial intelligence technology, the application of robots and drones in agriculture has gradually become an important part of smart agriculture. These devices, driven by multimodal sensor fusion and AI algorithms, have greatly improved the efficiency and intelligence of agricultural production. Grain storage operations are complex and dangerous, with issues such as high temperatures and dust, posing significant risks to manual operation. Against this backdrop, the application of robotics technology has become a key breakthrough. By introducing intelligent robots, grain storage operations can be automated, reducing reliance on manual labor, lowering labor intensity and safety risks, and improving operational efficiency. Using multi-sensor fusion, high-precision dynamic maps of the grain storage room are constructed through LiDAR, multi-view cameras, and inertial navigation; drones collect data on the distribution of grain within the storage room from the air, guiding suction and conveying robots to precise positioning, enabling the transfer and transport of grain.

[0003] In a grain depot, the operational mode varies depending on the state of the remaining grain. When the depot is nearly full, the grain suction and conveying robots are positioned around the perimeter, relying on gravity to transfer and transport the grain. When the remaining grain is scattered and irregularly distributed at the bottom, drones, along with the suction and conveying robots, can enter the depot to conduct multi-robot collaborative operations using a "ground main platform + aerial auxiliary unit." The drones are used for information monitoring, with a flight time of approximately 5-10 minutes. After completing the modeling of the remaining grain accumulation, they return to the ground to park. After the suction and conveying robots have altered the distribution of the remaining grain, the drones take off again to collect information, and this cycle repeats continuously.

[0004] Traditional visual positioning strategies suffer from significant inaccuracies in low-light environments inside grain silos, while lidar and inertial navigation systems (INS) are suitable for use in such conditions as grain silos because they are independent of ambient light. LiDAR, as an active sensor, measures distance by emitting a laser beam and receiving the reflected beam. It is independent of ambient light intensity and can operate stably in the low-light environment of a grain silo, providing accurate distance measurements and environmental information. Furthermore, it has strong anti-interference capabilities; even with interference factors such as dust and particulate matter present in a grain silo, lidar can still provide reliable measurement data. Inertial navigation (INS) is a navigation method that does not rely on external environmental information. It derives an object's position and attitude by measuring its acceleration and angular velocity. In the low-light environment of a grain silo, INS can provide stable positioning information; however, the navigation information is generated through integration, and the positioning error increases over time, leading to decreased accuracy with long-term use. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of the prior art by providing a multi-sensor fusion-based automated grain storage robot system. This system combines key technologies such as unmanned aerial vehicle (UAV) technology, robot path planning, multi-sensor fusion, collaborative operation, and 5G communication to achieve intelligent and efficient grain transportation in grain storage facilities. This reduces reliance on manual labor in traditional grain storage operations, lowers labor intensity, reduces safety risks, and improves operational efficiency.

[0006] Another object of the present invention is to provide an operation method based on the above-mentioned automated grain warehouse operation robot system.

[0007] The objective of this invention can be achieved by adopting the following technical solutions:

[0008] An automated grain storage robot system based on multi-sensor fusion includes a drone, a grain suction robot, a grain conveying robot, a sensor module, a communication module, and a cloud server. The sensor module and the communication module are installed on the drone, the grain suction robot, and the grain conveying robot. The drone, the grain suction robot, and the grain conveying robot are respectively connected to the cloud server through the communication module.

[0009] The drone collects environmental data inside the grain warehouse through its sensor module, constructs a grain warehouse map, estimates the remaining grain, and uploads the information to a cloud server. The grain suction robot and the grain conveying robot download the grain warehouse map from the cloud server and then work together. The cloud server coordinates and schedules the tasks of the drone, the grain suction robot, and the grain conveying robot, and monitors relevant values.

[0010] Furthermore, it also includes a user terminal, which is connected to the cloud server and is used to present the cloud server's data, decision results, task allocation, and environmental monitoring results to the user.

[0011] Furthermore, the user terminal includes a mobile terminal and a computer terminal, both of which include interactive components.

[0012] Furthermore, the sensor module includes a lidar, a multi-view camera, and an inertial sensor. The lidar is used for robot localization and environmental perception, the multi-view camera is used for robot target detection and recognition, and the inertial sensor is used for robot path correction.

[0013] Furthermore, the cloud server is equipped with a monitoring center, which is used to monitor the operating status, location information, and task execution of the robots in the grain warehouse.

[0014] Furthermore, each robot is equipped with a node to collect sensor data, location information, and instructions for performing tasks. The monitoring center communicates with the robot node to obtain robot status updates periodically.

[0015] Another objective of this invention can be achieved by adopting the following technical solution:

[0016] An automated grain storage operation method, implemented based on the aforementioned automated grain storage robot system, the method comprising:

[0017] Drones, grain suction robots, and grain delivery robots are assigned tasks through a cloud server. Each grain suction robot is assigned one drone and one grain delivery robot. The grain suction robot stops suctioning when the weight of grain on the delivery robot reaches a preset value. The delivery robot then transports the grain to the designated location as required. The grain suction robot waits for a new delivery robot to arrive before continuing to suction grain until the task is completed. The task assignment is established in the cloud server, and each grain suction robot is assigned a work area. The delivery robot is assigned according to the amount of grain collected.

[0018] When drones monitor grain warehouses, multi-view cameras perform preliminary positioning of the remaining grain, grain suction robots, and grain conveying robots in the grain warehouse. The weight and two-dimensional planar position of each remaining grain pile in the grain warehouse are marked by a pre-built grain remaining grain detection model. The lidar and inertial sensors, combined with the two-dimensional images from the multi-view cameras, construct a three-dimensional point cloud map of the grain warehouse. The cloud server receives the three-dimensional point cloud map uploaded by the drone and transmits the three-dimensional point cloud map to the grain suction robots and grain conveying robots.

[0019] The grain-suction robot collects grain using a hybrid ant colony-particle swarm optimization algorithm, while the grain-transporting robot maintains synchronization with it using an indoor low-light following model based on multi-camera, lidar, and inertial sensors. Once the suction robot reaches the designated location, it locates the grain pile, operates the working mechanism to suck up the grain, and transports it to the transporting robot via the transport mechanism. The 3D point cloud map is updated in real time, and changes are uploaded to a cloud server. When the transporting robot needs to move grain to a designated location, it downloads the updated 3D point cloud map from the cloud server, plans the global path for the transporting robot, and dynamically adjusts the path's local details in real time during its movement to adapt to environmental changes.

[0020] Furthermore, the construction of the 3D point cloud map within the grain warehouse specifically includes:

[0021] Based on the two-dimensional planar positions of each surplus grain pile in the granary marked by the two-dimensional image, a three-dimensional point cloud model of the surplus grain pile shape at that position is generated, and a three-dimensional point cloud map of the granary is constructed.

[0022] Furthermore, the method of maintaining synchronization distance with the grain-suction robot through an indoor low-light following model based on multi-view cameras, lidar, and inertial sensors specifically includes:

[0023] The system uses multi-camera to identify and track the position and attitude of the grain-suction robot, lidar to measure the distance to the grain-suction robot and surrounding obstacles, and inertial sensors to measure the position and attitude of the grain-delivery robot, providing high-precision positioning information in a short time.

[0024] Furthermore, the construction process of the grain surplus detection model includes:

[0025] Two-dimensional images of grain in a granary are collected by drones. The remaining grain piles in the two-dimensional images are marked and labeled according to the approximate weight of the remaining grain piles in the actual situation. A dataset is constructed and divided into a training set and a validation set. The target detection of the remaining grain piles on the two-dimensional images and the fitting of the relationship between the remaining grain piles and the weight are trained by the target detection model and the backpropagation model, respectively. The validation set is used to verify and adjust the parameters, and the combined results are used to obtain the grain remaining grain detection model.

[0026] The present invention has the following advantages over the prior art:

[0027] 1. This invention uses a multi-view camera mounted on a drone to initially locate the grain pile, the grain suction robot, and the grain conveying robot within the grain silo. Then, using onboard LiDAR and inertial sensors, it detects the volume and distribution of grain within the silo in real time, constructing a high-precision indoor dynamic map. This map information is transmitted to the grain suction and conveying robots. The grain suction robot creates a heatmap of grain transport demand using the 3D map information. Globally, it employs an improved ant colony algorithm for dynamic obstacle avoidance and path length compression. Locally, it uses a depth vision sensor to locate the grain pile and collect the grain. Before the grain is fully loaded, the grain conveying robot uses infrared-laser-inertial navigation multi-mode following combined with UWB ultra-wideband technology to follow the grain suction robot and collect the grain. Once the grain conveying robot is fully loaded, it plans a global-local path using a real-time updated 3D map to transport the grain to the designated location.

[0028] 2. This invention addresses multi-machine collaborative control communication between drones and robots, employing a cloud server as the overall communication framework of the system. This server is responsible for integrating information exchange between the various machines, scheduling tasks for each group of machines, and uploading the required data to the user terminal for easy viewing. Attached Figure Description

[0029] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying any creative work.

[0030] Figure 1 This is a block diagram of the overall structure of the automated grain storage robot system based on multi-sensor fusion according to an embodiment of the present invention.

[0031] Figure 2 This is a structural block diagram of the grain suction robot and grain conveying robot according to an embodiment of the present invention.

[0032] Figure 3 This is a schematic diagram illustrating the positioning process of the grain suction robot and the grain conveying robot according to an embodiment of the present invention.

[0033] Figure 4 This is a schematic diagram of the grain storage operation process according to an embodiment of the present invention.

[0034] Figure 5 This is a flowchart of the monitoring center according to an embodiment of the present invention.

[0035] Figure 6 This is a flowchart illustrating the operation of a robot inside a grain warehouse according to an embodiment of the present invention.

[0036] Figure 7 This is a flowchart illustrating the construction of the grain surplus detection model in an embodiment of the present invention. Detailed Implementation

[0037] To better understand the invention, various aspects of the invention will be described in more detail with reference to the accompanying drawings. It should be understood that these detailed descriptions are merely illustrative of exemplary embodiments of the invention and are not intended to limit the scope of the invention in any way. Throughout the specification, the same reference numerals refer to the same elements. The expression "and / or" includes any and all combinations of one or more of the associated listed items.

[0038] In the accompanying drawings, the size, dimensions, and shapes of the elements have been slightly adjusted for ease of illustration. The drawings are for illustrative purposes only and are not strictly to scale. As used herein, the terms “approximately,” “about,” and similar terms are used to indicate approximation, not degree, and are intended to illustrate inherent deviations in measured or calculated values ​​that will be recognized by those skilled in the art. Furthermore, the order in which the steps are described in this invention does not necessarily indicate the order in which these steps occur in actual operation, unless otherwise expressly defined or deduced from the context.

[0039] It should also be understood that expressions such as "comprising," "including," "having," "containing," and / or "comprising" are open-ended rather than closed-ended expressions in this specification, indicating the presence of the stated features, elements, and / or components, but not excluding the presence of one or more other features, elements, components, and / or combinations thereof. Furthermore, when expressions such as "at least one of..." appear after a list of listed features, they modify the entire list of features, not just individual elements in the list. Additionally, when describing embodiments of the invention, the word "may" is used to mean "one or more embodiments of the invention." And the term "exemplary" is intended to refer to examples or illustrations.

[0040] Unless otherwise specified, all terms used herein (including engineering and technical terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that, unless expressly stated herein, terms defined in common dictionaries shall be interpreted as having the meaning consistent with their meaning in the context of the relevant art, and not in an idealized or overly formalized sense.

[0041] It should be noted that, unless otherwise specified, the embodiments and features described in this invention can be combined with each other. The invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0042] Example:

[0043] like Figure 1 As shown, this embodiment provides an automated grain storage robot system based on multi-sensor fusion. This system enables efficient collaboration and accurate positioning of the grain suction robot and the grain conveying robot under the guidance of a drone, providing a feasible solution for intelligent and unmanned grain storage operations. It can be used for grain transportation and environmental monitoring in grain storage facilities. The system includes a drone, a grain suction robot, a grain conveying robot, sensor modules, a communication module, and a cloud server. The sensor and communication modules are mounted on the drone, the grain suction robot, and the grain conveying robot. The drone, the grain suction robot, and the grain conveying robot are connected to the cloud server via the communication modules. The drone collects environmental data within the grain storage facility through the sensor modules, constructs a grain storage map, estimates remaining grain, and uploads the information to the cloud server. The grain suction robot and the grain conveying robot download the grain storage map from the cloud server and then perform collaborative operations. The cloud server coordinates and schedules the tasks of the drone, the grain suction robot, and the grain conveying robot, and monitors relevant values.

[0044] Furthermore, the automated grain storage robot system may also include a user terminal, which is connected to a cloud server. The user terminal is used to present the cloud server's data, decision results, task allocation, and environmental monitoring results to the user. Specifically, the user terminal includes a mobile terminal and a computer terminal, both of which include interactive components. The mobile terminal uses an app, and the computer terminal uses a webpage. Users can view and download historical data, images, etc., through the interactive components.

[0045] Furthermore, drones used to guide robots need to possess a range of technologies and features, including navigation and positioning capabilities, remote sensing and monitoring capabilities, path planning and obstacle avoidance capabilities, and transmit high-altitude monitoring data to a cloud server to achieve efficient and precise guidance for grain delivery tasks. Grain-suction robots, used to extract grain, require a mobile chassis with autonomous navigation and positioning capabilities, equipped with a lifting mechanism and a suction nozzle. They create a negative pressure environment using a built-in fan or vacuum pump to suck the grain into a storage device, and also have a grain-transferring mechanism to accurately transfer the stored grain to the grain-transferring robot. Grain-transferring robots, used to transport grain, need to possess a range of capabilities, including positioning and navigation, collaborative communication, dynamic path planning, obstacle avoidance, speed adjustment, and synchronous motion control, to ensure they maintain a synchronized and coordinated relative position with the grain-suction robot and successfully complete the grain delivery task.

[0046] Furthermore, the cloud server serves as a 5G cloud control platform, enabling the monitoring, collaborative scheduling, and task management of multiple robots within the grain warehouse. This improves the operational efficiency of the grain warehouse, reduces human resource input, and provides grain warehouse managers with real-time operational status and data analysis to support decision-making and optimization. The cloud server can implement scheduling strategies and real-time monitoring of robots, considering factors such as the endurance and real-time location of grain suction and transport robots, and scheduling them based on factors such as the urgency and distance of the task. A monitoring center is established to monitor and control the position, status, and task execution of robots. Sensors and other devices are used to acquire robot information in real time, and the data is transmitted to the cloud server via wireless communication technology. Based on the monitoring data, path planning and scheduling are adjusted and optimized in a timely manner.

[0047] Furthermore, the sensor module includes LiDAR, multi-view cameras, and inertial sensors. LiDAR is used for robot localization and environmental perception, enabling real-time acquisition of 3D data of the surrounding environment to provide the robot with path planning and obstacle avoidance operations. The multi-view cameras are used for robot target detection and recognition, enabling the localization of grain piles, and are equipped with low-light image enhancement to reduce misidentification errors. The inertial sensors are used for robot path correction, reducing path deviation and improving positioning stability. The sensor modules on the grain suction robot and grain conveying robot can also include monitoring sensors responsible for monitoring the indoor environment of the grain silo, such as dust sensors, temperature and humidity sensors, and ambient moisture sensors.

[0048] Furthermore, the communication module is a 5G communication module, which is installed on the drone, the grain suction robot, and the grain delivery robot to connect them to the network.

[0049] like Figure 2 As shown, the grain suction robot and grain conveying robot use a tracked vehicle as a carrier, on which are installed a computer, LiDAR, multi-view camera, inertial sensor, communication module, actuator, and monitoring sensor. In order to save installation space while ensuring high performance of the computer, a high-performance, low-power embedded board NVIDIA Jetson TX2 is selected. The LiDAR, multi-view camera, inertial sensor, communication module, and monitoring sensor are all connected to the I / O port of the embedded board. The embedded board receives data from the sensors and feeds it back to the cloud server through the robot's communication module to monitor changes in the environment inside the grain silo. Before the robot is used, it needs to be initialized, including the initial parameter settings of the LiDAR, multi-view camera, inertial sensor, etc., and the communication number set on the cloud server for individual identification in multi-machine collaboration.

[0050] like Figure 3 As shown, this is the localization process of the grain suction robot and the grain conveying robot. After initial localization using the aerial vision of the UAV, the lidar on the grain suction robot and the grain conveying robot scans the surrounding point cloud and performs feature point matching with the surrounding point cloud of the initial localization position on the 3D map to correct the localization position. At the same time, the data on the inertial sensor estimates the position and attitude of the robot. After processing the data and filtering out noise using Kalman filtering, the data fusion is used to correlate the localization results to obtain the position of the grain suction robot and the grain conveying robot on the map.

[0051] like Figure 4The diagram shows the operation process of a drone, a grain suction robot, and a grain conveying robot inside a grain warehouse. After collecting map information, the drone uploads it to a cloud server. The grain suction robot and the grain conveying robot download the map from the cloud server. The grain suction robot uses an improved ant colony algorithm, namely the ant colony-particle swarm optimization (ACO-PSO) hybrid optimization algorithm, for grain collection path planning. The grain conveying robot uses indoor low light to follow the grain suction robot and uses A* and DTW grain transport path planning to transport the grain to the designated location.

[0052] like Figure 5 The diagram illustrates the monitoring process of the monitoring center. As the core component of the cloud server, the monitoring center is responsible for monitoring the operational status, location information, and task execution of the robots in the grain warehouse. Each robot is equipped with a node to collect sensor data, location information, and task execution instructions. The monitoring center periodically obtains robot status updates through communication with the robot nodes. This includes the robot's position, sensor data (such as vision and distance sensors), and task execution progress. The monitoring center uses real-time image transmission technology to receive image data acquired by the robot sensors to monitor the situation in the grain warehouse. Based on the grain warehouse's needs and the robot's status information, the monitoring center performs task scheduling and coordination. Tasks are assigned to available robots according to the grain warehouse's transportation requirements. By monitoring the robot's status and sensor data in real time, the monitoring center determines whether there are any alarm-prone events. Alarm-prone events, such as robot malfunctions or task delays, are reported by the alarm center to the user terminal and the cloud server, and an alarm is issued or the relevant maintenance personnel are notified for handling.

[0053] Furthermore, each robot communicates with the monitoring center to form a hybrid architecture multi-robot system. Robots can also communicate with each other to exchange information and work collaboratively. Appropriate communication mechanisms and collaboration strategies are formulated to ensure seamless collaboration between robots, sharing task status, location information, and sensor data. Robots cooperate with each other to complete tasks and achieve efficient workflows through reasonable collaboration strategies and communication mechanisms.

[0054] like Figure 6The diagram illustrates the robot's workflow within a grain silo. Before proceeding, the robot assesses the grain's accumulation status. When the silo is full, the robot operates from the perimeter, not entering the interior. Once the silo's interior space allows for robot entry, a drone first scans and maps the area, followed by a suction robot and a transport robot. When the transport robot is fully loaded, it detaches from the suction robot and transports the grain to a designated location. The suction robot then requests available transport robots from the cloud server. If no transport robots are available, the robot ceases operation, waiting for the transport robots to complete their transport tasks. The cloud server's task allocation determines whether the current grain silo task is complete and whether new task requests are pending.

[0055] like Figure 7 The diagram illustrates the construction process of the grain surplus detection model. Two-dimensional images of grain inside a grain warehouse are collected using drones. Annotation tools are used to mark the surplus grain piles in the two-dimensional images, and these piles are labeled according to their approximate weight in reality. A dataset is constructed and divided into training and validation sets. An object detection model and a backpropagation (BP) model are used to train the object detection of surplus grain piles in the two-dimensional images and the fitting of the relationship between surplus grain piles and weight, respectively. The validation set is used to validate and adjust parameters, resulting in the grain surplus detection model. The object detection model uses the YOLO-R object detection model. To ensure the accuracy of multi-camera object detection, low-light image enhancement preprocessing is used on the images acquired by the multi-camera. The image training set of the grain surplus detection model also undergoes low-light image enhancement processing to ensure the accuracy of the recognition network.

[0056] This embodiment also provides an automated grain storage operation method, which is based on the above-mentioned automated grain storage robot system and includes:

[0057] S1. Drones, grain suction robots, and grain conveying robots are assigned tasks via a cloud server. Each grain suction robot is assigned one drone and one grain conveying robot. The grain suction robot stops suctioning when the weight of grain on the conveying robot reaches a preset value, and the conveying robot transports the grain to the designated location as required. The grain suction robot waits for a new conveying robot to arrive before continuing to suction grain until the task is completed. The task assignment is established in the cloud server, and each grain suction robot is assigned a work area. The conveying robot is assigned according to the grain collection situation. In addition, the central server communicates with the grain suction robots through a communication module and monitors the grain collection situation in real time, providing feedback to the user terminal. Based on the collection situation, the system can better schedule the conveying robots and estimate the grain collection situation to guide the subsequent grain storage planning in the grain warehouse.

[0058] S2. When the drone monitors the grain warehouse, the multi-view camera performs preliminary positioning of the remaining grain, the grain suction robot, and the grain conveying robot in the grain warehouse. The weight and two-dimensional planar position of each remaining grain pile in the grain warehouse are marked by a pre-constructed grain remaining grain detection model. The lidar and inertial sensor, combined with the two-dimensional images of the multi-view camera, construct a three-dimensional point cloud map of the grain warehouse. The cloud server receives the three-dimensional point cloud map uploaded by the drone and transmits the three-dimensional point cloud map to the grain suction robot and the grain conveying robot.

[0059] In this embodiment, the construction of a three-dimensional point cloud map inside the grain warehouse is specifically as follows: based on the two-dimensional planar position of each surplus grain pile in the grain warehouse marked by the two-dimensional image, a three-dimensional point cloud model of the surplus grain pile shape at that position is generated, and a three-dimensional point cloud map inside the grain warehouse is constructed.

[0060] S3. The grain-suction robot collects grain using a hybrid optimization algorithm of ant colony and particle swarm optimization. The grain-transporting robot maintains a synchronized distance with the grain-suction robot using an indoor low-light following model based on multi-view cameras, lidar, and inertial sensors. When the grain-suction robot reaches the designated location, it locates the grain pile, operates the working mechanism to suck up the grain, and transports it to the grain-transporting robot via the grain-transporting mechanism. The 3D point cloud map is updated in real time, and changes are uploaded to the cloud server. When the grain-transporting robot needs to transport grain to a designated location, it downloads the updated 3D point cloud map from the cloud server, plans the global path of the grain-transporting robot, and adjusts the local details of the path in real time during the movement of the grain-transporting robot, dynamically adjusting the path to adapt to environmental changes.

[0061] In this embodiment, the Ant Colony Optimization (ACO) algorithm, by simulating ant foraging behavior, utilizes the accumulation and volatilization mechanism of pheromones for path optimization, exhibiting strong global search capabilities and good adaptability. However, the ACO algorithm is prone to getting trapped in local optima in the initial stage. The Particle Swarm Optimization (PSO) algorithm, on the other hand, simulates bird flock foraging behavior, possessing advantages such as fast convergence and strong global search capabilities. By combining the global search capability of the PSO algorithm with the local optimization capability of the ACO algorithm, the ant-particle swarm hybrid optimization algorithm can plan paths more efficiently. Based on the measurement results of surplus grain piles in the grain warehouse using the grain surplus detection model, combined with the 3D point cloud of the surplus grain pile morphology, the pheromone level of each surplus grain pile is assigned. The pheromone level is also related to the path length and obstacle density; the shorter the path, the higher the pheromone concentration, attracting the grain suction vehicle to prioritize that path; the fewer obstacles on the path, the higher the pheromone concentration. Simultaneously, the pheromone concentration is dynamically adjusted based on the size and location of the grain pile, prioritizing larger grain piles. As time progresses, the pheromone concentration along the path gradually decreases, simulating the evaporation process of pheromones in nature and preventing the algorithm from getting trapped in local optima. The particle swarm optimization algorithm uses the optimal positions of individual particles and the optimal positions of the swarm to update the particle speed and position, accelerating the ant colony algorithm's search for the globally optimal solution path, reducing path planning time, and improving operational efficiency. This allows for the construction of a movement path for the grain-suction robot to multiple target points.

[0062] In this embodiment, an indoor low-light following model based on multi-camera, lidar, and inertial sensors is used to maintain a synchronized distance with the grain-suction robot. Specifically, the multi-camera identifies and tracks the position and attitude of the grain-suction robot, the lidar measures the distance to the grain-suction robot and surrounding obstacles, and the inertial sensor measures the position and attitude of the grain-transporting robot, providing high-precision positioning information in a short time to compensate for the shortcomings of multi-camera and lidar in dynamic environments. The speed control of the grain-suction robot and the grain-transporting robot uses a master-slave collaborative control algorithm. This algorithm includes PD control logic with state feedback and disturbance feedforward to adjust the system's response speed and stability, thus constructing a master-slave mobile robot navigation system. The speed consistency decision component is used as PD control, which can avoid problems caused by the speed difference between the two mobile robots. Because the drive system of the two mobile robots has certain delay and nonlinear characteristics during cooperative driving, a dead zone is set during the control process to improve stability. When the longitudinal deviation is within the dead zone range, the speed needs to remain relatively stable. In order to ensure the response speed of master-slave cooperation, the grain suction robot and the grain conveying robot use UWB ultra-wideband technology for short-range wireless communication to share speed information and control commands, and correct speed differences through a feedback mechanism to ensure that the two mobile robots maintain an appropriate distance and relative position.

[0063] In this embodiment, when the grain-suction robot reaches the designated location, it uses a depth vision sensor to locate the grain pile, and the operating mechanism sucks up the grain and transports it to the grain-transporting robot through the grain-transporting mechanism. At the same time, the onboard LiDAR and multi-view camera update the 3D point cloud map in real time and upload the changes to the cloud server. When the grain-transporting robot needs to transport grain to the designated location, it downloads the updated map from the cloud server and uses the A* algorithm to plan the global path of the grain-transporting robot. Meanwhile, dynamic time warping (DTW) adjusts the local details of the path in real time during the movement of the grain-transporting robot, dynamically adjusting the path to adapt to environmental changes.

[0064] In summary, the automated grain storage robot system of this embodiment achieves highly efficient and intelligent grain storage operations through multi-sensor fusion, AI algorithm optimization, and 5G communication technology, and has the following advantages:

[0065] 1) High-precision positioning and environmental perception: By integrating LiDAR, multi-view cameras and inertial sensors, a high-precision three-dimensional dynamic map is constructed, which overcomes the limitations of traditional visual positioning in low-light environments and provides stable and reliable positioning and environmental perception capabilities.

[0066] 2) Multi-machine collaborative operation: Drones, grain suction robots, and grain delivery robots achieve efficient collaboration through a cloud server. Drones are responsible for monitoring and guidance, grain suction robots perform precise collection, and grain delivery robots use a multimodal following system to ensure synchronous operation, forming a complete automated operation chain.

[0067] 3) Real-time monitoring and intelligent scheduling: The cloud server monitors the robot's status and environmental data in real time, and performs intelligent scheduling based on the urgency of the task and the robot's location, improving system response speed and operational flexibility. Users can remotely monitor grain warehouse operations by monitoring the operational situation inside the grain warehouse through sensors mounted on the robots.

[0068] It should be noted that, in this invention, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus.

[0069] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A grain silo automated operation robot system based on multi-sensor fusion, characterized in that, It includes a drone, a grain suction robot, a grain conveying robot, a sensor module, a communication module, and a cloud server. The sensor module and the communication module are installed on the drone, the grain suction robot, and the grain conveying robot. The drone, the grain suction robot, and the grain conveying robot are connected to the cloud server through the communication module. The drone collects environmental data inside the grain warehouse through sensor modules, constructs a grain warehouse map, estimates the remaining grain, and uploads the information to a cloud server. The grain suction robot and the grain conveying robot download the grain warehouse map from the cloud server and then work together. The cloud server coordinates and schedules the tasks of the drones, grain suction robots, and grain delivery robots, and monitors relevant data.

2. The automated grain storage robot system according to claim 1, characterized in that, It also includes a user terminal, which is connected to a cloud server and is used to present the cloud server's data, decision results, task allocation, and environmental monitoring results to the user.

3. The automated grain storage robot system according to claim 2, characterized in that, The user terminal includes a mobile terminal and a computer terminal, and both the mobile terminal and the computer terminal include interactive components.

4. The automated grain storage robot system according to claim 1, characterized in that, The sensor module includes a lidar, a multi-view camera, and an inertial sensor. The lidar is used for robot localization and environmental perception, the multi-view camera is used for robot target detection and recognition, and the inertial sensor is used for robot path correction.

5. The automated grain storage robot system according to claim 1, characterized in that, The cloud server is equipped with a monitoring center, which is used to monitor the operating status, location information, and task execution of the robots in the grain warehouse.

6. The automated grain storage robot system according to claim 5, characterized in that, Each robot is equipped with a node to collect sensor data, location information, and instructions for performing tasks. The monitoring center communicates with the robot node to obtain robot status updates periodically.

7. A method for automated grain storage operations, implemented based on the automated grain storage robot system according to any one of claims 1-6, characterized in that, The method includes: Drones, grain suction robots, and grain delivery robots are assigned tasks through a cloud server. Each grain suction robot is assigned one drone and one grain delivery robot. The grain suction robot stops suctioning when the weight of grain on the delivery robot reaches a preset value. The delivery robot then transports the grain to the designated location as required. The grain suction robot waits for a new delivery robot to arrive before continuing to suction grain until the task is completed. The task assignment is established in the cloud server, and each grain suction robot is assigned a work area. The delivery robot is assigned according to the amount of grain collected. When drones monitor grain warehouses, multi-view cameras perform preliminary positioning of the remaining grain, grain suction robots, and grain conveying robots in the grain warehouse. The weight and two-dimensional planar position of each remaining grain pile in the grain warehouse are marked by a pre-built grain remaining grain detection model. The lidar and inertial sensors, combined with the two-dimensional images from the multi-view cameras, construct a three-dimensional point cloud map of the grain warehouse. The cloud server receives the three-dimensional point cloud map uploaded by the drone and transmits the three-dimensional point cloud map to the grain suction robots and grain conveying robots. The grain-suction robot collects grain using a hybrid ant colony-particle swarm optimization algorithm, while the grain-transporting robot maintains synchronization with it using an indoor low-light following model based on multi-camera, lidar, and inertial sensors. Once the suction robot reaches the designated location, it locates the grain pile, operates the working mechanism to suck up the grain, and transports it to the transporting robot via the transport mechanism. The 3D point cloud map is updated in real time, and changes are uploaded to a cloud server. When the transporting robot needs to move grain to a designated location, it downloads the updated 3D point cloud map from the cloud server, plans the global path for the transporting robot, and dynamically adjusts the path's local details in real time during its movement to adapt to environmental changes.

8. The automated grain storage operation method according to claim 7, characterized in that, The construction of the 3D point cloud map within the grain warehouse specifically includes: Based on the two-dimensional planar positions of each surplus grain pile in the granary marked by the two-dimensional image, a three-dimensional point cloud model of the surplus grain pile shape at that position is generated, and a three-dimensional point cloud map of the granary is constructed.

9. The automated grain storage operation method according to claim 7, characterized in that, The method of maintaining a synchronized distance with the grain-suction robot using an indoor low-light following model based on multi-camera-LiDAR-inertial sensors specifically includes: The system uses multi-camera to identify and track the position and attitude of the grain-suction robot, lidar to measure the distance to the grain-suction robot and surrounding obstacles, and inertial sensors to measure the position and attitude of the grain-delivery robot, providing high-precision positioning information in a short time.

10. The automated grain storage operation method according to claim 7, characterized in that, The construction process of the grain surplus detection model includes: Two-dimensional images of grain in a granary are collected by drones. The remaining grain piles in the two-dimensional images are marked and labeled according to the approximate weight of the remaining grain piles in the actual situation. A dataset is constructed and divided into a training set and a validation set. The target detection of the remaining grain piles on the two-dimensional images and the fitting of the relationship between the remaining grain piles and the weight are trained by the target detection model and the backpropagation model, respectively. The validation set is used to verify and adjust the parameters, and the combined results are used to obtain the grain remaining grain detection model.

Images