Automatic granary loading system and method based on visual and laser data fusion

Abstract

The invention discloses an automatic granary loading system and method based on vision and laser data fusion. The system comprises a discharging amount detection unit, a three-dimensional scanning unit, a multi-modal data fusion module and an automatic loading control system. The blanking amount detection unit is used for collecting video data of a target vehicle and detecting the blanking amount through a visual analysis deep learning model; the three-dimensional scanning unit is used for acquiring position and posture information of a truck container in real time and quickly calculating the grain volume through point cloud data; and the multi-modal data fusion module is used for receiving data information acquired by the blanking quantity detection unit and the three-dimensional scanning unit, dynamically adjusting the weight through a Kalman filtering algorithm, and fusing an optimal volume estimation value. The grain loading truck can automatically measure the volume of grains on the grain loading truck, accurate automatic grain loading is achieved, and the loading efficiency is improved.

Description

Technical Field

[0001] This invention relates to the field of automated loading technology, and in particular to an automated grain loading system and method based on the fusion of vision and laser data. Background Technology

[0002] Grain warehouses, as specialized buildings for storing grain, are used to store large quantities of grain. Traditional grain warehouse storage methods mainly rely on manual labor in conjunction with large transport vehicles for loading, which is inefficient and poses safety hazards during manual loading. Improper operation may lead to work-related accidents. With the development of the times, the efficiency and safety of traditional loading methods can no longer meet the needs of modern grain warehouse management.

[0003] Currently, the detection of cargo volume on large transport vehicles mainly relies on camera or ultrasonic measurement methods. However, due to the large differences in cargo capacity and models of transport vehicles, manual measurement cannot ensure accuracy. Furthermore, the current camera algorithms and ultrasonic methods for calculating cargo volume have problems such as inaccurate calculations and large errors.

[0004] With the increase in grain production and the development of the logistics industry, the demand for efficient, accurate and safe automated loading systems is growing. There is an urgent need for an automated loading system that can improve loading efficiency, reduce human error, and ensure loading uniformity and grain integrity. Summary of the Invention

[0005] To address the aforementioned problems, this invention provides an automated grain loading system and method based on the fusion of visual and laser data. This system can automatically measure the volume of grain on the grain truck, achieving precise loading and improving loading efficiency, reducing human error, and ensuring loading uniformity and grain integrity.

[0006] This invention provides an automated grain loading system based on the fusion of visual and laser data, the specific technical solution of which is as follows: The system includes a material feeding detection unit, a 3D scanning unit, a multimodal data fusion module, and an automatic loading control system; The material feeding detection unit is used to collect video data of the target vehicle and detect the material feeding amount through a visual analysis deep learning model. The three-dimensional scanning unit is used to acquire the position and orientation information of the truck's cargo box in real time and quickly calculate the grain volume using point cloud data. The multimodal data fusion module is used to receive data information collected by the material feeding detection unit and the three-dimensional scanning unit, and dynamically adjust the weights through the Kalman filter algorithm to fuse the optimal volume estimate. The automatic loading control system includes a license plate recognition unit and a data terminal; the license plate recognition unit is used to acquire the license plate information of the currently entering vehicle and transmit it to the data terminal; the data terminal retrieves the vehicle loading information based on the license plate information, calculates the unloading speed, and performs unloading anomaly detection.

[0007] Furthermore, the material feeding detection unit detects the material feeding amount, and the specific process is as follows: The collected video data of the target vehicle is input into the visual analysis deep learning model; The material feed rate is obtained based on the video data; The optical sensor is installed in a pre-defined position in the system, and the camera's intrinsic and extrinsic parameters are labeled. Calculate the material cutting height information based on the labeled parameters and the pre-trained model; When the system anticipates material overflow, it will automatically stop feeding.

[0008] Furthermore, the calculation of the material cutting height information based on the labeled parameters and the pre-trained model is as follows: The pre-trained model is used to detect and obtain the pixel positions of the feed inlet and the top of the feed pile in the video image; Based on the labeled parameters, perform spatial coordinate transformation to obtain the absolute spatial positions of the discharge port and the top of the discharge pile, and calculate the discharge height.

[0009] Furthermore, the three-dimensional scanning unit calculates the grain volume, and the specific process is as follows: The three-dimensional scanning unit acquires the pose information of the truck's cargo box in real time using lidar; Select scan lines to preprocess the point cloud data; After the point cloud data of the scan lines is preprocessed, data fusion is performed. Reconstruct the surface of the point cloud data and calculate the volume of the grain.

[0010] Furthermore, the volume of the grain is calculated using the slicing method, as detailed below:

[0011]

[0012] Wherein, the X-axis direction is used as the scan line direction, and the scan line spacing is... Where the Y-axis direction is used as the slice normal direction, and the slice spacing is... ; This represents the distance between two intersection points of the scan line and the point cloud data on the slice, where m represents the number of scan lines and n represents the number of slices. This represents the area of ​​the j-th slice.

[0013] Furthermore, after receiving the data information collected by the material feeding detection unit and the three-dimensional scanning unit, the multimodal data fusion module preprocesses the data information, including noise reduction, data alignment, and time synchronization.

[0014] Furthermore, the multimodal data fusion module uses the Kalman filter algorithm to input preprocessed data and dynamically adjusts the weights according to the measurement error of each sensor. Based on the dynamic weight fusion, a volume estimate is obtained, and the real-time loading amount is calculated based on the current estimate.

[0015] This invention also provides an automated grain loading method based on the fusion of visual and laser data. Based on the aforementioned automated loading system, the method includes: S1: Obtain the license plate information of the currently entering vehicle, and retrieve the vehicle's information data and loading data through the license plate information; S2: Vehicles enter the material positions in the yard in sequence. After the current vehicle enters the position, the real-time loading amount is obtained through the automated loading system to determine whether to unload. If so, the material is unloaded, and the unloading height detected by the unloading amount detection unit is used to determine whether the current material position is fully loaded. Unloading is automatically stopped after loading is completed. If not, the current vehicle moves to the next material position until the current vehicle has passed all material positions in the yard. S3: The three-dimensional scanning unit outputs the volume when loading is complete, and the multimodal data fusion module outputs the current loading amount based on the current volume; S4: Vehicle weighed and departs.

[0016] Furthermore, the real-time loading information of the vehicles entering the site is displayed and output in real time after the vehicles are in their designated positions.

[0017] Furthermore, in step S2, if it is determined from the material discharge height that loading has been completed at the current material position, then step S3 is executed directly.

[0018] The beneficial effects of this invention are as follows: This invention improves measurement accuracy and system robustness by integrating visual analysis and lidar scanning data for material unloading detection. The loading status data is displayed and output in real time throughout the loading process, automatically prompting the driver with loading steps, thus automating the loading process, effectively improving the automation level of grain silo loading, reducing manual operation, and lowering labor intensity. During loading, the loading status is monitored in real time, and through precise measurement and intelligent control, loading efficiency and accuracy are significantly improved, grain loss during loading is reduced, and unloading is automatically stopped when loading abnormalities are detected, ensuring loading safety. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the overall system architecture of the present invention.

[0020] Figure 2 This is a schematic diagram of the functional logic flow of the three-dimensional scanning unit of the present invention.

[0021] Figure 3 This is a schematic diagram of the method flow of the present invention. Detailed Implementation

[0022] The technical solutions in the embodiments of the present invention are clearly and completely described in the following description. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0023] In the description of the embodiments of the present invention, it should be noted that the indicated orientation or positional relationship is based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship in which the product of the invention is conventionally placed during use, or the orientation or positional relationship in which those skilled in the art conventionally understand it during use. This is only for the convenience of describing the present invention and simplifying the description, and is not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, it should not be construed as a limitation of the present invention. Furthermore, the terms "first" and "second" are only used to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0024] In the description of the embodiments of the present invention, it should also be noted that, unless otherwise explicitly specified and limited, the terms "set" and "connection" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a direct connection or an indirect connection through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms in the present invention based on the specific circumstances.

[0025] Example 1 Embodiment 1 of the present invention discloses an automated grain loading system based on the fusion of visual and laser data, such as... Figure 1 As shown, the details are as follows: The system includes a material feeding detection unit, a 3D scanning unit, a multimodal data fusion module, and an automatic loading control system; The material feeding detection unit is used to collect video data of the target vehicle and detect the material feeding amount through a visual analysis deep learning model. The specific process is as follows: The collected video data of the target vehicle is input into the visual analysis deep learning model; The material feed rate is obtained based on the video data; Install the optical sensor in the appropriate location in the system and label the camera's intrinsic and extrinsic parameters; Specifically, the optical sensor is installed at the front of the control room, angled downwards for easy viewing of the feed inlet and the entire material pile.

[0026] The material cutting height information is calculated based on the labeled parameters and the pre-trained model, as follows: The pre-trained model is used to detect and obtain the pixel positions of the feed inlet and the top of the feed pile in the video image; Based on the labeled parameters, perform spatial coordinate transformation to obtain the absolute spatial positions of the discharge port and the top of the discharge pile, and calculate the discharge height.

[0027] When the system anticipates material overflow, it automatically stops feeding to ensure that the maximum height of the material pile does not exceed the set height of the barrier.

[0028] The 3D scanning unit is used to acquire the cargo box pose information of the truck in real time, and quickly calculate the grain volume using point cloud data, such as... Figure 2 As shown, the specific process is as follows: The three-dimensional scanning unit acquires the pose information of the truck's cargo box in real time using lidar; Select scan lines to preprocess the point cloud data; After the point cloud data of the scan lines is preprocessed, data fusion is performed. Reconstruct the surface of the point cloud data and calculate the grain volume using the slicing method:

[0029]

[0030] Wherein, the X-axis direction is used as the scan line direction, and the scan line spacing is... Where the Y-axis direction is used as the slice normal direction, and the slice spacing is... ; This represents the distance between two intersection points of the scan line and the point cloud data on the slice, where m represents the number of scan lines and n represents the number of slices. The area of ​​the j-th slice is represented by the sum of the distances between the intersections of all scan lines on that slice and the scan line spacing. V represents the total volume, which is obtained by summing the areas of all slices by the slice spacing.

[0031] The multimodal data fusion module is used to receive data information collected by the material feeding detection unit and the three-dimensional scanning unit, and dynamically adjust the weights through the Kalman filter algorithm to fuse the optimal volume estimate. In a preferred embodiment, after receiving the data information collected by the material feeding detection unit and the three-dimensional scanning unit, the multimodal data fusion module preprocesses the data information, including noise reduction, data alignment and time synchronization.

[0032] In a preferred embodiment, the multimodal data fusion module uses a Kalman filter algorithm to input preprocessed data and dynamically adjusts the weights according to the measurement error of each sensor. Based on the dynamic weight fusion, a volume estimate is obtained, and the real-time loading amount is calculated based on the current estimate.

[0033] The automatic loading control system includes a license plate recognition unit, a data terminal, and a loading mechanism; the loading mechanism is an existing grain loading and unloading mechanism; the license plate recognition unit is used to obtain the license plate information of the currently entering vehicles and transmit it to the data terminal; the data terminal retrieves vehicle loading information based on the license plate information, calculates the unloading speed, and performs unloading anomaly detection.

[0034] Example 2 Embodiment 2 of the present invention discloses an automated grain loading system based on the fusion of vision and laser data, which is based on the automated loading system described in Embodiment 1, such as... Figure 3 As shown, the method includes: S1: Obtain the license plate information of the currently entering vehicle, and retrieve the vehicle's information data and loading data through the license plate information; S2: Vehicles enter the material positions in the yard in sequence. After the current vehicle enters the position, the real-time loading amount is obtained through the automated loading system to determine whether to unload. If so, the material is unloaded, and the unloading height detected by the unloading amount detection unit is used to determine whether the current material position is fully loaded. Unloading is automatically stopped after loading is completed. If not, the current vehicle moves to the next material position until the current vehicle has passed all material positions in the yard. The real-time loading information of the vehicles entering the site is displayed and output in real time after the vehicles are in place.

[0035] If the loading at the current material level is determined based on the material discharge height, then proceed directly to step S3.

[0036] S3: The three-dimensional scanning unit outputs the volume when loading is complete, and the multimodal data fusion module outputs the current loading amount based on the current volume; S4: Vehicle weighed and departs.

[0037] This invention is not limited to the specific embodiments described above. The invention extends to any new feature or combination disclosed in this specification, as well as any new method or process step or combination disclosed herein.

Claims

1. An automated grain loading system based on vision and laser data fusion, characterized in that, It includes a material feeding detection unit, a 3D scanning unit, a multimodal data fusion module, and an automatic loading control system; The material feeding detection unit is used to collect video data of the target vehicle and detect the material feeding amount through a visual analysis deep learning model. The three-dimensional scanning unit is used to acquire the position and orientation information of the truck's cargo box in real time and quickly calculate the grain volume using point cloud data. The multimodal data fusion module is used to receive data information collected by the material feeding detection unit and the three-dimensional scanning unit, and dynamically adjust the weights through the Kalman filter algorithm to fuse the optimal volume estimate. The automatic loading control system includes a license plate recognition unit and a data terminal; the license plate recognition unit is used to acquire the license plate information of the currently entering vehicle and transmit it to the data terminal; the data terminal retrieves the vehicle loading information based on the license plate information, calculates the unloading speed, and performs unloading anomaly detection.

2. The automated grain loading system based on vision and laser data fusion according to claim 1, characterized in that, The material feeding detection unit detects the material feeding amount, and the specific process is as follows: The collected video data of the target vehicle is input into the visual analysis deep learning model; The material feed rate is obtained based on the video data; The optical sensor is installed in a pre-defined position in the system, and the camera's intrinsic and extrinsic parameters are labeled. Calculate the material cutting height information based on the labeled parameters and the pre-trained model; When the system anticipates material overflow, it will automatically stop feeding.

3. The automated grain loading system based on vision and laser data fusion according to claim 2, characterized in that, The calculation of the material cutting height information based on the labeled parameters and the pre-trained model is as follows: The pre-trained model is used to detect and obtain the pixel positions of the feed inlet and the top of the feed pile in the video image; Based on the labeled parameters, perform spatial coordinate transformation to obtain the absolute spatial positions of the discharge port and the top of the discharge pile, and calculate the discharge height.

4. The automated grain loading system based on vision and laser data fusion according to claim 1, characterized in that, The three-dimensional scanning unit calculates the grain volume, and the specific process is as follows: The three-dimensional scanning unit acquires the pose information of the truck's cargo box in real time using lidar; Select scan lines to preprocess the point cloud data; After the point cloud data of the scan lines is preprocessed, data fusion is performed. Reconstruct the surface of the point cloud data and calculate the volume of the grain.

5. The automated grain loading system based on vision and laser data fusion according to claim 4, characterized in that, The volume of grain is calculated using the slicing method, as follows: Wherein, the X-axis direction is used as the scan line direction, and the scan line spacing is... Where the Y-axis direction is used as the slice normal direction, and the slice spacing is... ; This represents the distance between two intersection points of the scan line and the point cloud data on the slice, where m represents the number of scan lines and n represents the number of slices. This represents the area of ​​the j-th slice.

6. The automated grain loading system based on vision and laser data fusion according to claim 1, characterized in that, After receiving the data information collected by the material feeding detection unit and the three-dimensional scanning unit, the multimodal data fusion module preprocesses the data information, including noise reduction, data alignment and time synchronization.

7. The automated grain loading system based on vision and laser data fusion according to claim 6, characterized in that, The multimodal data fusion module uses the Kalman filter algorithm to input preprocessed data and dynamically adjusts the weights according to the measurement error of each sensor. Based on the dynamic weight fusion, it obtains a volume estimate and calculates the real-time loading amount based on the current estimate.

8. An automated grain loading method based on the fusion of visual and laser data, characterized in that, Based on the automated loading system according to any one of claims 1-7, the method includes: S1: Obtain the license plate information of the currently entering vehicle, and retrieve the vehicle's information data and loading data through the license plate information; S2: Vehicles enter the material positions in the yard in sequence. After the current vehicle enters the position, the real-time loading amount is obtained through the automated loading system to determine whether to unload. If so, the material is unloaded, and the unloading height detected by the unloading amount detection unit is used to determine whether the current material position is fully loaded. Unloading is automatically stopped after loading is completed. If not, the current vehicle moves to the next material position until the current vehicle has passed all material positions in the yard. S3: The three-dimensional scanning unit outputs the volume when loading is complete, and the multimodal data fusion module outputs the current loading amount based on the current volume; S4: Vehicle weighed and departs.

9. The automated grain loading method based on vision and laser data fusion according to claim 8, characterized in that, The real-time loading information of the vehicles entering the site is displayed and output in real time after the vehicles are in place.

10. The automated grain loading method based on vision and laser data fusion according to claim 8, characterized in that, In step S2, if it is determined from the material discharge height that loading has been completed at the current material position, then step S3 is executed directly.

Images