Cotton bale detection method based on multi-modal data timestamp synchronization and GPU / CPU collaborative acceleration

Abstract

The application discloses a cotton bale detection method based on multi-modal data timestamp synchronization and GPU / CPU collaborative acceleration, collects multi-modal data, marks timestamps, and performs GPU accelerated distortion correction on cotton bale images; the multi-modal data is synchronized and integrated according to timestamps; a main program adopts Nodelet combined with C++ multi-threading technology to process the reception of multi-modal data and the publication of multi-modal sensor coordinate systems in parallel, the main program running optimizes the transmission efficiency of multi-modal data; a cotton bale detection model is constructed, the optimized multi-modal data are input for training, the cotton bale detection model after training is optimized using TensorRT, and cotton bale images are input for inference, cotton bale target recognition data are obtained and transmitted to a CPU for post-processing operation, and OpenMP is used to accelerate point cloud traversal in the main program to match the cotton bale target recognition data with laser radar point cloud data; the matched multi-modal data are subjected to point cloud feature and plane fitting calculation, and the rectangular frame, pose information and score of the cotton bale target are obtained.

Description

Technical Field

[0001] This invention relates to the field of target tracking algorithm technology, and in particular to a cotton bale detection method based on multimodal data timestamp synchronization and GPU / CPU collaborative acceleration. Background Technology

[0002] The application of modern, fully mechanized, and intelligent technologies in cotton planting and harvesting has been increasing year by year. From sowing and field management to harvesting, fully automated operations have been achieved. This efficient management not only reduces reliance on labor but also effectively improves cotton production efficiency and quality stability. Furthermore, with the technological advancements in domestically produced agricultural machinery, the localization of agricultural equipment is gradually being realized, effectively reducing equipment costs and maintenance difficulties, and enhancing the sustainability of cotton production. In this vast cotton market, the increasing demand for agricultural automation has led to the deepening application of intelligent sensing technology in agricultural scenarios. During automated cotton harvesting and transportation, high-precision detection and multi-target tracking of cotton bales face numerous challenges. These issues not only affect the accuracy of cotton bale detection technology but also place stringent demands on the real-time performance and stability of the entire system.

[0003] In recent years, the fusion of vision and radar sensors has become a hot research topic in autonomous driving technology. By combining visual and radar information, the shortcomings of each sensor can be compensated for. Radar provides relatively accurate distance and speed information, while vision provides richer information on object categories and textures. Such fusion schemes can improve the accuracy of target recognition and tracking in complex environments and under poor lighting conditions.

[0004] For example, DeepFusion [Li Y, Yu AW, Meng T, et al. Deepfusion: Lidar-camera deepfusion for multi-modal 3d object detection[C] / / Proceedings of the IEEE / CVFconference on computer vision and pattern recognition.2022:17182-17191.] and CLOCs [Dhall A, Chelani K, Radhakrishnan V, et al. LiDAR-camera calibration using 3D-3D point correspondences[J]. arXiv preprint arXiv:1705.09785,2017] proposed a vision-radar deep fusion method, which enhances the robustness of object detection by jointly processing visual images and radar point cloud data. In the field of autonomous driving, deep learning models that combine visual and radar data can provide more comprehensive target detection and tracking capabilities. By using joint feature maps from vision and radar, objects can be located more accurately, improving the accuracy of multi-target detection and tracking.

[0005] While the fusion of vision and radar can significantly improve detection accuracy, it still faces many challenges. For example, how to efficiently synchronize timestamp data from different sensors, how to eliminate redundancy and noise in multi-sensor data fusion, and how to achieve efficient information sharing and processing between different data sources. In recent years, sensor fusion methods combining deep learning, such as BEVfusion [Liu Z, Tang H, Amini A, et al. Bevfusion: Multi-task multi-sensor fusion with unified bird's-eye view representation [C] / / 2023 IEEE international conference on robotics and automation (ICRA). IEEE, 2023: 2774-2781] and Fast-BEV [Li Y, Huang B, Chen Z, et al. Fast-BEV: A Fast and Strong Bird's-Eye View Perception Baseline [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.], utilize multi-sensor fusion and bird's-eye view technology to achieve more comprehensive perception and prediction of targets, further improving the accuracy and real-time performance of the fusion system.

[0006] With the continuous development of autonomous driving technology, vision-radar fusion schemes are playing an increasingly important role in target detection and multi-target tracking. Vision-radar fusion schemes have demonstrated great potential in improving multi-target tracking and reducing occlusion and separation problems. However, they also come with challenges: optimization of multimodal data fusion: further optimizing the fusion method of visual and radar data to improve the accuracy and real-time performance of multi-sensor systems. Robustness improvement: how to improve the system's robustness to factors such as occlusion, reflection, and low light in complex environments is an important research topic.

[0007] Therefore, the high-precision identification and detection of cotton bales in cotton ginning mills presents the following challenges: cotton bale detection systems not only require high accuracy but also sufficient real-time performance. Cotton bale detection involves multiple computational steps, including image acquisition, point cloud processing, feature extraction, target detection, and tracking, each with varying computational delays, making it difficult to guarantee system real-time performance. Especially in the case of multi-sensor fusion, the data frequencies from visual sensors and point cloud sensors differ, making efficient data synchronization and fusion crucial to ensure no critical information is lost during real-time decision-making a significant challenge.

[0008] Furthermore, given the complexity and massive computational demands of each stage, optimizing algorithms, reducing computational resource consumption, and ensuring the system can run in real-time under limited hardware conditions have become critical issues that must be addressed. How to achieve efficient real-time operation of the system through hardware acceleration and algorithm optimization, and further improve the real-time performance of detection and tracking, is a core problem that urgently needs to be solved. Summary of the Invention

[0009] The purpose of this invention is to address the technical deficiencies in the existing technology by providing a cotton bale detection method based on multimodal data timestamp synchronization and GPU / CPU collaborative acceleration.

[0010] The technical solution adopted to achieve the purpose of this invention is:

[0011] A method for detecting cotton bales based on multimodal data timestamp synchronization and GPU / CPU collaborative acceleration includes the following steps:

[0012] Step 1: Collect multimodal data using a multimodal sensor. The multimodal data includes images of cotton bales in the cotton ginning plant, depth information of the cotton bales and their surrounding environment, positioning and attitude data of the bale clamping vehicle, and pitch data of the fork arm. The collected multimodal data is timestamped.

[0013] Step 2: Based on the main thread, create a sub-thread to call the ROS message_filters module to synchronize the timestamps of the multimodal data marked in Step 1, and then integrate the synchronized multimodal data.

[0014] Step 3: The main program uses the ROS Nodelet framework and combines C++ multithreading technology to process the integrated multimodal data reception and multimodal sensor coordinate system publishing in parallel. The main program runs to optimize the data transmission efficiency of the integrated multimodal data.

[0015] Step 4: Construct a cotton bale detection model. Input the multimodal data optimized in Step 3 into the cotton bale detection model for training. Use TensorRT to optimize the trained cotton bale detection model. Input the cotton bale image into the optimized cotton bale detection model for inference to obtain cotton bale target recognition data, and store it in the GPU output buffer.

[0016] Step 5: Transfer the cotton bale target recognition data stored in the GPU output buffer to the CPU for post-processing, and use the C++ OpenMP framework to accelerate the point cloud traversal in the main program to match the cotton bale target recognition data with the LiDAR point cloud data.

[0017] Step 6: Perform point cloud feature calculation and plane fitting calculation on the multimodal data matched in Step 5 to obtain the bounding box, pose information and score of the cotton bag target.

[0018] In the above technical solution, the multimodal sensor includes multiple cameras, multiple lidar, a combined inertial navigation system, and a fork-arm pitch height sensor.

[0019] In the above technical solution, the camera is a fisheye camera. The fisheye camera is used for calibration to determine the camera's intrinsic parameters (camera matrix) and distortion coefficients. The intrinsic parameters are the acquired cotton bale images, and the distortion coefficients are used to correct the acquired cotton bale images to eliminate distortion.

[0020] In the above technical solution, step 2 includes: opening a sub-thread in the main thread to call the message_filters module to accurately synchronize the cotton bale image, the depth information of the cotton bale and its surrounding environment, the positioning and attitude data of the bag clamping vehicle, and the pitch data of the fork arm collected by the camera, lidar, combined inertial navigation system, and fork arm pitch height sensor, so that the error between the cotton bale image, the depth information of the cotton bale and its surrounding environment, the positioning and attitude data of the bag clamping vehicle, and the pitch data of the fork arm is within 0.01s-0.04s.

[0021] In the above technical solution, the optimization of the cotton bale detection model after training using TensorRT includes: optimizing the inference process of cotton bale target recognition data through techniques such as mixed precision calculation, layer fusion and memory reuse.

[0022] In the above technical solution, the post-processing operation includes filtering, sorting, and NMS of the cotton bale target recognition data.

[0023] In the above technical solution, step 6 includes the following steps:

[0024] S6.1: Calculate the centroid of the point cloud of the multimodal data after matching in step 5 to obtain the position information of the cotton bale in the global coordinate system;

[0025] S6.2: Calculate the width and height of the multimodal data point cloud to obtain the width and height of the cotton bale;

[0026] S6.3: Randomly select three non-coplanar points from the point cloud of the multimodal data to perform plane fitting, obtain the corresponding plane normal vector, normalize the plane normal vector, and calculate the dot product and cross product of the normalized plane normal vector with the reference axis.

[0027] S6.4: Construct a quaternion and convert the quaternion into Euler angles to extract the attitude information corresponding to the cotton bag.

[0028] In the above technical solution, the plane fitting formula is as follows:

[0029]

[0030]

[0031] In the formula, P1(x1, y1, z1), P2(x2, y2, z2), and P3(x3, y3, z3); ​​P1, P2, and P3 represent three non-coplanar points randomly selected from the point cloud of the multimodal data. These represent the corresponding direction vectors. This represents the corresponding plane normal vector.

[0032] In the above technical solution, the formula for calculating the dot product of the normalized plane normal vector and the reference axis is as follows:

[0033]

[0034] In the formula, The unit direction vector representing the reference axis, dot product This represents the dot product of the normalized plane normal vector and the reference axis. This represents the corresponding plane normal vector;

[0035] The formula for calculating the cross product of the normalized plane normal vector and the reference axis is as follows:

[0036]

[0037] In the formula, n represents the cross product of the normalized plane normal vector and the reference axis. x n represents the value of the normal vector along the x-axis. y n represents the value of the normal vector along the y-axis. z This represents the value of the normal vector along the z-axis. The unit direction vector representing the reference axis. This represents the corresponding plane normal vector.

[0038] In the above technical solution, the formula for extracting the posture information corresponding to the cotton bale is as follows:

[0039]

[0040] q=(cos(θ / 2),r x sin(θ / 2),r y sin(θ / 2),r z sin(θ / 2))

[0041] Roll = atan2(2(wx+yz), 1-2(x) 2 +y 2 ))

[0042] Pitch = asin(2(wy-zx))

[0043] Yaw=atan2(2(wz+xy),1-2(y 2 +z 2 ))

[0044] In the formula, θ represents the magnitude of the cross product, and θ represents the plane normal vector. Regarding the angle of the unit direction vector along the x-axis, the function can find the correct quadrant using both the sine and cosine values. The normalized unit vector representing the cross product vector is used to determine the rotation axis of the quaternion, where q represents the quaternion, and q = (w, x, y, z). x r represents the value of the unit direction vector along the x-axis. y r represents the value of the unit direction vector along the y-axis. z The z-axis value represents the unit direction vector, w represents the rotation angle or scale factor, x represents the x-axis direction and rotation angle, y represents the y-axis direction and rotation angle, z represents the z-axis direction and rotation angle, and Roll, Pitch, and Yaw represent the Euler angles calculated from the quaternions, which are roll, pitch, and yaw angles, respectively.

[0045] Compared with the prior art, the beneficial effects of the present invention are:

[0046] 1. This invention achieves precise synchronization of data streams from different sensors through the message_filters library, controlling the error between 0.01s and 0.04s, effectively avoiding data inconsistency caused by time delay.

[0047] 2. This invention uses the Nodelet framework to run the camera node and the cotton bale detection algorithm node in the same process. Data transmission does not require TCP communication and is performed directly in the same thread through C++ shared pointers, which significantly reduces ROS communication latency and improves data processing efficiency. By using C++ OpenMP technology to customize the number of threads, the data processing tasks are parallelized, further reducing computation time and improving system response speed.

[0048] 3. This invention uses TensorRT to optimize the cotton bale detection model, which can increase the detection speed of a single cotton bale image by 4 to 5 times, significantly reducing the detection time and improving the system's response speed. In practical applications, the cotton bale detection model can process more images in a shorter time, improving the system's throughput and real-time performance. Moreover, the accelerated cotton bale detection model has almost no loss in detection accuracy compared to the original detection model, ensuring high accuracy and high reliability of cotton bale detection. Attached Figure Description

[0049] Figure 1 The diagram illustrates the workflow of multimodal data timestamp synchronization and GPU / CPU collaborative acceleration described in this invention.

[0050] Figure 2 The diagram shown is a flowchart of the multimodal data processing described in this invention.

[0051] Figure 3 The image shown is a comparison diagram between the Nodelet node and the Node node described in this invention.

[0052] Figure 4 The diagram shown is a block diagram of the C++ multithreading technology described in this invention.

[0053] Figure 5 The diagram shown is a flowchart of the cotton bale detection model described in this invention.

[0054] Figure 6 The diagram shown is a flowchart of the TensorRT model inference acceleration process described in this invention. Detailed Implementation

[0055] The present invention will be further described in detail below with reference to specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0056] A cotton bale detection method based on multimodal data timestamp synchronization and GPU / CPU collaborative acceleration, see [link to relevant documentation]. Figure 1 This includes the following steps:

[0057] Step 1: Collect multimodal data using a multimodal sensor. The multimodal data includes images of cotton bales in the cotton ginning plant, depth information of the cotton bales and their surrounding environment, positioning and attitude data of the bale clamping vehicle, and pitch data of the fork arm. The collected multimodal data is timestamped.

[0058] This embodiment of the multimodal sensor includes multiple cameras, multiple LiDARs, an integrated inertial navigation system (INS), and a forklift pitch height sensor. The cameras are fisheye cameras, calibrated to determine their intrinsic parameters (camera matrix) and distortion coefficients. The intrinsic parameters are the acquired cotton bale images, and the distortion coefficients are used to correct the acquired cotton bale images to eliminate distortion. Based on this embodiment, the fisheye camera can perform GPU-accelerated distortion correction on the acquired cotton bale images in multimodal data in real time. Specifically, the cameras capture cotton bale images with a 15Hz update frequency; the LiDARs acquire depth information of the cotton bale and its surrounding environment at a 15Hz frequency to construct a 3D environment model; the integrated inertial navigation system provides accurate vehicle positioning and attitude data with a maximum update frequency of 10Hz for vehicle navigation; and the forklift pitch height sensor monitors the position and height data of the forklift, providing accurate pitch data at a high frequency of 30Hz. The camera and lidar are fixedly mounted on the fork arm for adjusting their dynamic coordinates.

[0059] Step 2: On top of the main thread (CPU), a sub-thread is created to call the message_filters module of ROS (Robot Operating System) to perform timestamp synchronization on the multimodal data timestamped in Step 1, in order to ensure the time consistency between each frame of cotton bag image and LiDAR data.

[0060] See Figure 2 The time-stamp synchronization of the multimodal data in step 1 includes: opening a sub-thread in the main thread to call the message_filters module to accurately synchronize the cotton bale images, depth information of the cotton bale and its surrounding environment, positioning and attitude data of the bag clamping vehicle, and pitch data of the fork arm collected by the camera, LiDAR, combined inertial navigation system (INS), and fork arm pitch height sensor. This ensures that the error between the cotton bale images, depth information of the cotton bale and its surrounding environment, positioning and attitude data of the bag clamping vehicle, and pitch data of the fork arm is within 0.01s-0.04s, avoiding data inconsistency caused by time delays received by the program.

[0061] Step 3: The main program uses the ROS Nodelet framework combined with C++ multithreading technology to process the reception of timestamp-synchronized multimodal data and the publication of multimodal sensor coordinate systems in parallel. The timestamp-synchronized multimodal data is obtained directly through shared pointers. In particular, obtaining the timestamp-synchronized multimodal data directly through shared pointers can optimize the transmission efficiency of multimodal data.

[0062] See Figure 3, Figure 4 The Nodelet framework in this embodiment is used to run camera nodes and cotton bale detection model nodes in the same process. It can achieve data transmission without TCP communication, and directly copy data within the same thread using C++ shared pointers (shared_ptr) (i.e., multiple pointers copy data within the same thread), which can reduce ROS communication latency and improve data processing efficiency.

[0063] Step 4: Construct a bale detection model (Yolov8s neural network model). Input the multimodal data optimized in Step 3 into the bale detection model for training (the trained bale detection model can achieve efficient and accurate bale detection in various environments, identifying the location information and category of bales in bale images). Use TensorRT (TensorRT is a deep learning inference acceleration library launched by NVIDIA) to optimize the trained bale detection model, which can improve the actual inference speed of the bale detection model on the GPU. See [link to relevant documentation]. Figure 5 The cotton bale image is input into the optimized cotton bale detection model for inference to obtain cotton bale target recognition data (the cotton bale target recognition data includes cotton bale target recognition box (rectangle), confidence value and category information), and the cotton bale target recognition data is stored in the GPU output buffer.

[0064] See Figure 6 The optimization of the cotton bale detection model after training using TensorRT includes: optimizing the inference process of cotton bale target recognition data through techniques such as mixed precision computation, layer fusion and memory reuse, reducing unnecessary computation and memory access, thereby greatly accelerating the inference process.

[0065] Step 5: The cotton bale target recognition data stored in the GPU output buffer is transferred to the CPU (Central Processing Unit) for post-processing. The C++ OpenMP (Open Multi-Processing) framework is used to accelerate the point cloud traversal in the main program to match the cotton bale target recognition data with the LiDAR point cloud data, which can improve the processing speed of multimodal data.

[0066] The post-processing operations include filtering, sorting, and performing NMS (non-maximum suppression) on the cotton bale target identification data, which can ensure the accuracy and completeness of the detection results.

[0067] The C++ OpenMP framework allows for custom allocation of the required number of threads for code blocks in the main program. By distributing the post-processing tasks of cotton bag target identification data to multiple sub-threads within these code blocks, the parallelism of data transmission and processing can be improved, thereby reducing the overall computation time and thus increasing the data processing speed.

[0068] Step 6: Perform point cloud feature calculation and plane fitting calculation on the multimodal data matched in Step 5 to obtain the bounding box, pose information and score of the cotton bag target.

[0069] Step 6 includes the following steps:

[0070] S6.1: Calculate the centroid of the point cloud of the multimodal data after matching in step 5 to obtain the position information (x, y, z) of the cotton bale in the global coordinate system.

[0071] S6.2: Calculate the width and height of the multimodal data point cloud to obtain the width and height of the cotton bale.

[0072] S6.3: Randomly select three non-coplanar points from the point cloud of the multimodal data to perform plane fitting, obtain the corresponding plane normal vector, normalize the plane normal vector, and calculate the dot product and cross product of the normalized plane normal vector with the reference axis (the reference axis is the x-axis).

[0073] The plane fitting formula is as follows:

[0074]

[0075] In the formula, P1(x1, y1, z1), P2(x2, y2, z2), and P3(x3, y3, z3); ​​P1, P2, and P3 represent three non-coplanar points randomly selected from the point cloud of the multimodal data. These represent the corresponding direction vectors. This represents the corresponding plane normal vector.

[0076] The formula for calculating the dot product of the normalized plane normal vector and the reference axis is as follows:

[0077]

[0078] In the formula, The unit direction vector representing the reference axis (x-axis), dot product This represents the dot product of the normalized plane normal vector and the reference axis. This represents the corresponding plane normal vector.

[0079] The formula for calculating the cross product of the normalized plane normal vector and the reference axis is as follows:

[0080]

[0081] In the formula, n represents the cross product of the normalized plane normal vector and the reference axis. x n represents the value of the normal vector along the x-axis. y n represents the value of the normal vector along the y-axis.z This represents the value of the normal vector along the z-axis. The unit direction vector representing the reference axis (x-axis) This represents the corresponding plane normal vector.

[0082] S6.4: Construct a quaternion and convert the quaternion into Euler angles to extract the pose information (pitch, roll, yaw) corresponding to the cotton bale.

[0083] The formula for extracting the posture information corresponding to the cotton bale is as follows:

[0084]

[0085] q=(cos(θ / 2),r x sin(θ / 2),r y sin(θ / 2),r z sin(θ / 2))

[0086] Roll = atan2(2(wx+yz), 1-2(x) 2 +y 2 ))

[0087] Pitch = asin(2(wy-zx))

[0088] Yaw=atan2(2(wz+xy),1-2(y 2 +z 2 ))

[0089] In the formula, θ represents the magnitude of the cross product, and θ represents the plane normal vector. Regarding the angle of the unit direction vector along the x-axis, the function can find the correct quadrant using both the sine and cosine values. The normalized unit vector representing the cross product vector is used to determine the rotation axis of the quaternion, where q represents the quaternion, and q = (w, x, y, z). x r represents the value of the unit direction vector along the x-axis. y r represents the value of the unit direction vector along the y-axis. z The z-axis value represents the unit direction vector, w represents the rotation angle or scale factor, similar to the scaling part in the rotation matrix, x represents the x-axis direction and rotation angle, y represents the y-axis direction and rotation angle, z represents the z-axis direction and rotation angle, and Roll, Pitch, and Yaw represent the Euler angles calculated from the quaternions, which are roll, pitch, and yaw angles, respectively.

[0090] The above description is only a preferred embodiment of the present invention. It should be noted that, for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for detecting cotton bales based on multimodal data timestamp synchronization and GPU / CPU collaborative acceleration, characterized in that, Includes the following steps: Step 1: Collect multimodal data using a multimodal sensor. The multimodal data includes images of cotton bales in the cotton ginning plant, depth information of the cotton bales and their surrounding environment, positioning and attitude data of the bale clamping vehicle, and pitch data of the fork arm. The collected multimodal data is timestamped. Step 2: Based on the main thread, create a sub-thread to call the ROS message_filters module to synchronize the timestamps of the multimodal data marked in Step 1, and then integrate the synchronized multimodal data. Step 3: The main program uses the ROS Nodelet framework and combines C++ multithreading technology to process the integrated multimodal data reception and multimodal sensor coordinate system publishing in parallel. The main program runs to optimize the data transmission efficiency of the integrated multimodal data. Step 4: Construct a cotton bale detection model. Input the multimodal data optimized in Step 3 into the cotton bale detection model for training. Use TensorRT to optimize the trained cotton bale detection model. Input the cotton bale image into the optimized cotton bale detection model for inference to obtain cotton bale target recognition data, and store it in the GPU output buffer. Step 5: Transfer the cotton bale target recognition data stored in the GPU output buffer to the CPU for post-processing, and use the C++ OpenMP framework to accelerate the point cloud traversal in the main program to match the cotton bale target recognition data with the LiDAR point cloud data. Step 6: Perform point cloud feature calculation and plane fitting calculation on the multimodal data matched in Step 5 to obtain the bounding box, pose information and score of the cotton bag target.

2. The cotton bale detection method according to claim 1, characterized in that, The multimodal sensor includes multiple cameras, multiple lidar, a combined inertial navigation system, and a fork-arm pitch height sensor.

3. The cotton bale detection method according to claim 2, characterized in that, The camera is a fisheye camera. The fisheye camera is used for calibration to determine the camera's intrinsic parameters and distortion coefficients. The intrinsic parameters are the acquired cotton bale images, and the distortion coefficients are used to correct the acquired cotton bale images to eliminate distortion.

4. The cotton bale detection method according to claim 1, characterized in that, Step 2 includes: creating a sub-thread in the main thread to call the message_filters module to accurately synchronize the cotton bale images, depth information of the cotton bale and its surrounding environment, positioning and attitude data of the bag clamping vehicle, and pitch data of the fork arm collected by the camera, LiDAR, combined inertial navigation system, and fork arm pitch height sensor, so that the error between the cotton bale images, depth information of the cotton bale and its surrounding environment, positioning and attitude data of the bag clamping vehicle, and pitch data of the fork arm is within 0.01s-0.04s.

5. The cotton bale detection method according to claim 1, characterized in that, The optimization of the cotton bale detection model after training using TensorRT includes optimizing the inference process of cotton bale target recognition data through mixed precision computation, layer fusion and memory reuse techniques.

6. The cotton bale detection method according to claim 1, characterized in that, The post-processing operations include filtering, sorting, and NMS of the cotton bale target recognition data.

7. The cotton bale detection method according to claim 1, characterized in that, Step 6 includes the following steps: S6.1: Calculate the centroid of the point cloud of the multimodal data after matching in step 5 to obtain the position information of the cotton bale in the global coordinate system; S6.2: Calculate the width and height of the multimodal data point cloud after matching in step 5 to obtain the width and height of the cotton bale; S6.3: Randomly select three non-coplanar points from the point cloud of the multimodal data after matching in step 5 to perform plane fitting, obtain the corresponding plane normal vector, normalize the plane normal vector, and calculate the dot product and cross product of the normalized plane normal vector with the reference axis. S6.4: Construct a quaternion and convert the quaternion into Euler angles to extract the attitude information corresponding to the cotton bag.

8. The cotton bale detection method according to claim 7, characterized in that, The plane fitting formula is as follows: In the formula, P1(x1, y1, z1), P2(x2, y2, z2), and P3(x3, y3, z3); ​​P1, P2, and P3 represent three non-coplanar points randomly selected from the point cloud of the multimodal data. These represent the corresponding direction vectors. This represents the corresponding plane normal vector.

9. The cotton bale detection method according to claim 7, characterized in that, The formula for calculating the dot product of the normalized plane normal vector and the reference axis is as follows: In the formula, The unit direction vector representing the reference axis, dot product This represents the dot product of the normalized plane normal vector and the reference axis. This represents the corresponding plane normal vector; The formula for calculating the cross product of the normalized plane normal vector and the reference axis is as follows: In the formula, n represents the cross product of the normalized plane normal vector and the reference axis. x n represents the value of the normal vector along the x-axis. y n represents the value of the normal vector along the y-axis. z This represents the value of the normal vector along the z-axis. The unit direction vector representing the reference axis. This represents the corresponding plane normal vector.

10. The cotton bale detection method according to claim 7, characterized in that, The formula for extracting the posture information corresponding to the cotton bale is as follows: q=(cos(θ / 2),r x sin(θ / 2),r y sin(θ / 2),r z sin(θ / 2)) Roll=atan2(2(wx+yz),1-2(x 2 +y 2 )) Pitch = asin(2(wy-zx)) Yaw=atan2(2(wz+xy),1-2(y 2 +z 2 )) In the formula, θ represents the magnitude of the cross product, and θ represents the plane normal vector. Regarding the angle of the unit direction vector along the x-axis, the function can find the correct quadrant using both the sine and cosine values. The normalized unit vector representing the cross product vector is used to determine the rotation axis of the quaternion, where q represents the quaternion, and q = (w, x, y, z). x r represents the value of the unit direction vector along the x-axis. y r represents the value of the unit direction vector along the y-axis. z The z-axis value represents the unit direction vector, w represents the rotation angle or scale factor, x represents the x-axis direction and rotation angle, y represents the y-axis direction and rotation angle, z represents the z-axis direction and rotation angle, and Roll, Pitch, and Yaw represent the Euler angles calculated from the quaternions, which are roll, pitch, and yaw angles, respectively.

Images