Agricultural data collection and decision-making method and device based on multi-sensor and cloud platform

Abstract

The present invention discloses a method and device for agricultural data collection and decision-making based on multiple sensors and a cloud platform, including: S1, collecting real-time data in farmland through multiple sensors deployed in farmland, obtaining the position and category of objects in the real-time data, and outputting them together with the farmland environment data in the real-time data; S2, inputting crop-related data into a crop growth model, adding farmland environment data corresponding to the crop-related data in real time, and obtaining the evaluation results of the current growth conditions of the crops; S3, generating tasks for target crops; S4, sending the tasks, the area and crop-related data of the target crops to an unmanned agricultural locomotive, and the unmanned agricultural locomotive switches agricultural implements to perform tasks; S5, performing local obstacle avoidance according to obstacle-related data, and monitoring the progress of task execution in real time. The present invention belongs to the field of intelligent agricultural perception technology, and can realize intelligent perception, full-process monitoring and real-time decision-making of the farm environment.

Description

Technical Field

[0001] The present invention relates to the field of intelligent agricultural perception technology, and in particular to an agricultural data collection and decision-making method and device based on multiple sensors and a cloud platform. Background Art

[0002] In traditional agriculture, the evaluation of crop growth conditions and the scheduling of agricultural machinery tasks mainly rely on human subjective judgment. This dependence has brought about a series of problems and limited the efficiency and sustainability of agricultural production. First, since human judgment is affected by subjective factors, there is a risk of inaccurate evaluation, resulting in the failure to obtain timely and accurate feedback on the growth status of crops. Secondly, the traditional scheduling of tasks is constrained by problems such as insufficient labor and difficulty in formulating work plans, which limits the timeliness and efficiency of operations. In addition, the data collection method of traditional agriculture is relatively backward, unable to provide real-time and comprehensive environmental information, and difficult to meet the needs of agricultural management for accurate decision-making. Summary of the invention

[0003] The purpose of the present invention is to provide an agricultural data collection and decision-making method and device based on multiple sensors and a cloud platform, which can realize intelligent perception, full-process monitoring and real-time decision-making of the farm environment.

[0004] To achieve the above objectives, the present invention provides an agricultural data collection and decision-making method based on multiple sensors and a cloud platform, which comprises:

[0005] S1, collects real-time data in the farmland through various sensors deployed in the farmland, obtains the location and category of objects in the real-time data, and outputs target-level perception data together with the farmland environment data in the real-time data; the target-level perception data includes crop-related data and obstacle-related data in the farmland;

[0006] S2, inputting crop-related data into a crop growth model, and adding farmland environment data corresponding to the crop-related data in real time, to obtain an evaluation result of the current growth status of the crops;

[0007] S3, generating tasks for target crops based on the evaluation results;

[0008] S4, sending the target crop’s mission, location, and crop-related data to the unmanned agricultural vehicle, planning the driving path based on the collected global map and the vehicle’s GPS location, and the unmanned agricultural vehicle switching the required agricultural implements to execute the sent mission;

[0009] S5, based on the current obstacle-related data, the unmanned agricultural vehicle performs local obstacle avoidance and monitors the progress of task execution in real time.

[0010] Furthermore, the multiple sensors include a laser radar and a camera, and step S1 specifically includes:

[0011] S11, acquiring real-time data in the farmland from various sensors deployed in the farmland, and calibrating the conversion relationship between the laser radar coordinate system and the camera coordinate system;

[0012] S12, based on the point cloud collected by the laser radar and the image collected by the camera, the pixel coordinates of the image are unified into the laser radar coordinate system by using the conversion relationship between the laser radar coordinate system and the camera coordinate system, so as to obtain preliminary obstacle-related data; S12 specifically includes:

[0013] S121, removing point cloud areas not related to crops, converting the point cloud into a three-dimensional voxel grid through a voxel downsampling operation, selecting the geometric center of the voxel grid as a representative point, retaining the representative point of each voxel grid, and removing the point cloud corresponding to the ground;

[0014] S122, extracting the point coordinates in the same cluster according to the distances between the point clouds retained in S121, and generating a plurality of clustered point clouds;

[0015] S123, obtain the clustered projection point set formed by the clustered point cloud in the xy plane of the laser radar coordinate system, find the polygon surrounding each clustered projection point set, search for each edge of the polygon to find the minimum circumscribed rectangle, and then combine the maximum coordinate z of the corresponding clustered projection point set on the z axis of the laser radar coordinate system max and the minimum value z min Get the height difference z of the corresponding cluster point cloud center1 , forming a first obstacle three-dimensional bounding box, and obtaining the position and obstacle category of the first obstacle three-dimensional bounding box;

[0016] S13, the type, color, area and height of crops obtained by the camera and lidar are classified into crop-related data.

[0017] Furthermore, in S123, the method of finding a polygon surrounding each cluster projection point set includes:

[0018] Select ((x min +x max ) / 2,(y min+ymax ) / 2) is used as the reference point of the cluster projection point set, and the tangent value of the line connecting the reference point and the remaining points is calculated in turn. The 360-degree area is divided into several tangent value intervals with the reference point as the origin. The farthest point in each tangent value interval is taken as the edge point, and the edge points in each tangent value interval are connected in turn to form a polygon; where (x min ,y min )、(x max ,y maxThey are the minimum and maximum coordinate points of the corresponding cluster point cloud in the plane.

[0019] Furthermore, in S123, the method of finding the minimum circumscribed rectangle through each side of the polygon includes:

[0020] S123a, determining whether the inner angle formed by two adjacent sides of the selected polygon is greater than 180°, if so, removing the two sides, otherwise retaining the two sides, traversing all two adjacent sides, obtaining the remaining sides of the polygon, and forming a new polygon;

[0021] S123b, first select the first side of the new polygon, calculate the distance from each vertex of the new polygon to the first side, and record the vertex with the largest distance as the first vertex; then project all points in the new polygon to the first side, take the first and second projection points that are farthest apart on the first side, obtain the required rectangle of the first side, and set the area of ​​the required rectangle to d 1 *d 2 , d 1 is the distance between the first vertex and the first edge, d 2 is the straight-line distance between the first and second projection points, traverse each edge of the new polygon, and convert d 1 *d 2 The required rectangle with the smallest area is taken as the minimum enclosing rectangle, and then the long side and the wide side of the minimum enclosing rectangle are obtained;

[0022] S123c, according to the coordinate values ​​of the first and second projection points in the xy plane of the laser radar coordinate system, obtain the slope of the first side of the minimum circumscribed rectangle, the slope is combined with the coordinates of the first vertex in the xy plane to obtain the opposite side of the first side, that is, the second side, the first and second projection points are projected onto the second side to obtain the third and fourth projection points, according to the coordinates of the first, second and third and fourth projection points in the xy plane, obtain three length values, after the three length values ​​are sorted, the length values ​​from small to large are used as the wide side, long side and hypotenuse of the minimum circumscribed matrix respectively, and the slope of the long side is the orientation angle yaw of the minimum circumscribed rectangle 1 , and get the center point coordinates of the minimum circumscribed rectangle (x center1 ,y center1 );

[0023] The method for obtaining the position of the three-dimensional bounding box of the first obstacle includes:

[0024] According to the minimum bounding rectangle of each cluster point cloud obtained by S123c, the center point coordinates of the first three-dimensional bounding box are (x center1 ,y center1 ,z center1 ), the heading angle is yaw 1, the long side, wide side, and height of the corresponding cluster point cloud on the z-axis of the laser radar coordinate system are set to the length l of the three-dimensional bounding box respectively. 1 、Width 1 , high h 1 ;

[0025] The method for obtaining the category of the first obstacle three-dimensional bounding box includes:

[0026] The deep learning algorithm is used to identify obstacles in the image, as well as the types and colors of crops. The pixel coordinates are unified to the lidar coordinate system using the conversion relationship between the lidar coordinate system and the camera coordinate system. The recognition result of the camera deep learning is matched with the coordinates of the first obstacle. For the part of the first obstacle that does not match the recognition result, the length l is used to calculate the coordinates of the first obstacle. 1 、Width 1 , high h 1 , combining the classification of obstacles with the corresponding threshold of the pre-set three-dimensional bounding box of the object, and determining the category of the three-dimensional bounding box of the first obstacle.

[0027] Furthermore, S12 specifically includes:

[0028] S124, detecting multiple obstacles in the point cloud through the deep learning model, and generating a three-dimensional bounding box of the second obstacle and its position: center point coordinates) x center2 ,y center2 ,z center2 (, the heading angle is yaw 2 , and length l 2 、Width 2 , high h 2 , according to the length l 2 、Width 2 , high h 2 ,Based on obstacle detection, the deep learning model will further classify the objects in each bounding box and determine the obstacle category of the second obstacle 3D bounding box;

[0029] S125, by matching the type and color of the crop in the recognition result with the first obstacle bounding box of the crop, the area and height of the crop are obtained, and the type, color, area and height of the crop are all used as crop-related data; using a multi-sensor target-level fusion algorithm, the position and obstacle category of the first obstacle three-dimensional bounding box and the position and obstacle category of the second obstacle three-dimensional bounding box are obtained to obtain obstacle-related data.

[0030] Furthermore, S2 also includes: displaying the current growth status of crops through a farmland environment digital twin visualization interface.

[0031] The present invention also provides an agricultural data collection and decision-making device based on multiple sensors and a cloud platform, which includes:

[0032] A data unit is used to collect real-time data in the farmland through various sensors deployed in the farmland, obtain the location and category of objects in the real-time data, and then output target-level perception data together with the farmland environment data in the real-time data; wherein the target-level perception data includes crop-related data and obstacle-related data in the farmland;

[0033] A crop growth assessment unit, which is used to input crop-related data into a crop growth model, and simultaneously add farmland environment data corresponding to the crop-related data in real time to obtain an assessment result of the current growth status of the crop;

[0034] A task generation unit, which is used to generate tasks for target crops according to the evaluation results;

[0035] The task issuing unit is used to send the task, the area and the crop-related data of the target crop to the unmanned agricultural vehicle, plan the driving path according to the collected global map and the position of the vehicle's GPS, and the unmanned agricultural vehicle switches the required agricultural implements to execute the issued task;

[0036] The task execution unit is used to perform local obstacle avoidance for the unmanned agricultural vehicle based on the current obstacle-related data and monitor the progress of task execution in real time.

[0037] Furthermore, the various sensors include lidar and cameras, and the data units specifically include:

[0038] A calibration subunit, which is used to obtain real-time data collected by various sensors deployed in the farmland, and calibrate the conversion relationship between the laser radar coordinate system and the camera coordinate system;

[0039] The obstacle data acquisition subunit is used to unify the pixel coordinates of the image into the lidar coordinate system based on the point cloud collected by the lidar and the image collected by the camera by using the conversion relationship between the lidar coordinate system and the camera coordinate system, so as to obtain preliminary obstacle-related data; the obstacle data acquisition subunit specifically includes:

[0040] The point cloud screening module is used to remove point cloud areas that are not related to crops. The point cloud is converted into a three-dimensional voxel grid through voxel downsampling operation, and the geometric center of the voxel grid is selected as the representative point. The representative point of each voxel grid is retained, and the point cloud corresponding to the ground is removed;

[0041] The clustering module is used to extract the point coordinates in the same cluster according to the distance between the point clouds retained by the point cloud screening module to generate several clustered point clouds;

[0042] The first obstacle three-dimensional encirclement acquisition module is used to obtain the clustered projection point set formed by the clustered point cloud in the xy plane of the laser radar coordinate system, by finding the polygon surrounding each clustered projection point set, searching for each edge of the polygon to find the minimum circumscribed rectangle, and then combining the corresponding clustered projection point set on the z axis of the laser radar coordinate system with the maximum coordinate z max and the minimum value z min Get the height difference z of the corresponding cluster point cloud center1 , forming a first obstacle three-dimensional bounding box, and obtaining the position and obstacle category of the first obstacle three-dimensional bounding box;

[0043] The preliminary crop data acquisition subunit is used to classify the type, color, area and height of crops obtained by cameras and lidar into crop-related data.

[0044] Furthermore, the first obstacle three-dimensional encirclement acquisition module specifically includes:

[0045] The polygon acquisition submodule is used to select ((x min +x max ) / 2,(y min+ymax ) / 2) as the reference point of the cluster projection point set, calculate the tangent value of the line connecting the reference point and the remaining points in turn, divide the 360-degree area into several tangent value intervals with the reference point as the origin, take the farthest point in each tangent value interval as the edge point, and connect the edge points in each tangent value interval in turn to form a polygon; where x min ,y min )、(x max ,y max ) are the minimum and maximum coordinate points of the corresponding cluster point cloud in the plane.

[0046] Furthermore, the first obstacle three-dimensional encirclement acquisition module specifically includes:

[0047] A new polygon acquisition submodule is used to determine whether the inner angle formed by two adjacent sides of the selected polygon is greater than 180°. If so, the two sides are removed, otherwise the two sides are retained, and all adjacent two sides are traversed to obtain the remaining sides of the polygon to form a new polygon;

[0048] The minimum circumscribed matrix acquisition submodule is used to first select the first edge of the new polygon, calculate the distance from each vertex of the new polygon to the first edge, and record the vertex with the largest distance as the first vertex; then project all points in the new polygon to the first edge, take the first and second projection points that are farthest apart on the first edge, obtain the required rectangle of the first edge, and set the area of ​​the required rectangle to d 1 *d 2 , d 1 is the distance between the first vertex and the first edge, d2 is the straight-line distance between the first and second projection points, traverse each edge of the new polygon, and convert d 1 *d 2 The rectangle with the smallest area is used as the minimum circumscribed rectangle, and then the long side and wide side of the minimum circumscribed rectangle are obtained; according to the coordinate values ​​of the first and second projection points in the xy plane of the laser radar coordinate system, the slope of the first side of the minimum circumscribed rectangle is obtained, and the slope is combined with the coordinates of the first vertex in the xy plane to obtain the opposite side of the first side, that is, the second side. The first and second projection points are projected onto the second side to obtain the third and fourth projection points. According to the coordinates of the first, second and third and fourth projection points in the xy plane, three length values ​​are obtained. After the three length values ​​are sorted, the length values ​​from small to large are used as the wide side, long side, and hypotenuse of the minimum circumscribed matrix respectively, and the slope of the long side is the orientation angle yaw of the minimum circumscribed rectangle 1 , and get the center point coordinates of the minimum circumscribed rectangle (x center1 ,y center1 );

[0049] The first obstacle three-dimensional bounding box position acquisition submodule is used to obtain the minimum bounding rectangle of each cluster point cloud obtained by the minimum bounding matrix acquisition submodule. The center point coordinates of the first three-dimensional bounding box are (x center1 ,y center1 ,z center1 ), the heading angle is yaw 1 , the long side, wide side, and height of the corresponding cluster point cloud on the z-axis of the laser radar coordinate system are set to the length l of the three-dimensional bounding box respectively. 1 、Width 1 , high h 1 ;

[0050] The category acquisition submodule of the first obstacle three-dimensional bounding box is used to use the deep learning algorithm to identify obstacles in the image, as well as the types and colors of crops, and use the conversion relationship between the laser radar coordinate system and the camera coordinate system to unify the pixel coordinates into the laser radar coordinate system, and match the recognition result of the camera deep learning with the coordinates of the first obstacle. For the part of the first obstacle that does not match the recognition result, the length l is used to obtain the category of the first obstacle. 1 、Width 1 , high h 1 , combining the classification of obstacles with the corresponding threshold of the pre-set three-dimensional bounding box of the object, and determining the category of the three-dimensional bounding box of the first obstacle.

[0051] Furthermore, the obstacle data acquisition subunit specifically includes:

[0052] The second obstacle 3D encirclement acquisition module is used to detect multiple obstacles in the point cloud through a deep learning model and generate a 3D encirclement frame of the second obstacle and its position: the center point coordinate x center2 ,y center2 ,z center2 ), the heading angle is yaw 2 , and length l 2 、Width 2 , high h 2 , according to the length l 2 、Width 2 , high h 2 ,Based on obstacle detection, the deep learning model will further classify the objects in each bounding box and determine the obstacle category of the second obstacle 3D bounding box;

[0053] The obstacle data fusion module is used to obtain the area and height of the crop by matching the type and color of the crop in the recognition result with the first obstacle bounding box of the crop. The type, color, area and height of the crop are all used as crop-related data; a multi-sensor target-level fusion algorithm is used to obtain obstacle-related data by the position and obstacle category of the first obstacle three-dimensional bounding box and the position and obstacle category of the second obstacle three-dimensional bounding box.

[0054] The present invention has the following beneficial effects:

[0055] 1. Since the present invention configures a variety of sensors in the farmland, the ability to perceive the state of the farmland is significantly improved. At the same time, laser radars and monocular cameras are installed at high places in the farmland to effectively realize the accurate identification of the position of agricultural machinery vehicles. Therefore, the unmanned agricultural machinery vehicles themselves do not need to be equipped with GPS (the full name of "Global Positioning System" in English, the full name of "Global Positioning System" in Chinese), IMU (the full name of "inertial measurement unit" in English, the full name of "inertial measurement unit" in Chinese), and other positioning devices, and can realize the vehicle-road-cloud collaboration of smart agriculture.

[0056] 2. Since the present invention establishes a growth model for farmland data and uses a real-time data update mechanism, it can form a digital twin interface of the farmland environment, which makes the growth simulation of crops more realistic and accurate. At the same time, users can obtain the changing trends of environmental parameters in real time through the interface, supporting more accurate farmland management and decision optimization.

[0057] 3. Since the present invention intelligently issues the operation task plan of the unmanned agricultural machinery vehicle through the evaluation of the current growth environment, and plans the driving path in combination with the global map, it helps to improve the operation efficiency of the agricultural machinery vehicle and ensure the operation quality. At the same time, the unmanned agricultural machinery vehicle does not need to be equipped with sensors. It can perform local obstacle avoidance through the real-time data of sensors installed in the farmland, forming a closed-loop feedback of the system, and can dynamically adjust the operation path to avoid collision with obstacles, so as to achieve safer and more efficient agricultural operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 The present invention is a flowchart of an agricultural data collection and decision-making method based on multiple sensors and a cloud platform according to an embodiment of the present invention.

[0059] Figure 2 A schematic diagram of finding a polygon surrounding each cluster projection point set according to an embodiment of the present invention.

[0060] Figure 3 This is a schematic diagram of finding the minimum circumscribed rectangle of each side of a new polygon according to an embodiment of the present invention. DETAILED DESCRIPTION

[0061] In the drawings, the same or similar reference numerals are used to represent the same or similar elements or elements with the same or similar functions. The embodiments of the present invention are described in detail below in conjunction with the drawings.

[0062] In the description of the present invention, the terms "center", "longitudinal", "lateral", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as limiting the scope of protection of the present invention.

[0063] like Figure 1 As shown, the agricultural data collection and decision-making method based on multiple sensors and a cloud platform provided in an embodiment of the present invention includes:

[0064] S1, collects real-time data in the farmland through various sensors deployed in the farmland, obtains the position and category of the object in the real-time data, and outputs the target-level perception data together with the farmland environment data in the real-time data, for example, all of which are transmitted to the cloud platform, and the transmission method can be a real-time transmission method such as TCP / IP communication protocol.

[0065] Among them, the various sensors in this embodiment include laser radar, camera, light sensor, soil temperature and humidity sensor, soil nutrient rapid tester and other sensing equipment. The laser radar, camera, and light sensor are preferably installed at a high place in the farmland, and the soil temperature and humidity sensor and the soil nutrient rapid tester need to be buried in the soil of the farmland to be tested. The farmland environmental data is directly obtained by relevant sensors, including moisture, soil state, and vegetation growth. Target-level perception data includes crop-related data and obstacle-related data in the farmland, which are obtained by identifying the data collected by the laser radar and camera. The identification process will be explained below.

[0066] S2, inputting the crop-related data into the crop growth model, and adding the farmland environment data corresponding to the crop-related data in real time, to obtain the evaluation result of the current growth condition of the crop.

[0067] S3, generates tasks for target crops based on the evaluation results.

[0068] S4, the mission, area and crop-related data of the target crops are sent to the unmanned agricultural vehicle, and the driving route is planned according to the collected global map and the position of the vehicle's GPS. The unmanned agricultural vehicle switches the required agricultural tools to execute the issued tasks.

[0069] S5, based on the current obstacle-related data, the unmanned agricultural vehicle performs local obstacle avoidance and monitors the progress of task execution in real time.

[0070] Since this embodiment introduces multiple sensors, it is possible to efficiently obtain real-time data on crop growth and farm environment, from soil conditions, vegetation growth to the operating conditions of operating vehicles, to achieve a comprehensive perception of the entire agricultural ecosystem. Combined with the use of digital twin technology, this embodiment can simulate and reflect the operating status of actual farms in a virtual environment, providing a more accurate reference for agricultural decision-making.

[0071] In one embodiment, the multiple sensors include a laser radar and a camera, and step S1 specifically includes:

[0072] S11, obtaining real-time data collected by various sensors deployed in the farmland, and calibrating the conversion relationship between the laser radar coordinate system and the camera coordinate system. The calibration method can be implemented by an existing method or by the calibration method provided in the following example.

[0073] For example, if the camera is a monocular camera, first use external parameters to calibrate the coordinate transformation relationship between the lidar and the monocular camera. Let the lidar coordinate system be L and the monocular camera coordinate system be C. The lidar point in the lidar coordinate system is referred to as the point coordinate below and is expressed in the lidar coordinate system L as (X L ,Y L,Z L ).

[0074] The pixel coordinates of the monocular camera are (u C ,v C ), the external parameters include the translation vector T from the laser radar to the monocular camera LC and the rotation matrix R LC , the monocular camera projection model can be expressed as formula (1):

[0075]

[0076] Among them, K C is the intrinsic parameter matrix of the monocular camera.

[0077] S12, based on the point cloud collected by the laser radar and the image collected by the camera, the pixel coordinates of the image are unified into the laser radar coordinate system by using the conversion relationship between the laser radar coordinate system and the camera coordinate system to obtain obstacle related data. S12 specifically includes:

[0078] S121 can reduce the amount of calculation by cutting the x-axis and y-axis of the point cloud to remove the point cloud area that is not related to crops. Through the voxel downsampling operation, the point cloud is converted into a three-dimensional voxel grid, the geometric center of the voxel grid is selected as the representative point, and the representative point of each voxel grid is retained. A height threshold is set from the point cloud data, and the height threshold is determined according to the actual distance from the laser radar to the ground to remove the point cloud on the ground. The average value of the point coordinates in the neighborhood of each point is calculated by weighted averaging such as Gaussian filtering, as the point cloud coordinate value after filtering of the point, so as to achieve smoothing of the laser radar point cloud. Among them, the point cloud is a collection composed of a large number of point coordinates, and each point coordinate contains the position information in the laser radar coordinate system. A voxel is a cubic unit in the laser radar coordinate system, and the point cloud in the laser radar coordinate system is divided into several cubic units, each of which may contain one or more points, represented by a voxel. The three-dimensional voxel grid is a three-dimensional grid structure composed of a series of voxels, which is divided into many small cubes in the laser radar coordinate system.

[0079] S122, according to the distance between the point clouds, the point coordinates in the same cluster are extracted to generate a number of clustered point clouds. The clustering method can be, for example, the Euclidean clustering DBSCAN algorithm, combined with existing algorithms such as Kd-Tree, Ball Tree, and R-Tree, to quickly search for the nearest neighbor of each point in the point cloud outputted in step S121 to complete the clustering operation of the point cloud.

[0080] S123, for example, by using pcl::getMinMax3D(cluster,minPoint,maxPoint), obtain the minimum coordinate point and the maximum coordinate point in each cluster point cloud. Where cluster is the cluster point cloud, minPoint is the minimum coordinate point (x min ,y min ,z min ), maxPoint is the maximum value coordinate point x max ,y max ,z max ). pcl::getMinMax3D is a function in the PointCloud Library (PCL) that is used to obtain the minimum and maximum values ​​of each dimension in the point cloud. Specifically, this function will traverse all points in the clustered point cloud and find the minimum and maximum coordinate values ​​in the x, y, and z dimensions of the laser radar coordinate system. After obtaining the minimum coordinate point minPoint and the maximum coordinate point maxPoint, the clustered point cloud is projected from the laser radar coordinate system to the plane to obtain the clustered projection point set formed by each clustered point cloud in the xy plane. Among them, the x-axis is the scanning direction of the solid-state laser radar set at a high place, that is, from front to back along the ground, the y-axis is the horizontal direction of the solid-state laser radar, that is, perpendicular to the scanning direction, from left to right, and the z-axis is the vertical direction of the ground, from bottom to top. Projecting to the xy plane here means discarding the z-axis height and projecting the point to the xy plane.

[0081] Of course, the cluster projection point set formed by each cluster point cloud in the xy plane can also be obtained by other existing methods, which will not be described one by one here.

[0082] After obtaining the cluster projection point set, we then find the polygon that surrounds each cluster projection point set, search for each edge of the polygon to find the minimum circumscribed rectangle, and then combine the corresponding cluster projection point set with the maximum coordinate z on the z-axis of the laser radar coordinate system. max and the minimum value z min Get the height difference z of the corresponding cluster point cloud center1 , forming a first obstacle three-dimensional bounding box, and obtaining the position of the first obstacle three-dimensional bounding box.

[0083] S123 provided in this embodiment obtains the position of the three-dimensional bounding box of the first obstacle and performs preliminary point cloud segmentation. At this time, the point cloud is divided into several clusters. For example, agricultural tools and a piece of crops are separated separately. Because the point cloud of an object is dense, there will be a certain distance between the point clouds of two objects. The purpose of finding the minimum circumscribed rectangle is to wrap the cluster with a three-dimensional bounding box, so that the coordinates of the center point of the cluster point cloud can be obtained.

[0084] In one embodiment, in S123, the method of finding a polygon surrounding each cluster projection point set includes: Figure 2 As shown:

[0085] In a cluster projection point set, select ((x min +x max ) / 2,(y min+ymax ) / 2) as the reference point A of the clustered projection point set, calculate the tangent value of the line connecting the reference point A and the remaining points in turn, and divide the 360-degree area into several tangent value intervals with the reference point A as the origin. Take the farthest point in each tangent value interval as the edge point, and connect the edge points in each tangent value interval in turn to form a polygon. Figure 2 , the blue points are cluster projection points, the green points are the origin A, the blue line segments divide the 360-degree area into 6 tangent value intervals, the red points are the farthest points in each interval, and the red line segments connect these 6 edge points in sequence to form a polygon surrounding each cluster projection point set. min ,y min )、(x max ,y max ) are the minimum and maximum coordinate points of the corresponding cluster point cloud in the plane.

[0086] S123 can also find a method to surround each clustered projection point set by existing methods such as Graham scanning algorithm or Jarvis stepping method. The advantage of clustered point cloud detection provided by S123 is that the detection is more comprehensive and no missed detection occurs.

[0087] In one embodiment, in S123, the method of finding the minimum circumscribed rectangle through each side of the polygon includes:

[0088] S123a, determine whether the inner angle formed by two adjacent sides of the selected polygon is greater than 180°, if so, remove the two sides, otherwise keep the two sides, traverse all two adjacent sides, obtain the remaining sides of the polygon, and form a new polygon.

[0089] For example, Figure 2 As shown in the figure, from L 0 Start, judge and L 1 Is the internal angle formed greater than 180°? If it is less than 180°, keep these two sides and then determine L 1 With L 2 Is the internal angle greater than 180°? If it is greater than 180°, remove L 1 and L 2 These two edges continue to judge L 3 and L 4 Until L is determined 5 To L 0, get the remaining edges of the polygon and form a new polygon.

[0090] Obtaining a new polygon can also be achieved by other existing methods, for example: taking the point with the smallest y coordinate value in the cluster projection point set as the starting point, and sorting the point set in ascending order according to the polar coordinate angle with the starting point. First, connect the second point in the sorted point set with the starting point to obtain the first edge, then connect the third point with the second point to obtain the second edge, and determine whether the angle formed by the two edges is greater than 180°. If so, remove the third point, and continue to connect the fourth point with the second point and then determine; if not, retain the third point, the third point is a vertex, and continue to connect the fourth point with the third point and then determine with the previous edge. Consider each point in turn, and finally obtain the point set of the polygon.

[0091] S123b, after determining the new polygon, find the minimum circumscribed rectangle of each side of the new polygon:

[0092] like Figure 3 As shown, first select the first edge L in the new polygon 0 , calculate the distance from each vertex of the new polygon to the first edge L 0 The vertex with the largest distance is recorded as the first vertex p 0 ; Then project all the points in the new polygon to the first edge L 0 , take the first edge L 0 The first and second projection points p that are farthest apart 1 、p 2 , get the first edge L 0 The area of ​​the required rectangle is set to d 1 *d 2 , d 1 is the first vertex p 0 With the first edge L 0 The distance, d 2 The first and second projection points p 1 、p 2 Then we select the next side of the polygon and repeat this process until we find the side with the smallest area of ​​the rectangle. That is, d 1 *d 2 The smallest rectangle to be found is taken as the minimum circumscribed matrix, and the edge of the minimum circumscribed matrix is ​​recorded as the first edge L of the minimum circumscribed rectangle. 1 .

[0093] S123c, according to the first and second projection points p 1 、p 2 Obtain the coordinate value of the first side L of the minimum circumscribed rectangle in the xy plane of the laser radar coordinate system 1 The slope of the first vertex p0 Obtain the opposite side of the first side, i.e. the second side L, from the coordinates of the xy plane 2 , project the first and second projection points onto the second edge L 2 On the third and fourth projection points p 3 、p 4 , according to the first and second projection points p 1 、p 2 And the third and fourth projection points p 3 、p 4 In the coordinates of the xy plane, three length values ​​are obtained. After the three length values ​​are sorted, the largest length value is used as the hypotenuse of the minimum circumscribed matrix, the smallest length value is used as the wide side of the minimum circumscribed matrix, and the middle length value is used as the long side of the minimum circumscribed matrix, where the slope of the long side is the orientation angle yaw of the minimum circumscribed rectangle 1 , from the four endpoints of the minimum circumscribed rectangle, we can get the center point coordinates (x center1 ,y center1 ).

[0094] The method for obtaining the position of the first obstacle three-dimensional bounding box includes:

[0095] Position: According to the minimum bounding rectangle of each cluster point cloud obtained by S123c, the center point coordinates of the first three-dimensional bounding box are (x center1 ,y center1 ,z center1 ), the heading angle is yaw 1 , the long side, wide side, and height of the corresponding cluster point cloud on the z-axis of the laser radar coordinate system are set to the length l of the three-dimensional bounding box respectively. 1 、Width 1 , high h 1 .

[0096] The method for obtaining the category of the first obstacle three-dimensional bounding box includes:

[0097] The deep learning algorithm is used to identify obstacles in the image, as well as the types and colors of crops. The pixel coordinates are unified to the lidar coordinate system using the conversion relationship between the lidar coordinate system and the camera coordinate system. The recognition result of the camera deep learning is matched with the coordinates of the first obstacle. For the part of the first obstacle that does not match the recognition result, the length l is used to calculate the coordinates of the first obstacle. 1 、Width 1 , high h 1 , combined with the corresponding threshold of the pre-set three-dimensional bounding box of the object to classify the obstacles, determine the category of the three-dimensional bounding box of the first obstacle. Among them, the deep learning algorithm can be, for example, the YOLOV5 target detection algorithm, which can identify the type and color of land and crops in the image. Matching can be understood as corresponding.

[0098] In one embodiment, further, in order to increase the accuracy of point cloud recognition, S12 specifically includes:

[0099] S124, detect multiple obstacles in the point cloud through a deep learning model (such as PointNet, PointPillar, CenterPoint, etc.), and generate a three-dimensional bounding box of the second obstacle and its position: the center point coordinates (x center2 ,y center2 ,z center2 ), the heading angle is yaw 2 , and length l 2 、Width 2 , high h 2 , according to the length l 2 、Width 2 , high h 2 ,Based on obstacle detection, the deep learning model will further classify the objects in each bounding box and determine the obstacle category of the second obstacle 3D bounding box.

[0100] Through S124 provided in this embodiment, obstacles are further identified through point cloud deep learning, which can improve the accuracy of point cloud identification of obstacles such as people and farm tools. The point cloud of the lidar is identified using a deep learning algorithm to continue identifying obstacles. As further obstacle-related data, the obstacle classification can be more accurate, which complements the obstacle category of the first obstacle three-dimensional bounding box obtained in S123.

[0101] S125, by matching the type and color of the crop in the recognition result with the first obstacle bounding box of the crop, the area and height of the crop are obtained, and the type, color, area and height of the crop are all used as crop-related data.

[0102] By using a multi-sensor target-level fusion algorithm such as an extended Kalman filter, obstacle-related data is obtained for the position of the first obstacle three-dimensional bounding box and the obstacle category in S123c and the position of the second obstacle three-dimensional bounding box and the obstacle category in S124.

[0103] Based on crop-related data and obstacle-related data, the geometric characteristics of the object can be obtained, which will then be used for obstacle avoidance in subsequent unmanned agricultural vehicles.

[0104] Through S125 provided in this embodiment, the positions of obstacles such as people and farm tools identified by S123 and S124 can be fused, and the fused positions are used for subsequent unmanned farm tools vehicles to perform operations. The type, color, area, height, etc. of the crops can also be obtained by S124. The type and color can be identified by the camera, while the area and height are obtained by the area and height of point cloud clustering. The information of the crops is input into the growth model through the cloud server to perform digital twin and automated scheduling operations.

[0105] In one embodiment, S2 also includes: displaying the current growth status of crops through a farmland environment digital twin visualization interface.

[0106] In one embodiment, step 2 includes:

[0107] S21, establish a data storage structure on the cloud platform and synchronize the timestamps of each sensor data.

[0108] S22, prior knowledge such as the geographical location and soil type of farmland can be used to simulate the growth, development and yield of different crops using the existing DSSAT (Decision Support System for Agrotechnology Transfer) model, and select the corresponding crop growth model in combination with the crop types identified in S13. Among them, the DSSAT model is a comprehensive crop model system used to simulate the growth, development and yield of different crops. The DSSAT model includes multiple sub-models, each of which uses a series of formulas to describe the different processes of crop growth.

[0109] S23, transmits the real-time collected data to the DSSAT model, and the real-time data includes light, soil temperature and humidity, nutrient concentration, etc. The crop growth model is an existing model, a comprehensive crop model system, which can simulate the growth process of various crops, including corn, wheat, rice, etc.

[0110] S24, using the computing resources of the cloud platform, runs the DSSAT model to simulate the growth of crops.

[0111] S25, combined with the output results and real-time data of the DSSAT model, with the help of the graphics rendering capability and scalability of the Unity software, creates an efficient and intuitive visualization interface to display the real-time changes of crop growth status and environmental parameters, thereby building a digital twin interface of the farmland environment. The interface visualizes the growth status of crops such as area and height, and the real-time change trend of environmental parameters.

[0112] In one embodiment, step 3 includes:

[0113] S31, process and analyze the data acquired in real time, including trend analysis and anomaly detection of environmental parameters. Compare the growth simulation results output by the DSSAT model in S24 with the real-time data to find out the key parameters that do not meet the standards. For example, if the growth stage predicted by the DSSAT model does not meet the standards, check whether environmental parameters such as soil moisture and nutrients are lower than the historical average level to evaluate the current growth status of the farmland, including the growth stage of crops, pests and diseases, etc.

[0114] S32, based on the evaluation results of the current growth environment, automatically generates task scheduling for unmanned agricultural vehicles, and the task scheduling includes agricultural production activities such as sowing, watering, and fertilizing.

[0115] Step 4: Send the target crop area to the unmanned agricultural vehicle, plan the driving route based on the collected global map and the GPS position of the vehicle, and the unmanned agricultural vehicle switches the required agricultural tools to perform the assigned tasks.

[0116] Specifically, step 4 includes:

[0117] S41, automatically sends the generated task scheduling to the unmanned agricultural vehicle through the TCP / IP communication protocol of the cloud platform. The transmitted information includes the coordinates, shape, area, etc. of the target crop obtained in S126, so that the agricultural vehicle can accurately locate and operate.

[0118] S42, using the collected global map information, plans the driving path of the unmanned agricultural vehicle to ensure that it can efficiently reach the area where the target crops are located.

[0119] S43, generating a switching task of the agricultural implements required by the agricultural vehicle according to the target crops and operation requirements, and sending the task to the unmanned agricultural vehicle to ensure that it can switch and perform the corresponding agricultural implement operation after reaching the target crop area.

[0120] Step 5: Send the target-level perception data to the unmanned agricultural vehicle through the cloud platform for local obstacle avoidance, monitor the progress of task execution in real time, and form a closed-loop feedback loop of the system.

[0121] Specifically, step 5 includes:

[0122] S51, agricultural machinery vehicles do not need to be equipped with sensors and positioning equipment such as GPS and IMU. The global coordinates of the vehicle can be obtained through the laser radar and monocular camera installed high above the farmland.

[0123] S52, after receiving the object target-level perception data, the cloud platform uses the local obstacle avoidance algorithm to analyze and process the data, adjusts the path planning in a timely manner, and issues control instructions to the unmanned agricultural machinery vehicle, including adjusting the driving speed, changing the direction of travel, etc., to avoid collisions between agricultural machinery vehicles during operation.

[0124] S53 monitors the progress of the unmanned agricultural vehicle in the target crop area in real time, including sowing coverage, fertilization uniformity, etc. By monitoring abnormal conditions during the operation, the task plan is dynamically adjusted to form a closed-loop intelligent agricultural management system.

[0125] This embodiment also provides an agricultural data collection and decision-making device based on multiple sensors and a cloud platform, which includes a data unit, a crop growth assessment unit, a task generation unit, a task issuing unit, and a task execution unit, wherein:

[0126] The data unit is used to collect real-time data in the farmland through various sensors deployed in the farmland, obtain the location and category of objects in the real-time data, and output target-level perception data together with the farmland environment data in the real-time data. Among them, the target-level perception data includes crop-related data and obstacle-related data in the farmland.

[0127] The crop growth assessment unit is used to input crop-related data into the crop growth model, and at the same time add farmland environment data corresponding to the crop-related data in real time to obtain an assessment result of the current growth status of the crops.

[0128] The task generation unit is used to generate tasks for target crops according to the evaluation results.

[0129] The task issuing unit is used to send the target crop's task, location and crop-related data to the unmanned agricultural vehicle, plan the driving route according to the collected global map and the vehicle's GPS position, and the unmanned agricultural vehicle switches the required agricultural tools to perform the issued task.

[0130] The task execution unit is used to perform local obstacle avoidance for the unmanned agricultural vehicle based on the current obstacle-related data and monitor the progress of task execution in real time.

[0131] In one embodiment, the multiple sensors include a laser radar and a camera, and the data unit specifically includes a calibration subunit, an obstacle data acquisition subunit, and a preliminary crop data acquisition subunit, wherein:

[0132] The calibration subunit is used to obtain real-time data in the farmland collected by various sensors deployed in the farmland, and to calibrate the conversion relationship between the lidar coordinate system and the camera coordinate system.

[0133] The obstacle data acquisition subunit is used to obtain preliminary obstacle-related data by unifying the pixel coordinates of the image to the laser radar coordinate system based on the point cloud collected by the laser radar and the image collected by the camera using the conversion relationship between the laser radar coordinate system and the camera coordinate system. The obstacle data acquisition subunit specifically includes a point cloud screening module, a clustering module, and a first obstacle three-dimensional encirclement acquisition module, wherein:

[0134] The point cloud screening module is used to remove point cloud areas that are not related to crops. Through voxel downsampling operations, the point cloud is converted into a three-dimensional voxel grid. The geometric center of the voxel grid is selected as the representative point, the representative point of each voxel grid is retained, and the point cloud corresponding to the ground is removed.

[0135] The clustering module is used to extract the point coordinates in the same cluster according to the distance between the point clouds retained by the point cloud screening module to generate several clustered point clouds.

[0136] The first obstacle three-dimensional encirclement acquisition module is used to obtain the clustered projection point set formed by the clustered point cloud in the xy plane of the laser radar coordinate system, by finding the polygon surrounding each clustered projection point set, searching for each edge of the polygon to find the minimum circumscribed rectangle, and then combining the corresponding clustered projection point set on the z axis of the laser radar coordinate system to obtain the maximum coordinate z max and the minimum value z min Get the height difference z of the corresponding cluster point cloud center1 , forming a first obstacle three-dimensional bounding box, and obtaining the position and obstacle category of the first obstacle three-dimensional bounding box.

[0137] The preliminary crop data acquisition subunit is used to classify the type, color, area and height of crops obtained by cameras and lidar into crop-related data.

[0138] In one embodiment, the first obstacle three-dimensional encirclement acquisition module specifically includes a new polygon acquisition submodule, a minimum circumscribed matrix acquisition submodule, a first obstacle three-dimensional encirclement frame position acquisition submodule and a first obstacle three-dimensional encirclement frame category acquisition submodule, wherein:

[0139] The new polygon acquisition submodule is used to determine whether the internal angle formed by two adjacent sides of the selected polygon is greater than 180°. If so, these two sides are removed, otherwise these two sides are retained, and all adjacent two sides are traversed to obtain the remaining sides of the polygon to form a new polygon.

[0140] The minimum circumscribed matrix acquisition submodule is used to first select the first edge of the new polygon, calculate the distance from each vertex of the new polygon to the first edge, and record the vertex with the largest distance as the first vertex; then project all points in the new polygon to the first edge, take the first and second projection points that are farthest apart on the first edge, obtain the required rectangle of the first edge, and set the area of ​​the required rectangle to d 1 *d 2 , d 1 is the distance between the first vertex and the first edge, d 2 is the straight-line distance between the first and second projection points, traverse each edge of the new polygon, and convert d 1 *d 2The rectangle with the smallest area is used as the minimum circumscribed rectangle, and then the long side and wide side of the minimum circumscribed rectangle are obtained; according to the coordinate values ​​of the first and second projection points in the xy plane of the laser radar coordinate system, the slope of the first side of the minimum circumscribed rectangle is obtained, and the slope is combined with the coordinates of the first vertex in the xy plane to obtain the opposite side of the first side, that is, the second side. The first and second projection points are projected onto the second side to obtain the third and fourth projection points. According to the coordinates of the first, second and third and fourth projection points in the xy plane, three length values ​​are obtained. After the three length values ​​are sorted, the length values ​​from small to large are used as the wide side, long side, and hypotenuse of the minimum circumscribed matrix respectively, and the slope of the long side is the orientation angle yaw of the minimum circumscribed rectangle 1 , and get the center point coordinates of the minimum circumscribed rectangle (x center1 ,y center1 ).

[0141] The first obstacle three-dimensional bounding box position acquisition submodule is used to obtain the minimum bounding rectangle of each cluster point cloud according to the minimum bounding matrix acquisition submodule. The center point coordinates of the first three-dimensional bounding box are (x center1 ,y center1 ,z center1 ), the heading angle is yaw 1 , the long side, wide side, and height of the corresponding cluster point cloud on the z-axis of the laser radar coordinate system are set to the length l of the three-dimensional bounding box respectively. 1 、Width 1 , high h 1 .

[0142] The category acquisition submodule of the first obstacle three-dimensional bounding box is used to use the deep learning algorithm to identify obstacles in the image, as well as the types and colors of crops, and use the conversion relationship between the laser radar coordinate system and the camera coordinate system to unify the pixel coordinates into the laser radar coordinate system, and match the recognition result of the camera deep learning with the coordinates of the first obstacle. For the part of the first obstacle that does not match the recognition result, the length l is used to obtain the category of the first obstacle. 1 、Width 1 , high h 1 , combining the classification of obstacles with the corresponding threshold of the pre-set three-dimensional bounding box of the object, and determining the category of the three-dimensional bounding box of the first obstacle.

[0143] In one embodiment, the obstacle data acquisition subunit further includes a second obstacle three-dimensional encirclement acquisition module and an obstacle data fusion module, wherein:

[0144] The second obstacle 3D encirclement acquisition module is used to detect multiple obstacles in the point cloud through the deep learning model and generate the 3D encirclement frame of the second obstacle and its position: the center point coordinate x center2 ,y center2 ,zcenter2 ), the heading angle is yaw 2 , and length l 2 、Width 2 , high h 2 , according to the length l 2 、Width 2 , high h 2 ,Based on obstacle detection, the deep learning model will further classify the objects in each bounding box and determine the obstacle category of the second obstacle 3D bounding box.

[0145] The obstacle data fusion module is used to obtain the area and height of the crop by matching the type and color of the crop in the recognition result with the first obstacle bounding box of the crop. The type, color, area and height of the crop are all used as crop-related data; a multi-sensor target-level fusion algorithm is used to obtain obstacle-related data by the position and obstacle category of the first obstacle three-dimensional bounding box and the position and obstacle category of the second obstacle three-dimensional bounding box.

[0146] Finally, it should be pointed out that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them. Those skilled in the art should understand that the technical solutions described in the above embodiments may be modified, or some of the technical features thereof may be replaced by equivalents; these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An agricultural data collection and decision-making method based on multi-sensors and cloud platform, characterized in that: include: S1, collects real-time data in the farmland through various sensors deployed in the farmland, obtains the location and category of objects in the real-time data, and then outputs target-level perception data together with the farmland environment data in the real-time data; the target-level perception data includes crop-related data and obstacle-related data in the farmland; the various sensors include lidar, camera, light sensor, soil temperature and humidity sensor and soil nutrient rapid tester, and the farmland environment data includes moisture, soil state and vegetation growth; S2, establish a data storage structure on the cloud platform, synchronize the timestamps of each sensor data, input crop-related data into the crop growth model, use the computing resources of the cloud platform to run the crop growth model, simulate the growth, development and yield of different crops, and add farmland environmental data that corresponds to the crop-related data in real time to obtain the evaluation results of the current growth status of crops and display the real-time changes of crop growth status and environmental parameters; S3, generating tasks for target crops based on the evaluation results; S4, sending the target crop’s mission, location, and crop-related data to the unmanned agricultural vehicle, planning the driving route based on the collected global map and the vehicle’s GPS location, and the unmanned agricultural vehicle switching the required agricultural implements to execute the sent mission; S5, based on the current obstacle-related data, the unmanned agricultural vehicle performs local obstacle avoidance and monitors the task execution progress in real time; Step S1 specifically includes: S12, based on the point cloud collected by the laser radar and the image collected by the camera, the pixel coordinates of the image are unified into the laser radar coordinate system by using the conversion relationship between the laser radar coordinate system and the camera coordinate system, so as to obtain preliminary obstacle-related data; S12 specifically includes: S121, removing point cloud areas not related to crops, converting the point cloud into a three-dimensional voxel grid through a voxel downsampling operation, selecting the geometric center of the voxel grid as a representative point, retaining the representative point of each voxel grid, and removing the point cloud corresponding to the ground; S122, extracting the point coordinates in the same cluster according to the distances between the point clouds retained in S121, and generating a plurality of clustered point clouds; S123, obtain the clustered projection point set formed by the clustered point cloud in the xy plane of the laser radar coordinate system, find the polygon surrounding each clustered projection point set, search for each edge of the polygon to find the minimum circumscribed rectangle, and then combine the maximum coordinate z of the corresponding clustered projection point set on the z axis of the laser radar coordinate system max and the minimum value z min Get the height difference z of the corresponding cluster point cloud center1 , forming a first obstacle three-dimensional bounding box, and obtaining the position and obstacle category of the first obstacle three-dimensional bounding box; S124, multiple obstacles are detected in the point cloud through the deep learning model, and a three-dimensional bounding box of the second obstacle and its position are generated: the center point coordinates (x center2 ,y center2 ,z center2 ), the orientation angle is yaw2, and the length is l2, the width is w2, and the height is h2. Based on the length l2, the width is w2, and the height is h2, the deep learning model further classifies the objects in each bounding box on the basis of obstacle detection to determine the obstacle category of the second obstacle three-dimensional bounding box; S125, by matching the type and color of the crop in the recognition result with the first obstacle bounding box of the crop, the area and height of the crop are obtained, and the type, color, area and height of the crop are all used as crop-related data; using a multi-sensor target-level fusion algorithm, the position and obstacle category of the first obstacle three-dimensional bounding box and the position and obstacle category of the second obstacle three-dimensional bounding box are obtained to obtain obstacle-related data; In S123, the method of finding the minimum circumscribed rectangle through each side of the polygon includes: S123a, determining whether the inner angle formed by two adjacent sides of the selected polygon is greater than 180°, if so, removing the two sides, otherwise retaining the two sides, traversing all two adjacent sides, obtaining the remaining sides of the polygon, and forming a new polygon; S123b, first select the first side of the new polygon, calculate the distance from each vertex of the new polygon to the first side, and record the vertex with the largest distance as the first vertex; then project all points in the new polygon to the first side, take the first and second projection points that are farthest apart on the first side, obtain the required rectangle of the first side, set the area of ​​the required rectangle to d1*d2, d1 is the distance between the first vertex and the first side, d2 is the straight-line distance between the first and second projection points, traverse each side of the new polygon, and take the required rectangle with the smallest area of ​​d1*d2 as the minimum circumscribed rectangle, and then obtain the long side and wide side of the minimum circumscribed rectangle; S123c, according to the coordinate values ​​of the first and second projection points in the xy plane of the laser radar coordinate system, obtain the slope of the first side of the minimum circumscribed rectangle, combine the slope with the coordinates of the first vertex in the xy plane to obtain the opposite side of the first side, that is, the second side, project the first and second projection points onto the second side to obtain the third and fourth projection points, and according to the coordinates of the first, second and third and fourth projection points in the xy plane, obtain three length values, after sorting the three length values, use the length values ​​from small to large as the wide side, long side and hypotenuse of the minimum circumscribed matrix respectively, the slope of the long side is the orientation angle yaw1 of the minimum circumscribed rectangle, and at the same time obtain the coordinates of the center point (x center1 ,y center1 ); The method for obtaining the position of the three-dimensional bounding box of the first obstacle includes: According to the minimum bounding rectangle of each cluster point cloud obtained by S123c, the center point coordinates of the first three-dimensional bounding box are (x center1 ,y center1 ,z center1 ), the orientation angle is yaw1, the long side, wide side, and height of the corresponding cluster point cloud on the z-axis of the laser radar coordinate system are set to the length l1, width w1, and height h1 of the three-dimensional bounding box respectively; The method for obtaining the category of the first obstacle three-dimensional bounding box includes: A deep learning algorithm is used to identify obstacles, types and colors of crops in the image. The pixel coordinates are unified into the lidar coordinate system using the conversion relationship between the lidar coordinate system and the camera coordinate system. The recognition result of the camera deep learning is matched with the coordinates of the first obstacle. For the part of the first obstacle that does not match the recognition result, the obstacle is classified according to the length l1, width w1 and height h1, combined with the corresponding threshold of the pre-set three-dimensional bounding box of the object, to determine the category of the three-dimensional bounding box of the first obstacle.

2. The agricultural data collection and decision-making method based on multi-sensors and cloud platform according to claim 1, characterized in that: Prior to S12, this also included: S11, acquiring real-time data in the farmland from various sensors deployed in the farmland, and calibrating the conversion relationship between the laser radar coordinate system and the camera coordinate system; After S12, it also includes: S13, the type, color, area and height of crops obtained by the camera and lidar are classified into crop-related data.

3. The agricultural data collection and decision-making method based on multi-sensor and cloud platform as claimed in claim 1, characterized in that: In S123, the method of finding a polygon surrounding each cluster projection point set includes: Select As the reference point of the cluster projection point set, the tangent value of the line connecting the reference point and the remaining points is calculated in turn. The 360-degree area is divided into several tangent value intervals with the reference point as the origin. The farthest point in each tangent value interval is taken as the edge point, and the edge points in each tangent value interval are connected in turn to form a polygon; where (x min ,y min )、(x max ,y max They are the minimum and maximum coordinate points of the corresponding cluster point cloud in the plane.

4. An agricultural data collection and decision-making device based on multiple sensors and cloud platform, characterized in that: include: The data unit is used to collect real-time data in the farmland through various sensors deployed in the farmland, obtain the location and category of the object in the real-time data, and then output the target-level perception data together with the farmland environment data in the real-time data; the target-level perception data includes crop-related data and obstacle-related data in the farmland; the various sensors include lidar, camera, light sensor, soil temperature and humidity sensor and soil nutrient rapid tester, and the farmland environment data includes water content, soil status and vegetation growth; A crop growth assessment unit, which is used to input crop-related data into a crop growth model, and simultaneously add farmland environment data corresponding to the crop-related data in real time to obtain an assessment result of the current growth status of the crop; A task generation unit, which is used to generate tasks for target crops according to the evaluation results; The task issuing unit is used to send the task, the area and the crop-related data of the target crop to the unmanned agricultural vehicle, plan the driving path according to the collected global map and the position of the vehicle's GPS, and the unmanned agricultural vehicle switches the required agricultural implements to execute the issued task; A task execution unit, which is used to perform local obstacle avoidance for the unmanned agricultural vehicle based on current obstacle-related data and monitor the progress of task execution in real time; The data units specifically include: The obstacle data acquisition subunit is used to unify the pixel coordinates of the image into the lidar coordinate system based on the point cloud collected by the lidar and the image collected by the camera by using the conversion relationship between the lidar coordinate system and the camera coordinate system, so as to obtain preliminary obstacle-related data; the obstacle data acquisition subunit specifically includes: The point cloud screening module is used to remove point cloud areas that are not related to crops. The point cloud is converted into a three-dimensional voxel grid through voxel downsampling operation, and the geometric center of the voxel grid is selected as the representative point. The representative point of each voxel grid is retained, and the point cloud corresponding to the ground is removed; The clustering module is used to extract the point coordinates in the same cluster according to the distance between the point clouds retained by the point cloud screening module to generate several clustered point clouds; The first obstacle three-dimensional encirclement acquisition module is used to obtain the clustered projection point set formed by the clustered point cloud in the xy plane of the laser radar coordinate system, by finding the polygon surrounding each clustered projection point set, searching for each edge of the polygon to find the minimum circumscribed rectangle, and then combining the corresponding clustered projection point set on the z axis of the laser radar coordinate system with the maximum coordinate z max and the minimum value z min Get the height difference z of the corresponding cluster point cloud center1 , forming a first obstacle three-dimensional bounding box, and obtaining the position and obstacle category of the first obstacle three-dimensional bounding box; The second obstacle 3D encirclement acquisition module is used to detect multiple obstacles in the point cloud through a deep learning model and generate a 3D encirclement frame of the second obstacle and its position: the center point coordinate x center2 ,y center2 ,z center2 ), the orientation angle is yaw2, and the length is l2, the width is w2, and the height is h2. Based on the length l2, the width is w2, and the height is h2, the deep learning model further classifies the objects in each bounding box on the basis of obstacle detection to determine the obstacle category of the second obstacle three-dimensional bounding box; The obstacle data fusion module is used to obtain the area and height of the crop by matching the crop type and color in the recognition result with the first obstacle bounding box of the crop, and the crop type, color, area and height are all used as crop-related data; a multi-sensor target-level fusion algorithm is used to obtain obstacle-related data by the position and obstacle category of the first obstacle three-dimensional bounding box and the position and obstacle category of the second obstacle three-dimensional bounding box; The first obstacle three-dimensional encirclement acquisition module specifically includes: A new polygon acquisition submodule is used to determine whether the inner angle formed by two adjacent sides of the selected polygon is greater than 180°. If so, the two sides are removed, otherwise the two sides are retained, and all adjacent two sides are traversed to obtain the remaining sides of the polygon to form a new polygon; The minimum circumscribed matrix acquisition submodule is used to first select the first side of the new polygon, calculate the distance from each vertex of the new polygon to the first side, and record the vertex with the largest distance as the first vertex; then project all points in the new polygon to the first side, take the first and second projection points that are farthest apart on the first side, obtain the required rectangle of the first side, set the area of ​​the required rectangle to d1*d2, d1 is the distance between the first vertex and the first side, d2 is the straight-line distance between the first and second projection points, traverse each side of the new polygon, and take the required rectangle with the smallest area of ​​d1*d2 as the minimum circumscribed rectangle, and then obtain the long side and wide side of the minimum circumscribed rectangle. ; According to the coordinate values ​​of the first and second projection points in the xy plane of the laser radar coordinate system, the slope of the first side of the minimum circumscribed rectangle is obtained. The slope is combined with the coordinates of the first vertex in the xy plane to obtain the opposite side of the first side, that is, the second side. The first and second projection points are projected onto the second side to obtain the third and fourth projection points. According to the coordinates of the first, second and third and fourth projection points in the xy plane, three length values ​​are obtained. After the three length values ​​are sorted, the length values ​​from small to large are used as the wide side, long side and hypotenuse of the minimum circumscribed matrix respectively. The slope of the long side is the orientation angle yaw1 of the minimum circumscribed rectangle. At the same time, the coordinates of the center point of the minimum circumscribed rectangle (x center1 ,y center1 ); The first obstacle three-dimensional bounding box position acquisition submodule is used to obtain the minimum bounding rectangle of each cluster point cloud obtained by the minimum bounding matrix acquisition submodule. The center point coordinates of the first three-dimensional bounding box are (x center1 ,y center1 ,z center1 ), the orientation angle is yaw1, the long side, wide side, and height of the corresponding cluster point cloud on the z-axis of the laser radar coordinate system are set to the length l1, width w1, and height h1 of the three-dimensional bounding box respectively; The category acquisition submodule of the three-dimensional bounding box of the first obstacle is used to use a deep learning algorithm to identify obstacles in the image, as well as the types and colors of crops, and use the conversion relationship between the laser radar coordinate system and the camera coordinate system to unify the pixel coordinates into the laser radar coordinate system, and match the recognition result of the camera deep learning with the coordinates of the first obstacle. For the part of the first obstacle that does not match the recognition result, the obstacle is classified according to the length l1, width w1, and height h1, combined with the corresponding threshold of the pre-set three-dimensional bounding box of the object, to determine the category of the three-dimensional bounding box of the first obstacle.

5. The agricultural data collection and decision-making device based on multiple sensors and cloud platform as claimed in claim 4, characterized in that: The data units specifically include: A calibration subunit, which is used to obtain real-time data collected by various sensors deployed in the farmland, and calibrate the conversion relationship between the laser radar coordinate system and the camera coordinate system; The preliminary crop data acquisition subunit is used to classify the type, color, area and height of crops obtained by cameras and lidar into crop-related data.

6. The agricultural data collection and decision-making device based on multiple sensors and cloud platform as claimed in claim 4, characterized in that: The first obstacle three-dimensional encirclement acquisition module specifically includes: The polygon acquisition submodule is used to select As the reference point of the cluster projection point set, the tangent value of the line connecting the reference point and the remaining points is calculated in turn. The 360-degree area is divided into several tangent value intervals with the reference point as the origin. The farthest point in each tangent value interval is taken as the edge point, and the edge points in each tangent value interval are connected in turn to form a polygon; where x min ,y min )、(x max ,y max ) are the minimum and maximum coordinate points of the corresponding cluster point cloud in the plane.

Images