A pepper harvester motion parameter control method based on point cloud image combination

Abstract

The application discloses a pepper harvester motion parameter control method based on point cloud image combination. The method comprises the following steps: ①spatial calibration of radar and camera, construction of data acquisition platform and preprocessing of image and point cloud data; ②acquiring the real-time pepper quantity in front of the harvester and the average height of the pepper plant canopy to the soil through point cloud; acquiring the cutting width in front of the harvester and the pepper fruit proportion through image; acquiring the comprehensive lowest position of the pepper fruit through point cloud image combination; ③acquiring the real-time motion parameters of the pepper harvester; ④adopting the fuzzy PID control method to control the real-time roller speed, working speed and roller height based on the comprehensive lowest position of the pepper fruit and the predicted value of the next time feeding amount predicted by the improved BP neural network model. The control method can control the motion parameters of the harvester in real time according to the height, density, growth and position distribution of the pepper plants.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of pepper harvesting machines, and particularly relates to a pepper harvesting machine motion parameter control method based on point cloud image combination. BACKGROUND

[0002] At present, the intelligence maturity of the pepper harvesting machine is not high, and the operation state of the machine is determined by manual experience during operation, which is low in operation efficiency and difficult to realize the reasonable matching of external load and power, and is not conducive to the intelligent development of the pepper harvesting machine. The motion parameters of the pepper harvesting machine are closely related to the density of the pepper plants, the operation width, the roller height, the pepper fruit proportion and the comprehensive lowest position of the pepper fruits. Overloading of the key parts of the pepper harvesting machine will cause a series of problems such as roller blockage, increased pepper damage rate, reduced net picking rate and difficult harvesting machine driving.

[0003] In view of the above problems, the existing technology mainly analyzes the force on a key part of the pepper harvesting machine to adjust the motion parameters of the harvesting machine, for example, by analyzing the size of the torque on the roller shaft and the impact force on the spring tooth. However, such adjustment is made under overload conditions, and the adjustment has serious hysteresis, the control is single, the intelligence degree is not high, and the research on intelligent identification and intelligent prediction of the motion parameters of the pepper harvesting machine is insufficient. The present application provides a pepper harvesting machine motion parameter control method based on point cloud image combination, which can obtain the number of pepper plants, the average distance from the pepper plant canopy to the soil, the pepper fruit proportion, the operation width and the comprehensive lowest position of the pepper fruits in the front harvesting area of the pepper harvesting machine through images and point clouds, predict the real-time feeding amount of the roller according to these pepper plant parameters, and obtain the real-time motion parameters of the pepper harvesting machine combined with the Hall sensor, the speed sensor and the roller height sensor. The method can realize real-time control of the operation speed, the roller speed and the roller height of the harvesting machine, avoid the problem of serious hysteresis, significantly improve the net picking rate and reduce the damage rate of the harvested peppers, reduce the operation intensity of the pepper harvesting machine operator, and meet the development trend of intelligent agricultural machinery. SUMMARY

[0004] In order to solve the above technical problems, the application provides a pepper harvester motion parameter control method based on image point cloud combination, which can predict the feeding amount of the harvester at the next moment, obtain the growth of the pepper and the distribution of the pepper through the combination of image and point cloud, and then control the roller speed, operation speed and header height of the pepper harvester in real time. The method can reasonably match the power output of the hydraulic system of the harvester and the external load, improve the real-time performance, improve the net picking rate of the pepper, reduce the damage rate and failure rate of the pepper, realize automatic control, and greatly improve the intelligent level.

[0005] The application is realized through the following technical solutions.

[0006] The application provides a pepper harvester motion parameter control method based on point cloud image combination:

[0007] A pepper harvester motion parameter control method based on point cloud image combination comprises the following steps:

[0008] S1 obtains laser point cloud data and image data from a laser radar and a camera fixedly installed on the top of the driver's cabin of the harvester;

[0009] S2 uses a multi-source sensor time synchronization method based on frequency self-matching to synchronize the camera and the laser radar in time, and then uses a joint calibration method to realize spatial synchronization on the basis of time synchronization, so that the point cloud data and the image data are synchronized in time and space after processing; the point cloud data and the image data collected by the camera are subjected to denoising and enhancement processing and other preprocessing operations, the area of the non-working area is reduced, the noise points are reduced, the running speed of the system is improved, the point cloud data collected by the radar is subjected to coordinate system transformation, so that the ROI area of the processed point cloud data and the image data is the same; finally, the conversion relationship between the image pixel coordinate system and the laser radar coordinate system, the conversion relationship between the laser radar coordinate system and the vehicle coordinate system, and the conversion relationship between the vehicle coordinate system and the geodetic coordinate system are obtained;

[0010] S3 obtains the real-time number of pepper plants in front of the pepper harvester And the average height of the pepper plant canopy to the soil ;

[0011] S4 obtains the real-time cutting width in front of the pepper harvester , and the proportion of pepper fruits ;

[0012] S5 obtains the comprehensive lowest position of the pepper fruits through the point cloud and image combination method;

[0013] S6 measures the real-time speed of the picking roller by using a Hall sensor The sensor probe is symmetrically fixed and installed on the left and right ends of the roller shaft, and the magnetic steel is fixed on the left and right elastic tooth installation plates. The roller height sensor is installed on the left and right sides of the roller support shaft to obtain the real-time height of the roller, and the roller height sensor is an ultrasonic sensor.

[0014] S7 predicts the pepper plant density by the improved BP neural network model , the number of pepper plants , the header height , the average height of the pepper plant canopy to the soil , the proportion of pepper fruits As the input of the improved BP neural network, the total mass in the fixed area As the output of the BP neural network, according to the total mass , the plant density is calculated by the formula, and the predicted value of the feeding amount at the next moment is further calculated.

[0015] Plant density calculation formula:

[0016]

[0017] Wherein: is the total mass in the specified area predicted by the BP neural network, is the actual area processed

[0018] Feeding amount calculation formula:

[0019]

[0020] Wherein: is the predicted value of the feeding amount, is the cutting width, is the working speed of the harvester, is the plant density;

[0021] S8 controls the speed of the roller and the working speed of the pepper harvester according to the predicted value of the feeding amount, and controls the height of the header according to the comprehensive lowest position of the pepper fruits, and the control method adopts fuzzy PID control.

[0022] According to steps S1 and S6, the laser radar, camera and navigation speed measurement system are solid-state laser radar, drive-free USB high-definition camera and BD / GPS dual-system navigation speed measurement module. The navigation speed measurement system is a real-time speed acquisition method based on extended Kalman filter, aiming to improve the measurement accuracy and noise suppression ability of the speed measurement.

[0023] As described in step S2, the spatiotemporal synchronization of point cloud data and image data is achieved by first using a multi-source sensor time synchronization method based on frequency self-matching for temporal synchronization, and then using a joint calibration method to achieve spatial synchronization based on the temporal synchronization. The calibration process requires defining a vehicle coordinate system, ultimately obtaining the laser point cloud coordinates from the image pixel coordinates, and then obtaining the corresponding geodetic coordinates from the laser point cloud coordinates.

[0024] As described in step S3, the real-time number of chili plants in front of the harvester is obtained. and the average height of the chili plant canopy from the soil To obtain the number of chili pepper plants in the area to be harvested, Euclidean distance clustering segmentation algorithm and DBSCAN-based adaptive point cloud clustering algorithm were used respectively, based on the collected point cloud data. and the average height of the chili plant canopy from the soil This was used in the later improved BP neural network model to predict the plant density of the harvester. .

[0025] As described in step S4, obtain the cutting width. The method involves collecting images of chili plants in front of the cutter head, processing the images, marking the cutter head in the images, and then performing row spacing template matching to mark the row of plants closest to the cutter head. Then, using the length of the cutter head as the total width of the cut, the method further determines how many plants are in front of the cutter head and obtains the distance between the two furthest plants at this time, thus obtaining the cut width.

[0026] As described in step S4, the percentage of chili pepper fruits is obtained. The process involves processing images captured by a camera to obtain a binary image of the chili pepper fruit and a binary image of the entire plant. Then, an iterator is used to access the two binary images to obtain the number of pixels. The ratio of these two images represents the proportion of the chili pepper fruit. Image processing includes the following steps:

[0027] S41 performs histogram equalization on the image processed by S1, further increasing the image contrast and making the image clearer.

[0028] S42 performs filtering. Taking all factors into consideration, bilateral filtering is used to remove noise from the image and better preserve the image edges.

[0029] S43 image segmentation uses the Otsu's method (OTSU) to segment the image;

[0030] S44 morphological processing uses erosion and dilation to eliminate small particle noise in the image;

[0031] S45 respectively obtains the binary image of the pepper fruit and the binary image of the whole pepper plant.

[0032] According to step S5, the comprehensive lowest position of the pepper fruit is obtained by scanning and detecting the pixel points of the obtained binary image of the pepper plant to obtain a set of pepper pixel coordinates of the pepper fruit located at the lowest layer of the pepper plant image, then a set of corresponding laser radar coordinate points is obtained through the conversion relationship between the image pixel coordinates and the laser radar coordinates, a set of corresponding geodetic coordinate points is obtained through the conversion relationship between the laser radar coordinates and the geodetic coordinates, and finally the average value of the Z coordinate values of the points in the set of geodetic coordinate points is calculated as the comprehensive lowest position of the pepper fruit.

[0033] According to step S8, the working speed is controlled by setting a reasonable feeding amount range in advance according to the torque of the roller shaft. When the predicted feeding amount exceeds the reasonable feeding amount of the machine, the speed should be appropriately reduced, and when the predicted feeding amount is lower than the reasonable feeding amount of the machine, the speed should be appropriately increased, thereby reducing the machine failure rate while ensuring the working efficiency of the combine harvester. The calculation formula of the target working speed is as follows:

[0034]

[0035] Among them: is the target working speed, is the measured real-time working speed of the harvester, is the rated feeding amount of the harvester, is the predicted feeding amount, is the plant density.

[0036] According to step S8, the roller speed is controlled by setting a reasonable feeding amount range in advance according to the torque of the roller shaft. When the predicted value is within the reasonable range and does not change the speed of the roller, when the predicted feeding amount is lower than the threshold value, the speed of the roller is appropriately increased, and vice versa. The adjustment range of the roller speed is consistent with the change range of the feeding amount.

[0037] According to step S8, the cutter height is controlled by presetting a reasonable cutter height adjustment range according to the average height of the measured pepper plant canopy to the soil, and determining the adjustment size of the cutter according to the comprehensive lowest position of the pepper fruit. BRIEF DESCRIPTION OF DRAWINGS

[0038] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art are briefly introduced below. The drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort and still fall within the scope of the present invention.

[0039] Figure 1 A flowchart illustrating at least one embodiment of the present invention;

[0040] Figure 2 A schematic diagram of the sensor's installation location in this invention;

[0041] Figure 3 A flowchart of parameter acquisition and control for a chili harvester based on point cloud image fusion in this invention;

[0042] Figure 4 Flowchart of the spatiotemporal registration algorithm between the camera and the lidar in this invention;

[0043] Figure 5 A flowchart of the algorithm for obtaining the average height from the canopy of a chili plant to the soil based on point cloud in this invention;

[0044] Figure 6 Flowchart of the algorithm for obtaining the number of chili pepper plants based on point cloud in this invention;

[0045] Figure 7 Flowchart of the algorithm for obtaining the proportion of chili pepper fruits based on images in this invention;

[0046] Figure 8 The flowchart of the algorithm for obtaining the lowest integrated position of a chili pepper fruit based on the image point cloud joint method in this invention;

[0047] Figure 9 Flowchart of the BP neural network processing in this invention;

[0048] In the diagram: 1. Camera; 2. Solid-state LiDAR; 3. Navigation and speed measurement system; 4. Roller height sensor (ultrasonic sensor); 5. Hall sensor. Detailed Implementation

[0049] The technical solution of the present invention is further described below, but the scope of protection is not limited to what is described.

[0050] Example 1

[0051] like Figures 1-9 The method for controlling the motion parameters of a chili harvester based on image point clouds, as shown, includes:

[0052] S1 Obtain laser point cloud data and image data from a laser radar and a camera fixedly installed on the top of the driver's cabin of the harvester;

[0053] S2 Use a multi-source sensor time synchronization method based on frequency self-matching to synchronize the camera and the laser radar in time, and then use a joint calibration method to realize spatial synchronization on the basis of time synchronization, so that the point cloud data and the image data are synchronized in time and space after processing; carry out denoising and enhancement processing on the point cloud data and the image data collected by the camera, and carry out coordinate system transformation on the point cloud data collected by the radar, so that the ROI regions of the processed point cloud data and the image data are the same; finally, the conversion relationship between the image pixel coordinate system and the laser radar coordinate system, the conversion relationship between the laser radar coordinate system and the vehicle coordinate system, and the conversion relationship between the vehicle coordinate system and the geodetic coordinate system are obtained;

[0054] S3 Obtain the real-time number of pepper plants in front of the pepper harvester through the processed laser point cloud data and the average height of the pepper plant canopy to the soil ;

[0055] S4 Obtain the real-time cutting width in front of the pepper harvester through the processed image , the proportion of pepper fruits ;

[0056] S5 Obtain the comprehensive lowest position of the pepper fruits through the joint method of point cloud and image

[0057] S6 Measure the real-time rotating speed of the picking roller by using a Hall sensor , the sensor probes are fixedly installed symmetrically on the left and right ends of the roller shaft, and the magnetic steels are fixed to the left and right elastic tooth mounting plates; a navigation speed measurement system measures the real-time working speed of the harvester , which is installed on the top of the driver's cabin of the pepper harvester; a roller height sensor is installed on the both sides of the roller support shaft to obtain the real-time height of the roller, and the roller height sensor is an ultrasonic sensor;

[0058] S7 Predict the pepper plant density through a BP neural network model , wherein the number of pepper plants , the cutting table height , the average height of the pepper plant canopy to the soil , and the proportion of pepper fruits are the inputs of the improved BP neural network, the total mass in a fixed area is the output of the BP neural network, and the plant density is calculated according to the total mass through a formula, and the predicted value of the feeding amount at the next moment is further calculated.

[0059] Plant density calculation formula:

[0060] Wherein: is the total mass under the specified area of the BP neural network prediction, is the actual area processed

[0061] Feeding amount calculation formula:

[0062]

[0063] Wherein: is the predicted value of the feeding amount, is the cutting width, is the working speed of the harvester, is the plant density;

[0064] S8 controls the speed of the drum and the working speed of the pepper harvester according to the predicted value of the feeding amount, and controls the height of the header according to the comprehensive lowest position of the pepper fruits. Both control methods use fuzzy PID control.

[0065] Example 2

[0066] First, the laser radar and camera are time and space calibrated, the coordinate system is defined and converted, and the image and point cloud data are preprocessed:

[0067] As Figure 4 shown, a multi-source sensor time synchronization method based on frequency self-matching is used to synchronize the camera and laser radar in time, and the specific process is as follows:

[0068] Assume that the interval of each frame of point cloud data of the laser radar is seconds, and the interval of each frame of image of the camera is seconds. Since the data acquisition frequency of the laser radar is less than that of the camera, i.e. is less than , the threshold should be / 2. Both sensors are set to start working at zero time; after the same time, the laser radar obtains m frames of point cloud for each point cloud, and the camera obtains ( ) frames of image for each image. The next frame of image is , wherein all square brackets in this section indicate that they are integers.

[0069] When , m frames of point cloud and frames of image form a set of point pairs. Wherein represents The difference between the time used by the frame image and the corresponding m-frame lidar data. If the difference is less than a set threshold, the image data of the frame can be retained; otherwise, they will be discarded. When [ m T 1 T 2 +1]- m T 1 T 2 < T 2 / 2 The point cloud of the m-frame and The frame image forms a set of point pairs. Among them [ m T 1 T 2 +1]- m T 1 T 2 Indicates The difference between the time used by the frame image and the corresponding m-frame lidar data. If the difference is less than a set threshold, the image data of the frame can be retained; otherwise, they will be discarded. Eliminate the remaining image information that does not satisfy the above two inequalities, finally, the LiDAR and camera data are combined into a new data packet, and the time synchronization of the information collected by the two sensors is realized through the above operation process.

[0070] On the basis of realizing the time synchronization of the camera and the laser radar, the spatial synchronization of the camera and the laser radar is realized by joint calibration. The camera is calibrated by using a checkerboard calibration board to obtain the internal and external parameters of the camera. By setting a circular hole calibration board and taking the center of the circular hole on the calibration board as a feature point, the camera and the laser radar detect the position of the circular hole on the calibration board, respectively, and calculate the rotation matrix R and the translation matrix T between the laser radar coordinate system and the camera coordinate system, thereby completing the joint calibration. The conversion relationship between the laser radar coordinate system and the image pixel coordinate system is as follows:

[0071]

[0072] In the formula: the unknown quantity is the internal parameter, the rotation matrix R and the translation matrix T between the laser radar coordinate system and the camera coordinate system, is the radar coordinate system, and is the image pixel coordinate system, K= is the internal parameter of the camera, is the distance depth information

[0073] The specific calibration process is as follows:

[0074] S1 first calibrates the camera to obtain its internal and external parameters

[0075] S2 the camera collects the circular hole calibration board image, and extracts the center coordinates of the circular hole in the two-dimensional image and the radius (i=1,2,3,4)

[0076] The S3 lidar scans the circular aperture calibration plate to obtain the coordinates of the center of the circle in the point cloud. and radius

[0077] S4 uses the four centers of the circles on the calibration board as feature points, and the coordinates of the center of the circles in the lidar coordinate system are used as the reference points. Transform to the center coordinates in the image coordinate system Establish constraints and calculate the rotation and translation matrix from the lidar coordinate system to the camera coordinate system.

[0078] To achieve the conversion between laser point cloud coordinates and vehicle coordinates, the vehicle coordinate system is defined as follows:

[0079] Vehicle coordinate system This refers to a relative coordinate system fixed on the vehicle body. The coordinates of its origin change with the position of the harvester. The horizontal zero plane passes through the front drive axle of the harvester and is parallel to the ground; the lateral zero plane passes through the midpoint of the front drive axle and is perpendicular to the horizontal plane; and the longitudinal zero plane passes through the center of the entire track length and is perpendicular to the horizontal plane. Therefore... The point where the harvester's horizontal zero plane, lateral zero plane, and longitudinal zero plane intersect. The axis is the intersection of the transverse zero plane and the horizontal zero plane, with the driver's right side being positive; The line of intersection between the vertical zero plane and the horizontal plane is denoted by the axis, with forward being positive; The axis is the intersection of the horizontal zero plane and the vertical zero plane, with upward being positive.

[0080] Since the lidar is mounted on the harvester, the lidar point cloud coordinate system needs to be transformed into the vehicle coordinate system of the chili harvester. This transformation can be achieved using a translation matrix L and a rotation matrix F. The two coordinate systems are translated and rotated until their origins coincide. Let... L=[ L x L y L z ] F is calculated using Euler angles. Definition plane and The lines of intersection of the planes are ; for and The included angle; for and The included angle; for and The angle between the lidar point cloud coordinates and the chili harvester vehicle coordinates is:

[0081] =F + =

[0082] in:

[0083] ;

[0084] ;

[0085] ;

[0086] Since the chili harvester's vehicle coordinate system is fixed to the vehicle body, its position information and desired path relationship during movement are established in a geodetic coordinate system. Therefore, it is necessary to implement the harvester's vehicle coordinate system... Geodetic coordinate transformation The formula is as follows:

[0087]

[0088] In the formula: , , It represents the rotation angles along the x, y, and z axes between two coordinate systems. , , It represents the translation distance of each axis of the two coordinate systems.

[0089] like Figure 2 The solid-state lidar shown is fixedly mounted on the top of the harvester, and the camera is fixed above the lidar using a two-axis gimbal. The centers of the two are roughly on the same vertical line, which can acquire image information and point cloud information in front of the chili harvester.

[0090] Example 3

[0091] The method for obtaining chili plant parameters based on the joint point cloud and image is as follows:

[0092] S1 is a method for obtaining the average distance from the canopy of chili pepper plants to the soil based on point cloud analysis.

[0093] like Figure 5As shown in Example 2, the 3D lidar coordinates of the point cloud obtained can be transformed into geodetic coordinates through a transformation relationship. Adaptive point cloud clustering based on the DBSCAN method is performed in the Z-axis direction of the geodetic coordinate system, that is, the direction perpendicular to the ground. The lidar point clouds from different height ranges are layered, and then the validity of the cluster data that meets the plant height range is judged. The judgment rule is: whether the height distribution variance of the cluster is less than a certain set threshold. If it is, it means that the planarity of this cluster is good and it is a point cloud from pepper plants; otherwise, it is discarded.

[0094] The final height is determined based on the highest average height among the point cloud clusters of effective chili plant regions, which is used as the final chili plant population height. The formula is as follows:

[0095]

[0096] In the formula: This represents the vertical height from the origin of the world coordinate system to the ground. This indicates that the point cloud cluster belongs to the one with the largest average height among all cluster categories; Indicates the number of point clouds in the clustered region; This represents the k-th coordinate in the ground coordinate system; This represents the unit vector along the Z-axis.

[0097] S2 method for obtaining the number of chili plants based on point cloud:

[0098] like Figure 6 As shown in Example 2, the number of chili pepper plants is obtained from the collected point cloud. The main idea is to extract stems of a certain height and use the Euclidean distance clustering method to segment the extracted crop population stems into clusters, obtaining cluster centers. The number of cluster centers is then counted to represent the number of chili pepper plants. The core of the Euclidean distance clustering algorithm is the calculation of the Euclidean distance in the point cloud. A threshold is set to meet the requirements to complete the segmentation of the point cloud data. Let G be the spatial dataset of the point cloud, and let the three-dimensional coordinates of the point cloud data be represented as follows: The Euclidean distance between any two points in the point cloud dataset G can be obtained by the following formula:

[0099]

[0100] The specific process of the Euclidean distance clustering and segmentation algorithm is as follows:

[0101] S21 establishes a KD-tree topology data structure for point cloud data P;

[0102] S22. Create an empty set A as the cluster set, and establish an initial queue B.

[0103] S23 for each point in the point cloud data ), the following operations are performed: adding the initial queue B, for all points ), the following operations are performed: for ) establishing a k-neighborhood with a search radius R, and the searched point set is ) calculating the Euclidean distance between and , and the two points with the smallest distance are classified into one class, and determining whether all points perform the above operations, and adding the points in B to the set A;

[0104] S24 when the above operations are performed on all points , if is part of the set A, the algorithm ends.

[0105] S3 image acquisition algorithm of pepper fruit proportion

[0106] As shown in Figure 7 , based on example 2, the pepper fruit proportion is obtained by image processing on the image collected by the camera, and then the pixel point number of the two binary images is obtained by using the iterator access method, and the ratio of the two is the pepper fruit proportion.

[0107] The image processing process is as follows:

[0108] S31 histogram equalization processing is performed on the preprocessed picture, which further increases the contrast of the picture and makes the image clearer;

[0109] S32 filtering processing is performed, and after comprehensive consideration, bilateral filtering is adopted to remove the noise of the image and better retain the edges of the image;

[0110] S33 image segmentation, the maximum inter-class variance method (OTSU) is adopted to segment the image;

[0111] S34 morphological processing, erosion and expansion are adopted to eliminate small particle noise in the image;

[0112] S35 binary images of pepper fruits and binary images of whole pepper plants are obtained respectively

[0113] S4 comprehensive lowest position of pepper fruits

[0114] As shown in Figure 8 ​​​​​​​As shown, based on example 2, the laser radar and the camera are jointly calibrated and the vehicle coordinate system and the geodetic coordinate system are defined, and the conversion relationship of the laser radar point cloud coordinate system and the image pixel coordinate system is obtained as:

[0115]

[0116] On the basis of obtaining the conversion relationship of the laser radar point cloud coordinate system and the image coordinate system, the specific steps of jointly obtaining the comprehensive lowest position of the pepper by the laser point cloud and the image are as follows:

[0117] S41, based on the binary image obtained in the foregoing, pixel point scanning detection is performed, and the purpose is to obtain the pepper pixel coordinates of the lowest layer pepper fruit in the pepper plant image, and the specific process is as follows:

[0118] S411, on the basis of the binary image of the image of the pepper plant obtained after processing, an edge detection is performed by using a canny operator to obtain an edge image of the pepper fruit.

[0119] S412, the whole image is traversed point by point, and it is judged whether the pixel value of each point is 255. If yes, the pixel coordinates of the point are recorded, otherwise the next point is searched. The recorded points are stored in an array.

[0120] S413, until the whole image is traversed, the pixel coordinates in the array are classified according to whether the horizontal coordinates are the same, then the vertical coordinates of the pixel coordinates with the same horizontal coordinates are compared, and the pixel coordinate point with the minimum vertical coordinate in each category is reserved. The obtained pixel coordinate points are the pixel coordinates of the lowest layer pepper fruit.

[0121] S42, the laser radar coordinates of the lowest layer pepper fruit obtained in the foregoing can be obtained through the conversion relationship of the laser radar and the image pixel coordinates,

[0122] S43, the pixel coordinates obtained are mapped to the corresponding laser radar coordinates, and then converted into the vehicle coordinates through the formula, and finally converted into the geodetic coordinates through the formula. The z direction value of the point set of the coordinate points obtained in the geodetic coordinate system is added and averaged, and the average value is the lowest comprehensive height of the pepper fruit.

[0123] Example 4

[0124] Obtain the real-time motion parameters of the pepper harvesting machine:

[0125] S1, the Hall sensor obtains the rotating speed:

[0126] The Hall sensor mainly comprises sensor probes and magnetic steels, the sensor probes are symmetrically fixed and installed on left and right ends of the roller shaft respectively, the magnetic steels are fixed on the left and right elastic tooth mounting plates respectively, N poles of the magnetic steels are uniformly coated with AB glue and adhered to the elastic tooth mounting plates, four magnetic steels are adhered for one circle for the precision requirement, the Hall sensor probe generates a pulse signal every time passing one magnetic steel, four pulse signals are acquired for one circle of the component, and finally the pulse signals are transmitted to the main controller through the signal output end for counting and display.

[0127] The S2 navigation speed measurement system acquires real-time operation speed, and specifically comprises the following steps:

[0128] S21 resets the system correctly after the system is powered on;

[0129] S22 acquires the initial operation speed of the system

[0130] The navigation speed measurement system can obtain relevant information such as time, latitude and longitude, and speed of the machine operation, and the position at the moment of and is , then the operation speed at the moment t is:

[0131]

[0132] In the formula: is the sampling time (s), is the speed (m / s).

[0133] S23 uses the extended Kalman filter algorithm to improve the speed measurement accuracy and noise suppression capability

[0134] The formula of the discrete nonlinear speed measurement model is as follows:

[0135]

[0136] In the formula: , represent the transfer function of the nonlinear system; Z, z represent system noise and measurement noise, wherein the system noise is generated by the inaccuracy of the harvester system parameters, and the measurement noise is generated by the basic error of the sensor, and O and D are taken as the variance matrix of the system respectively; is the state variable of the system, is the output value, is the control variable, which is 0 here.

[0137] In the middle, the nonlinear function is expanded by one section Taylor, and the formula is as follows:

[0138]

[0139] In the formula: In the formula, represents the state estimate of the harvester at the next moment, sampled by the speed measurement system. Let the longitudinal acceleration and the rate of change of longitudinal acceleration of the harvester be denoted as . and ,but Motion state components of longitudinal acceleration The formula is as follows:

[0140]

[0141] X m =[ V m -1 , a m , i m ]

[0142] If the harvester is at t, t+2 The distance traveled at each moment is respectively , , Then we have the following formula:

[0143]

[0144] From the formula above, we can obtain t+ Speed ​​of time for:

[0145]

[0146] The state equation for the longitudinal velocity can be obtained from the above formula:

[0147]

[0148] Then substitute it into the formula (first order Taylor expansion), have to:

[0149]

[0150] Z represents the system noise of the harvester, and its main influencing factor is the sampling period of the sensor. The formula for its variance matrix D is as follows:

[0151]

[0152] Similarly, C=[1,1,1], the observation noise Z is mainly determined by the velocity measurement module. Finally, the speed of the combine harvester is estimated by the update equation of the extended Kalman filter algorithm, as shown in the following formula:

[0153] & P m | m -1 = A P m | m -1 A T + D & X m | m = X m | m -1 + K m | g ∙( Y m - C ∙ X m | m -1 ) & K m | g = P m | m -1 ∙ C T / ( C ∙ P m | m -1 ∙ C T + O ) & P m | m =[ I - K m | g ∙ C ]∙ P m | m -1

[0154] In the formula: Let m be the predicted value of the error covariance from time m-1 to time m. Given the Kalman gain, the optimal estimate at time m using the Kalman filter model can be obtained according to (the previous formula). This allows us to obtain the real-time speed of the harvester.

[0155] Real-time acquisition of S3 roller height:

[0156] The cutting table height is primarily determined by using two roller height sensors (ultrasonic sensors) on either side of the roller support shaft to obtain the height of the roller support shaft from the ground. Then, the actual height of the bottom of the roller from the ground is calculated using a formula. This height of the roller bottom from the ground is the total height of the roller. The formula is as follows:

[0157]

[0158]

[0159] H in the formula , , and These represent the real-time height of the roller to be measured, the height measured by the roller height sensor, the height difference between the roller support shaft and the bottom of the roller, the ground height measured by the sensor on the left side of the roller support shaft, and the ground height measured by the sensor on the right side of the roller support shaft, respectively.

[0160] Example 5

[0161] like Figure 9 The diagram illustrates the process of using the acquired chili plant parameters to train an improved backpropagation (BP) neural network model, including:

[0162] Traditional backpropagation (BP) neural network models suffer from slow convergence and susceptibility to local minima during training. Therefore, particle swarm optimization (PSO) is proposed to improve the traditional BP neural network model, enhancing its global search capability and thus improving prediction accuracy. The improved BP neural network model based on PSO is described in detail below:

[0163] S1 Particle Swarm Optimization Algorithm:

[0164] Particle swarm optimization (PSO) uses the continuous updating of particle positions and velocities to obtain the optimal fitness value. The algorithm checks if the PSO has reached its maximum value. If so, it outputs the global optimum and its corresponding global optimum fitness value. If both are true, training continues. Finally, the global optimum (i.e., the optimal initial value) and a threshold are assigned to the neural network for training, as shown in the following formula:

[0165]

[0166]

[0167] where w represents the inertial weight; d = 1, 2, …, D; k is the iteration number of the system; represents the velocity of the particle, represents the position of the particle; and is an acceleration factor, usually a non-negative constant; and is a random number between 0 and 1;

[0168] S2 Improved BP neural network model:

[0169] S21 In order to shorten the training time, it is proved by referring to existing theories that when the number of hidden layer nodes is sufficient, even if there is only one hidden layer, it can also achieve arbitrary precision approximation of nonlinear functions, so a three-layer BP neural network model with a hidden layer and fewer nodes is established first, and then the number of nodes is increased until the desired result is met. If it cannot be achieved, the hidden layer is increased to continue training

[0170] S22 After the previous analysis, four input layer nodes (pepper plant quantity, header height, pepper fruit proportion, and average height of pepper plant canopy to soil) and one output node (total mass under fixed area) are selected. The total mass under the fixed area is the feeding density mentioned earlier.

[0171] S23 Number of nodes in hidden layer

[0172] Too few hidden nodes will result in insufficient precision and poor fitting effect; too many nodes will result in too long training time and low efficiency. However, there is no reliable theory to guide the selection of the number of nodes, because we usually use the empirical formula as follows:

[0173]

[0174] where n is the number of nodes in the hidden layer, i is the number of input layer nodes, j is the number of output layer nodes, is a constant, usually taken as 0~10. After multiple verifications, the number of hidden nodes is selected as 8.

[0175] S24 Transfer function

[0176] The hidden layer selects the logsig function, and the output layer selects the purelin function. Their formulas are as follows:

[0177]

[0178]

[0179] S25 improving training and simulation of neural network model

[0180] S251 determination of improved neural network model structure

[0181] A three-layer BP neural network model is adopted, and the number of pepper plants, header height, pepper fruit proportion, and the average distance from the pepper plant canopy to the soil are used as the input layer, and the total mass under the fixed area is used as the output layer The number of nodes in the hidden layer is set to 8

[0182] S252 data collection and preprocessing

[0183] 300 groups of data are collected in the field, of which 70% of the samples are used as the training set, 15% of the samples are used as the validation set, and the rest are used as the test set.

[0184] In order to prevent the phenomenon of saturation, the collected data is normalized by using the mapminmax function

[0185] S253 sample training and evaluation of training model accuracy

[0186] After normalization, the data is input into the model for training, the number of iterations is 1000 times, the learning factor = = 2, and the minimum error of the training target is 0.000001.

[0187] In order to estimate the accuracy of the model, the evaluation indexes mean absolute error (MAE) and root mean square error (RMSE) are used to judge the accuracy of the model. When the mean square error is very small and the generalization ability of the training set is not improved, it means that the accuracy of the model is the best.

[0188] S254 simulation of the model

[0189] MATLAB is used to simulate the model. The closer the model determination coefficient (goodness of fit) value is to 1, the better the fitting effect. By analyzing the fitting effect of the data of the training set, test set and validation set, the model determination coefficient of the entire data set is compared comprehensively. When the determination coefficient of the model is close to 1, it means that the prediction effect of the fitting model of the prediction model is good.

[0190] S255 calculation of plant density

[0191] The predicted total mass m under the fixed area of the model is brought into the pepper plant density calculation formula to obtain the real-time plant density q in front of the pepper harvester. The pepper plant density calculation formula is as follows:

[0192]

[0193] Calculation of the feed amount predicted by the S256 model at the next time step.

[0194] Substitute the predicted plant density q into the basic formula. This will determine the feed amount for the harvester at the next moment.

[0195] Example 6

[0196] like Figure 1 and Figure 3 This illustrates a method for real-time control of the motion parameters of a chili harvester by combining the predicted feed rate with the overall lowest position of the chili fruit. Specifically, this includes:

[0197] S1 uses fuzzy PID control to control the harvesting speed:

[0198] Fuzzy PID control is used to control the operating speed of a chili harvester. This control system primarily involves exploring the fuzzy relationship between the PID parameters and the deviation e and deviation rate ec, and then modifying and tuning these parameters to achieve the best control effect. The specific steps are as follows:

[0199] S11 determines the input and output. The navigation speed measurement system acquires the real-time operating speed of the harvester, and the total mass under a fixed area can be obtained through an improved BP neural network model. The predicted value was then used to determine the plant density. The plant density is obtained from the formula, and the target operating speed is finally obtained from the target speed formula. The operating speed deviation e and its deviation rate ec are calculated. The operating speed deviation e and deviation rate ec are used as input linguistic variables, representing the three parameters of the fuzzy PID controller. , , As an output language variable.

[0200] The formula for calculating plant density is as follows:

[0201]

[0202] Formula for calculating target operating speed:

[0203]

[0204] S12 performs fuzzification processing on the input and output variables. The operation speed is normalized, and the fuzzy domain of deviation e and deviation rate ec is quantized to {-3, -2, -1, 0, 1, 2, 3}. The fuzzy set of the language is set to seven levels, including {NB, NM, NS, ZO, PS, PM, PB}.

[0205] S13 Determine membership function. Since the triangular membership function is simple, fast response and can improve the sensitivity of the system, so choose the triangular membership function as the membership function in this paper.

[0206] S14 Establish fuzzy control rule table. The fuzzy control rules in the speed control system are summarized according to the expert experience and converted into fuzzy language, and the parameter self-tuning rules are as follows: when the system deviation e is large, the value of Kp is usually increased, Kp is zero, and Kp takes a smaller value; when the deviation e and the rate of change of the deviation ec are small, Kp is appropriately increased, Kp is zero, and Kp takes a smaller value; when the deviation e and the rate of change of the deviation ec are large, Kp takes a smaller value, and Kp takes a larger value when ec is small. According to the positive definite rule, off-line editing is carried out in MATLAB software, and stored in the data module of the control unit.

[0207] S15 Inference and defuzzification: this paper uses Zadeh approximate reasoning method to complete the reasoning by fuzzy control rules and obtain fuzzy control quantity, in addition, defuzzification processing is also needed, in order to operate quickly, usually weighted average method is adopted, the formula is as follows:

[0208]

[0209] In the formula: is the accurate value of output; is the value in the domain of fuzzy control quantity, is the membership value of

[0210] S16 Anti-fuzzy. The control quantity is changed from fuzzy quantity to accurate quantity, and the algorithm is as follows:

[0211]

[0212] In the formula, T is the sampling time of the system, , , is the proportional, integral and differential adjustment coefficient of fuzzy PID controller, and its linear combination constitutes the output of the control quantity, so as to realize the control of the working speed of the pepper harvesting machine.

[0213] S2 adopts fuzzy PID control method to realize the control of the drum height of the pepper harvesting machine:

[0214] According to the height of the pepper plant canopy to the soil, a reasonable drum height adjustment range is set, which is ​​​​​​​​Before control, it is necessary to determine whether the lowest position of the chili pepper fruit is within this range. If it is within the range, the original roller height remains unchanged. If it is not within the range, control is performed as follows:

[0215] Fuzzy PID control is used to control the drum height of a chili harvester. This control system primarily involves exploring the fuzzy relationship between the PID parameters and the deviation e and deviation rate ec, and then modifying and tuning it to achieve the best control effect. The specific steps are as follows:

[0216] S21 determines the input and output. The real-time height of the drum is obtained by an ultrasonic sensor located on the end of the drum support shaft. The overall lowest position of the pepper fruit is obtained by combining image and laser point cloud methods. The drum height deviation e and its deviation rate ec are calculated. The drum height deviation e and deviation rate ec are used as input linguistic variables, and the three parameters of the fuzzy PID controller are... , , As an output language variable.

[0217] S22 performs fuzzification processing on the input and output variables. The operation speed is normalized, and the fuzzy domain of the deviation e and deviation rate ec is quantized to {-3, -2, -1, 0, 1, 2, 3}. The fuzzy set of the language is set to seven levels, including {NB, NM, NS, ZO, PS, PM, PB}.

[0218] S23 Determine the membership function. Since the triangular membership function is simple to calculate, has a fast response speed, and can improve the system's sensitivity, it is chosen as the membership function in this paper.

[0219] S24 establishes the fuzzy control rule table. The fuzzy control rules in the speed control system are summarized based on expert experience and converted into fuzzy language. The parameter self-tuning rules are as follows: When the system deviation e is large, it is usually... The value increases, Zero, Take the smaller value; when the deviation e and the rate of change of deviation ec are small, increase them appropriately. , When ec is large, Take the smaller value; when ec is smaller, The larger value is then selected. Based on the positive definiteness rule, the data is edited offline in MATLAB software and stored in the control unit's data module.

[0220] S25 Inference and Defuzzification: This paper uses the Zadeh approximate inference method to complete the inference from the fuzzy control rules and obtain the fuzzy control quantity. In addition, defuzzification processing is required. For the sake of speed of calculation, the weighted average method is usually adopted, and the formula is shown below:

[0221]

[0222] wherein: is the precise value of the output; is the value within the domain of the fuzzy control variable, is the membership value of

[0223] S26 defuzzification. The control variable is changed from a fuzzy variable to a precise variable, and the algorithm is as follows:

[0224]

[0225] wherein T is the sampling time of the system, , , are the proportional, integral, and differential adjustment coefficients of the fuzzy PID controller, and the linear combination thereof constitutes the output of the control variable, thereby achieving control of the roller height of the pepper harvester.

[0226] S3 control of the roller speed of the pepper harvester is achieved using a fuzzy PID control method:

[0227] When the harvester starts to work, a normal roller speed is set, the size of the feeding amount at the next moment is predicted according to the measured plant parameters and in combination with the BP neural network model, and it is determined whether the predicted feeding amount size is within the set reasonable feeding amount range. If yes, the roller speed is not changed, otherwise the change amplitude of the roller speed should be adjusted according to the change amplitude of the predicted feeding amount at the next moment compared to the real-time feeding amount, that is, when , the roller speed should be increased, and the roller speed should be adjusted to ; when , the roller speed should be decreased, and the roller speed should be adjusted to ; here is the predicted current feeding amount size, is the feeding amount size at the next moment predicted by the model. The adjustment of the speed is controlled using a fuzzy PID control method, and the roller speed of the pepper harvester is controlled using fuzzy PID control. In this control system, the fuzzy relationship between the PID parameters and the deviation e and the deviation rate ec is mainly explored and modified to achieve the best control effect. The specific steps are as follows:

[0228] ​S31 determines the input and output. The real-time rotating speed of the cylinder is obtained by a Hall sensor installed on the spring tooth support plate of the spring tooth cylinder, and the target adjusting rotating speed of the cylinder is calculated by the change of the feeding amount. The deviation e and the deviation rate ec of the cylinder rotating speed are taken as the input language variables, and the three parameters of the fuzzy PID controller are taken as the output language variables. 、 、 .

[0229] S32 performs fuzzy processing on the input and output variables. The deviation e and the deviation rate ec are quantized to {-3, -2, -1, 0, 1, 2, 3} by normalizing the working speed, and the fuzzy set of the language is set to seven levels, including {NB, NM, NS, ZO, PS, PM, PB}.

[0230] S33 determines the membership function. Since the triangular membership function is simple to calculate, fast in response, and can improve the sensitivity of the system, it is selected as the membership function in this paper.

[0231] S34 establishes a fuzzy control rule table. The fuzzy control rules in the speed control system are summarized according to expert experience and converted into fuzzy language. The parameter self-tuning rules are as follows: when the deviation e of the system is large, the value of Kp is increased, KI is zero, and KD takes a small value; when the deviation e and the deviation change rate ec are small, Kp is appropriately increased, and KD takes a small value, while ec takes a large value. According to the positive rule, off-line editing is performed in MATLAB software and stored in the data module of the control unit. 、

[0232] S35 reasoning and defuzzification: Zadeh's approximate reasoning method is used to complete reasoning and obtain fuzzy control quantity from fuzzy control rules. In addition, defuzzification processing is also required. In order to operate quickly, weighted average method is usually used, and the formula is as follows:

[0233]

[0234] In the formula: is the accurate value of the output; is the value in the domain of the fuzzy control quantity, is the membership degree value of

[0235] S36 anti-fuzzification. The control quantity is changed from fuzzy quantity to accurate quantity, and the algorithm is as follows: ​​​​​​​

[0236]

[0237] where T is the system sampling time, 、 、 are the proportional, integral, and derivative tuning coefficients of the fuzzy PID controller, whose linear combination constitutes the output of the controller to the control variable, thus achieving the control of the roller speed of the pepper harvester.

Claims

1. A pepper harvester motion parameter control method based on point cloud image association, characterized in that, The method comprises the following steps: S1, acquiring laser point cloud data and image data of pepper plants in a to-be-harvested area from a laser radar and a camera fixedly installed on the top of a driver's cabin of a harvester; S2, using a multi-source sensor time synchronization method based on frequency self-matching to synchronize the camera and the laser radar in time, and then using a joint calibration method to realize spatial synchronization on the basis of time synchronization, so that the point cloud data and the image data are synchronized in time and space after processing; S3, acquiring the number n of pepper plants and the average height L of a pepper plant canopy to soil in a to-be-harvested area in front of the pepper harvester in real time through the processed laser point cloud data; S4, acquiring the cutting width w and the proportion p of pepper fruits in the to-be-harvested area in front of the pepper harvester in real time through the processed image; S5, acquiring the comprehensive lowest position of the pepper fruits in the to-be-harvested area through the joint method of point cloud and image; S6, measuring the real-time rotating speed N of the picking roller by using a Hall sensor, the sensor probe is fixedly installed on the left and right ends of the roller shaft, and the magnetic steel is fixed to the left and right elastic tooth mounting plates; a navigation speed measurement system is used to measure the real-time working speed V0 of the harvester, which is installed on the top of the driver's cabin of the pepper harvester; a roller height sensor is installed on the both sides of the roller support shaft to acquire the real-time height of the roller, and the roller height sensor is an ultrasonic sensor; S7, predicting the pepper plant density q through an improved BP neural network model, wherein the number n of pepper plants, the cutting table height H, the average height L of the pepper plant canopy to soil, and the proportion p of pepper fruits are input into the improved BP neural network, and the total mass m in a fixed area is output from the BP neural network, the plant density q is calculated according to the total mass m, and the predicted value of the feeding amount at the next moment is further calculated, The calculation formula of the feeding amount is: Plant density calculation formula: wherein: m is the total mass under the predicted area of the BP neural network, s ’ is the actual area of the treatment, Q=wvq Wherein: Q is the predicted value of the feeding amount, w is the cutting width, and v is the working speed of the harvester; S8, controlling the rotating speed of the roller and the working speed of the pepper harvester according to the predicted value of the feeding amount, and controlling the height of the cutting table according to the comprehensive lowest position of the pepper fruits, and the control methods are both fuzzy PID control. ​ The comprehensive lowest position of the pepper fruits in the area to be picked is obtained by scanning and detecting pixel points of a binary image of the pepper plants, obtaining a pepper pixel coordinate set of the lowest layer of the pepper fruits in the pepper plant image, obtaining a corresponding laser radar coordinate point set through a conversion relationship between the image pixel coordinates and the laser radar coordinates, obtaining a corresponding geodetic coordinate point set through a conversion relationship between the laser radar coordinates and the geodetic coordinates, and finally taking an average value of Z coordinate values of the geodetic coordinate point set as the comprehensive lowest position of the pepper fruits; The operation speed control is based on the size of the torque of the roller shaft to set a reasonable feeding amount range in advance, and when the feeding amount prediction value exceeds the reasonable feeding amount of the machine, the speed should be appropriately reduced, and when the feeding amount prediction value is lower than the reasonable feeding amount of the machine, the speed should be appropriately increased, so as to reduce the machine failure rate while ensuring the operation efficiency of the combine harvester, and the calculation formula of the target operation speed is as follows: where: V is the target operating speed, V0 is the measured real-time operating speed of the harvester, Q 额 is the rated feed rate of the harvester, and Q is the predicted feed rate.

2. The method according to claim 1, wherein: The laser radar, the camera and the navigation speed measurement system are solid-state laser radars, drive-free USB high-definition cameras and BD / GPS double-system navigation speed measurement systems, wherein the navigation speed measurement system is a real-time speed acquisition method based on an extended Kalman filter, and aims to improve the measurement accuracy and noise suppression capability of the speed measurement.

3. The method according to claim 1, wherein: The method for time synchronization of multiple source sensors based on frequency self-matching is used to realize the time synchronization of the camera and the laser radar, and then the joint calibration method is used to realize the spatial synchronization on the basis of the time synchronization, a vehicle coordinate system is defined in the calibration process, the laser point cloud coordinates are obtained through the image pixel coordinates, and the corresponding geodetic coordinates are obtained through the laser point cloud coordinates.

4. The method according to claim 1, wherein: The number of the pepper plants in the area to be picked and the average height L of the pepper plant canopy to the soil in the area to be picked are obtained by using the Euclidean distance clustering segmentation algorithm and the adaptive point cloud clustering based on the DBSCAN method on the processed point cloud data, and are used for the improved BP neural network model to predict the pepper plant density q of the harvester.

5. The method for controlling motion parameters of a pepper harvester based on point cloud image association according to claim 1, characterized in that: The cutting width w of the pepper harvester is obtained by collecting the pictures of the pepper plants in front of the harvesting device, processing the pictures, marking the harvesting device in the processed pictures, performing row spacing template matching again, marking the nearest row of plants in front of the cutting table, taking the length of the cutting table as the total width of the cutting width, further determining how many plants in front of the cutting table are within the range of the cutting table, obtaining the distance between the farthest two plants, and then obtaining the cutting width w.

6. The method of claim 1, wherein: The pepper fruit proportion p of the pepper plants in the area to be picked is obtained by processing the images collected by the camera to obtain binary images of the pepper fruits and the whole plants, and then using an iterator access method to obtain the pixel point numbers of the two binary images, and the ratio of the two is the pepper fruit proportion p, and the image processing includes the following steps: S41: the pre-processed picture is subjected to histogram equalization again to further increase the contrast of the picture and make the image clearer; S42: filtering processing is performed, and according to comprehensive consideration, a bilateral filter is adopted to remove the noise of the image and better retain the edges of the image; S43: image segmentation, the maximum inter-class variance method (OTSU) is adopted to segment the image; S44: morphological processing, erosion and expansion are adopted to eliminate small particle noise in the image; S45: binary images of the pepper fruits and the whole pepper plants are obtained respectively.

7. The method of claim 1, wherein: The rotation speed of the roller is controlled according to the size of the torque borne by the roller shaft, and a reasonable feeding amount range is set in advance; when the predicted value is within the reasonable range, the speed of the roller is not changed; when the predicted feeding amount is lower than a threshold value, the rotation speed of the roller is appropriately increased; otherwise, the rotation speed of the roller is decreased; the adjustment range of the rotation speed of the roller is consistent with the change range of the predicted feeding amount.

8. The method of claim 1, wherein: The height of the cutting table is controlled according to the average height of the pepper plant canopy to the soil, and a reasonable cutting table height adjustment range is preset; and the adjustment size of the cutting table is determined according to the comprehensive lowest position of the pepper fruits.

Images