A method, device and storage medium for detecting and locating leaked seedlings

Abstract

The present invention discloses a method, device and storage medium for detecting and locating missing seedlings, comprising the steps of: S1: marking samples, S2: training a detection model, S3: position calibration, S4: using process, S5: calculating the missing seedling rate and S6: outputting results; the present invention monitors and detects rice seedlings in real time through image deep learning technology, utilizes image deep learning technology to monitor and detect rice seedlings in real time, calculates the missing seedling ratio and the position of the missing seedlings, so that artificial seedlings can be supplemented, and the missing seedling ratio is calculated, which can help farmers or operators to understand the growth of rice seedlings in time and take corresponding measures as soon as possible to improve the efficiency and yield of rice planting, can accurately determine the position of the missing seedlings and calculate the missing seedling rate, and provide effective decision support for agricultural production, thereby improving the yield and quality of crops.

Description

Technical Field

[0001] The invention relates to the technical field of agriculture, and in particular to a method, a device and a storage medium for detecting and locating leaking seedlings. Background Art

[0002] Rice planting usually requires the following steps: seed selection, seedling raising, transplanting, field management, harvesting and processing. Currently, many places use rice transplanters to automatically transplant rice seedlings, but the machines may miss some rice seedlings. To address this phenomenon, cameras are installed on the rice transplanters for image detection to obtain the missed seedling rate and location, so that manual transplanting can be performed. Summary of the invention

[0003] The purpose of the present invention is to provide a method, device and storage medium for detecting and locating missing rice seedlings, which utilizes image deep learning technology to monitor and detect rice seedlings in real time, calculate the proportion of missing rice seedlings and the location of the missing rice seedlings, and thus enable artificial seedling replenishment.

[0004] The purpose of the present invention can be achieved by the following technical solutions:

[0005] The present application provides a method for detecting and locating leaking seedlings, comprising the following steps:

[0006] S1: Label samples, manually label samples of rice seedlings in the field and obtain the positions of rice seedlings in the image;

[0007] S2: Training the detection model. After collecting a large number of manually annotated samples, a segmentation model is trained using threshold binarization or deep learning segmentation algorithms. The model can accurately segment the location of rice seedlings in the image.

[0008] S3: Position calibration: a camera device and an RTK or GPS device are installed on the transplanter to see the newly inserted seedlings. The relative position relationship between the camera and the RTK device is calibrated in advance to ensure that the camera height is preset and the transformation relationship is calibrated to convert the pixel position of the target in the image into the world coordinate system position.

[0009] S4: Usage process: using the trained segmentation model to detect the pixel position of the rice seedlings through the images collected by the camera, and combining the calibration parameters, the transformation relationship between the camera and RTK, and the position information obtained by RTK to convert the pixel position of the rice seedlings into the position in the world coordinate system;

[0010] S5: Calculate the seedling leakage rate. According to the preset straight line and interval, sample the straight line in the world coordinate system, calculate the distance between the sampling point and the position of the rice seedling, and judge the seedling leakage situation by judging whether there is a sampling point whose distance is less than a certain threshold, and finally calculate the seedling leakage rate;

[0011] S6: Output the results. According to the calculation results of the seedling leakage rate, the area where the seedlings are missing is displayed so as to facilitate the artificial seedling filling operation.

[0012] Preferably, according to the labeled sample in step S1, a segmentation method using binary segmentation or deep learning is used, including:

[0013] Image preprocessing: preprocessing the collected camera images;

[0014] Threshold segmentation: select the threshold value and convert the image into a binary image. By setting the threshold value, the rice seedlings are separated from the background to form an obvious black and white contrast.

[0015] Connected region analysis: Conduct connected region analysis on binary images to identify the connected regions where rice seedlings are located;

[0016] The position information is obtained, and the position of the rice seedling in the image is determined according to the position and size characteristics of the connected area.

[0017] Preferably, according to the detection model described in step S2, threshold binarization or deeplab v3 algorithm is used to prepare data, and a large number of manually annotated samples are collected, including images of rice seedlings and corresponding pixel-level annotations; then model training is performed, and the annotated sample data is used to train the deep learning segmentation model, and the relationship between learning image features and pixel-level annotations is enabled to enable the model to accurately segment the rice seedling area; finally, model inference is performed, and the preprocessed image is input into the trained deep learning segmentation model, and the pixel-level segmentation result of the rice seedling is obtained by forward propagation.

[0018] Preferably, according to the relative position relationship between the participating cameras and the RTK device in the calibration camera described in step S3, in the camera coordinate system, the three-dimensional point P = [X, Y, Z] is projected to the pixel point p = [u, v, 1] on the image plane, where u and v are pixel coordinates, and 1 represents homogeneous coordinates;

[0019] Define the camera intrinsic parameter matrix K as:

[0020] K=[[fx,0,cx],

[0021] [0,fy,cy],

[0022] [0,0,1]]

[0023] Where fx and fy are the focal lengths in the horizontal and vertical directions respectively, and (cx, cy) are the coordinates of the principal point;

[0024] According to the imaging principle of the camera, the following relationship is obtained:

[0025] s*u=fx*X / Z+cx;

[0026] s*v=fy*Y / Z+cy;

[0027] Where s is the pixel scale factor, we get:

[0028] u=(fx*X / Z+cx) / s

[0029] v=(fy*Y / Z+cy) / s

[0030] Substituting the camera intrinsic parameter matrix K into the above equation, we get the final camera model:

[0031] u=(fx*X / Z+cx) / s

[0032] v=(fy*Y / Z+cy) / s.

[0033] Preferably, the transformation relationship is calibrated, points in a known world coordinate system are collected, and pixel positions of these points in the camera image and corresponding position information obtained by the RTK device are simultaneously obtained;

[0034] Using camera calibration technology, we get the camera's intrinsic parameter matrix K, including focal length and principal point coordinates. We use the camera's intrinsic parameter matrix K to convert the pixel position (u, v) on the image into coordinates (Xc, Yc, Zc) in the camera coordinate system. Specifically,

[0035] Xc=(u-cx)*Z / fx

[0036] Yc=(v-cy)*Z / fy

[0037] Zc=Z

[0038] Where (cx, cy) is the coordinate of the principal point, fx and fy are the focal lengths in the horizontal and vertical directions respectively, and Z is the depth or distance of the point in the known world coordinate system;

[0039] Then, based on the known position information of the point in the world coordinate system and the corresponding position information obtained by the RTK device, the rotation matrix R and translation vector T between the camera and the RTK device are calculated by solving the following formula:

[0040] s*[u'v'1]'=K*[RT]*[Xc Yc Zc 1]'

[0041] Where (u',v') is the pixel position of the corresponding position information obtained by the RTK device in the image, [RT] represents the external parameter matrix from the camera to the RTK device, and [Xc Yc Zc 1]' is the coordinate in the camera coordinate system;

[0042] Using the rotation matrix R and translation vector T from the camera to the RTK device and the known coordinates (Xc, Yc, Zc) in the camera coordinate system, the coordinates in the camera coordinate system are converted to the coordinates in the RTK device coordinate system (Xr, Yr, Zr), which is achieved by the following formula:

[0043] [Xr Yr Zr 1]'=[RT]*[Xc Yc Zc 1]'

[0044] Where [Xr Yr Zr 1]' is the coordinate of the target in the RTK device coordinate system.

[0045] Preferably, calculating the seedling leakage rate according to step S5 includes the following steps:

[0046] S51: Use the model to detect the image captured by the camera to obtain the pixel position of the rice seedling in the image;

[0047] S52: Get the position of the rice seedling on the image, and obtain the position Pw of the rice seedling in the world coordinate system according to the previously calibrated relationship between the image pixel and the camera internal parameter, the transformation relationship between the camera and the RTK, and the RTK itself.

[0048] S53: The camera is fixed relative to the rice transplanter, and the rice transplanter basically moves in a straight line in the field, so a straight line is preset on the image, and the user can adjust the position of the straight line at any time;

[0049] S54: converting the straight line in step S53 to the world coordinate system, and sampling the straight line in a manner of sampling points from near to far relative to the rice transplanter, wherein the interval size is related to the rice transplanter's transplanting interval and is known;

[0050] S55: according to the point Q obtained in step S54, traverse each point Qi to calculate the distance between the point Qi and the position Pw of the rice seedling detected in step S52 in the world coordinate system in the world coordinate system;

[0051] S56: If the distance is less than a threshold, it means that there are seedlings at the Qi position. If the distance is greater than a preset threshold, it means that there are seedlings missing at the Qi position. The threshold is preset according to the transplanting interval of the rice transplanter.

[0052] Preferably, according to steps S51-S56, the calculation of the missing seedling rate is to take the points in Q with the rice seedling detection result greater than the threshold as the number of missed seedlings, and divide it by the total number of Q to obtain the missing seedling rate, and the calculation formula is:

[0053] Seedling leakage rate = number of seedlings leaked / |Q|

[0054] Among them, |Q| represents the total number of point set Q.

[0055] A device for detecting and locating leaking seedlings includes at least one processor, at least one memory, and computer program instructions stored in the memory. When the computer program instructions are executed by the processor, the above-mentioned method is implemented.

[0056] A storage medium stores computer program instructions, which implement the above-mentioned method when the computer program instructions are executed by a processor.

[0057] The beneficial effects of the present invention are:

[0058] (1) Using deep learning technology and camera equipment, rice seedlings are detected and located in real time. By annotating samples and training detection models, the positions of rice seedlings in images can be accurately detected, including key points, rectangular frames, or segmentation results. The images captured by the camera are combined with the pre-calibrated camera internal parameters and the transformation relationship between the camera and the RTK equipment to convert the pixel positions of rice seedlings into positions in the world coordinate system.

[0059] (2) According to the preset straight lines and intervals, the straight lines are sampled in the world coordinate system, and the distance between the sampling point and the position of the rice seedlings is calculated. By comparing the distance with the threshold, it is possible to determine whether the seedlings are missed, count the number of missed seedlings, and calculate the missed seedling rate. In this way, the growth of the rice seedlings can be understood in a timely manner, and decision support such as seedling replacement can be provided. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] For better understanding and implementation, the technical solution of the present application is described in detail below with reference to the accompanying drawings.

[0061] Figure 1 A flowchart of the steps of a method for detecting and locating leaking seedlings provided in Example 1 of the present application;

[0062] Figure 2 A flow chart of the steps for calculating the leakage rate of a method for leakage detection and positioning provided in the present application;

[0063] Figure 3 A schematic diagram of a detection algorithm for key points of a method for leaking seedlings detection and location provided in this application;

[0064] Figure 4 A schematic diagram of a rectangular frame target detection algorithm for a method for leaking seedlings detection and positioning provided in this application;

[0065] Figure 5 A schematic diagram of a segmentation method using binary segmentation or deep learning for a method for leaking seedlings detection and positioning provided in the present application;

[0066] Figure 6 A schematic diagram of a preset straight line on an image of a method for leaking seedlings detection and positioning provided in the present application;

[0067] Figure 7 A schematic diagram of a linear conversion to a world coordinate system for a method for leaking seedlings detection and positioning provided in the present application;

[0068] Figure 8 A schematic diagram of linear sampling from near to far intervals for a method for leaking seedlings detection and positioning provided in the present application. DETAILED DESCRIPTION

[0069] In order to further explain the technical means and effects taken by the present invention to achieve the predetermined invention purpose, exemplary embodiments will be described in detail here, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are only examples of methods and systems consistent with some aspects of the present application as detailed in the attached claims.

[0070] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0071] The specific implementation methods, features and effects of the present invention are described in detail below in conjunction with the accompanying drawings and preferred embodiments.

[0072] Example 1

[0073] See also Figure 1-Figure 8 The present embodiment provides a method, device and storage medium for detecting and locating missing rice seedlings, which utilizes image deep learning technology to monitor and detect rice seedlings in real time, calculate the ratio of missing rice seedlings and the location of the missing rice seedlings, and thus enable artificial seedling replacement.

[0074] The present invention provides a method for detecting and locating rice seedling leakage, which uses image deep learning technology to detect rice seedlings and calculate the rice seedling leakage ratio and the position of the rice seedling leakage. A camera is mounted on the rice transplanter to take images of the rice seedlings after transplanting, and RTK is used to obtain the position of the world coordinate system, which includes the following steps:

[0075] S1: Label samples. Using deep learning technology, we need to manually label samples of rice seedlings in the field. We can use key point detection algorithms to label samples such as Figure 3As shown in the figure, the contact points of the rice seedlings on the soil are marked. The rectangular box target detection algorithm can also be used as Figure 4 The rectangular box with rice seedlings inserted in the soil can be segmented using binary segmentation or deep learning methods such as Figure 5 As shown, the purpose is to obtain the position of the rice seedlings in the image;

[0076] S2: Train the detection model, obtain a large number of manually annotated samples, and use algorithms such as openpose for key point detection, yolo series for rectangular box target detection, and threshold binarization or deeplab v3 for segmentation;

[0077] S3: Position calibration: a camera device and an RTK or GPS device are installed on the transplanter to see the newly inserted seedlings. The relative position relationship between the camera and the RTK device is calibrated in advance. Because the camera height is preset in advance, the target pixel position in the image can be converted to the world coordinate system position after the transformation relationship is calibrated.

[0078] S4: Usage process: using the trained segmentation model to detect the pixel position of the rice seedlings through the images collected by the camera, and combining the calibration parameters, the transformation relationship between the camera and RTK, and the position information obtained by RTK to convert the pixel position of the rice seedlings into the position in the world coordinate system;

[0079] S5: Calculate the seedling leakage rate, such as Figure 6 As shown, according to the preset straight line and interval, the straight line is sampled in the world coordinate system, and the distance between the sampling point and the position of the rice seedling is calculated. By judging whether there are sampling points with a distance less than a certain threshold, the leakage of rice seedlings is judged, and finally the leakage rate is calculated;

[0080] S6: Output the results. According to the calculation results of the seedling leakage rate, the area where the seedlings are missing is displayed so as to facilitate the artificial seedling filling operation.

[0081] For example, 100 sample images of rice seedlings were labeled, among which the contact points of the rice seedlings were marked by the key point detection algorithm, and the pixel position of each contact point was recorded. 1,000 sample images of rice seedlings were used for training, and the YOLO algorithm was used for rectangular frame target detection. A camera device was installed on the rice transplanter and connected to the RTK device. The camera intrinsic parameter calibration and the relative position calibration from the camera to the RTK device were performed. The pixel positions of 10 rice seedlings were detected from the images collected by the camera, and these pixel positions were converted to positions in the world coordinate system through the calibration parameters and the position information obtained by RTK. Then, the straight lines were sampled at intervals of 1 meter in the world coordinate system, and the distance between each sampling point and the nearest rice seedling was calculated. It was found that the distance between 2 sampling points and the nearest rice seedling exceeded the preset threshold, so the seedling missing rate was 20%.

[0082] Through real-time monitoring and calculation of the seedling leakage rate, precise management and regulation of the rice planting process can be achieved, and planting efficiency and yield can be improved. At the same time, it can also provide timely decision-making references, such as seedling supplementation or other measures to solve the seedling leakage problem, ensure the healthy growth of rice seedlings, and improve the quality of crops and the sustainability of agricultural production.

[0083] In this embodiment, according to the labeled sample in step S1, a segmentation method using binary segmentation or deep learning is used, including:

[0084] Image preprocessing: preprocessing the acquired camera images, such as resizing, normalization, etc.

[0085] Threshold segmentation: select a suitable threshold value to convert the image into a binary image. By setting the threshold value, the rice seedlings are separated from the background to form an obvious black and white contrast.

[0086] Connected region analysis: Conduct connected region analysis on binary images to identify the connected regions where rice seedlings are located;

[0087] The position information is obtained, and the position of the rice seedlings in the image is determined according to the location and size of the connected area.

[0088] The accurate acquisition of the position of rice seedlings in the image can provide basic data for the subsequent calibration of the transformation relationship between the camera and RTK equipment and the calculation of the missed seedling rate. At the same time, the use of deep learning segmentation methods may be able to more accurately segment rice seedlings and improve the accuracy and robustness of detection.

[0089] In this embodiment, according to the detection model described in step S2, threshold binarization or deeplab v3 algorithm is used to prepare data, and a large number of manually annotated samples are collected, including images of rice seedlings and corresponding pixel-level annotations; then model training is performed, and the annotated sample data is used to train a deep learning segmentation model (such as DeepLab v3), and the relationship between learning image features and pixel-level annotations is enabled to enable the model to accurately segment the area of ​​rice seedlings; finally, model inference is performed, and the preprocessed image is input into the trained deep learning segmentation model, and the pixel-level segmentation result of the rice seedlings is obtained through forward propagation.

[0090] It can achieve pixel-level segmentation of rice seedlings and effectively separate the rice seedling area from the background in the image. Using threshold binarization or DeepLab v3 algorithm can improve the accuracy and robustness of rice seedling segmentation, obtain the label of each pixel, and thus obtain the precise position and contour information of the rice seedlings in the image. Using deep learning segmentation models, such as DeepLab v3, it can use convolutional neural networks (CNN) and other technologies to perform end-to-end processing of images, automatically extract image features and perform pixel-level classification. Compared with traditional threshold binarization methods, deep learning models have stronger expressive and generalization capabilities, can handle various complex scenes and lighting changes, and further improve the accuracy and stability of rice seedling segmentation.

[0091] In this embodiment, according to the relative position relationship between the participating cameras and the RTK device in the calibration camera described in step S3, in the camera coordinate system, the three-dimensional point P = [X, Y, Z] is projected to the pixel point p = [u, v, 1] on the image plane, where u and v are pixel coordinates, and 1 represents a homogeneous coordinate;

[0092] Define the camera intrinsic parameter matrix K as:

[0093] K=[[fx,0,cx],

[0094] [0,fy,cy],

[0095] [0,0,1]]

[0096] Where fx and fy are the focal lengths in the horizontal and vertical directions respectively, and (cx, cy) are the coordinates of the principal point;

[0097] According to the imaging principle of the camera, the following relationship is obtained:

[0098] s*u=fx*X / Z+cx;

[0099] s*v=fy*Y / Z+cy;

[0100] Where s is the pixel scale factor, we get:

[0101] u=(fx*X / Z+cx) / s

[0102] v=(fy*Y / Z+cy) / s

[0103] Substituting the camera intrinsic parameter matrix K into the above equation, we get the final camera model:

[0104] u=(fx*X / Z+cx) / s

[0105] v=(fy*Y / Z+cy) / s.

[0106] According to the relative position relationship between the participating cameras and the RTK device in the calibration camera described in step S3, in the camera coordinate system, set the three-dimensional point P = [X, Y, Z] to be projected to the pixel point p = [u, v, 1] on the image plane, where u and v are pixel coordinates and 1 represents homogeneous coordinates;

[0107] Suppose a camera calibration experiment is conducted, images at different distances are collected, and the grid size on the calibration board is known to be 10cm×10cm. A specific image is selected, in which the pixel coordinates of the center point of one grid are (100,150);

[0108] The camera intrinsic parameter matrix is: K = [[500,0,320], [0,500,240], [0,0,1]]

[0109] According to the camera model formula, the world coordinates of the grid can be calculated as: X = (u-cx)*Z / (fx*s)Y = (v-cy)*Z / (fy*s). Substituting the known data, we can get: X = (100-320)*Z / (500*s)Y = (150-240)*Z / (500*s);

[0110] The selected distance is 1 meter, then Z=1000, s=1, and substituting into the above formula we get: X=(100-320)*1000 / (500*1)=-440 Y=(150-240)*1000 / (500*1)=-180 Therefore, the world coordinates of the grid are P=[-440,-180,1000].

[0111] Through calibration experiments and calculations, we can obtain the camera's intrinsic parameter matrix K, and further obtain the focal lengths fx and fy, as well as the principal point coordinates (cx, cy), thereby completing the solution of the camera's intrinsic parameters.

[0112] The camera's imaging principle and geometric relationship are used to derive the formula for the camera's intrinsic parameters. By collecting images at different angles and distances and calibrating objects of known size, these formulas can be used to solve the camera's intrinsic parameter matrix to obtain parameters such as focal length and principal point coordinates.

[0113] In this embodiment, the transformation relationship is calibrated, points in a known world coordinate system are collected, and the pixel positions of these points in the camera image and the corresponding position information obtained by the RTK device are simultaneously obtained;

[0114] Using camera calibration technology, we get the camera's intrinsic parameter matrix K, including focal length and principal point coordinates. We use the camera's intrinsic parameter matrix K to convert the pixel position (u, v) on the image into coordinates (Xc, Yc, Zc) in the camera coordinate system. Specifically,

[0115] Xc=(u-cx)*Z / fx

[0116] Yc=(v-cy)*Z / fy

[0117] Zc=Z

[0118] Where (cx, cy) is the coordinate of the principal point, fx and fy are the focal lengths in the horizontal and vertical directions respectively, and Z is the depth or distance of the point in the known world coordinate system;

[0119] Then, based on the known position information of the point in the world coordinate system and the corresponding position information obtained by the RTK device, the rotation matrix R and translation vector T between the camera and the RTK device are calculated by solving the following formula:

[0120] s*[u'v'1]'=K*[RT]*[Xc Yc Zc 1]'

[0121] Where (u',v') is the pixel position of the corresponding position information obtained by the RTK device in the image, [RT] represents the external parameter matrix from the camera to the RTK device, and [Xc Yc Zc 1]' is the coordinate in the camera coordinate system;

[0122] Using the rotation matrix R and translation vector T from the camera to the RTK device and the known coordinates (Xc, Yc, Zc) in the camera coordinate system, the coordinates in the camera coordinate system are converted to the coordinates in the RTK device coordinate system (Xr, Yr, Zr), which is achieved by the following formula:

[0123] [Xr Yr Zr 1]'=[RT]*[Xc Yc Zc 1]'

[0124] Where [Xr Yr Zr 1]' is the coordinate of the target in the RTK device coordinate system.

[0125] Suppose four corner points A, B, C and D are selected, their world coordinates are (Xw1, Yw1, Zw1), (Xw2, Yw2, Zw2), (Xw3, Yw3, Zw3), (Xw4, Yw4, Zw4), and the corresponding pixel positions are (u1, v1), (u2, v2), (u3, v3), (u4, v4), and the real positions are (Xr1, Yr1, Zr1), (Xr2, Yr2, Zr2), (Xr3, Yr3, Zr3), (Xr4, Yr4, Zr4).

[0126] Through the world coordinates and pixel positions of these known points, suppose a corner point A is collected, its world coordinates are (1,0,0), the corresponding pixel position is (100,200), the corresponding position information obtained by the RTK device is at the pixel position (120,220) in the image, and the camera's intrinsic parameter matrix K is:

[0127] K=[fx,0,cx;

[0128] 0,fy,cy;

[0129] 0,0,1] According to the formula Xc=(u-cx)*Z / fx, Yc=(v-cy)*Z / fy, Zc=Z, the pixel position (100,200) can be converted to the coordinates in the camera coordinate system. Assuming the depth or distance Z is 10, the coordinates in the camera coordinate system are: Xc=(100-cx)*10 / fxYc=(200-cy)*10 / fyZc=10 According to the formula s*'=K*R*[Xc,Yc,Zc,1]+T, the rotation matrix R and translation vector T can be solved.

[0130] Finally, according to the formula [Xr, Yr, Zr, 1]' = [R, T] * [Xc, Yc, Zc, 1]', the coordinates in the camera coordinate system can be converted to the coordinates in the RTK device coordinate system.

[0131] In this embodiment, the calculation of the seedling leakage rate according to step S5 includes the following steps:

[0132] S51: Use the model to detect the image captured by the camera to obtain the pixel position of the rice seedling in the image;

[0133] S52: Get the position of the rice seedling on the image, and according to the previously calibrated relationship between the image pixel and the camera internal parameter, the transformation relationship between the camera and the RTK, the RTK obtains its own position in the world coordinate system, so as to obtain the position Pw of the rice seedling in the world coordinate system;

[0134] S53: The camera is fixed relative to the rice transplanter, and the rice transplanter basically moves in a straight line in the field, so the image is preset as follows: Figure 6For such a straight line, the user can also adjust the position of the straight line at any time;

[0135] S54: Convert the straight line in step S53 to the world coordinate system. Figure 7 , and collect points from near to far relative to the transplanter. Figure 7 The straight line sampling in Figure 8 The size of the interval is related to the transplanting interval of the rice transplanter and is known in advance;

[0136] S55: According to step S54 Figure 8 point Q in the world coordinate system, traverse each point Qi to calculate the distance between the position Pw of the rice seedling detected in step S52 in the world coordinate system;

[0137] S56: If the distance is less than a threshold, it means that there are seedlings at the Qi position. If the distance is greater than a preset threshold, it means that seedlings are missing at the Qi position. The threshold is preset according to the transplanting interval of the rice transplanter.

[0138] For example, assuming that the pixel position of the rice seedling detected by the camera on the image is (100, 200), based on the pre-calibrated camera intrinsic parameters and the transformation relationship between the camera and RTK, and the position of the RTK itself in the world coordinate system (1, 2, 3), the position Pw of the rice seedling in the world coordinate system can be calculated to be (4, 5, 6), the positions of the preset straight lines on the image are (50, 100) and (200, 100), and based on the method of sampling points from near to far relative to the rice transplanter, the sampling points of the straight line in the world coordinate system are (7, 8, 9) and (10, 11, 12).

[0139] According to the above data, traverse Figure 8 For example, suppose Figure 8 The point Q in is (5,5,5), and the distance between this point and the position Pw of the rice seedling detected in step S52 in the world coordinate system is calculated to be 73. The preset threshold value of the transplanting interval of the rice transplanter is 2. Since 73 is less than the threshold value 2, it means that there are seedlings at the position of point Q and no seedlings are missed.

[0140] In a similar way, all points Q are traversed, the distance between each point and the rice seedling is calculated, and whether the seedling is missed is determined based on a preset threshold. Finally, the proportion of missed seedlings, i.e., the missed seedling rate, is calculated.

[0141] In this embodiment, according to steps S51-S56, the calculation of the missing seedling rate is to take the points in Q with the rice seedling detection result greater than the threshold as the number of missed seedlings, and divide it by the total number of Q to obtain the missing seedling rate, and the calculation formula is:

[0142] Seedling leakage rate = number of seedlings leaked / |Q|

[0143] Among them, |Q| represents the total number of point set Q.

[0144] A device for detecting and locating leaking seedlings includes at least one processor, at least one memory, and computer program instructions stored in the memory. When the computer program instructions are executed by the processor, the above-mentioned method is implemented.

[0145] A storage medium stores computer program instructions, which implement the above-mentioned method when the computer program instructions are executed by a processor.

[0146] Example 2

[0147] This embodiment uses a key point detection algorithm to obtain the position of the rice seedling in the image, including:

[0148] Data preparation: collect a set of image samples containing rice seedlings and annotate the key points of the rice seedlings for each image;

[0149] Model selection and training: Select a suitable deep learning model, such as a convolutional neural network (CNN), and train the model using labeled sample data. During the training process, the model will learn to extract image features and predict the key point locations of rice seedlings;

[0150] Image preprocessing: During the testing phase, the captured camera images are preprocessed, such as resizing, normalizing, or contrast enhancement, to improve the accuracy of the model.

[0151] Model inference: The preprocessed image is input into the trained model, and the prediction results of key points are obtained through forward propagation. These key points represent the position of rice seedlings in the image.

[0152] Example 3

[0153] This embodiment uses a rectangular frame target detection algorithm to obtain the position of the rice seedlings in the image, including:

[0154] Data preparation: Also collect a set of image samples containing rice seedlings, and mark the rectangular box position of the rice seedlings in each image;

[0155] Model selection and training: Select a suitable object detection model, such as Faster R-CNN, YOLO, or SSD, and use the labeled sample data to train the model. During the training process, the model will learn to detect the position of the rectangular box of rice seedlings;

[0156] Image preprocessing, performing preprocessing steps similar to key point detection on the acquired camera images;

[0157] Model inference, the preprocessed image is input into the trained target detection model, and the position of the rectangular box where the rice seedling is located is obtained through forward propagation.

[0158] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment as above, it is not used to limit the present invention. Any technical personnel in this field can make some changes or modify the technical contents disclosed above into equivalent embodiments without departing from the scope of the technical solution of the present invention. However, any brief modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention are still within the scope of the technical solution of the present invention.

Claims

1. A method for detecting and locating leaking seedlings, characterized in that: The steps include: S1: Label samples, manually label samples of rice seedlings in the field and obtain the positions of rice seedlings in the image; S2: Training the detection model. After collecting a large number of manually annotated samples, a segmentation model is trained using threshold binarization or deep learning segmentation algorithms. The model can accurately segment the location of rice seedlings in the image. S3: Position calibration: a camera device and an RTK or GPS device are installed on the transplanter to see the newly inserted seedlings. The relative position relationship between the camera and the RTK device is calibrated in advance to ensure that the camera height is preset and the transformation relationship is calibrated to convert the pixel position of the target in the image into the world coordinate system position. S4: Usage process: using the trained segmentation model to detect the pixel position of the rice seedlings through the images collected by the camera, and combining the calibration parameters, the transformation relationship between the camera and RTK, and the position information obtained by RTK to convert the pixel position of the rice seedlings into the position in the world coordinate system; S5: Calculate the seedling leakage rate. According to the preset straight line and interval, sample the straight line in the world coordinate system, calculate the distance between the sampling point and the position of the rice seedling, and judge the seedling leakage situation by judging whether there is a sampling point whose distance is less than a certain threshold, and finally calculate the seedling leakage rate; S6: Result output, according to the calculation result of the seedling leakage rate, the area where the seedlings are missing is displayed so as to carry out the artificial seedling filling operation; The transformation relationship is calibrated, points in a known world coordinate system are collected, and the pixel positions of these points in the camera image and the corresponding position information obtained by the RTK device are simultaneously obtained; Using camera calibration technology, we get the camera's intrinsic parameter matrix K, including focal length and principal point coordinates. We use the camera's intrinsic parameter matrix K to convert the pixel position (u, v) on the image into coordinates (Xc, Yc, Zc) in the camera coordinate system. Specifically, Xc=(u-cx)*Z / fx Yc=(v-cy)*Z / fy Zc=Z Where (cx, cy) is the coordinate of the principal point, fx and fy are the focal lengths in the horizontal and vertical directions respectively, and Z is the depth or distance of the point in the known world coordinate system; Then, based on the known position information of the point in the world coordinate system and the corresponding position information obtained by the RTK device, the rotation matrix R and translation vector T between the camera and the RTK device are calculated by solving the following formula: s*[u'v'1]'=K*[RT]*[Xc Yc Zc 1]' Where (u',v') is the pixel position of the corresponding position information obtained by the RTK device in the image, [RT] represents the external parameter matrix from the camera to the RTK device, and [Xc Yc Zc 1]' is the coordinate in the camera coordinate system; Using the rotation matrix R and translation vector T from the camera to the RTK device and the known coordinates (Xc, Yc, Zc) in the camera coordinate system, the coordinates in the camera coordinate system are converted to the coordinates in the RTK device coordinate system (Xr, Yr, Zr), which is achieved by the following formula: [Xr Yr Zr 1]'=[RT]*[Xc Yc Zc 1]' Where [Xr Yr Zr 1]' is the coordinate of the target in the RTK device coordinate system.

2. A method for detecting and locating leaking seedlings according to claim 1, characterized in that: According to the labeled sample in step S1, a segmentation method using binary segmentation or deep learning is used, including: Image preprocessing: preprocessing the collected camera images; Threshold segmentation: select the threshold value and convert the image into a binary image. By setting the threshold value, the rice seedlings are separated from the background to form an obvious black and white contrast. Connected region analysis: Conduct connected region analysis on binary images to identify the connected regions where rice seedlings are located; The position information is obtained, and the position of the rice seedling in the image is determined according to the position and size characteristics of the connected area.

3. The method for detecting and locating leaking seedlings according to claim 1, characterized in that: According to the detection model described in step S2, threshold binarization or deeplab v3 algorithm is used to prepare data and collect a large number of manually annotated samples, including images of rice seedlings and corresponding pixel-level annotations; then model training is performed, and the annotated sample data is used to train the deep learning segmentation model, and the relationship between the image features and the pixel-level annotations is learned so that the model can accurately segment the area of ​​the rice seedlings; finally, model inference is performed, and the preprocessed image is input into the trained deep learning segmentation model, and the pixel-level segmentation result of the rice seedlings is obtained through forward propagation.

4. The method for detecting and locating leaking seedlings according to claim 1, characterized in that: According to the relative position relationship between the participating cameras and the RTK device in the calibration camera described in step S3, in the camera coordinate system, set the three-dimensional point P = [X, Y, Z] to be projected to the pixel point p = [u, v, 1] on the image plane, where u and v are pixel coordinates and 1 represents homogeneous coordinates; Define the camera intrinsic parameter matrix K as: K=[[fx,0,cx], [0,fy,cy], [0,0,1]] Where fx and fy are the focal lengths in the horizontal and vertical directions respectively, and (cx, cy) are the coordinates of the principal point; According to the imaging principle of the camera, the following relationship is obtained: s*u=fx*X / Z+cx; s*v=fy*Y / Z+cy; Where s is the pixel scale factor, we get: u=(fx*X / Z+cx) / s v=(fy*Y / Z+cy) / s Substituting the camera intrinsic parameter matrix K into the above equation, we get the final camera model: u=(fx*X / Z+cx) / s v=(fy*Y / Z+cy) / s.

5. The method for detecting and locating leaking seedlings according to claim 1, characterized in that: Calculating the seedling leakage rate according to step S5 includes the following steps: S51: Use the model to detect the image captured by the camera to obtain the pixel position of the rice seedling in the image; S52: Get the position of the rice seedling on the image, and obtain the position Pw of the rice seedling in the world coordinate system according to the previously calibrated relationship between the image pixel and the camera internal parameter, the transformation relationship between the camera and the RTK, and the RTK itself. S53: The camera is fixed relative to the rice transplanter, and the rice transplanter basically moves in a straight line in the field, so a straight line is preset on the image, and the user can adjust the position of the straight line at any time; S54: converting the straight line in step S53 to the world coordinate system, and sampling the straight line in a manner of sampling points from near to far relative to the rice transplanter, wherein the interval size is related to the rice transplanter's transplanting interval and is known; S55: according to the point Q obtained in step S54, traverse each point Qi to calculate the distance between the point Qi and the position Pw of the rice seedling detected in step S52 in the world coordinate system in the world coordinate system; S56: If the distance is less than a threshold, it means that there are seedlings at the Qi position. If the distance is greater than a preset threshold, it means that there are seedlings missing at the Qi position. The threshold is preset according to the transplanting interval of the rice transplanter.

6. A method for detecting and locating leaking seedlings according to claim 5, characterized in that: According to steps S51-S56, the calculation of the missing seedling rate is to take the points in Q with the rice seedling detection result greater than the threshold as the number of missed seedlings, and divide it by the total number of Q to obtain the missing seedling rate. The calculation formula is: Seedling leakage rate = number of seedlings leaked / |Q| Among them, |Q| represents the total number of point set Q.

7. A device for detecting and locating leaking seedlings, characterized in that: The method comprises at least one processor, at least one memory and computer program instructions stored in the memory, and when the computer program instructions are executed by the processor, the method according to any one of claims 1 to 6 is implemented.

8. A storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images