Rice transplanter operation quality detection method and system

Abstract

A method and system for detecting the quality of rice transplanter operations belongs to the field of image recognition technology, and particularly relates to detecting the quality of rice transplanter transplanting. The method addresses the problems of existing rice seedling detection methods, such as low seedling tracking and counting accuracy, low computational efficiency, unsuitability for portable embedded devices, and poor real-time performance. The method comprises the following steps: S6, utilizing a distance mapping algorithm based on trigonometric transformation to convert the pixel distance between two adjacent rice seedlings in a straight line path of each row of rice seedlings to obtain a spatial distance as the spacing between the two adjacent rice seedlings; S7, obtaining the number of missing seedlings between each seedling and its subsequent adjacent seedling based on the spacing between the two adjacent rice seedlings. The method and system for detecting the quality of rice transplanter operations are suitable for real-time monitoring of rice transplanter operation quality.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, and more particularly to the detection of rice transplanting quality using a rice transplanter. Background Technology

[0002] Rice is a staple food crop with a long history of cultivation, wide planting area, and high yield. As a major food crop, rice yield is closely related to cultivation techniques.

[0003] Currently, common rice transplanting methods are divided into manual transplanting and mechanical transplanting. Manual transplanting is inefficient, labor-intensive, and the inconsistent depth and size of planting results in a low seed setting rate, which has significant limitations. Mechanical transplanting, on the other hand, uses standardized seedling cultivation and mechanized planting techniques. Its main components include seedling cultivation suitable for mechanical transplanting, operation of the transplanter, and supporting agronomical measures for field management. By rationally controlling the row and plant spacing, adopting a wide-row, narrow-spacing approach, ventilation and light penetration are improved, further enhancing the photosynthesis of the rice plant population. This greatly improves seedling quality and transplanting efficiency, while saving manpower and resources. A common type of rice transplanter is... Figure 10 As shown.

[0004] According to data, a good rice transplanting standard is a row spacing of 25cm and a plant spacing of 12-15cm, ensuring 20,000-21,000 planting holes per mu (approximately 0.16 acres), with a seedling uniformity of over 85% in each hole, and a rate of seedling drift, missed planting, and damaged seedlings all below 5%. The transplanting effect of the rice transplanter directly affects the quality of seedling growth.

[0005] In recent years, with the rapid improvement of rice transplanter technology, the transplanting quality of rice transplanters has gradually gained recognition from farmers. However, during use, the reliability, transplanting efficiency, and transplanting quality of rice transplanters still have certain problems and are not entirely satisfactory. For example, current rice transplanters generally suffer from low cost-effectiveness and poor adaptability to different agronomic practices. Furthermore, during the transplanting process, issues such as missing seedlings (i.e., insufficient seedlings) and crooked seedlings are also prone to occur.

[0006] To date, no portable and widely applicable solution has been found for the problem of seedling shortage. Currently, the solution mainly relies on manual identification or years of work experience, with limited effectiveness. Meanwhile, while computer vision technologies such as image recognition have made great progress in transportation and security, their application in agriculture remains limited.

[0007] In conclusion, timely and accurate monitoring of various problems encountered during rice transplanter operation is crucial to ensuring transplanting quality. To supplement the existing monitoring functions of rice transplanters, there is an urgent need for a method or system capable of real-time monitoring or detection of rice transplanting operations. Summary of the Invention

[0008] This invention proposes a method and system for detecting the quality of rice transplanter operations, which solves the problems of low seedling tracking accuracy, low counting accuracy, low computational efficiency, unsuitability for portable embedded devices, and poor real-time performance in existing seedling detection methods.

[0009] The technical solution of the rice transplanter operation quality detection method of the present invention is as follows:

[0010] The method includes the following steps:

[0011] S1. Acquire multiple rice transplanter operation videos taken in the direction in front of the rice transplanter; each rice transplanter operation video includes multiple frames of rice transplanter operation footage;

[0012] S2. In each frame of the rice transplanter operation, a rectangular area in front of the rice transplanter is extracted as the frame to be processed.

[0013] S3. Use a target detector to detect each seedling in each frame of the image to be processed, and obtain the pixel coordinates and detection box size of each seedling in each frame of the image to be processed.

[0014] S4. Based on the pixel coordinates of the seedlings in the image to be processed, use the random sampling consistency algorithm to fit the straight path of each row of seedlings that the transplanter can plant at the same time, and assign each seedling to the straight path of each row of seedlings it forms.

[0015] S5. Based on the pixel distance in the x-direction between the intersection of the path lines of two adjacent rows of seedlings and the bottom of the image to be processed, obtain the row spacing of the path lines of two adjacent rows of seedlings.

[0016] S6. Using a distance mapping algorithm based on triangular transformation, the spatial distance obtained by converting the pixel distance between two adjacent seedlings in the straight line of the path of each row of seedlings is used as the plant distance between two adjacent seedlings.

[0017] S7. Based on the plant spacing between two adjacent seedlings, obtain the number of missing seedlings between each seedling and its next adjacent seedling.

[0018] S8. Use the seedling tracking algorithm to track the same seedling in each frame of the image to be processed, and assign a tracking ID to the same seedling.

[0019] The central region counting method was used to count the number of seedlings with different tracking IDs, the plant spacing between each seedling with a tracking ID and its next adjacent seedling, and the number of missing seedlings.

[0020] Based on the above statistical results, store and update the unit information of the tracking target for each frame to be processed;

[0021] By summarizing the unit information of the tracked targets in all rice transplanter operation videos, the total number of rice transplanted and the total number of missing seedlings can be obtained.

[0022] Furthermore, a preferred embodiment is provided, wherein the target detector is an improved model based on YOLOv5-Lite;

[0023] The improved model based on YOLOv5-Lite uses YOLOv5-Lite as its base model and makes two improvements on top of YOLOv5-Lite:

[0024] The first improvement is the addition of the CBAM attention mechanism module to YOLOv5-Lite to enhance the extraction of seedling features and suppress background information in the image to be processed.

[0025] The second improvement is to replace the loss function of YOLOv5-Lite with the EIOU bounding box loss function.

[0026] Furthermore, in a preferred embodiment, step S4 includes:

[0027] S4.1. Perform super-green feature grayscale conversion on the image to be processed to obtain a grayscale image of the seedlings;

[0028] S4.2 Binarize the grayscale image of the seedlings to obtain a black and white image of the seedlings;

[0029] S4.3. Use the pixel coordinates of the white pixels in the black and white image of the rice seedlings as the initial sampling dataset for the RANSAC algorithm; use an iterative method to obtain multiple fitting paths; the iteration process of each iteration is as follows:

[0030] The RANSAC algorithm is used to identify the interior points of the current iteration from the sampled dataset of the previous iteration, and a fitting path is fitted using the interior points of the current iteration.

[0031] Remove the interior points of the current iteration from the sampled dataset of the previous iteration to obtain the sampled dataset of the current iteration;

[0032] Based on the sampled dataset from this iteration, determine whether all paths have been properly fitted:

[0033] Once all paths have been fitted, the iterative method ends.

[0034] Otherwise, continue to the next iteration.

[0035] Furthermore, in a preferred embodiment, step S5 is as follows:

[0036] Obtain the pixel distance in the x-direction between the intersection of the straight lines of the paths of two adjacent rows of rice seedlings and the bottom of the image to be processed;

[0037] Based on the position of the straight lines of the paths of two adjacent rows of rice seedlings in the image to be processed, a correction coefficient is used to correct the pixel distance in the x-direction, and the corrected result is used as the row spacing of the straight lines of the paths of two adjacent rows of rice seedlings.

[0038] d = C × dp

[0039] Where, d r The result is the corrected result; c is the correction factor; d p The pixel distance in the x-direction;

[0040] The correction coefficient ranges from [1.0, 1.1]. The closer the straight lines of the paths of two adjacent rows of seedlings are to the sides of the image to be processed, the larger the correction coefficient becomes.

[0041] Furthermore, in a preferred embodiment, step S6 includes:

[0042] S6.1. Based on the image to be processed, obtain the spatial geometric relationship between the shooting point of the image to be processed and the rice seedlings; in the spatial geometric relationship:

[0043] The seedlings are located in the same horizontal line segment; the shooting point of the image to be processed is above the horizontal line segment, and the vertical distance from the shooting point of the image to be processed to the horizontal line segment is height.

[0044] Let R be the vertical projection point of the shooting point of the image to be processed onto the horizontal line segment. Then R is located at the bottom of the image to be processed.

[0045] In the image to be processed, seedlings are arranged in order from near to far above R, where the i-th seedling is called Seedling. i , where i is an integer greater than or equal to 0;

[0046] Starting from the shooting point of the image to be processed, draw line segments connecting each seedling, which serve as diagonal lines connecting each seedling.

[0047] Draw a perpendicular line from the previous seedling to the line connecting the diagonal lines of the next seedling, where the seedling preceding the 0th seedling is denoted as R;

[0048] The perpendicular line to the line connecting the diagonal lines of the i-th seedling is l. i ;

[0049] The angle between the oblique line connecting the i-th seedling and the horizontal line segment is θ. i ;

[0050] S6.2. Based on the spatial geometric relationship and the trigonometric transformation formula, obtain the pixel distance between the i-th seedling and the seedling in front of it on the horizontal line segment, and use the pixel distance on the horizontal line segment as the plant distance between the i-th seedling and the seedling in front of it.

[0051] The pixel distance between the i-th seedling and the seedling preceding it on the horizontal line segment is obtained using the following trigonometric transformation formula:

[0052]

[0053] Furthermore, in a preferred embodiment, step S7 includes:

[0054] Let l be the distance between any seedling in a straight line of any row of seedlings and its next adjacent seedling; let t be the distance threshold.

[0055] Compare the magnitudes of l and t:

[0056] If l≤t, then the number of missing seedlings between any seedling and its next adjacent seedling is 0;

[0057] Otherwise, let s be the number of missing seedlings between any given seedling and its next adjacent seedling:

[0058]

[0059] Here, represents rounding up.

[0060] Furthermore, a preferred embodiment is provided, wherein the seedling tracking algorithm is an improved algorithm based on the Bytetrack algorithm; the improved algorithm based on the Bytetrack algorithm has made two improvements to the Bytetrack algorithm;

[0061] The first improvement is: in the matching process between the detection box and the predicted box in the Bytetrack algorithm, the similarity between the detection box and the predicted box is calculated by combining pixel information and IoU.

[0062] The images of the predicted bounding box obtained by the Kalman filter algorithm and the detection bounding box of the seedlings in the next frame to be processed are respectively extracted, and the two images are converted to the same size;

[0063] Perform super-green feature grayscale conversion on both images respectively;

[0064] Obtain the pixel matrix from each of the two images, and concatenate the odd-numbered rows of the two pixel matrices into a row vector. Calculate the cosine similarity between the two row vectors.

[0065] The similarity between the detection box and the predicted box is calculated using IoU.

[0066] The final similarity between the detection boxes and predicted boxes is obtained by weighting the similarity calculated using IoU with the cosine similarity of the two row vectors:

[0067] s = ws1 + (1 - w)s0;

[0068] Where s represents the final similarity, s1 represents the cosine similarity between the two row vectors, s0 represents the similarity between the detection box and the prediction box calculated using IoU, and w represents the weighting coefficient.

[0069] The second improvement is: in the Kalman filtering algorithm of the Bytetrack algorithm, the measurement noise covariance matrix is ​​adaptively adjusted.

[0070]

[0071] Among them, R k Here, is the measurement noise covariance matrix in the Kalman filter algorithm of the Bytetrack algorithm; conf is the confidence score of the detected object; k is a constant; This is the adjusted measurement noise covariance matrix.

[0072] Furthermore, a preferred embodiment is provided, wherein the central region counting method is as follows: in each frame of the image to be processed, the central region of the image is selected as the counting region; only within the counting region, the number of seedlings with different tracking IDs and the number of missing seedlings between each tracking ID and its next adjacent seedling are counted.

[0073] Furthermore, a preferred embodiment is provided, wherein the unit information of the tracking target includes: the position of the seedling, the size of the target box, the plant spacing between the seedling of each tracking ID and its next adjacent seedling, and the number of missing seedlings.

[0074] This invention also proposes a quality inspection system for rice transplanters, the technical solution of which is as follows:

[0075] The system includes image acquisition equipment, embedded devices, data transmission equipment, and a cloud server;

[0076] The image acquisition device is used to acquire real-time video of the rice transplanter operating within a fixed range in front of the rice transplanter, and send the rice transplanter operating video to an embedded device.

[0077] The embedded device is used to obtain monitoring information based on the rice transplanter operation video using the above-mentioned rice transplanter operation quality detection method;

[0078] The data transmission device is used to send the positioning information sent by the cloud server to the embedded device, and also to send the monitoring information sent by the embedded device to the cloud server.

[0079] The present invention has the following beneficial effects:

[0080] 1. The rice transplanter operation quality detection method of the present invention has an average seedling detection accuracy of 81.9%, a seedling tracking accuracy of 56.56%, and a counting accuracy of 96.75%, which can meet the statistical work of the number of seedlings sown and the number of missing seedlings during the operation of the rice transplanter.

[0081] 2. The rice transplanter operation quality detection method of the present invention has a detection speed of up to 15 frames / second and high real-time performance. It is applicable to portable embedded devices and is a portable and widely applicable technical solution that effectively solves the problem of difficulty in efficiently and quickly detecting the number of missing seedlings.

[0082] 3. The rice transplanter operation quality detection system of the present invention is used to be installed on a rice transplanter to monitor the sowing status of seedlings in real time. During the operation of the transplanter, it effectively identifies missing seedlings using built-in methods. All the corresponding functions are integrated into a small portable device, which is inexpensive and easy to implement.

[0083] 4. The rice transplanter operation quality detection system described in this invention uses small hardware devices with 4G wireless transmission capabilities, does not rely on network cables or Wi-Fi, and is convenient for field operations.

[0084] 5. The rice transplanter operation quality detection system of the present invention is based on video images collected by the rice transplanter. It utilizes a deep learning-based target detector, multi-target tracking algorithm, and image processing algorithm to integrate the identification of multiple parameters such as transplanting quantity, missing seedling rate, row spacing, and plant spacing, providing an efficient and comprehensive solution for rice transplanting operation monitoring.

[0085] The rice transplanter operation quality detection method and system described in this invention are suitable for real-time monitoring of rice transplanter operation quality. Attached Figure Description

[0086] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0087] Figure 1 This is a flowchart illustrating a method for detecting the operational quality of a rice transplanter, as described in one embodiment of the present invention.

[0088] Figure 2 This is a schematic diagram of the structure of YOLOv5-Lite in one embodiment of the present invention;

[0089] Figure 3 This is a flowchart illustrating the RANSAC algorithm in one embodiment of the present invention.

[0090] Figure 4 This is a flowchart illustrating step s4 of the rice transplanter operation quality detection method in one embodiment of the present invention.

[0091] Figure 5 This is a schematic diagram illustrating the spatial geometric relationship between the shooting point and the rice seedlings in one embodiment of the present invention.

[0092] Figure 6 This is a schematic diagram of the plant spacing and row spacing of seedlings in one embodiment of the present invention;

[0093] Figure 7 This is a flowchart illustrating the Bytetrack algorithm in one embodiment of the present invention.

[0094] Figure 8 This is a schematic diagram of the counting region in the central region counting method according to one embodiment of the present invention;

[0095] Figure 9 A signal connection diagram of a rice transplanter operation quality detection system is shown in one embodiment of the present invention.

[0096] Figure 10 This is a schematic diagram of a commonly used rice transplanter in one embodiment of the present invention;

[0097] Figure 11 This is a schematic diagram of the external structure of the hardware component of the rice transplanter operation quality detection system in one embodiment of the present invention;

[0098] Figure 12 This is a schematic diagram of the internal structure of the hardware component of the rice transplanter operation quality detection system in one embodiment of the present invention;

[0099] Figure 13 This is a physical schematic diagram of the hardware components of a rice transplanter operation quality detection system, as shown in one embodiment of the present invention.

[0100] Figure 14 This is a physical schematic diagram of an embedded development board according to one embodiment of the present invention;

[0101] Figure 15 This is a physical schematic diagram of a 4G DTU module in one embodiment of the present invention;

[0102] Figure 16 This is a schematic diagram of an onboard camera with a USB interface, as shown in one embodiment of the present invention.

[0103] Reference numerals: 1. GPS module; 2. 4G antenna module; 3. Camera; 4. Heat dissipation hole; 5. Wiring hole; 6. Development board interface; 7. 4G DTU module; 8. Development board power supply; 9. Embedded development board; 10. 4G DTU power supply. Detailed Implementation

[0104] In order to make the technical solutions and advantages of the present invention more clearly described, the specific embodiments of the present invention will be further described in detail and completely in conjunction with the accompanying drawings. The various embodiments described below are only part of the preferred embodiments of the present invention, rather than all implementation plans; the various embodiments described below are intended to explain the present invention and cannot be understood as limiting the present invention; the reasonable combination of the technical features defined in the various embodiments of the present invention, as well as all other implementation plans obtained by ordinary technicians in this field based on the embodiments of the present invention without making creative work, all fall within the scope of protection of the present invention.

[0105] Implementation Method 1: Combination Figures 1 to 16 This embodiment describes a method for detecting the operational quality of a rice transplanter. The specific implementation details are as follows:

[0106] A method for inspecting the operational quality of rice transplanters, comprising the following steps:

[0107] S1. Acquire multiple rice transplanter operation videos taken in the direction in front of the rice transplanter; each rice transplanter operation video includes multiple frames of rice transplanter operation footage;

[0108] S2. In each frame of the rice transplanter operation, a rectangular area in front of the rice transplanter is extracted as the frame to be processed.

[0109] S3. Use a target detector to detect each seedling in each frame of the image to be processed, and obtain the pixel coordinates and detection box size of each seedling in each frame of the image to be processed.

[0110] S4. Based on the pixel coordinates of the seedlings in the image to be processed, use the random sampling consistency algorithm to fit the straight path of each row of seedlings that the transplanter can plant at the same time, and assign each seedling to the straight path of each row of seedlings it forms.

[0111] S5. Based on the pixel distance in the x-direction between the intersection of the path lines of two adjacent rows of seedlings and the bottom of the image to be processed, obtain the row spacing of the path lines of two adjacent rows of seedlings.

[0112] S6. Using a distance mapping algorithm based on triangular transformation, the spatial distance obtained by converting the pixel distance between two adjacent seedlings in the straight line of the path of each row of seedlings is used as the plant distance between two adjacent seedlings.

[0113] S7. Based on the plant spacing between two adjacent seedlings, obtain the number of missing seedlings between each seedling and its next adjacent seedling.

[0114] S8. Use the seedling tracking algorithm to track the same seedling in each frame of the image to be processed, and assign a tracking ID to the same seedling.

[0115] The central region counting method was used to count the number of seedlings with different tracking IDs, the plant spacing between each seedling with a tracking ID and its next adjacent seedling, and the number of missing seedlings.

[0116] Based on the above statistical results, store and update the unit information of the tracking target for each frame to be processed;

[0117] By summarizing the unit information of the tracked targets in all rice transplanter operation videos, the total number of rice transplanted and the total number of missing seedlings can be obtained.

[0118] In this embodiment, the rice transplanter operation video is obtained by a camera. Since the camera has a wide field of view, there are a lot of useless parts in the video. By cropping the rectangular area in front of the rice transplanter, the interference of useless parts can be reduced and the processing efficiency can be improved.

[0119] In this embodiment, the x-direction refers to the direction perpendicular to the rice transplanting operation direction (i.e., the direction in which the rice transplanter moves forward). The rice transplanting operation direction (i.e., the direction in which the rice transplanter moves forward) is also called the y-direction.

[0120] Implementation Method Two: Combination Figures 1 to 16 This embodiment further defines the rice transplanter operation quality detection method described in Embodiment 1. The specific implementation details are as follows:

[0121] The target detector is an improved model based on YOLOv5-Lite;

[0122] The improved model based on YOLOv5-Lite uses YOLOv5-Lite as its base model and makes two improvements on top of YOLOv5-Lite:

[0123] The first improvement is the addition of the CBAM attention mechanism module to YOLOv5-Lite to enhance the extraction of seedling features and suppress background information in the image to be processed.

[0124] The second improvement is to replace the loss function of YOLOv5-Lite with the EIOU bounding box loss function.

[0125] In this embodiment, the rice transplanter operation quality detection method can be summarized into three parts: seedling detection algorithm, seedling information extraction algorithm, and seedling tracking and counting algorithm.

[0126] The seedling detection algorithm involves using a target detector to detect each seedling in each frame of the image to be processed.

[0127] Before using seedling information extraction algorithms to monitor seedling information, seedling detection algorithms are needed to identify or detect the (sown) seedlings in the image.

[0128] In this embodiment, traditional image processing methods for identifying or detecting seedlings typically involve manually designing feature extractors and classifiers, which requires a large number of people with specialized knowledge and is inefficient.

[0129] Meanwhile, manually designed feature extractors and classifiers may have unstable performance and poor robustness in complex scenarios.

[0130] Furthermore, traditional image processing methods require significant computational resources and time when processing large-scale image data.

[0131] In this embodiment, compared with traditional image processing methods, the deep learning-based object detection model (or object detector) can automatically learn the features in the image without the need for manual feature extraction, and has better accuracy, generalization ability and adaptability.

[0132] In this embodiment, the improved model based on YOLOv5-Lite is an improvement on the YOLOv5-Lite base model to improve detection performance.

[0133] In this embodiment, YOLOv5-Lite is a lightweight version of YOLOv5.

[0134] In this embodiment, YOLOv5 (You Only Look Once) is a single-stage object detection algorithm (or object detection model, object detector) released by Ultralytics LLC. Compared with YOLOv4, although the average detection accuracy of YOLOv5 is slightly lower, its mean weight file is smaller, the inference speed is faster, and the required training time is shorter.

[0135] In this embodiment, YOLOv5-Lite comprises four parts: input, backbone, neck, and prediction network.

[0136] The input components of YOLOv5-Lite mainly include three parts: Mosaic data augmentation, image size processing, and adaptive anchor box calculation.

[0137] Mosaic data augmentation is used to combine four images to enrich the background of an image.

[0138] Image resizing is used to adaptively add minimal black borders to original images of different lengths and widths, and to uniformly scale the original images to a standard size;

[0139] Adaptive anchor box calculation is used to compare the output predicted box with the ground truth box based on the initial anchor box, calculate the difference, and then update the parameters in reverse, continuously iterating the parameters to obtain the most suitable anchor box value.

[0140] The backbone of YOLOv5-Lite replaces BottleneckCSP and Focus in the original YOLOv5 with ShuffleNetv2, and also removes the 1024conv and 5×5 pooling of the ShuffleNetv2 backbone, which greatly improves the detection speed.

[0141] In the Neck section of YOLOv5-Lite, a structure combining FPN and PAN is adopted. The conventional FPN layer is combined with a bottom-up feature pyramid to fuse the extracted semantic and positional features. At the same time, the backbone layer is fused with the detection layer to enable the model to obtain richer feature information.

[0142] The Prediction part of YOLOv5-Lite consists of three detection layers (Head). Feature maps of different sizes are used to detect target objects of different sizes. Each Head outputs a corresponding vector, which contains the target object's class probability, object score, and the location of the object's bounding box. Finally, the predicted bounding box and class of the target in the original image are generated and labeled.

[0143] In this embodiment, adding the CBAM attention mechanism module to the YOLOv5-Lite feature extraction network can enhance the extraction of rice seedling features and suppress background information, thereby enabling the improved model (or network) to adaptively focus on important feature channels and spatial locations, thus improving the model's detection performance.

[0144] In this embodiment, the attention mechanism has advantages such as plug-and-play functionality, shared weights, and effective information enhancement, making it suitable for application in the field of deep learning-based object detection.

[0145] In this implementation, CBAM (Convolutional Block Attention Module)

[13] It is an attention mechanism module for image classification tasks, which can enhance the image perception and classification performance of convolutional neural networks.

[0146] In this embodiment, the CBAM attention mechanism module comprises two parts: a channel attention module and a spatial attention module. Wherein:

[0147] The channel attention module focuses on identifying meaningful features. Since each channel of a feature represents a specific detector, global average pooling and max pooling are used to aggregate spatial features, each utilizing different information. The formulas are as follows:

[0148] M c (F)=σ(MLP(AvgPool(F))+MLP(MaxPool(F)));

[0149] In the formula, F represents an H×W×C feature of the input, AvgPool represents global average pooling, MaxPool represents global max pooling, MLP represents a specific neural network, and σ represents the sigma activation function.

[0150] As can be seen from the formula, the processing flow of the channel attention module is as follows: First, the input F is subjected to spatial global average pooling and max pooling to obtain two 1×1×C channel descriptions; then, they are fed into a two-layer neural network. The first layer has C / r neurons, where r is the scaling ratio and the activation function is ReLU. The second layer has C neurons. This two-layer neural network is shared; then, the two features are added together and passed through a Sigmoid activation function to obtain the weight coefficient Mc; finally, the weight coefficient is multiplied by the original feature F to obtain the scaled new feature.

[0151] The spatial attention module focuses on where features are meaningful. To summarize spatial features, it employs both global average pooling and max pooling to utilize different types of information, as shown in the following formula:

[0152] M s (F)=σ(f 7*7 ([AvgPool(F),MaxPool(F)]));

[0153] As can be seen from the formula, the processing flow of the spatial attention module is as follows: First, similar to the channel attention module, given a feature F of H×W×C, average pooling and max pooling are performed on one channel dimension to obtain two channel descriptions of H×W×1, and these two descriptions are concatenated together according to the channel; then, it goes through a 7×7 convolutional layer with the sigmoid activation function to obtain the weight coefficients Ms; finally, the weight coefficients are multiplied by the feature F to obtain the scaled new feature.

[0154] Combining channel attention modules and spatial attention modules can further improve the expressive power of features and classification performance.

[0155] In this embodiment, the loss function used in the YOLOv5-Lite model is the CIOU bounding box loss function. This loss function is used to measure the difference between the predicted bounding box and the ground truth box. The regression process is achieved by minimizing the distance between the center points of the predicted box and the ground truth box. At the same time, the CIOU bounding box loss function adds a loss term for aspect ratio to ensure the aspect ratio of the predicted box and the ground truth box as much as possible.

[0156] In this embodiment, to improve the target localization capability of the improved model in complex paddy field environments, the EIOU bounding box loss function is adopted. The EIOU bounding box loss function, based on the CIOU bounding box loss function penalty term (meaning that the principle of EIOU is based on CIOU), decomposes the influence factor of the aspect ratio between the predicted box and the ground truth box into the differences between the width and height of the predicted box and the height and width of the minimum bounding box, respectively. By minimizing these differences, the improved model achieves faster convergence.

[0157] In this embodiment, the EIOU bounding box loss function is:

[0158]

[0159] Where, L IoI L represents the overlap loss between the predicted bounding box and the ground truth bounding box. dis L represents the loss due to the center distance between the predicted bounding box and the ground truth bounding box. asp This represents the width and height loss between the predicted bounding box and the ground truth bounding box;

[0160] C ω C represents the width of the smallest bounding box that covers both the ground truth bounding box and the predicted bounding box. h ρ represents the height of the smallest bounding box that covers both the ground truth bounding box and the predicted bounding box; c represents the diagonal distance of the smallest closed region that can simultaneously contain both the predicted and ground truth bounding boxes; ρ represents the Euclidean distance between the center points of the predicted and ground truth bounding boxes; b represents the center point of the predicted bounding box; b gtThe center point of the ground truth bounding box is represented by ω; IoU represents the ratio of the area of ​​the intersection of the predicted bounding box and the ground truth bounding box to the area of ​​their union; ω represents the width of the predicted bounding box; ω gt h represents the width of the ground truth bounding box; h represents the height of the predicted bounding box; gt This represents the height of the actual bounding box.

[0161] Furthermore, an implementation method is provided, including an ablation test to verify the effectiveness of the improvements to YOLOv5-Lite, specifically verifying the effectiveness of adding the CBAM attention mechanism module and replacing the loss function with the EIOU bounding box loss function on the model's detection performance. The ablation test, based on YOLOv5s as a benchmark and incorporating different improvement strategies, yielded multiple models, and performance comparison tests were performed on these models. The ablation test comparison results are shown in the table below:

[0162] Table 1 Comparison Results of Ablation Tests

[0163]

[0164]

[0165] The performance of the YOLOv5-Lite model, as shown in the comparison results of the ablation experiments presented in the table, is as follows:

[0166] The processing speed is 3.5ms, the model size is 0.821MB, the mAP@0.5 is 0.808, the mAP@0.95 is 0.33, and the number of parameters is 3.01M.

[0167] After adding the CBAM attention mechanism module to YOLOv5-Lite, the performance of the improved model compared to the original model is as follows:

[0168] The model size increased by 3.2%, the number of parameters increased by 2.6%, the recall rate improved by 3.6%, and the mAP@0.5 improved by 1.1%.

[0169] This demonstrates that adding the CBAM attention mechanism module to YOLOv5-Lite can effectively improve the model's feature representation capabilities with minimal increase in model size and parameter count, thereby enhancing detection accuracy.

[0170] Furthermore, the table also shows that after replacing the loss function of YOLOv5-Lite with the EIOU bounding box loss function, the improved model obtained is 1.4% more accurate and 1.2% more mAP@0.5 than the original YOLOv5-Lite. The inference speed is reduced to 3.4ms, and the model size and number of parameters are not significantly changed.

[0171] Finally, the table also shows the improved model obtained by simultaneously adding the CBAM attention mechanism module and replacing the loss function with the EIOU bounding box loss function:

[0172] Compared to models that only add the CBAM attention mechanism module, the model size and number of parameters are not significantly different, but the accuracy is further improved.

[0173] Compared to the original YOLOv5-Lite, it improves accuracy by 1.7% and mAP@0.5 by 1.36%.

[0174] The ablation test showed that the two improvements mentioned above can effectively improve the model's detection performance for rice seedlings, and the improvements are effective.

[0175] Implementation Method 3: Combination Figures 1 to 16 This embodiment further defines the rice transplanter operation quality detection method described in Embodiment 1. The specific implementation details are as follows:

[0176] Step S4 includes:

[0177] S4.1. Perform super-green feature grayscale conversion on the image to be processed to obtain a grayscale image of the seedlings;

[0178] S4.2 Binarize the grayscale image of the seedlings to obtain a black and white image of the seedlings;

[0179] S4.3. Use the pixel coordinates of the white pixels in the black and white image of the rice seedlings as the initial sampling dataset for the RANSAC algorithm; use an iterative method to obtain multiple fitting paths; the iteration process of each iteration is as follows:

[0180] The RANSAC algorithm is used to identify the interior points of the current iteration from the sampled dataset of the previous iteration, and a fitting path is fitted using the interior points of the current iteration.

[0181] Remove the interior points of the current iteration from the sampled dataset of the previous iteration to obtain the sampled dataset of the current iteration;

[0182] Based on the sampled dataset from this iteration, determine whether all paths have been properly fitted:

[0183] Once all paths have been fitted, the iterative method ends.

[0184] Otherwise, continue to the next iteration.

[0185] In this embodiment, the process steps of step S4 are as follows: Figure 4 As shown.

[0186] In this embodiment, it is determined whether all paths have been fitted based on whether the number of fitted paths has reached the preset number of rows for the rice transplanter. For example, if a rice transplanter has 6 rows for planting, then when the number of fitted paths reaches 6, it is determined that all paths have been fitted.

[0187] In this embodiment, the image to be processed is the image to be fitted.

[0188] In this embodiment, after obtaining the image to be fitted, the image to be fitted is first preprocessed, namely, ultra-green grayscale conversion, binarization and other operations.

[0189] In this embodiment, the initial sampling dataset is the sampling dataset of the 0th iteration process.

[0190] In this embodiment, each row of seedlings in the image approximately forms a straight line segment, and the RANSAC algorithm can be used to fit the straight line path of each row of seedlings.

[0191] In this embodiment, the RANSAC (Random Sample Consensus) algorithm is a model fitting algorithm based on random sampling, commonly used to handle datasets with noise and outliers. Its main idea is to fit a model by randomly sampling a portion of the dataset, and then use that model to predict the remaining data in the dataset. By calculating the error between the predicted and actual values, and considering data points with errors less than a certain threshold as inliers, RANSAC can identify inliers in the dataset and fit a more robust model.

[0192] In this embodiment, the steps of the RANSAC algorithm are as follows: Figure 3 As shown:

[0193] A certain number of data points are randomly selected from the point cloud dataset (or sampled dataset) as interior points, and the remaining data points are selected as exterior points;

[0194] The initial model of the model to be fitted is used by randomly selected interior points (i.e., the parameters are fitted by randomly selected points);

[0195] An iterative approach is used to update the interior points and the initial model to obtain the final model of the model to be fitted. Each iteration of this iterative approach is as follows:

[0196] Using the interior points updated in the previous iteration, a model is fitted and used as the model to be verified in this iteration.

[0197] For each outlier, calculate its distance to the model to be validated. If the distance is less than the threshold, update the outlier to an inlier; otherwise, it remains an outlier (i.e., validate the model using the remaining set of points).

[0198] Once all outer points have been updated, obtain the current number of inner points;

[0199] Determine if the current number of interior points has reached the threshold, and determine if the current iteration count has reached the maximum iteration count:

[0200] If the number of inliers reaches the threshold, or the number of iterations reaches the maximum number of iterations, the iteration ends, and the model to be verified in this iteration becomes the final model to be fitted (i.e., the model parameters are obtained).

[0201] Otherwise, continue to the next iteration.

[0202] In this embodiment, the advantage of the RANSAC algorithm is that it can still fit a relatively accurate model even in the presence of noise and outliers.

[0203] In this embodiment, the super-green feature grayscale (or super-green grayscale) is an image grayscale method for green images. It uses specific pixel values ​​to calculate and highlight the green parts of the image while suppressing other parts. The calculation formula is as follows.

[0204] ExG = 2G - RB;

[0205] Where ExG represents the pixel value of a point in the image after the super-green feature is grayscaled, and R, G, and B represent the pixel values ​​of the red, green, and blue channels of that point in the image, respectively.

[0206] In this embodiment, using ultra-green feature grayscale can separate green seedlings from complex non-green backgrounds, avoiding interference from irrelevant features.

[0207] Implementation Method 4: Combination Figures 1 to 16 This embodiment further defines the rice transplanter operation quality detection method described in Embodiment 1. The specific implementation details are as follows:

[0208] Step S5 is as follows:

[0209] Obtain the pixel distance in the x-direction between the intersection of the straight lines of the paths of two adjacent rows of rice seedlings and the bottom of the image to be processed;

[0210] Based on the position of the straight lines of the paths of two adjacent rows of rice seedlings in the image to be processed, a correction coefficient is used to correct the pixel distance in the x-direction, and the corrected result is used as the row spacing of the straight lines of the paths of two adjacent rows of rice seedlings.

[0211] d r =c×d p ;

[0212] Where, dr The result is the corrected result; c is the correction factor; d p The pixel distance in the x-direction;

[0213] The correction coefficient ranges from [1.0, 1.1]. The closer the straight lines of the paths of two adjacent rows of seedlings are to the sides of the image to be processed, the larger the correction coefficient becomes.

[0214] In this embodiment, since the straight lines of the seedling paths obtained by each fitting are not parallel, the intersection of the straight line of the path and the bottom of the screen is uniformly taken as the basis for calculating the row spacing.

[0215] In this embodiment, the camera's shooting angle may not be perpendicular to the ground, but is likely to have a certain tilt angle. This will cause the pixel distance between adjacent straight lines on the left and right sides of the captured image to be smaller than the actual distance.

[0216] In this embodiment, considering that a single rice transplanter can sow a limited number of rows at the same time, and that the actual operation requires a wide row spacing, a correction coefficient is used to obtain more accurate row spacing data in order to simplify the calculation.

[0217] Implementation Method 5: Combination Figures 1 to 16 This embodiment further defines the rice transplanter operation quality detection method described in Embodiment 1. The specific implementation details are as follows:

[0218] Step S6 includes:

[0219] S6.1. Based on the image to be processed, obtain the spatial geometric relationship between the shooting point of the image to be processed and the rice seedlings; in the spatial geometric relationship:

[0220] The seedlings are located in the same horizontal line segment; the shooting point of the image to be processed is above the horizontal line segment, and the vertical distance from the shooting point of the image to be processed to the horizontal line segment is height.

[0221] Let R be the vertical projection point of the shooting point of the image to be processed onto the horizontal line segment. Then R is located at the bottom of the image to be processed.

[0222] In the image to be processed, seedlings are arranged in order from near to far above R, where the i-th seedling is called Seedling. i , where i is an integer greater than or equal to 0;

[0223] Starting from the shooting point of the image to be processed, draw line segments connecting each seedling, which serve as diagonal lines connecting each seedling.

[0224] Draw a perpendicular line from the previous seedling to the line connecting the diagonal lines of the next seedling, where the seedling preceding the 0th seedling is denoted as R;

[0225] The perpendicular line to the line connecting the diagonal lines of the i-th seedling is l. i ;

[0226] The angle between the oblique line connecting the i-th seedling and the horizontal line segment is θ. i ;

[0227] S6.2. Based on the spatial geometric relationship and the trigonometric transformation formula, obtain the pixel distance between the i-th seedling and the seedling in front of it on the horizontal line segment, and use the pixel distance on the horizontal line segment as the plant distance between the i-th seedling and the seedling in front of it.

[0228] The pixel distance between the i-th seedling and the seedling preceding it on the horizontal line segment is obtained using the following trigonometric transformation formula:

[0229]

[0230] In this embodiment, when calculating the plant spacing between seedlings, since the camera's shooting angle may not be perpendicular to the ground and is likely to have a certain tilt angle, the pixel distance between adjacent seedlings that are on the same straight line in the image cannot be simply taken as the plant spacing.

[0231] In this embodiment, the spatial geometric relationship is as follows: Figure 5 As shown, the method for calculating plant spacing using trigonometric transformation is as follows:

[0232] exist Figure 5 In this context, let the camera be the shooting point of the image to be processed. Then, let R be the vertical projection point of the camera on the horizontal line segment (i.e., the ground where the seedlings are located), and let height be the vertical distance from the camera to the ground.

[0233] In the image to be processed, let the seedling closest to the bottom of a row of seedlings be denoted as Seedling. 0 The remaining seedlings above are seedlings in order. 1 Seedling 2 Seedling 3 wait;

[0234] Since the rice transplanter always sows in a straight line, the pixel distance between two seedlings in the same row is simplified to the y-coordinate difference between the two seedlings (i.e., the coordinate in the sowing direction, or the horizontal line segment).

[0235] Starting from the camera, draw a line segment connecting each seedling, and then draw a perpendicular line from the previous seedling to the line connecting the next seedling to the camera (Seedling). 0 The preceding seedling is R), namely l0, l1, l2, l3 in the figure;

[0236] The angles between each connecting line and the horizontal line are θ0, θ1, θ2, and θ3, respectively.

[0237] The pixel distance between the bottom seedling in the row and the bottom of the screen (i.e., R) is d0, and the pixel distances between the other two adjacent seedlings are d1, d2, d3, etc.; the actual distance between the bottom seedling in the screen and the bottom of the screen is l0, and the actual distances between the other two adjacent seedlings are l1, l2, l3, etc.

[0238] From the content of trigonometric transformations, we can obtain the following relationship:

[0239]

[0240] From the above relationship, θ1 and l1 can be solved. Then, θ1 is assigned to θ0, and d2 is copied to d1. By repeating the above process, the actual distance between each adjacent seedling can be solved.

[0241] By using the algorithm described above, the plant distance between each identified seedling and its adjacent seedling below can be calculated.

[0242] Implementation Method Six: Combination Figures 1 to 16 This embodiment further defines the rice transplanter operation quality detection method described in Embodiment 1. The specific implementation details are as follows:

[0243] Step S7 includes:

[0244] Let l be the distance between any seedling in a straight line of any row of seedlings and its next adjacent seedling; let t be the distance threshold.

[0245] Compare the magnitudes of l and t:

[0246] If l≤t, then the number of missing seedlings between any seedling and its next adjacent seedling is 0;

[0247] Otherwise, let s be the number of missing seedlings between any given seedling and its next adjacent seedling:

[0248]

[0249] Here, represents rounding up.

[0250] In this embodiment, the value of the distance threshold t is selected according to the actual situation.

[0251] In this embodiment, the number of missing seedlings can be calculated by observing the operation of the rice transplanter. It can be seen from the video feed that the distance between two adjacent seedlings in a missing row is significantly greater than that in a row without missing seedlings. Based on the spacing between two seedlings, the number of missing seedlings between them can be effectively calculated.

[0252] Implementation Method Seven: Combination Figures 1 to 16 This embodiment further defines the rice transplanter operation quality detection method described in Embodiment 1. The specific implementation details are as follows:

[0253] The seedling tracking algorithm is an improved algorithm based on the Bytetrack algorithm; the improved algorithm based on the Bytetrack algorithm has two improvements on the Bytetrack algorithm;

[0254] The first improvement is: in the matching process between the detection box and the predicted box in the Bytetrack algorithm, the similarity between the detection box and the predicted box is calculated by combining pixel information and IoU.

[0255] The images of the predicted bounding box obtained by the Kalman filter algorithm and the detection bounding box of the seedlings in the next frame to be processed are respectively extracted, and the two images are converted to the same size;

[0256] Perform super-green feature grayscale conversion on both images respectively;

[0257] Obtain the pixel matrix from each of the two images, and concatenate the odd-numbered rows of the two pixel matrices into a row vector. Calculate the cosine similarity between the two row vectors.

[0258] The similarity between the detection box and the predicted box is calculated using IoU.

[0259] The final similarity between the detection boxes and predicted boxes is obtained by weighting the similarity calculated using IoU with the cosine similarity of the two row vectors:

[0260] s = ws1 + (1 - w)s0;

[0261] Where s represents the final similarity, s1 represents the cosine similarity between the two row vectors, s0 represents the similarity between the detection box and the prediction box calculated using IoU, and w represents the weighting coefficient.

[0262] The second improvement is: in the Kalman filtering algorithm of the Bytetrack algorithm, the measurement noise covariance matrix is ​​adaptively adjusted.

[0263]

[0264] Among them, R k Here, is the measurement noise covariance matrix in the Kalman filter algorithm of the Bytetrack algorithm; conf is the confidence score of the detected object; k is a constant; This is the adjusted measurement noise covariance matrix.

[0265] In this embodiment, the improved algorithm based on Bytetrack is obtained by improving upon the Bytetrack algorithm.

[0266] In this embodiment, the Bytetrack algorithm is a lightweight multi-target tracking algorithm based on the Tracking-By-Detection paradigm. By introducing a (target) detector, the Bytetrack algorithm can re-detect and update the target during the target tracking process, thereby tracking the target more accurately.

[0267] In this embodiment, the Bytetrack algorithm flow is as follows: Figure 7 As shown, the main steps can be summarized as follows:

[0268] Step 1: The detection boxes detected by the target detector are used as part of the input to the Bytetrack algorithm to initialize or update the (tracking) trajectory of the target box; the trajectory usually includes the initial position, velocity and other possible attributes (such as size, shape, etc.) of the target (i.e., seedlings);

[0269] Step two: Use the Kalman filter algorithm to estimate the motion of the target in the image and obtain the size and position of the prediction box; in this process, measuring the noise covariance matrix plays a crucial role; the prediction box contains the position of the existing trajectory in the next frame;

[0270] Step 3: Match the detection boxes and prediction boxes: Use IoU (overlap) to calculate the similarity between each prediction box and each detection box, and then use the Hungarian matching algorithm to match the updated detection boxes with the prediction boxes of the target in the previous frame to determine whether the target in the current frame is the same target as the target in the previous frame.

[0271] Step four: By continuously repeating the above process, real-time multi-target tracking of video can be achieved.

[0272] The specific steps for matching detection boxes and prediction boxes are as follows:

[0273] Detection box classification: All detection boxes in the current frame are classified into high-confidence and low-confidence detection boxes based on their confidence level (i.e., score). The purpose of this step is to distinguish which detection boxes are more reliable and which may contain noise or uncertainty. Detection boxes of different confidence levels are matched separately. During matching, the similarity between the detection box and the trajectory prediction position is usually calculated, and the IOU (Intersection over Union) can be used to calculate the similarity.

[0274] Trajectory matching of high-confidence detection boxes: For high-confidence detection boxes, attempt to match them with previous trajectories (i.e., initial matching);

[0275] Processing of low-confidence detection boxes: A secondary matching process will be performed, attempting to match them with the trajectories that failed to match in the initial matching.

[0276] Track update and creation: Based on the matching results, update the information of the matched tracks; for high-confidence detection boxes that do not match the tracks, they will be used as new tracks to start tracking; for tracks that do not match low-score detection boxes (i.e., low-confidence detection boxes), they will be temporarily retained for 30 frames and treated as missing tracks. If no match is found after 30 frames, the tracks will be deleted.

[0277] It should be noted that the measurement noise covariance matrix (usually denoted as R) describes the statistical characteristics of noise in the observed data (i.e., the detection box). In practical applications, due to various factors (such as sensor error, illumination changes, occlusion, etc.), the position and size of the detection box may have certain errors. These errors can be regarded as noise, and they have a significant impact on the accuracy of target tracking.

[0278] The measurement noise covariance matrix R is used to quantify the magnitude and correlation of these measurement noises. Its diagonal elements represent the noise variance of each measurement dimension, i.e., the magnitude of the noise in each dimension; the off-diagonal elements represent the noise correlation between different measurement dimensions. By appropriately setting R, the Kalman filter algorithm can better handle these noises, thereby improving the accuracy and robustness of target tracking.

[0279] Specifically, in the update step of the Kalman filter, the measurement noise covariance matrix R is used to calculate the Kalman gain K. The Kalman gain is a weighted average of the predicted and observed values, determined based on prediction uncertainty and measurement noise. By adjusting the value of R, the calculation method of the Kalman gain can be altered, thus affecting the degree to which the filtering algorithm trusts the observed data.

[0280] If R is set too large, it means that there is a lot of noise in the observed data, and the filtering algorithm will rely more on the predicted values, reducing its confidence in the observed values. Conversely, if R is set too small, the filtering algorithm will trust the observed values ​​more, which may lead to overfitting to the noise. Therefore, choosing an appropriate R value is crucial to the performance of Kalman filtering in the ByteTrack algorithm.

[0281] It should be noted that in the ByteTrack algorithm, the target box, detection box, and prediction box each have specific meanings:

[0282] Bounding box: In an image or video, this typically refers to the bounding box of the object (such as a seedling) that you want to track. The bounding box defines the object's position and size. In object tracking tasks, algorithms need to continuously update and predict the positions of these bounding boxes in order to track the object correctly.

[0283] Detection boxes are generated by object detection algorithms (such as object detectors) and are used to represent the location and size of objects detected in an image. Detection boxes are typically computed using deep neural networks or other machine learning models, and they may contain scores with varying confidence levels, indicating the probability of the detected object's presence.

[0284] Predicted bounding boxes: In the ByteTrack algorithm, predicted bounding boxes typically refer to the bounding boxes that predict the possible locations of the target in the current frame based on previous trajectory information, using Kalman filtering or other prediction algorithms. Predicted bounding boxes provide prior knowledge during the target bounding box update process, helping the algorithm to track the target more accurately.

[0285] The relationship between target boxes and detection boxes: Target boxes and detection boxes are not conceptually identical. A target box is a broader concept, referring to the boundary of the object of interest in a tracking task. A detection box, on the other hand, is more specific; it refers to the actual boundary of an object found in an image by a detection algorithm. In the ByteTrack algorithm, the detection box is part of the algorithm's input, used to initialize or update the target box's trajectory. The algorithm updates the target box's position based on the detection boxes and generates new trajectories or deletes non-existent ones as needed.

[0286] In this embodiment, the advantage of the Bytetrack algorithm is that it retains both high-scoring and low-scoring detection boxes. By performing secondary matching on the low-scoring boxes, it effectively reduces the ID switching problem caused by screen occlusion or jitter.

[0287] In this implementation, the Bytetrack algorithm still has some problems:

[0288] First, the algorithm does not utilize the appearance features of the target during the matching process, so the tracker's performance largely depends on the performance of the (target) detector. If the detector's performance is poor, the tracker's performance will also be greatly affected.

[0289] Secondly, during the matching process, the algorithm only uses Kalman filtering to predict the position of the target trajectory in the current frame in the next frame, and then calculates the similarity between the two matches by the IoU between the predicted box and the actual detection box, and completes the matching using the Hungarian matching algorithm. However, since the distance between different seedlings is relatively close, and as the rice transplanter moves, the same seedling in the picture will gradually become smaller, and its detection box will also gradually become smaller. After experimentation, it was found that when the ByteTrack algorithm is used to track seedlings planted by the rice transplanter, frequent ID switching occurs, which leads to the number of tracked IDs far exceeding the actual number of seedlings, and cannot meet the requirement of accurate seedling counting.

[0290] Finally, when using Kalman filtering to predict the movement of seedlings, the algorithm sets the same measurement noise covariance matrix for objects with different confidence scores, ignoring information about the noise scale.

[0291] It should be noted that, according to existing research, there are currently many multi-object tracking algorithms that utilize appearance learning, which can overcome the shortcoming of the Bytetrack algorithm that does not utilize the appearance features of the target. However, most of these algorithms are based on deep learning to extract the appearance features of the target, such as DeepSORT, which places high demands on the computational performance of the equipment. The rice transplanter operation quality detection method described above needs to be implemented using embedded devices with limited performance. If existing multi-object tracking algorithms that utilize appearance learning are applied to embedded devices with limited performance, it will lead to problems such as long computation time and inability to meet real-time requirements.

[0292] Therefore, existing multi-target tracking algorithms that utilize appearance learning cannot be used for rice transplanter operation quality inspection methods.

[0293] In this embodiment, by improving the first point, namely by using a combination of pixel information and IoU to calculate the similarity between the detection box and the prediction box, appearance features are introduced into the ByteTrack algorithm. This reduces the error caused by using only IoU to calculate similarity and overcomes the shortcoming of ByteTrack not utilizing appearance features.

[0294] It should be noted that although the appearance of each seedling is similar, the specific posture of different seedlings is still quite different, and the posture similarity of the same seedling in adjacent frames is very high. Therefore, using the row vectors of the pixel matrix to calculate the similarity can accurately track each seedling. At the same time, taking an odd number of rows can reduce the amount of computation and speed up the method's operation.

[0295] In this embodiment, the value of the weighting coefficient w is determined experimentally.

[0296] In this embodiment, by adaptively adjusting the measurement noise covariance matrix, the problem of setting the same measurement noise covariance matrix for objects with different confidence scores is solved. This allows objects with higher confidence scores to have lower measurement noise covariance, and objects with lower confidence scores to have higher measurement noise covariance, which helps to improve the accuracy of Kalman filter state updates.

[0297] In this embodiment, the constant k is determined experimentally.

[0298] Furthermore, an implementation method is provided, specifically a comparative experiment, to verify the effectiveness of the improved algorithm based on the Bytetrack algorithm.

[0299] The comparative experiment was set up as follows:

[0300] (1) An improved model based on YOLOv5-Lite was used as the target detector;

[0301] (2) Compare the improved algorithm based on the Bytetrack algorithm with the following algorithms:

[0302] The DeepSORT algorithm, Bytetrack algorithm, Bytetrack algorithm combined with NSA Kalman algorithm, and Bytetrack algorithm combined with appearance feature extraction algorithm are used to extract target appearance features using deep learning methods.

[0303] (3) Four videos of rice transplanter operation were used as evaluation videos to evaluate the performance of the two tracking algorithms (or trackers);

[0304] (4) The evaluation metrics adopted are those used in the multi-object tracking benchmark, including explicitly balancing the performance of accurate detection, association and localization into a single unified metric, the high-order accuracy (HOTA) used to compare trackers, the multiple object tracking accuracy (MOTA) metric for measuring the accuracy of multi-object tracking under a single camera, the identification F1-score (IDF1), recall, the number of actual target boxes (GT), the total number of missed targets (FN), and the frame rate (FPS).

[0305]

[0306] Where t is the time sequence variable of the tracking video sequence, FP is the number of false positives, IDSW is the number of times the target reappears after being occluded. If the ID number changes, it is recorded as one IDSW. IDTP, IDFP, and IDFN are true positives, false positives, and false negatives, respectively.

[0307] The comparison results of the two algorithms are as follows:

[0308] Table 2 Comparison Results of Tracking Algorithms

[0309] HOTA MOTA IDF1 IDSW ByteTrack 39.11％ 32.13％ 50.04％ 992 Deepsort 13.09％ -61.21％ 11.87％ 1212 ByteTrack + NSA Kalman 47.33％ 45.52％ 65.51％ 485 ByteTrack+ Appearance Features 49.04％ 49.35％ 66.11％ 422 An improved algorithm based on the Bytetrack algorithm 53.05％ 56.56％ 73.85％ 233

[0310] As shown in the table above, compared to the original ByteTrack algorithm, the "ByteTrack + NSA Kalman filter" improved HOTA by 8.22%, MOTA by 13.39%, IDF1 by 15.47%, and IDSW by 51.11%. By correlating the measurement noise covariance with the confidence level of the detected target, it significantly improved the tracking accuracy of seedlings. The "ByteTrack + appearance features" algorithm improved HOTA by 9.93%, MOTA by 17.22%, IDF1 by 16.07%, and IDSW by 57.46%, demonstrating the significant improvement in seedling tracking accuracy through the introduction of appearance features. Feature extraction can effectively improve tracking accuracy. When using the improved algorithm based on the Bytetrack algorithm, all indicators are further improved: HOTA is improved by 13.94%, MOTA by 24.43%, IDF1 by 23.81%, and IDSW is reduced by 76.51%. In contrast, the DeepSORT algorithm uses deep learning methods to extract target appearance features. Due to the complex background and terrain of rice planting scenes, the feature extraction effect is limited, resulting in poor tracking performance. Only MOTA is 4.65% higher than the improved Bytetrack algorithm; all other indicators are far inferior, making it difficult to meet practical needs. Therefore, the improved Bytetrack algorithm can effectively improve the multi-target tracking effect of rice seedlings.

[0311] Implementation Method 8: Combination Figures 1 to 16 This embodiment further defines the rice transplanter operation quality detection method described in Embodiment 1. The specific implementation details are as follows:

[0312] The central region counting method is as follows: in each frame of the image to be processed, the central region of the image is selected as the counting region; only within the counting region, the number of seedlings with different tracking IDs and the number of missing seedlings between each tracking ID and its next adjacent seedling are counted.

[0313] It should be noted that traditional tracking methods for counting seedlings use the number of all unique IDs appearing in the frame. However, in this invention, the camera on the rice transplanter captures the seedlings at an angle. The closer to the top of the frame, the denser the seedling distribution, and the slower the seedlings move in the frame as the transplanter moves. Conversely, the closer to the bottom of the frame, the more dispersed the seedling distribution, and the faster they move in the frame. Therefore, the tracker's predictions will have larger errors in the upper and lower parts of the frame, resulting in more frequent ID switching in actual experiments.

[0314] In this embodiment, by using a region-based counting strategy, that is, by cropping the central area of ​​the image and counting only the number of all different IDs appearing in that area, the number of incorrect counts can be effectively reduced.

[0315] It should be noted that during actual rice transplanting, the transplanter moves and plants seedlings simultaneously. Already planted seedlings gradually move away from the camera, while newly planted seedlings continuously enter the camera's view. To count the number of transplanted and missing seedlings, tracking and counting are necessary after identifying the seedlings and calculating the required information. The improved multi-target tracking algorithm and counting method described above can effectively count the number of transplanted and missing seedlings.

[0316] Furthermore, an implementation method is provided, specifically a comparative experiment, to verify the effectiveness of combining the improved Bytetrack-based algorithm with the central region counting method. Four videos of rice transplanter operations are used as evaluation videos to assess the accuracy of seedling counting. The reference transplanting count for each video is the result of manual counting. The calculation formula for counting accuracy is as follows:

[0317]

[0318] Where Accuracy represents the accuracy of the statistical transplanting quantity, Tr represents the transplanting quantity counted by the algorithm using a comparative experiment method, and Ref represents the transplanting quantity obtained by manual counting.

[0319] The comparison results of counting seedlings using different multi-target tracking methods and combinations of different counting methods are shown in the table below:

[0320] Table 3 Comparison of Counting Results

[0321]

[0322] As shown in the table above, compared to "ByteTrack + traditional counting," the counting accuracy of "ByteTrack + central region counting method" is improved by 24.71%, and compared to "DeepSORT + traditional counting," the counting accuracy of "DeepSORT + central region counting method" is improved by 33.58%. This is because the central region counting method eliminates the areas in the upper and lower parts of the video frame where seedling tracking is inaccurate, greatly reducing the number of seedling ID switching times within the counting area, thereby improving the counting accuracy. Compared to "ByteTrack + central region counting method,"... While the "DeepSORT + Central Region Counting Method" improved the counting accuracy by 0.11%, the counting rate (FPS) decreased by 64.54%, which is too slow to meet practical needs. Compared to "ByteTrack + Traditional Counting," the "Improved Algorithm Based on Bytetrack + Central Region Counting Method" still achieved an average calculation of 55.75 frames per second with a slight decrease in FPS of 11.15%, and the counting accuracy was improved by 36.06%. It can be seen that the above improvements can effectively improve the accuracy of the tracking counting algorithm and ensure the reliability of seedling counting.

[0323] Implementation Method Nine: Combination Figures 1 to 16 This embodiment further defines the rice transplanter operation quality detection method described in Embodiment 8. The specific implementation details are as follows:

[0324] The unit information of the tracked target includes: the location of the seedling, the size of the target box, the plant distance between the seedling of each tracking ID and its next adjacent seedling, and the number of missing seedlings.

[0325] In this embodiment, after the seedlings are identified by the target detector on the screen, a unit containing information such as the seedling position, detection size, and confidence score is obtained.

[0326] It should be noted that in traditional target tracking algorithms, after tracking the target, a unit information of the tracked target is calculated. The traditional unit information of the tracked target only includes information such as the seedling position and the size of the detection box, and does not include information such as the plant spacing and the number of missing seedlings in each frame.

[0327] As previously described, the calculation of seedling spacing and missing seedlings is performed using the spatial distance between the seedling and its neighboring seedlings in the current frame. However, the position of the same seedling differs across frames. In consecutive frames, some seedling data may be calculated repeatedly, meaning the total number of missing seedlings cannot be simply the sum of the two calculated numbers. Using traditional unit information for tracking targets makes it difficult to accurately determine the total number of missing seedlings.

[0328] By improving the unit information of traditional tracking targets, the plant spacing and number of missing seedlings between each tracking ID and its next adjacent seedling are also stored or updated in the unit information, which makes it easier to calculate the total number of transplanted seedlings and the number of missing seedlings over a period of time.

[0329] Implementation Method 10: Combination Figures 1 to 16 This embodiment describes a rice transplanter operation quality detection system, and the specific implementation details are as follows:

[0330] The system includes image acquisition equipment, embedded devices, data transmission equipment, and a cloud server;

[0331] The image acquisition device is used to acquire real-time video of the rice transplanter operating within a fixed range in front of the rice transplanter, and send the rice transplanter operating video to an embedded device.

[0332] The embedded device is used to obtain monitoring information based on the rice transplanter operation video using the above-mentioned rice transplanter operation quality detection method;

[0333] The data transmission device is used to send the positioning information sent by the cloud server to the embedded device, and also to send the monitoring information sent by the embedded device to the cloud server.

[0334] In this embodiment, the embedded device includes a processor and a memory; the memory is used to store executable instructions of the processor, and the processor is configured to execute the above-described rice transplanter operation quality detection method by executing the executable instructions.

[0335] In this embodiment, the system is used to realize real-time monitoring or detection of the operation quality of rice transplanters.

[0336] In this embodiment, the rice transplanter operation quality detection method is used to process the video captured by the camera, and to perform target detection, target tracking, and statistics on the rice seedlings in the video, as well as the row spacing, plant spacing, transplanting quantity, and missing seedling quantity, thereby obtaining monitoring information.

[0337] Furthermore, a preferred embodiment is provided in which the image acquisition device is a camera 3; more specifically, an onboard camera with a USB interface.

[0338] It should be noted that the rice transplanter operation quality detection system needs to control the size of the device so that it can be installed and disassembled on the rice transplanter. The acquired images only need to include the field of view within a fixed range in front of the rice transplanter, and the dynamic range requirements of the equipment are low. Therefore, a USB interface onboard camera is selected as the image acquisition device.

[0339] Furthermore, a preferred embodiment is provided in which the embedded device includes an embedded development board 9 and a development board power supply 8.

[0340] In this embodiment, the aim is to establish a portable, integrated real-time detection system for rice transplanting operations. Therefore, the carrier on which the method runs is an embedded development board, specifically the NVIDIA Jetson Nano. This development board is small in size, has high performance, and uses the Ubuntu 18.04 system, which facilitates program deployment.

[0341] Furthermore, a preferred embodiment is provided, wherein the data transmission device includes a GPS module 1, a 4G antenna module 2, a 4G DTU module 7, and a 4G DTU power supply 10.

[0342] It should be noted that farmland scenarios are typically located in remote areas, making wired transmission difficult and WiFi signals weak. Therefore, a 4G DTU is used for the data transmission module. A 4G DTU wireless communication module is a special communication technology that utilizes a 4G network to achieve remote data transmission between the cloud server and local devices. The DTU model used in this implementation is the USR G780s, which acts as an intermediary between the monitoring unit and the cloud server in the hardware system.

[0343] In this embodiment, the system workflow is as follows: Figure 9 As shown:

[0344] First, a camera captures real-time video of rice seedlings being planted by the rice transplanter and transmits the video to the embedded development board. Then, a method on the embedded development board calculates the rice transplanter's operational quality monitoring information. Next, the 4G DTU transmits GPS positioning information to the embedded development board via a serial port. Then, the program on the embedded development board packages the monitoring and positioning information into a single monitoring data set and sends it to the 4G DTU. Finally, the 4G DTU transmits the packaged data to the cloud server.

[0345] Furthermore, in a preferred embodiment, the system further includes a housing; the housing is a rectangular box.

[0346] The embedded device, 4G DTU module 7, and 4G DTU power supply 10 are fixedly placed inside the housing;

[0347] The GPS module 1 and the 4G antenna module 2 are fixedly mounted on the outside of the housing;

[0348] The housing has heat dissipation holes 4, wiring holes 5, and a development board interface 6.

[0349] The above further describes the technical solution provided by the present invention in detail through several specific embodiments in order to highlight the advantages and benefits of the technical solution provided by the present invention. However, the several specific embodiments described above are not intended to limit the present invention. Any reasonable changes and improvements to the present invention, reasonable combinations of implementation methods and equivalent replacements based on the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for inspecting the operational quality of a rice transplanter, characterized in that, The method comprises the following steps: S1. Acquire multiple rice transplanter operation videos taken in the direction in front of the rice transplanter; each rice transplanter operation video includes multiple frames of rice transplanter operation footage; S2. In each frame of the rice transplanter operation, a rectangular area in front of the rice transplanter is extracted as the frame to be processed. S3. Use a target detector to detect each seedling in each frame of the image to be processed, and obtain the pixel coordinates and detection box size of each seedling in each frame of the image to be processed. S4. Based on the pixel coordinates of the seedlings in the image to be processed, use the random sampling consistency algorithm to fit the straight path of each row of seedlings that the transplanter can plant at the same time, and assign each seedling to the straight path of each row of seedlings it forms. S5. Based on the pixel distance in the x-direction between the intersection of the path lines of two adjacent rows of seedlings and the bottom of the image to be processed, obtain the row spacing of the path lines of two adjacent rows of seedlings. S6. Using a distance mapping algorithm based on triangular transformation, the spatial distance obtained by converting the pixel distance between two adjacent seedlings in the straight line of the path of each row of seedlings is used as the plant distance between two adjacent seedlings. S7. Based on the plant spacing between two adjacent seedlings, obtain the number of missing seedlings between each seedling and its next adjacent seedling. S8. Use the seedling tracking algorithm to track the same seedling in each frame of the image to be processed, and assign a tracking ID to the same seedling. The central region counting method was used to count the number of seedlings with different tracking IDs, the plant spacing between each seedling with a tracking ID and its next adjacent seedling, and the number of missing seedlings. Based on the above statistical results, store and update the unit information of the tracking target for each frame to be processed; By summarizing the unit information of the tracked targets in all rice transplanter operation videos, the total transplanted rice quantity and total missing seedling quantity can be obtained; Step S6 includes: S6.

1. Based on the image to be processed, obtain the spatial geometric relationship between the shooting point of the image to be processed and the rice seedlings; in the spatial geometric relationship: The seedlings are positioned within the same horizontal line segment; the shooting point of the image to be processed is located above the horizontal line segment, and the vertical distance from the shooting point of the image to be processed to the horizontal line segment is... ; Let R be the vertical projection point of the shooting point of the image to be processed onto the horizontal line segment. Then R is located at the bottom of the image to be processed. In the image to be processed, seedlings are arranged in order from near to far above R, where the i-th seedling is... , It is an integer greater than or equal to 0; Starting from the shooting point of the image to be processed, draw line segments connecting each seedling, which serve as diagonal lines connecting each seedling. Draw a perpendicular line from the previous seedling to the line connecting the diagonal lines of the next seedling, where the seedling preceding the 0th seedling is denoted as R; The perpendicular line to the line connecting the diagonal lines of the i-th seedling is ; The angle between the oblique line connecting the i-th seedling and the horizontal line segment is . ; S6.

2. Based on the spatial geometric relationship and the trigonometric transformation formula, obtain the pixel distance between the i-th seedling and the seedling in front of it on the horizontal line segment, and use the pixel distance on the horizontal line segment as the plant distance between the i-th seedling and the seedling in front of it. The pixel distance between the i-th seedling and the seedling preceding it on the horizontal line segment is obtained using the following trigonometric transformation formula: ； ； ； The seedling tracking algorithm is an improved algorithm based on the Bytetrack algorithm; the improved algorithm based on the Bytetrack algorithm has made two improvements to the Bytetrack algorithm; The first improvement is: in the matching process between the detection box and the predicted box in the Bytetrack algorithm, the similarity between the detection box and the predicted box is calculated by combining pixel information and IoU. The images of the predicted bounding box obtained by the Kalman filter algorithm and the detection bounding box of the seedlings in the next frame to be processed are respectively extracted, and the two images are converted to the same size; Perform super-green feature grayscale conversion on both images respectively; Obtain the pixel matrix from each of the two images, and concatenate the odd-numbered rows of the two pixel matrices into a row vector. Calculate the cosine similarity between the two row vectors. The similarity between the detection box and the predicted box is calculated using IoU. The final similarity between the detection boxes and predicted boxes is obtained by weighting the similarity calculated using IoU with the cosine similarity of the two row vectors: ； Where s represents the final similarity, The cosine similarity between two row vectors is denoted as . This represents the similarity between the detected bounding box and the predicted bounding box calculated using IoU, where w represents the weighting coefficient. The second improvement is: in the Kalman filtering algorithm of the Bytetrack algorithm, the measurement noise covariance matrix is ​​adaptively adjusted. ； in, Here, is the measurement noise covariance matrix in the Kalman filter algorithm of the Bytetrack algorithm; conf is the confidence score of the detected object; k is a constant; This is the adjusted measurement noise covariance matrix.

2. The method for detecting the operational quality of a rice transplanter according to claim 1, characterized in that, The target detector is an improved model based on YOLOv5-Lite; The improved model based on YOLOv5-Lite uses YOLOv5-Lite as its base model and makes two improvements on top of YOLOv5-Lite: The first improvement is the addition of the CBAM attention mechanism module to YOLOv5-Lite to enhance the extraction of seedling features and suppress background information in the image to be processed. The second improvement is to replace the loss function of YOLOv5-Lite with the EIOU bounding box loss function.

3. The method for detecting the operational quality of a rice transplanter according to claim 1, characterized in that, Step S4 includes: S4.

1. Perform super-green feature grayscale conversion on the image to be processed to obtain a grayscale image of the seedlings; S4.2 Binarize the grayscale image of the seedlings to obtain a black and white image of the seedlings; S4.

3. Use the pixel coordinates of the white pixels in the black and white image of the rice seedlings as the initial sampling dataset for the RANSAC algorithm; use an iterative method to obtain multiple fitting paths; the iteration process of each iteration is as follows: The RANSAC algorithm is used to identify the interior points of the current iteration from the sampled dataset of the previous iteration, and a fitting path is fitted using the interior points of the current iteration. Remove the interior points of the current iteration from the sampled dataset of the previous iteration to obtain the sampled dataset of the current iteration; Based on the sampled dataset from this iteration, determine whether all paths have been properly fitted: Once all paths have been fitted, the iterative method ends. Otherwise, continue to the next iteration.

4. The method for detecting the operational quality of a rice transplanter according to claim 1, characterized in that, Step S5 is as follows: Obtain the pixel distance in the x-direction between the intersection point of the straight lines of the paths of two adjacent rows of seedlings and the bottom of the image to be processed; Based on the position of the straight lines of the paths of two adjacent rows of rice seedlings in the image to be processed, a correction coefficient is used to correct the pixel distance in the x-direction, and the corrected result is used as the row spacing of the straight lines of the paths of two adjacent rows of rice seedlings. ； in, This is the corrected result; c is the correction factor; The pixel distance in the x-direction; The correction coefficient ranges from [1.0, 1.1]. The closer the straight lines of the paths of two adjacent rows of seedlings are to the sides of the image, the larger the correction coefficient becomes.

5. The method for detecting the operational quality of a rice transplanter according to claim 1, characterized in that, Step S7 includes: Let l be the distance between any seedling in a straight line of any row of seedlings and its next adjacent seedling; let t be the distance threshold. Compare the magnitudes of l and t: like If the number of missing seedlings between any seedling and its next adjacent seedling is 0; Otherwise, let s be the number of missing seedlings between any given seedling and its next adjacent seedling: ； in, This indicates rounding up to the nearest integer.

6. The method for detecting the operational quality of a rice transplanter according to claim 1, characterized in that, The central region counting method is as follows: in each frame of the image to be processed, the central region of the image is selected as the counting region; only within the counting region, the number of seedlings with different tracking IDs and the number of missing seedlings between each tracking ID and its next adjacent seedling are counted.

7. The method for detecting the operational quality of a rice transplanter according to claim 6, characterized in that, The unit information of the tracked target includes: the location of the seedling, the size of the target box, the plant distance between the seedling of each tracking ID and its next adjacent seedling, and the number of missing seedlings.

8. A rice transplanter operation quality inspection system, characterized in that, The system includes image acquisition equipment, embedded devices, data transmission equipment, and a cloud server; The image acquisition device is used to acquire real-time video of the rice transplanter operating within a fixed range in front of the rice transplanter, and send the rice transplanter operating video to an embedded device. The embedded device is used to obtain monitoring information based on the rice transplanter operation video using the rice transplanter operation quality detection method according to any one of claims 1-7. The data transmission device is used to send the positioning information sent by the cloud server to the embedded device, and also to send the monitoring information sent by the embedded device to the cloud server.

Images