Crop detection and weed root location apparatus and method

Abstract

The invention provides crop detection and weed root localization devices and methods. A computing device for crop detection and weed root localization is described, the computing device comprising: a computing resource; and a target detection framework that invokes the computing resource for crop detection and weed root localization, wherein the target detection framework is configured to: receive an image captured from a scene comprising crops and weeds; perform a bounding box detection and object classification task to detect crops and weeds from the image; and perform a root coordinate regression task to determine weed root coordinates from the image.

Description

Technical Field

[0001] This invention relates generally to the field of computer vision technology, and more specifically to devices and methods for crop detection and weed root location. Background Technology

[0002] Weed control is a significant challenge in modern agriculture. Weeds compete with crops for critical resources such as water, nutrients, and sunlight, significantly impacting crop yield and quality. With the rapid development of deep learning and computer vision technologies, image recognition-based intelligent weeding technology has gradually become a research hotspot.

[0003] Deep learning-based object detection frameworks have been widely applied to various object detection and localization applications. In particular, the development of deep convolutional neural networks has significantly improved image classification, segmentation, and detection performance, enabling the identification of specific objects within images, the classification of objects contained in images, and the localization of specific objects within images. Applying this deep learning-based object detection framework to agriculture, using intelligent agricultural machinery to accurately identify the location of weeds, can save significant manpower, material resources, and financial resources for subsequent weeding.

[0004] However, current target detection frameworks for field weed detection and localization are based on bounding box classification and localization. While this bounding box-based approach has achieved good results in weed detection, its localization accuracy is low due to the limitation of bounding box size, which only allows for a rough determination of the weed's extent. This results in inefficient weed removal by weed control devices based on this framework. Summary of the Invention

[0005] This invention proposes a technical solution for crop detection and weed root localization. Based on the proposed target detection framework, it performs crop and / or weed detection and weed root localization, effectively addressing the problems of low weed localization accuracy and low weeding efficiency. In this invention, the target detection framework can perform both object detection and classification tasks as well as root coordinate regression tasks, thus accurately detecting and distinguishing crops and weeds in farmland scenes and precisely locating weed roots. Furthermore, during weeding, lasers can be precisely emitted towards the weed roots, efficiently removing weeds from the field, effectively preventing damage to nearby crops, and significantly reducing energy consumption.

[0006] In a first aspect of the invention, a computing device for crop detection and weed root localization is provided, comprising: computing resources; and a target detection framework that invokes the computing resources to perform crop detection and weed root localization, wherein the target detection framework is configured to: receive an image acquired from a scene including crops and weeds; perform bounding box detection and object classification tasks to detect crops and weeds from the image; and perform root coordinate regression tasks to determine weed root coordinates from the image.

[0007] The computing device described above, wherein the target detection framework is configured to perform the root coordinate regression task based on image features corresponding to weeds detected in the image, in order to determine the root coordinates of the weeds.

[0008] The computing device as described in any of the above, wherein the execution of the root coordinate regression task by the target detection framework is related to the root coordinate regression loss L. reg Correspondingly, the root coordinate regression loss L reg Based on the Euclidean distance between the actual coordinates of the weed roots and the determined coordinates of the weed roots.

[0009] The computing device as described in any of the above embodiments, wherein the object detection framework is further configured to: be optimized based on a comprehensive loss L through multi-task learning, wherein the comprehensive loss L is based on a classification loss L cls Bounding box detection loss L bbox and the root coordinate regression loss L reg The object detection framework's performance on the bounding box detection and object classification tasks is related to the bounding box detection loss L. bbox and the classification loss L cls Related.

[0010] In any of the computing devices described above, the overall loss L is calculated as: L = α·L cls +β·L bbox +γ·L reg Wherein, the hyperparameters α, β, and γ indicate the classification loss L cls The bounding box detection loss L bbox and root coordinate regression loss L reg The weights in the overall loss.

[0011] The computing device as described in any of the above, wherein the hyperparameter α is set to be greater than or equal to the threshold weight, and the hyperparameters β and γ are adjusted and traversed based on the setting of the hyperparameter α via multi-task learning performed by the object detection framework.

[0012] The computing device as described in any of the above, wherein the object detection framework is further configured to: employ a teacher-student framework by which a teacher model and a student model are optimized through semi-supervised learning on labeled image training sets and unlabeled image training sets.

[0013] The computing device as described in any of the above embodiments, wherein the teacher model is obtained by the object detection framework through multi-task learning based on the comprehensive loss L on the labeled image training set, and wherein the teacher model is configured to: perform the bounding box detection and object classification tasks on the unlabeled image training set to obtain inferred crop and weed detection results, the inferred crop and weed detection results having corresponding classification confidence; perform the root coordinate regression task on the unlabeled image training set to obtain inferred weed root coordinates, the inferred weed root coordinates having corresponding root location confidence; and output the inferred crop and weed detection results with classification confidence higher than the classification threshold confidence, and the inferred weed root coordinates with root location confidence higher than the root location confidence threshold, as pseudo-labels for the unlabeled image training set.

[0014] The computing device as described in any of the above, wherein the student model is obtained by configuring the object detection framework with reference to the model parameters of the teacher model, and wherein the student model is configured to be trained on a pseudo-labeled image training set and a labeled image training set based on the comprehensive loss, wherein the pseudo-labeled image training set is obtained by labeling the unlabeled image training set using the pseudo-labels.

[0015] The computing device as described in any of the preceding claims, wherein the labeled image training set is enhanced using a first image enhancement algorithm to obtain a first enhanced image training set, and the unlabeled image training set is enhanced using a second image enhancement algorithm to obtain a second enhanced image training set, the second image enhancement algorithm being different from the first image enhancement algorithm, wherein the teacher model is obtained by the object detection framework performing the multi-task learning based on the comprehensive loss L on the first enhanced image training set, and is configured to perform the bounding box detection and object classification tasks and the root coordinate regression task on the second enhanced image training set to output the pseudo-labels of the second enhanced image training set, wherein the pseudo-labeled image training set is obtained by labeling the second enhanced image training set using the pseudo-labels, and the student model is configured to be trained on the pseudo-labeled image training set and the first enhanced image training set based on the comprehensive loss.

[0016] The computing device as described in any of the above, wherein the target detection framework is based on a single-stage target detection framework with anchored frames.

[0017] In a second aspect of the invention, a weeding device is provided, comprising: a computing device as described in any of the preceding claims; a camera for capturing the image; and a laser emitting device for emitting a laser beam toward the roots of the weeds based on determined coordinates of the weed roots.

[0018] The weeding equipment as described in any of the above-mentioned items further includes: a moving device for moving the weeding equipment.

[0019] In a third aspect of the invention, a method for detecting crops and locating weed roots is provided, comprising: receiving an image acquired from a scene including crops and weeds by a target detection framework; performing bounding box detection and object classification tasks by the target detection framework to detect crops and weeds from the image; and performing root coordinate regression tasks by the target detection framework to determine the coordinates of weed roots from the image.

[0020] As described in any of the above methods, the root coordinate regression task performed by the target detection framework includes: the target detection framework performing the root coordinate regression task based on image features corresponding to weeds detected in the image to determine the root coordinates of the weeds.

[0021] The method described in any of the above embodiments, wherein the object detection framework performs the root coordinate regression task and the root coordinate regression loss L reg Correspondingly, the root coordinate regression loss L reg Based on the Euclidean distance between the actual coordinates of the weed roots and the determined coordinates of the weed roots.

[0022] The method described in any of the above embodiments, wherein the object detection framework is a first optimized object detection framework obtained by performing multi-task learning based on the object detection framework using a comprehensive loss L, wherein the comprehensive loss L is based on a classification loss L. cls Bounding box detection loss L bbox and the root coordinate regression loss L reg The object detection framework performs the bounding box detection and object classification tasks with the bounding box detection loss L. bbox and the classification loss L cls Related.

[0023] As described in any of the above methods, the comprehensive loss L is calculated as: L = α·L cls +β·L bbox +γ·L regWherein, the hyperparameters α, β, and γ indicate the classification loss L cls The bounding box detection loss L bbox and root coordinate regression loss L reg The weights in the overall loss.

[0024] The method described in any of the above embodiments, wherein the hyperparameter α is set to be greater than or equal to the threshold weight, and the hyperparameters β and γ are adjusted and traversed based on the setting of the hyperparameter α by performing multi-task learning via the object detection framework.

[0025] The method described in any of the above embodiments, wherein the object detection framework is a second optimized object detection framework, which is obtained by further employing a teacher-student framework in the object detection framework and performing semi-supervised learning on labeled image training sets and unlabeled image training sets through a teacher model and a student model.

[0026] The method described in any of the above embodiments, wherein the teacher model is obtained by the object detection framework performing the multi-task learning on the labeled image training set based on the comprehensive loss L, and the student model is obtained by the object detection framework configuring the model parameters of the teacher model, and wherein the optimization using the teacher-student framework on the labeled and unlabeled image training sets through semi-supervised learning includes: the teacher model performing the bounding box detection and object classification tasks on the unlabeled image training set to obtain inferred crop and weed detection results, the inferred crop and weed detection results having corresponding classification confidence; the teacher model performing the multi-task learning on the labeled image training set to obtain ... The model performs the root coordinate regression task on the unlabeled image training set to obtain inferred weed root coordinates, which have corresponding root location confidence scores. The inferred crop and weed detection results with classification confidence scores higher than the classification threshold, and the inferred weed root coordinates with root location confidence scores higher than the root location confidence threshold, are used as pseudo-labels for the unlabeled image training set to label it, thus obtaining a pseudo-labeled image training set. The student model then performs multi-task learning on both the pseudo-labeled and labeled image training sets based on the comprehensive loss to obtain the second optimized target detection framework.

[0027] The method described in any of the above embodiments, wherein the labeled image training set is enhanced using a first image enhancement algorithm to obtain a first enhanced image training set, and the unlabeled image training set is enhanced using a second image enhancement algorithm to obtain a second enhanced image training set, wherein the second image enhancement algorithm is different from the first image enhancement algorithm, wherein the teacher model is obtained by the object detection framework performing the multi-task learning based on the comprehensive loss L on the first enhanced image training set, and is configured to perform the bounding box detection and object classification tasks and the root coordinate regression task on the second enhanced image training set to output the pseudo-label of the second enhanced image training set, wherein the pseudo-labeled image training set is obtained by labeling the second enhanced image training set using the pseudo-label, and the student model is configured to be trained on the pseudo-labeled image training set and the first enhanced image training set based on the comprehensive loss.

[0028] In a fourth aspect of the invention, a machine-readable medium is provided, including machine-executable instructions stored thereon, which, when executed, cause a machine to perform the method as described in any of the preceding claims.

[0029] In a fifth aspect of the invention, a computer program product is provided, comprising computer instructions that, when executed, implement the method as described in any of the preceding embodiments. Attached Figure Description

[0030] Some implementations of the present invention will be described below by way of example only and with reference to the accompanying drawings. In the drawings, every identical or nearly identical component shown throughout the various figures may be denoted by the same reference numerals. For clarity, not every component is labeled in each figure. In the drawings:

[0031] Figure 1 A schematic diagram of an object detection framework is shown as an illustrative example;

[0032] Figure 2 A schematic diagram of a target detection framework according to an embodiment of the present invention is shown;

[0033] Figure 3 A schematic block diagram of a computing device for crop detection and weed root location according to an embodiment of the present invention is shown;

[0034] Figure 4 A schematic block diagram of a weeding device according to an embodiment of the present invention is shown;

[0035] Figure 5A schematic flowchart illustrating a method for crop detection and weed root location according to an embodiment of the present invention is shown.

[0036] Figure 6 A schematic flowchart illustrating a method for crop detection and weed root location according to another embodiment of the present invention is shown.

[0037] Figure 7 A schematic flowchart illustrating the process of training an object detection framework based on a teacher-student framework according to an embodiment of the present invention is shown.

[0038] Figure 8 A schematic diagram illustrating the architecture of a teacher-student framework according to an embodiment of the present invention is shown;

[0039] Figure 9 A comparison of weed root localization errors using different target detection frames according to embodiments of the present invention is shown;

[0040] Figure 10 This illustrates an exemplary application scenario of a crop detection and weed root location scheme according to an embodiment of the present invention;

[0041] Figures 11-12 This demonstrates the effectiveness of a crop detection and weed root location scheme according to an embodiment of the present invention. Detailed Implementation

[0042] Exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings, wherein, unless otherwise expressly stated, the same or similar reference numerals in different drawings denote the same or similar elements. Furthermore, it should be noted that exemplary embodiments of the present invention may perform the steps of the corresponding methods in a different order than that shown and described in the specification. In some embodiments of the present invention, the method may include more or fewer steps than those described in the specification and shown in the accompanying drawings. Moreover, a single step used in some embodiments of this specification may be broken down into multiple steps in other embodiments, or multiple steps used in some embodiments of this specification may be combined into a single step in other embodiments.

[0043] Numerous specific details are set forth in the following description. However, it should be understood that embodiments of the invention may be practiced without these specific details. In other instances, well-known circuits, structures, and techniques have not been shown in detail so as not to obscure the understanding of this description.

[0044] References to "an embodiment," "an embodiment," "an exemplary embodiment," etc., in the specification indicate that the described embodiment may include a specific feature, structure, or characteristic; however, not every embodiment necessarily includes that specific feature, structure, or characteristic. Furthermore, such phrases do not necessarily refer to the same embodiment. Additionally, when a specific feature, structure, or characteristic is described in connection with an embodiment, it is believed that the influence of such feature, structure, or characteristic on such feature, structure, or characteristic in conjunction with other embodiments, whether explicitly described or not, is within the knowledge of those skilled in the art.

[0045] For the purposes of this disclosure, the phrase "A and / or B" means (A), (B), or (A and B). For the purposes of this disclosure, the phrase "A, B, and / or C" means (A), (B), (C), (A and B), (A and C), (B and C), or (A, B, and C).

[0046] In the following specification and claims, the terms “coupled” and “connected” and their derivatives may be used. It should be understood that these terms are not intended to be synonyms for each other. Rather, in certain embodiments, “connected” may be used to indicate that two or more elements are in direct physical or electrical contact with each other. “Coupled” may mean that two or more elements are in direct physical or electrical contact. However, “coupled” may also mean that two or more elements are not in direct contact with each other, but still cooperate or interact with each other.

[0047] Embodiments of the present invention provide a computing device for crop detection and weed root localization, which can be used to detect crops and weeds in farmland and locate the roots of weeds. The crop detection and weed root localization technology solution according to embodiments of the present invention is based on a target detection framework.

[0048] Figure 1 The image schematically illustrates an example of an object detection framework. For example... Figure 1 As shown, the basic components of the object detection framework 10 include a backbone network 102, a neck network 104, and a head network 106, which are typically implemented using neural networks.

[0049] In the object detection framework 10, the backbone network 102 can typically be implemented as a pre-trained convolutional neural network with pre-trained parameters on a large dataset, which is used to extract image features from the input image.

[0050] In the object detection framework 10, the neck network 104, as the intermediate part connecting the backbone network 102 and the head network 106, can be configured to process the feature maps extracted by the backbone network 102 and combine feature maps from different levels, thereby providing the object detection framework 10 with multi-scale object detection capabilities.

[0051] In the object detection framework 10, the head network 106 receives processed feature maps from the neck network 104 and performs object detection and localization based on these processed feature maps. Generally, the head network of different frameworks may vary due to the specific requirements of their object detection tasks. As an exemplary implementation, in single-stage object detection frameworks such as YOLO (YouOnly Look Once) and SSD (Single Shot MultiBox Detector), each grid unit of the head network 106 predicts multiple bounding boxes (e.g., anchor boxes of a specific size) based on the received feature maps and fits the bounding boxes to the actual targets through a regression process based on a loss function, thereby directly outputting the location of the bounding boxes, the class and probability of the objects contained in the bounding boxes, and the corresponding confidence scores from the feature maps.

[0052] According to embodiments of the present invention, the object detection framework 10 can be trained to perform weed detection and localization. As an example, the object detection framework 10 can be trained on a large number of training images containing and / or not containing weeds (e.g., one or more types of weeds) to enable it to be used to detect and localize specific types of weeds (and / or crops).

[0053] However, as described above, the object detection framework 10 detects and locates one or more categories of weeds (and / or crops) based on bounding boxes, through bounding box regression and object classification processes. Therefore, the weed location output by the object detection framework 10 is actually the location of the bounding box; that is, it can only roughly determine the extent of the weeds and cannot further pinpoint the exact location of the weed roots. When the weeding device uses the weed location results from the object detection framework 10 to remove weeds, it needs to aim the laser at a larger area to ensure effective weeding, which leads to high energy consumption and may damage nearby crops.

[0054] Figure 2 A target detection framework 20 according to an embodiment of the present invention is schematically shown. Figure 2 As shown, compared to Figure 1 The target detection framework 10 and target detection framework 20 shown include additional head network components (e.g., such as...). Figure 2 The head network 208 shown is illustrated. Figure 1 The bounding box regression and object classification processes performed by the head network 106 shown in the figure are different. Figure 2 The head network 208 shown can perform a point regression process that fits the predicted points to the true points.

[0055] According to an embodiment of the present invention, in the target detection framework 20, the head network 206 can receive the processed feature map from the neck network 204 and infer the position of the bounding box, the category of the object contained in the bounding box and its probability, etc., based on the bounding box regression and object classification process; the head network 208 can receive the processed feature map from the neck network 204 and predict the position of the point based on the point regression process.

[0056] According to an embodiment of the present invention, in the object detection framework 20, the head network 206 performs bounding box regression and object classification processes, which may be associated with bounding box loss and classification loss. The bounding box loss indicates the difference between the inferred bounding box position and the actual bounding box position, and the classification loss indicates the difference between the inferred object category and the actual object category. According to an embodiment of the present invention, the head network 208 performs point regression processes, which may be associated with point location loss. The point location loss indicates the distance between the point location predicted by the head network 208 and the actual point location. In an embodiment of the present invention, the object detection framework 20 may perform bounding box regression and object classification processes and point location regression processes based on a loss function that includes at least bounding box loss, classification loss, and point location loss, through the head networks 206 and 208.

[0057] According to other embodiments of the present invention, the head network 208 can perform a point regression process based on the inference results of the head network 206. As an exemplary implementation, the head network 208 can perform a point regression process based on the object category and its probability inferred by the head network 206, targeting the image features corresponding to that object.

[0058] According to embodiments of the present invention, the object detection framework 20 can be configured to perform crop detection and weed root localization. As an exemplary implementation, the head network 206 of the object detection framework 20 can be configured to detect crops and / or weeds in an image by performing bounding box detection and object classification tasks, and the head network 208 can be configured to determine the root location of weeds in the image by performing root coordinate regression tasks.

[0059] According to an embodiment of the present invention, the head network 206 can be configured to perform bounding box detection and object classification tasks on feature maps extracted by the backbone network 202 and the neck network 204 via bounding box regression and object classification processes as described above, to infer the category of each object contained in the image and its probability (e.g., the probability that the object contained in the inferred bounding box is (e.g., a specific category) a crop and the object is (e.g., a specific category) a crop), and the corresponding confidence score. According to an embodiment of the present invention, the head network 206 performing the bounding box detection and object classification tasks can be coupled with a bounding box detection loss L. bboxAnd classification loss L cls Correlated, where bounding box detection loss L bbox The classification loss L indicates the difference between the inferred bounding box and the actual bounding box (e.g., it can be calculated based on the vertex coordinates of the inferred bounding box and the actual bounding box). cls It indicates the difference between the inferred object category and the actual object category for the corresponding bounding box (e.g., it can be calculated based on the probability distribution of the inferred category and the probability distribution of the actual category).

[0060] According to an embodiment of the present invention, the head network 208 can be configured to perform a root coordinate regression task on the feature maps extracted by the backbone network 202 and the neck network 204 via a root coordinate regression process as described above, to infer the root location of weeds in an image. According to an embodiment of the present invention, the head network 208 performing the root coordinate regression task can be combined with a root coordinate regression loss L. reg Related. As an example implementation, the root coordinate regression loss L reg The distance can be calculated based on the Euclidean distance between the actual coordinates of the weed roots and the inferred coordinates of the weed roots: Where n indicates the number of input images, y i Indicates the actual coordinates of the weed roots in the i-th input image. Indicates the coordinates of the inferred weed roots in the i-th input image.

[0061] According to an embodiment of the present invention, the object detection framework 20 is an end-to-end model that receives an input image at the receiving end and outputs crop detection and weed root localization results at the output end. As an example, the object detection framework 20 can receive an image captured from a scene containing crops and weeds, and output the category and probability of each object in the scene (e.g., (crop, 0.8), indicating that the object in the scene is a crop and the probability of the object being a crop is 0.8) and the root coordinates of the weeds.

[0062] When using the target detection framework 20 described above for crop detection and weed root location, the weeding device can concentrate the laser beams emitted at the weed root locations output by the target detection framework 20. This can improve the accuracy of weeding, avoid damage to nearby crops, reduce energy consumption, and improve the weeding effect.

[0063] According to embodiments of the present invention, head network 208 may be further configured to perform a root coordinate regression task based on the inferences of head network 206. As an exemplary implementation, head network 206 may be configured to detect crops and weeds from a feature map, and head network 208 may be configured to perform a root coordinate regression process based on image features corresponding to the weeds detected by head network 206 to determine the root coordinates of the weeds. As an alternative implementation, where head network 206 classifies objects in the detected boxes as weeds by performing bounding box detection and object classification tasks on the feature map, since head network 206 has already inferred the positions of the bounding boxes (e.g., the vertex coordinates of the bounding boxes), head network 208 can obtain the positions of the bounding boxes in which weeds were detected from head network 206 and perform the root coordinate regression task based on these bounding box positions.

[0064] On the one hand, compared to the root coordinate regression task directly on the feature maps extracted by the backbone network 202 and the neck network 204, performing the root coordinate regression process based on the weeds detected by the head network 206 from the feature maps can further reduce the error of the determined weed root coordinates and improve the accuracy of determining the weed root coordinates. On the other hand, when the head network 206 classifies the object in the detection box as a weed, the situation where crops and / or weeds in the detection box are also occluded by other objects (e.g., crops) can be effectively excluded (because in these cases, the head network 206 will not classify the object in the detection box as a weed or the probability of the object in the detection box being a weed is low). That is, in the embodiments of the present invention, the head network 208 only further determines the root coordinate position of the object when the head network 206 classifies the object in the detection box as a weed and the probability of the object being a weed is high enough. This can save computing resources and improve the accuracy and safety of weeding operations, thereby further reducing energy consumption and improving weeding effect.

[0065] An embodiment of the present invention provides a computing device for crop detection and weed root location, which can detect crops and weeds and locate weed roots based on the target detection framework 20 provided by the present invention.

[0066] Figure 3 A schematic block diagram of a computing device 30 for crop detection and weed root location according to an embodiment of the present invention is shown.

[0067] like Figure 3 As shown, the computing device 30 may include computing resources 302 and a target detection framework 304. According to an embodiment of the present invention, the computing resources 302 may include various hardware and software resources for performing data processing, storage, transmission, etc.

[0068] According to an embodiment of the present invention, the target detection framework 304 may be obtained by end-to-end training of the target detection framework 20 on a large number of training images that include and / or do not include (e.g., specific categories) crops. The target detection framework 304 is capable of synchronously outputting the detected crops and weeds, as well as the root locations of the weeds, from the input images.

[0069] According to an embodiment of the present invention, the target detection framework 304 can call computing resources 302 to perform crop detection and weed root location.

[0070] According to embodiments of the present invention, the object detection framework 304 can be configured to receive images (e.g., field images) captured by image acquisition devices inside and / or outside the computing device 30 from a scene containing crops and / or weeds, and perform bounding box detection and object classification tasks on these images to detect crops and weeds, and perform root coordinate regression tasks on these images to determine the root coordinates of weeds. As an exemplary implementation, the object detection framework 304 can be configured to perform bounding box regression and object classification processes on image features based on a loss function to infer the category and probability (e.g., the probability that the bounding box contains an object that is a crop and that the object is a crop) and the corresponding confidence level of each object in the received image; and can be configured to perform weed root coordinate regression processes on image features based on a loss function to infer the root location of weeds in the received image.

[0071] According to an embodiment of the present invention, the object detection framework 304 can perform bounding box detection and object classification tasks with bounding box detection loss L. bbox And classification loss L cls Correlated, where bounding box detection loss L bbox The classification loss L indicates the difference between the inferred bounding box and the actual bounding box. cls This indicates the difference between the inferred object category and the actual object category for the corresponding bounding box. As an example implementation, the bounding box detection loss L... bbox The classification loss can be calculated as the Euclidean distance between the inferred bounding box coordinates (e.g., vertex coordinates, center point coordinates, feature point coordinates, etc.) and the actual bounding box coordinates; the classification loss can be calculated as the difference between the inferred object class probability distribution for the corresponding bounding box and the actual object class probability distribution (e.g., cross-entropy).

[0072] According to an embodiment of the present invention, the target detection framework 304 can perform the weed root coordinate regression process with the root coordinate regression loss L. reg Related. As an example implementation, the root coordinate regression loss L regThe distance can be calculated based on the Euclidean distance between the actual coordinates of the weed roots and the inferred coordinates of the weed roots: Where n indicates the number of input images, y i Indicates the actual coordinates of the weed roots in the i-th input image. This indicates the coordinates of the inferred weed roots in the i-th input image. According to an embodiment of the invention, the loss function can be based at least on the bounding box detection loss L. bbox Classification loss L cls and root coordinate regression loss L reg .

[0073] According to other embodiments of the present invention, the object detection framework 304 may be configured to receive images captured from a scene containing crops and / or weeds, and perform bounding box detection and object classification tasks on these images to detect crops and weeds; and perform root coordinate regression tasks based on image features corresponding to the weeds detected in these images to determine the root coordinates of the weeds. As an exemplary implementation, the object detection framework 304 may be configured to perform bounding box regression and object classification processes on all image features extracted from the received images based on a loss function to infer the categories and probabilities of objects contained in these images, as well as the corresponding confidence levels; additionally, the object detection framework 304 may be configured to perform root coordinate regression processes only on image features corresponding to objects identified as weeds (e.g., inferring that the category of the object is weeds and the probability of the object being a weed is higher than a threshold) based on a loss function to infer the root location for that object.

[0074] Unlike existing weed detection and localization schemes that focus on the overall detection and localization of weeds, the computing device 30 according to an embodiment of the present invention can accurately detect and distinguish crops and weeds in a farmland scene using a target detection framework 304, and precisely locate the roots of the weeds. Furthermore, according to an embodiment of the present invention, the target detection framework 304 can also perform a root coordinate regression process based on its crop and weed detection, which can further reduce the error of the determined weed root coordinates and improve the accuracy of determining the weed root coordinates.

[0075] According to an embodiment of the present invention, the object detection framework 304 can be implemented as an object detection framework 20 that is further fine-tuned based on the comprehensive loss L through a multi-task learning process. In other words, the object detection framework 304 can be configured to be optimized based on the comprehensive loss L through multi-task learning.

[0076] As an example implementation, considering the losses of each head network included in the object detection framework 20, the comprehensive loss L can be defined as including the classification loss L. cls Bounding box detection loss L bboxand root coordinate regression loss L reg Therefore, optimizing the target detection framework 20 based on the comprehensive loss L can simultaneously improve the crop detection capability and weed root localization capability of the target detection framework. As an exemplary implementation, the comprehensive loss L can be calculated as L = α·L cls +β·L bbox +γ·L reg Wherein, the hyperparameters α, β, and γ represent the classification loss L, respectively. cls Bounding box detection loss L bbox and root coordinate regression loss L reg The weights in the comprehensive loss L can be dynamically adjusted during multi-task learning. As an example implementation, the hyperparameter α can be set to a weight greater than a threshold, and the hyperparameters β and γ can be adjusted based on the setting of hyperparameter α through multi-task learning performed by the object detection framework 20. For example, the sum of hyperparameters α, β, and γ can be set to 1, and hyperparameter α can be set to a value greater than 0.5 (or another appropriate threshold weight value), allowing the values ​​of hyperparameters α, β, and γ to be adjusted through multi-task learning performed by the object detection framework 20.

[0077] According to an embodiment of the present invention, through multi-task learning, the target detection framework 304 can be better adapted to crop detection and weed root location, and its detection and location accuracy is further improved.

[0078] According to an embodiment of the present invention, the object detection framework 304 can be further configured to be optimized by employing a teacher-student framework on top of multi-task learning, through semi-supervised learning on labeled and unlabeled image training sets. As an exemplary implementation, an object detection framework 20 finely tuned through multi-task learning on labeled image training sets based on a comprehensive loss L can be used as the teacher model Teacher(·), and the student model Student(·) can be configured with reference to the model parameters of the teacher model Teacher(·).

[0079] As an exemplary implementation, a target detection framework 20, fine-tuned through multi-task learning on an labeled image training set based on a comprehensive loss L, can be used as the teacher model Teacher(·), and an untrained target detection framework 20 can be used as the student model Student(·). According to an embodiment of the present invention, the unlabeled image training set can be input into the teacher model Teacher(·), which performs bounding box detection and object classification tasks on the unlabeled image training set to infer the crops and weeds contained in these unlabeled training images. It should be noted that the difference between the unlabeled image training set and the labeled image training set referred to herein is only whether they are labeled; that is, the labeled image training set is the version of the unlabeled image training set after being labeled with real labels. (Refer to the above...) Figure 2 As described, each inference for crops or weeds has a corresponding classification confidence level. As an example implementation, this classification confidence level can be calculated as ConfScore = Max(Softmax(Teacher(E)). u ))), where E u The image features corresponding to the unlabeled training image are indicated. According to an embodiment of the present invention, the inference that the ConfScore is higher than the classification confidence threshold τ can be used as the pseudo-label of the corresponding unlabeled training image. That is, the category of the corresponding object in the unlabeled training image is labeled based on the inference that the classification confidence is higher than the classification confidence threshold τ (e.g., (weed, 0.86), which indicates that the object contained in the bounding box is weed and the probability that the object is weed is 0.86).

[0080] According to embodiments of the present invention, the teacher model Teacher(·) can also perform a root coordinate regression task on an unlabeled image training set to infer the root coordinates of weeds in these unlabeled training images. In embodiments of the present invention, each inference for the weed root coordinates has a corresponding root location confidence score SimScore, and the root locations of weeds in the corresponding unlabeled training images can be labeled based on inferences where the SimScore is higher than the root location confidence threshold ξ. As an exemplary implementation, the root location confidence score SimScore can be calculated as the weed image features extracted by the teacher model Teacher(·) from the unlabeled training images. Compared to the corresponding weed image features of the labeled training image corresponding to the unlabeled training image. Similarity between them, for example, calculating the SimScore confidence score of the root location as a cosine similarity:

[0081] According to embodiments of the present invention, the student model Student(·) can be trained on both labeled image training sets and pseudo-labeled image training sets based on a comprehensive loss L. In embodiments of the present invention, the student model Student(·) can perform supervised learning on the labeled image training set based on the comprehensive loss L, and unsupervised learning on the pseudo-labeled image training set based on the comprehensive loss L.

[0082] The following will refer to Figure 8 The overall architecture of the object detection framework, which employs a teacher-student framework for semi-supervised learning to optimize the object detection framework, will be described in further detail according to an embodiment of the present invention.

[0083] According to an embodiment of the present invention, the object detection framework 304 is trained on a larger training set via semi-supervised learning, thereby learning more general image features and thus having higher accuracy, generalization ability and robustness.

[0084] According to an embodiment of the present invention, before the teacher model Teacher(·) and the student model Student(·) perform semi-supervised learning on labeled and unlabeled image training sets based on a teacher-student framework, image augmentation can be performed on the labeled and unlabeled image training sets first.

[0085] According to embodiments of the present invention, different image enhancement algorithms can be used to enhance both labeled and unlabeled image training sets. According to an embodiment of the present invention, a first image enhancement algorithm can be used to enhance the labeled image training set. Specifically, the first image enhancement algorithm is used to process each labeled training image in the labeled image training set to obtain enhanced labeled training images, thereby obtaining a first enhanced image training set, which includes the enhanced labeled training images. According to an embodiment of the present invention, a second image enhancement algorithm, different from the first image enhancement algorithm, can be used to enhance the unlabeled image training set. Specifically, the second image enhancement algorithm is used to process each unlabeled training image in the unlabeled image training set to obtain enhanced unlabeled training images, thereby obtaining a second enhanced image training set, which includes the enhanced unlabeled training images. As an exemplary implementation, the first image enhancement algorithm may include brightness adjustment, contrast adjustment, image cropping, and image flipping, while the second image enhancement algorithm may only include brightness adjustment and contrast adjustment.

[0086] According to an embodiment of the present invention, the teacher model Teacher(·) can be configured to employ a target detection framework 20 finely tuned through multi-task learning on a first enhanced image training set based on a comprehensive loss L. It can also be configured to perform bounding box detection and object classification tasks, as well as root coordinate regression tasks, on a second enhanced image training set, and to filter the obtained inferences based on confidence levels to obtain pseudo-labels for the second enhanced image training set. Thus, according to an embodiment of the present invention, these pseudo-labels can be used to label each enhanced unlabeled training image in the second enhanced image training set to obtain a pseudo-labeled image training set. For the student model Student(·), according to an embodiment of the present invention, it can be configured to be trained on this pseudo-labeled image training set and the first enhanced image training set based on a comprehensive loss L.

[0087] According to embodiments of the present invention, the generalization ability of the target detection framework can be further improved through image enhancement.

[0088] Embodiments of the present invention provide a weeding device that can be used to remove weeds in farmland by detecting crops and weeds and locating the roots of weeds based on the target detection framework provided by the present invention.

[0089] The following text refers to Figure 4 The weeding device 40 according to an embodiment of the present invention will be described in detail below.

[0090] like Figure 4 As shown, the weeding device 40 may include a camera 402, as referenced above. Figure 3 The computing device 30 and laser emitting device 404 are described, with the computing device 30 being coupled to the camera 402 and the laser emitting device 404. According to an embodiment of the invention, the camera 402 can be configured to capture field images, which are provided to the computing device 30 for crop detection and weed root localization. The computing device 30 performs bounding box detection and object classification tasks, as well as root coordinate regression tasks, to detect crops and weeds from these images and determine the coordinates of weed roots; the related operations have been described above. Figure 3 The invention has been described in detail and will not be repeated here to avoid obscuring the understanding of the invention. According to an embodiment of the invention, the laser emitting device 404 can emit a laser based on the detection of crops and weeds and the location of weed roots by the computing device 30. As an exemplary implementation, the laser emitting device 404 can receive the determined coordinates of the weed roots from the computing device 30 and emit a laser at the location indicated by those coordinates.

[0091] like Figure 4As shown, in an embodiment of the present invention, the weeding device 40 may optionally include a moving device 406 for moving the weeding device 40. As an exemplary implementation, the moving device 406 may be implemented using tracks, wheels, outriggers, etc., so that the weeding device 40 can be easily moved to any location in the field.

[0092] As described above, the computing device 30 can accurately identify crops and weeds in the farmland and precisely locate the roots of the weeds. Based on this, the weeding device 40 according to an embodiment of the present invention can accurately emit lasers to the roots of the weeds to remove them, which can efficiently remove weeds in the field, effectively prevent damage to nearby crops, and significantly reduce energy consumption.

[0093] Table 1 below shows the performance of a conventional weeding device based on a conventional target detection framework and a weeding device 40 according to an embodiment of the present invention.

[0094] Table 1 Comparison of performance of weeding equipment

[0095] Root distance error False positive rate accuracy Energy costs Traditional weeding equipment 2.9770 10％ 75.37％ 1.55 Weeding equipment 40 2.4838 0 80.42％ 1.05

[0096] As shown in Table 1, under the same weeding conditions, compared to using a conventional target detection framework based on bounding boxes for weed detection and localization in traditional weeding equipment, the weeding equipment 40 according to the embodiments of the present invention reduces energy consumption by approximately 32.3%, while improving weeding accuracy by 5.05%. Using the weeding equipment 40 according to the embodiments of the present invention, based on the target detection framework 304 proposed in this invention, can significantly improve the accuracy of crop and / or weed detection and reduce energy costs for weeding operations. Furthermore, as shown in Table 1, the target detection framework 304 according to the embodiments of the present invention can reduce the false weed detection rate to 0, which particularly ensures that the weeding equipment 40 will not misidentify crops as weeds in actual operation, thus improving the safety of weeding operations.

[0097] Embodiments of the present invention provide a method for detecting crops and locating weed roots, which can detect crops and weeds and locate weed roots based on the target detection framework 20 provided by the present invention.

[0098] Figure 5 A schematic flowchart of a method 50 for crop detection and weed root location according to an embodiment of the present invention is shown.

[0099] like Figure 5As shown, method 50 may include, at block 502, receiving images acquired from a scene including crops and weeds by an object detection framework. As an exemplary implementation, the object detection framework may receive field images captured by an image acquisition device in real time, and / or field images stored therein from an image storage device.

[0100] like Figure 5 As shown, method 50 may further include: at box 504, a bounding box detection and object classification task performed by an object detection framework to detect crops and weeds from the image. As an exemplary implementation, the object detection framework may perform the tasks described above on the received image. Figure 2 The described bounding box regression and object classification process infers the category and probability of each object contained therein, with a corresponding confidence level. According to embodiments of the invention, the bounding box detection and object classification tasks performed by the object detection framework can be coupled with the bounding box detection loss L. bbox And classification loss L cls Correlated, where bounding box detection loss L bbox The classification loss L indicates the difference between the inferred bounding box and the actual bounding box (e.g., it can be calculated as the Euclidean distance between the inferred bounding box location and the actual bounding box location). cls It indicates the gap between the inferred object category and the actual object category for the corresponding bounding box (e.g., it can be calculated as the cross-entropy between the inferred object category probability distribution and the actual object category probability distribution).

[0101] like Figure 5 As shown, method 50 may further include: at block 506, the object detection framework performs a root coordinate regression task to determine the coordinates of weed roots from the image. As an exemplary implementation, the object detection framework may perform the task as described above on the received image. Figure 2 The described root coordinate regression process is used to determine the root coordinates of weeds. According to an embodiment of the invention, the root coordinate regression task performed by the target detection framework can be related to the root coordinate regression loss L. reg Related. As an example implementation, the root coordinate regression loss L reg The distance can be calculated based on the Euclidean distance between the actual coordinates of the weed roots and the inferred coordinates of the weed roots: Where n indicates the number of input images, y i Indicates the actual coordinates of the weed roots in the i-th input image. Indicates the coordinates of the inferred weed roots in the i-th input image.

[0102] Unlike existing weed detection and location schemes that focus on the overall detection and location of weeds, the method 50 according to an embodiment of the present invention can accurately detect and distinguish crops and weeds in farmland scenarios using the target detection framework proposed in this invention, and can accurately locate the roots of weeds.

[0103] Figure 6 A schematic flowchart of a method 60 for crop detection and weed root location according to other embodiments of the present invention is shown.

[0104] like Figure 6 As shown, method 60 may include: at box 602, receiving an image from a scene including crops and weeds by an object detection framework.

[0105] like Figure 6 As shown, method 60 may further include: at box 604, a target detection framework performs bounding box detection and object classification tasks to detect crops and weeds from the image.

[0106] like Figure 6 As shown, method 60 may further include: at box 606, the object detection framework performs a root coordinate regression task based on image features corresponding to the weeds detected in the image to determine the root coordinates of the weeds. For example, if the object detection framework detects object 1 as a weed at box 604, then subsequently, at box 606, the object detection framework may perform a root coordinate regression task on the image features corresponding to object 1 to determine the root coordinates of the weeds for object 1.

[0107] According to an embodiment of the present invention, in method 60, the target detection framework performs a root coordinate regression process based on its crop and weed detection, which can further reduce the error of the determined weed root coordinates and improve the accuracy of determining the weed root coordinates.

[0108] According to embodiments of the present invention, the above references Figure 5 The described method 50 and reference Figure 6 The object detection framework used in the described method 60 can be as described above. Figure 2 The described object detection framework 20 is trained end-to-end on a large number of training images that contain and / or do not contain (e.g., specific categories) crops, and is able to synchronously output detected crops and weeds, as well as the root locations of weeds, from input images.

[0109] According to an embodiment of the present invention, the target detection framework can be implemented as a first optimized target detection framework obtained by further fine-tuning the target detection framework 20 based on a multi-task learning process using a comprehensive loss L.

[0110] As an example implementation, considering the losses of each head network included in the object detection framework 20, the comprehensive loss L can be defined as including the classification loss L. cls Bounding box detection loss L bbox and root coordinate regression loss L reg Therefore, optimizing the target detection framework 20 based on the comprehensive loss L can simultaneously improve the crop detection capability and weed root localization capability of the target detection framework. As an exemplary implementation, the comprehensive loss L can be calculated as L = α·L cls +β·L bbox +γ·L reg Wherein, the hyperparameters α, β, and γ represent the classification loss L, respectively. cls Bounding box detection loss L bbox and root coordinate regression loss L reg The weights in the comprehensive loss L can be dynamically adjusted during multi-task learning. As an example implementation, the hyperparameter α can be set to a weight greater than a threshold, and the hyperparameters β and γ can be adjusted based on the setting of hyperparameter α through multi-task learning performed by the object detection framework 20. For example, the sum of hyperparameters α, β, and γ can be set to 1, and hyperparameter α can be set to a value greater than 0.5 (or another appropriate threshold weight value), allowing the values ​​of hyperparameters α, β, and γ to be adjusted through multi-task learning performed by the object detection framework 20.

[0111] According to an embodiment of the present invention, the first optimized target detection framework obtained by the target detection framework 20 through multi-task learning can be better adapted to crop detection and weed root location, and its detection and location accuracy is further improved.

[0112] According to embodiments of the present invention, the object detection framework can also be implemented as a second optimized object detection framework obtained by further employing a teacher-student framework to perform semi-supervised learning on labeled image training sets and unlabeled image training sets based on the multi-task learning described above.

[0113] Figure 7The diagram schematically illustrates a process 70 according to an embodiment of the present invention, in which a teacher model and a student model perform semi-supervised learning on labeled and unlabeled image training sets based on a teacher-student framework to obtain a second optimized object detection framework. It should be noted that the difference between the unlabeled and labeled image training sets referred to herein lies only in whether they are labeled; that is, the labeled image training set is a version of the unlabeled image training set after being labeled with real labels. As an exemplary implementation, an object detection framework 20 finely tuned through multi-task learning on the labeled image training set based on a comprehensive loss L can be used as the teacher model Teacher(·), and the student model Student(·) can be configured with reference to the model parameters of the teacher model Teacher(·).

[0114] like Figure 7 As shown, process 70 may include: at box 702, the teacher model Teacher(·) performs bounding box detection and object classification tasks on an unlabeled image training set to infer crops and weeds contained in these unlabeled training images. According to an embodiment of the invention, each inference for crops or weeds has a corresponding classification confidence score ConfScore. As an exemplary implementation, this classification confidence score can be calculated as ConfScore = Max(Softmax(Teacher(E)). u ))), where E u Indicates the image features corresponding to unlabeled training images. Further, such as... Figure 7 As shown, process 70 may further include: at box 704, determining whether the classification confidence score ConfScore for the inference regarding crops or weeds is higher than the classification confidence threshold τ. According to an embodiment of the invention, inferences with a classification confidence score ConfScore higher than the classification confidence threshold τ can be retained as pseudo-labels for the corresponding unlabeled training images. According to an embodiment of the invention, for inferences with a classification confidence score ConfScore not higher than the classification confidence threshold τ, such as... Figure 7 As shown, at box 705, these inferences are discarded instead of being treated as pseudo-labels.

[0115] like Figure 7As shown, process 70 may further include: at box 706, a root coordinate regression task is performed by the teacher model Teacher(·) on an unlabeled image training set to infer the root coordinates of weeds in these unlabeled training images. According to embodiments of the invention, each inference of weed root coordinates also has a corresponding root location confidence score SimScore. As an exemplary implementation, the root location confidence score SimScore can be calculated as the weed image features extracted by the teacher model Teacher(·) from the unlabeled training images. Compared to the corresponding weed image features of the labeled training image corresponding to the unlabeled training image. Similarity between them, for example, calculating the SimScore confidence score of the root location as a cosine similarity: Furthermore, such as Figure 7 As shown, process 70 may further include: at box 708, determining whether the root location confidence score SimScore for the inferred weed root coordinates is higher than the root location confidence threshold ξ. According to an embodiment of the invention, inferences where the root location confidence score SimScore is higher than the root location confidence threshold ξ can be retained as pseudo-labels for the corresponding unlabeled training images. According to an embodiment of the invention, for inferences where the root location confidence score SimScore is not higher than the root location confidence threshold ξ, such as... Figure 7 As shown, at box 709, these inferences are discarded instead of being treated as pseudo-labels.

[0116] like Figure 7 As shown, process 70 may further include: at box 710, using the inferences regarding object categories where the classification confidence score ConfScore determined at box 706 is higher than the classification confidence threshold τ, and the inferences regarding weed root coordinates where the root location confidence score SimScore determined at box 708 is higher than the root location confidence threshold ξ, as pseudo-labels to label the unlabeled image training set, thereby obtaining a pseudo-labeled image training set. As an exemplary implementation, the teacher model Teacher(·) can label the object category in the corresponding unlabeled training image based on the inferences regarding object categories where the classification confidence score ConfScore determined at box 706 is higher than the classification confidence threshold (e.g., labeling the object as a crop or a weed), and can label the weed root location in the corresponding unlabeled training image based on the inferences regarding the root location confidence score SimScore determined at box 708 being higher than the root location confidence threshold.

[0117] like Figure 7As shown, process 70 may further include: at box 712, the student model performs multi-task learning based on the comprehensive loss L on the labeled image training set and the pseudo-labeled image training set to obtain a second optimized object detection framework. In an embodiment of the present invention, the student model Student(·) can perform supervised learning based on the comprehensive loss L on the labeled image training set and unsupervised learning based on the comprehensive loss L on the pseudo-labeled image training set to obtain the second optimized object detection framework.

[0118] According to embodiments of the present invention, through semi-supervised learning, the object detection framework can be trained on a larger training set, thereby learning more general image features and thus having higher accuracy, generalization ability and robustness.

[0119] According to an embodiment of the present invention, before the teacher model Teacher(·) and the student model Student(·) perform semi-supervised learning on labeled and unlabeled image training sets based on a teacher-student framework, image augmentation can be performed on the labeled and unlabeled image training sets first.

[0120] According to embodiments of the present invention, different image enhancement algorithms can be used to enhance both labeled and unlabeled image training sets. According to an embodiment of the present invention, a first image enhancement algorithm can be used to enhance the labeled image training set. Specifically, the first image enhancement algorithm is used to process each labeled training image in the labeled image training set to obtain enhanced labeled training images, thereby obtaining a first enhanced image training set, which includes the enhanced labeled training images. According to an embodiment of the present invention, a second image enhancement algorithm, different from the first image enhancement algorithm, can be used to enhance the unlabeled image training set. Specifically, the second image enhancement algorithm is used to process each unlabeled training image in the unlabeled image training set to obtain enhanced unlabeled training images, thereby obtaining a second enhanced image training set, which includes the enhanced unlabeled training images. As an exemplary implementation, the first image enhancement algorithm may include brightness adjustment, contrast adjustment, image cropping, and image flipping, while the second image enhancement algorithm may only include brightness adjustment and contrast adjustment.

[0121] According to an embodiment of the present invention, the teacher model Teacher(·) can be configured to employ a target detection framework 20 finely tuned through multi-task learning on a first enhanced image training set based on a comprehensive loss L. It can also be configured to perform bounding box detection and object classification tasks, as well as root coordinate regression tasks, on a second enhanced image training set, and to filter the obtained inferences based on confidence levels to obtain pseudo-labels for the second enhanced image training set. Thus, according to an embodiment of the present invention, these pseudo-labels can be used to label each enhanced unlabeled training image in the second enhanced image training set to obtain a pseudo-labeled image training set. For the student model Student(·), according to an embodiment of the present invention, it can be configured to be trained on this pseudo-labeled image training set and the first enhanced image training set based on a comprehensive loss L.

[0122] According to embodiments of the present invention, the generalization ability of the target detection framework can be further improved through image enhancement.

[0123] In addition, as mentioned above Figures 5-7 As described, the method for crop detection and weed root location provided by the present invention can accurately locate the root position of weeds. Based on this, during weeding operations, a laser beam can be precisely emitted towards the point indicated by the root coordinates, thereby reducing the energy consumption of weeding operations.

[0124] Figure 8 The architecture of the teacher-learning framework according to an embodiment of the present invention is shown in general.

[0125] like Figure 8 As shown, teacher model 802 and student model 804 are used in unlabeled data (e.g., as mentioned above). Figure 3 , Figure 7 The unlabeled image training set described above and labeled data (e.g., as mentioned above) Figure 3 , Figure 7 The training is performed on the labeled image training set described herein.

[0126] According to an embodiment of the present invention, labeled data can be input into teacher model 802. Subsequently, as shown in box 8022, teacher model 802 extracts image features from the labeled data and for objects marked as weeds in the labeled data, and stores the obtained weed feature vectors in weed feature vector library 806.

[0127] According to an embodiment of the present invention, unlabeled data is also input into teacher model 802. Subsequently, as shown in box 8024, teacher model 802 performs bounding box detection and object classification tasks on the unlabeled data, and performs bounding box filtering based on the corresponding classification confidence score ConfScore, retaining inferences with a classification confidence score ConfScore higher than a threshold τ (i.e., ConfScore > τ) as pseudo-labels for the unlabeled data. For inferences satisfying ConfScore > τ, as shown in box 8026, it is further determined whether the inferred object category is weeds. If the object is determined to be weeds at box 8026, then, as shown in box 8028, teacher model 802 further performs root coordinate regression on the image features corresponding to the object, and performs weed root coordinate filtering based on the corresponding root position confidence score SimScore, retaining inferences with a root position confidence score SimScore higher than a threshold ξ (i.e., SimScore > ξ) as pseudo-labels for the unlabeled data. Figure 8 As shown, pre-stored weed feature vectors can be obtained from a weed vector library, and the similarity between these vectors and the feature vectors of the corresponding image features of objects inferred as weeds by teacher model 802 can be calculated as the root location confidence score (SimScore). The process described here of teacher model 802 performing bounding box detection and object classification tasks and root coordinate regression tasks on unlabeled data to obtain and filter pseudo-labels is similar to the process described above. Figure 3 , Figure 7 (For example, the descriptions in boxes 702-710) correspond to this.

[0128] According to embodiments of the present invention, such as Figure 8 As shown, labeled data is input into student model 804, and student model 804 performs supervised learning based on the real labels of the labeled data; and unlabeled data is input into student model 804, and student model 804 performs unsupervised learning based on the pseudo labels of the unlabeled data. The training process of the student model on labeled and unlabeled data described here is the same as described above. Figure 3 , Figure 7 (For example, the description in box 712) corresponds to this.

[0129] Furthermore, the inventors discovered that single-stage models, especially the YOLO series models, outperform multi-stage detection models such as Faster-RCNN and SSD in tasks requiring high-speed inference and competitive accuracy. Therefore, the object detection framework proposed in this invention (e.g., referred to above) Figure 2 The described object detection framework 20, reference Figure 3 The described object detection framework 304, and the reference Figures 5-6The object detection framework used in methods 50 and 60 described herein can preferably be based on a single-stage object detection framework, such as YOLOv7, YOLOv8, and YOLOv10. The inventors have also found that anchor-free object detection frameworks (such as YOLOv8 and YOLOv10) may exhibit drift in root prediction, while anchor-bound object detection frameworks (such as YOLOv7) effectively prevent drift by confining loss calculations within their respective grids. Therefore, the object detection framework proposed in this invention can be further preferably based on a single-stage anchor-bound object detection framework (such as YOLOv7). Based on this, the object detection framework can be optimized via semi-supervised learning, thereby further improving model performance.

[0130] Figure 9 The diagram illustrates a comparison of the root distance errors between the inferred weed root locations and the actual weed root locations, presented by a YOLOv7-based target detection framework and a YOLOv8-M-based target detection framework according to embodiments of the present invention. It further shows that these two target detection frameworks are implemented via semi-supervised learning (e.g., ...). Figure 9 (as illustrated by the shaded bar chart) and unsupervised learning (such as...) Figure 9 The comparison of weed root distance errors under the condition shown in the blank bar chart is as follows.

[0131] like Figure 9 As shown, compared to the YOLOv8-M-based object detection framework, the YOLOv7-based object detection framework has a significantly lower error in the distance to weed roots, and as... Figure 9 As shown, regardless of whether the target detection framework is based on YOLOv7 or YOLOv8-M, it exhibits lower weed root distance error after semi-supervised learning. For example, as shown in Table 1 above, without semi-supervised learning, the error between the inferred weed root location and the actual weed root location using the YOLOv7-based target detection framework is 2.4838. After semi-supervised learning, this root distance error is reduced to 2.1485 (e.g., ...). Figure 9 (As shown in the image). This demonstrates that semi-supervised learning can effectively improve the performance of object detection frameworks.

[0132] Figure 10 An exemplary application scenario 1000 of a crop detection and weed root location scheme according to an embodiment of the present invention is illustrated schematically. (Refer to the above text) Figure 3 The described computing device 300 for crop detection and weed root location, reference Figures 5-6 The methods 500 and 600 described for crop detection and weed root location can be found in reference [reference]. Figure 10The described application scenario is implemented in 1000.

[0133] like Figure 10 As shown, at sub-scene 1000A, an automated robot 1002 captures images to obtain field images containing crops and / or weeds. According to an embodiment of the invention, the automated robot 1002 can be implemented as described above. Figure 4 The described weeding equipment 40 is a portion thereof. According to an embodiment of the invention, at sub-scene 1000B, an image sensor 1004 acquires field images captured by an automated robot 1002 to extract image data (such as...). Figure 10 (As shown). Those skilled in the art will understand that the image sensor 1004 according to embodiments of the present invention can be implemented using any image sensor suitable for image acquisition. For image data acquired by the image sensor 1004, such as Figure 10 As shown, in sub-scene 1000C, the neural network 1006 processes these image data to perform crop detection and weed root localization. According to an embodiment of the present invention, the above references... Figure 2 The described object detection framework 20, reference Figure 3 The described object detection framework 304, and the reference Figures 5-6 The object detection frameworks used in methods 50 and 60 described herein can all be implemented using the neural network 1006 described herein. According to embodiments of the present invention, the neural network 1006 can be configured to have a head network that performs bounding box detection and object classification (e.g., as shown in the image). Figure 10 (as shown at midframe 10062) and a head network having a function for performing weed root localization (e.g., as shown in the middle frame 10062) Figure 10 (As shown at position 10064 in the middle frame). Figure 10 The output of the neural network 1006 performing crop detection and weed root localization is shown in subscene 1000C. As shown, the neural network 1006 can output the category and probability of each object in the image data, as well as the root location of the object classified as a weed.

[0134] To further illustrate the beneficial effects of the crop detection and weed root location scheme proposed in this invention, the following will combine... Figures 11-12 To further compare the results of crop detection and weed location using a conventional target detection framework with the results of crop detection and weed root location using the crop detection and weed root location scheme proposed in this invention.

[0135] exist Figure 11The first row shows the crop detection and weed localization results obtained by processing different field image data using a conventional object detection framework; the second row shows the crop detection and weed localization results obtained by processing different field image data using a crop detection and weed root localization scheme according to an embodiment of the present invention; the third row shows the corresponding ground truth labels indicating the category of the object and the location of the weed roots. Figure 11 The vertical comparison of each column shows that, compared with the conventional target detection framework, the crop detection and weed root localization scheme according to the embodiment of the present invention can well match the object category and weed root location inferred by crop detection and weed root localization with the true label in the third row. As an example, as shown in column six, the inference results using the conventional object detection framework are (weed, 0.19), (weed, 0.21), (weed, 0.17), (weed, 0.24), (weed, 0.22), and (weed, 0.32), along with the corresponding bounding box positions. The inference results using the crop detection and weed root localization scheme according to an embodiment of the present invention are (weed, 0.73), (weed, 0.24), and the corresponding root positions. The ground truth labels indicate two objects classified as weeds and their root positions. It can be seen that the inference results (weed, 0.73), (weed, 0.24), and the corresponding root positions using the crop detection and weed root localization scheme according to an embodiment of the present invention significantly match the ground truth labels better. Therefore, the crop detection and weed root localization scheme proposed in this invention can accurately detect and distinguish crops and weeds in farmland scenes.

[0136] Figure 12 The location results are shown in more detail, where diamond-shaped dots indicate the inferred location and circular dots indicate the actual location of weed roots. Figure 12 In the diagram, the first row shows the weed locations inferred using a conventional target detection framework, and the second row shows the weed root locations inferred using a crop detection and weed root localization scheme according to an embodiment of the present invention. Figure 12 As shown, the inferred locations shown in the first row deviate from the actual weed root locations, while the inferred locations shown in the second row match the actual weed root locations well. Therefore, the crop detection and weed root location scheme proposed in this invention can accurately locate the roots of weeds.

[0137] Based on this, when carrying out weeding operations, the weeding scheme according to embodiments of the present invention can accurately emit lasers to the roots of weeds, thereby efficiently removing weeds in the field, effectively preventing damage to nearby crops, and significantly reducing energy consumption.

[0138] One or more aspects of at least one embodiment of the present invention can be implemented as a computer program product, wherein the computer program product includes computer instructions. Program code is applied to the input instructions to perform the functions described herein and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, the present invention also provides a computing device comprising any system having a processor, such as, for example, a digital signal processor (DSP), a microcontroller, an application-specific integrated circuit (ASIC), or a microprocessor. The computing device also includes a memory coupled to the processor for storing program code. The program code can be implemented using a high-level procedural programming language or an object-oriented programming language.

[0139] One or more aspects of at least one embodiment of the present invention may be implemented by representational instructions stored on a machine-readable medium representing various logic in a processor, which, when read by a machine, cause the machine to produce logic for performing the techniques described herein.

[0140] This invention provides a technical solution for crop detection and weed root localization, which is based on the proposed target detection framework for crop and / or weed detection and weed root localization. In this invention, the target detection framework can perform both object detection and classification tasks as well as root coordinate regression tasks, thus accurately detecting and distinguishing crops and weeds in farmland scenes and precisely locating weed roots. Furthermore, this invention proposes to further optimize the proposed target framework using multi-task learning and a teacher-student framework for semi-supervised learning. This allows the proposed target detection framework to better adapt to crop detection and weed root localization, learn more general image features, and therefore has higher detection accuracy, generalization ability, and robustness to new data. Based on this, during weeding, lasers can be precisely emitted towards the roots of weeds, thus efficiently removing weeds from the field, effectively preventing damage to nearby crops, and significantly reducing energy consumption.

[0141] The preferred embodiments of the present invention have been described in detail above. However, it should be understood that the present invention can be implemented and modified in various ways without departing from its broad spirit and scope. Those skilled in the art can make many modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of the prior art should fall within the protection scope defined by the claims of the present invention.

Claims

1. A computing device for crop detection and weed root location, comprising: Computing resources; as well as A target detection framework, comprising a backbone network, a neck network, a head network, and an additional head network, is used to utilize the computing resources for crop detection and weed root localization. The target detection framework is configured as follows: Receive images from scenes including crops and weeds; Perform bounding box detection and object classification tasks to detect crops and weeds from the image; and Perform a root coordinate regression task to determine the coordinates of weed roots from the image. The object detection framework is further configured to employ a teacher-student framework, where a teacher model and a student model are optimized through semi-supervised learning on labeled and unlabeled image training sets. The teacher model is the object detection framework based on a comprehensive loss on the labeled image training set. L The teacher model, obtained through multi-task learning, is configured as follows: The bounding box detection and object classification tasks are performed on the unlabeled image training set to obtain inferred crop and weed detection results, which have corresponding classification confidence levels. The root coordinate regression task is performed on the unlabeled image training set to obtain inferred weed root coordinates, which have corresponding root location confidence scores; and The output includes the inferred crop and weed detection results with classification confidence scores higher than the classification threshold, and the inferred weed root coordinates with root location confidence scores higher than the root location confidence threshold, which are used as pseudo-labels for the unlabeled image training set. Furthermore, the student model is obtained by configuring the object detection framework with reference to the model parameters of the teacher model, and the student model is configured as follows: Training is performed on the pseudo-labeled image training set and the labeled image training set based on the comprehensive loss, wherein the pseudo-labeled image training set is obtained by labeling the unlabeled image training set using the pseudo labels.

2. The computing device as claimed in claim 1, wherein, The target detection framework is configured as follows: The root coordinate regression task is performed based on image features corresponding to the weeds detected in the image to determine the root coordinates of the weeds.

3. The computing device as claimed in claim 2, wherein, The target detection framework's execution of the root coordinate regression task and the root coordinate regression loss L reg Relatedly, the root coordinate regression loss L reg Based on the Euclidean distance between the actual coordinates of the weed roots and the determined coordinates of the weed roots.

4. The computing device as claimed in claim 3, wherein, The comprehensive loss L Based on classification loss L cls Bounding box detection loss L bbox and the root coordinate regression loss L reg The object detection framework's performance on the bounding box detection and object classification tasks is related to the bounding box detection loss. L bbox and the classification loss L cls Related.

5. The computing device as claimed in claim 4, wherein, The comprehensive loss L Calculated as: L = α L cls + β L bbox + γ L reg ， Among them, hyperparameters α , β , γ Indicates the classification loss L cls The bounding box detection loss L bbox and root coordinate regression loss L reg The weights in the overall loss.

6. The computing device as claimed in claim 5, wherein, Hyperparameters α The weight is set to be greater than or equal to the threshold, and the hyperparameter is... β , γ Based on hyperparameters α The settings are adjusted and traversed through multi-task learning performed by the object detection framework.

7. The computing device as claimed in claim 1, in, The labeled image training set is enhanced using a first image enhancement algorithm to obtain a first enhanced image training set, and the unlabeled image training set is enhanced using a second image enhancement algorithm to obtain a second enhanced image training set. The second image enhancement algorithm is different from the first image enhancement algorithm. The teacher model is the object detection framework based on the comprehensive loss on the first enhanced image training set. L The data obtained through the multi-task learning process is configured to perform the bounding box detection and object classification tasks, as well as the root coordinate regression task, on the second enhanced image training set to output the pseudo-labels for the second enhanced image training set. Furthermore, the pseudo-annotated image training set is obtained by annotating the second enhanced image training set using the pseudo-labels, and the student model is configured to be trained on the pseudo-annotated image training set and the first enhanced image training set based on the comprehensive loss.

8. The computing device as claimed in claim 1, wherein, The target detection framework is based on a single-stage target detection framework with anchored frames.

9. A weeding device, comprising: The computing device as described in any one of claims 1-8; Camera, used to capture the image; as well as A laser emitting device used to emit a laser beam toward the roots of weeds based on the determined coordinates of the weed roots.

10. The weeding device as described in claim 9, further comprising: A mobile device for moving the weeding equipment.

11. A method for detecting crops and locating weed roots, comprising: The object detection framework receives images from a scene including crops and weeds, and the object detection framework includes a backbone network, a neck network, a head network, and an additional head network. The target detection framework performs bounding box detection and object classification tasks to detect crops and weeds from the image; as well as The target detection framework performs a root coordinate regression task to determine the coordinates of weed roots from the image. The object detection framework further employs a teacher-student framework, using a teacher model and a student model to perform semi-supervised learning on labeled and unlabeled image training sets to obtain an optimized object detection framework. The teacher model is derived from the object detection framework based on a comprehensive loss. L The student model is obtained by performing multi-task learning on the labeled image training set, and is configured by the object detection framework with reference to the model parameters of the teacher model. Furthermore, the optimization using a teacher-student framework on both labeled and unlabeled image training sets through semi-supervised learning includes: The teacher model performs the bounding box detection and object classification tasks on the unlabeled image training set to obtain inferred crop and weed detection results, which have corresponding classification confidence levels. The teacher model performs the root coordinate regression task on the unlabeled image training set to obtain inferred weed root coordinates, which have corresponding root location confidence. The crop and weed detection results inferred with classification confidence scores higher than the classification threshold, and the weed root coordinates inferred with root location confidence scores higher than the root location confidence threshold, are used as pseudo-labels for the unlabeled image training set. This is used to label the unlabeled image training set, resulting in a pseudo-labeled image training set. The student model performs the multi-task learning on the pseudo-labeled image training set and the labeled image training set based on the comprehensive loss to obtain the optimized object detection framework.

12. The method of claim 11, wherein the root coordinate regression task is performed by the object detection framework, comprising: The target detection framework performs the root coordinate regression task based on image features corresponding to the weeds detected in the image to determine the root coordinates of the weeds.

13. The method of claim 12, wherein, The object detection framework performs the root coordinate regression task and the root coordinate regression loss. L reg Relatedly, the root coordinate regression loss L reg Based on the Euclidean distance between the actual coordinates of the weed roots and the determined coordinates of the weed roots.

14. The method of claim 13, wherein, The comprehensive loss L Based on classification loss L cls Bounding box detection loss L bbox and the root coordinate regression loss L reg The object detection framework performs the bounding box detection and object classification tasks with the bounding box detection loss. L bbox and the classification loss L cls Related.

15. The method of claim 14, wherein, The comprehensive loss L Calculated as: L = α L cls + β L bbox + γ L reg Among them, hyperparameters α , β , γ Indicates the classification loss L cls The bounding box detection loss L bbox and root coordinate regression loss L reg The weights in the overall loss.

16. The method of claim 15, wherein, Hyperparameters α The weight is set to be greater than or equal to the threshold, and the hyperparameter is... β , γ Based on hyperparameters α The settings are adjusted and traversed through multi-task learning performed by the object detection framework.

17. The method as described in claim 11, in, The labeled image training set is enhanced using a first image enhancement algorithm to obtain a first enhanced image training set, and the unlabeled image training set is enhanced using a second image enhancement algorithm to obtain a second enhanced image training set. The second image enhancement algorithm is different from the first image enhancement algorithm. The teacher model is the object detection framework based on the comprehensive loss on the first enhanced image training set. L The data obtained through the multi-task learning process is configured to perform the bounding box detection and object classification tasks, as well as the root coordinate regression task, on the second enhanced image training set to output the pseudo-labels for the second enhanced image training set. Furthermore, the pseudo-annotated image training set is obtained by annotating the second enhanced image training set using the pseudo-labels, and the student model is configured to be trained on the pseudo-annotated image training set and the first enhanced image training set based on the comprehensive loss.

18. A machine-readable medium comprising machine-executable instructions stored thereon, which, when executed, cause a machine to perform the method as claimed in any one of claims 11 to 17.

19. A computer program product comprising computer instructions that, when executed, implement the method as described in any one of claims 11 to 17.

Images