Crop detection and weed root positioning apparatus and method

Abstract

The invention provides crop detection and weed root localization apparatus and methods. A computing device for crop detection and weed root localization is described, the computing device comprising: a computing resource; and a target detection framework invoking computing resources for crop detection and weed root localization, where the target detection framework is configured to: receive an image acquired from a scene including a crop and a weed; performing bounding box detection and object classification tasks to detect crops and weeds from the image; and performing a root coordinate regression task to determine weed root coordinates from the image.

Description

Technical Field

[0001] The present invention generally relates to the field of computer vision technology, and more particularly to an apparatus and method for performing crop detection and weed root location. Background Art

[0002] Weed control is an important challenge in modern agriculture. Weeds compete with crops for key resources such as water, nutrients, and sunlight, significantly affecting crop yield and quality. With the rapid development of deep learning and computer vision technology, intelligent weed control technology based on image recognition has gradually become a research hotspot.

[0003] The deep learning-based target detection framework has been widely used in various object detection and positioning. In particular, the development of deep convolutional neural networks has greatly improved the classification, segmentation and detection performance of images, and can identify specific objects in images, classify objects contained in images, and locate specific objects in images. Applying this deep learning-based target detection framework to the agricultural field and accurately identifying the location of weeds through intelligent agricultural machinery can save a lot of manpower, material and financial resources for subsequent weeding.

[0004] However, the current target detection framework used for field weed detection and localization is based on the classification and localization of bounding boxes. It detects and localizes weeds in images by providing bounding boxes. Although this weed detection and localization method based on bounding boxes has achieved good results in weed detection, it can only roughly determine the range of weeds in terms of weed localization due to the limitation of bounding box size. This results in low accuracy in weed localization, and the weed removal device based on this target detection framework cannot efficiently remove field weeds. Summary of the invention

[0005] The present invention proposes a technical solution for crop detection and weed root location, which performs crop and / or weed detection and weed root location based on the target detection framework proposed in the present invention, and can at least solve the above-mentioned problems of low weed location accuracy and low weeding efficiency. In the present invention, the target detection framework can perform both object detection and classification tasks and root coordinate regression tasks, so it can accurately detect and distinguish crops and weeds in farmland scenes, and accurately locate the roots of weeds. On this basis, when weeding, lasers can be accurately emitted to the roots of weeds, so that weeds in the field can be efficiently removed, damage to nearby crops can be effectively prevented, and energy consumption can be significantly reduced.

[0006] In a first aspect of the present invention, a computing device for crop detection and weed root location is provided, comprising: computing resources; and a target detection framework, calling the computing resources to perform crop detection and weed root location, wherein the target detection framework is configured to: receive an image collected from a scene including crops and weeds; perform bounding box detection and object classification tasks to detect crops and weeds from the image; and perform a root coordinate regression task to determine the weed root coordinates from the image.

[0007] The computing device as described above, wherein the target detection framework is configured to: perform the root coordinate regression task based on image features corresponding to the weeds detected in the image to determine the weed root coordinates.

[0008] A computing device as described in any of the above items, wherein the execution of the root coordinate regression task by the object detection framework is related to the root coordinate regression loss L reg Associated, the root coordinate regression loss L reg Based on the Euclidean distance between the true weed root coordinates and the determined weed root coordinates.

[0009] A computing device as described in any of the above items, wherein the target detection framework is further configured to: optimize through multi-task learning based on a comprehensive loss L, wherein the comprehensive loss L is based on the classification loss L cls , bounding box detection loss L bbox , and the root coordinate regression loss L reg , the object detection framework performs the bounding box detection and object classification tasks with the bounding box detection loss L bbox and the classification loss L cls Associated.

[0010] A computing device as described in any one of the above items, wherein the comprehensive loss L is calculated as: L = α·L cls +β·L bbox +γ·L reg , where the hyperparameters α, β, and γ indicate the classification loss L cls , the bounding box detection loss L bbox , and the root coordinate regression loss L reg The weight in the composite loss.

[0011] A computing device as described in any of the above items, wherein the hyperparameter α is set to be greater than or equal to a threshold weight, and the hyperparameters β and γ are adjusted based on the setting of the hyperparameter α and multi-task learning is performed via the target detection framework.

[0012] A computing device as described in any of the above items, wherein the target detection framework is further configured to: further adopt a teacher-student framework to optimize the teacher model and the student model through semi-supervised learning on a labeled image training set and an unlabeled image training set.

[0013] A computing device as described in any of the above items, wherein the teacher model is obtained by the target detection framework performing the multi-task learning on the labeled image training set based on the comprehensive loss L, and wherein the teacher model is configured to: perform the bounding box detection and object classification tasks on the unlabeled image training set to obtain inferred crop and weed detection results, and the inferred crop and weed detection results have corresponding classification confidences; perform the root coordinate regression task on the unlabeled image training set to obtain inferred weed root coordinates, and the inferred weed root coordinates have corresponding root location confidences; and output the inferred crop and weed detection results whose classification confidences are higher than a classification threshold confidence, and the inferred weed root coordinates whose root location confidences are higher than a root location confidence threshold, as pseudo labels for the unlabeled image training set.

[0014] A computing device as described in any of the above items, wherein the student model is obtained by configuring the target detection framework with reference to the model parameters of the teacher model, and wherein the student model is configured to: be trained on a pseudo-annotated image training set and the annotated image training set based on the comprehensive loss, wherein the pseudo-annotated image training set is obtained by annotating the unlabeled image training set with the pseudo-label.

[0015] A computing device as described in any of the above items, wherein the labeled image training set is enhanced by a first image enhancement algorithm to obtain a first enhanced image training set, and the unlabeled image training set is enhanced by a second image enhancement algorithm to obtain a second enhanced image training set, the second image enhancement algorithm is different from the first image enhancement algorithm, wherein the teacher model is obtained by the target detection framework performing the multi-task learning on the first enhanced image training set based on the comprehensive loss L, and is configured to perform the bounding box detection and object classification tasks and the root coordinate regression task on the second enhanced image training set to output the pseudo-label of the second enhanced image training set, and wherein the pseudo-labeled image training set is obtained by annotating the second enhanced image training set with the pseudo-label, and the student model is configured to be trained on the pseudo-labeled image training set and the first enhanced image training set based on the comprehensive loss.

[0016] A computing device as described in any of the above items, wherein the object detection framework is based on a single-stage object detection framework with an anchor box.

[0017] In a second aspect of the present invention, a weeding device is provided, comprising: a computing device as described in any one of the above; a camera for capturing the image; and a laser emitting device for emitting a laser beam to the roots of weeds based on the determined coordinates of the roots of weeds.

[0018] The weeding device as described in any one of the above items further comprises: a moving device for moving the weeding device.

[0019] In a third aspect of the present invention, a method for crop detection and weed root location is provided, comprising: receiving, by a target detection framework, an image collected from a scene including crops and weeds; performing, by the target detection framework, bounding box detection and object classification tasks to detect crops and weeds from the image; and performing, by the target detection framework, a root coordinate regression task to determine the weed root coordinates from the image.

[0020] The method as described in any of the above items, wherein the target detection framework performs the root coordinate regression task, comprising: the target detection framework performs the root coordinate regression task based on image features corresponding to the weeds detected in the image, so as to determine the weed root coordinates.

[0021] A method as described in any of the above, wherein the target detection framework performs the root coordinate regression task and the root coordinate regression loss L reg Associated, the root coordinate regression loss L reg Based on the Euclidean distance between the true weed root coordinates and the determined weed root coordinates.

[0022] A method as described in any of the above items, wherein the target detection framework is a first optimized target detection framework obtained by performing multi-task learning on the target detection framework based on a comprehensive loss L, wherein the comprehensive loss L is based on a classification loss L cls , bounding box detection loss L bbox , and the root coordinate regression loss L reg , and the object detection framework performs the bounding box detection and object classification tasks with the bounding box detection loss L bbox and the classification loss L cls Associated.

[0023] A method as described in any one of the above, wherein the comprehensive loss L is calculated as: L = α·L cls +β·L bbox +γ·L reg, where the hyperparameters α, β, and γ indicate the classification loss L cls , the bounding box detection loss L bbox , and the root coordinate regression loss L reg The weight in the composite loss.

[0024] A method as described in any of the above items, wherein the hyperparameter α is set to be greater than or equal to a threshold weight, and the hyperparameters β and γ are adjusted based on the setting of the hyperparameter α and multi-task learning is performed via the target detection framework.

[0025] A method as described in any of the above items, wherein the target detection framework is a second optimized target detection framework, and the second optimized target detection framework is obtained by further adopting a teacher-student framework in the target detection framework to perform semi-supervised learning on a labeled image training set and an unlabeled image training set by using a teacher model and a student model.

[0026] A method as described in any of the above items, wherein the teacher model is obtained by the target detection framework performing the multi-task learning on the labeled image training set based on the comprehensive loss L, and the student model is obtained by configuring the target detection framework with reference to the model parameters of the teacher model, and wherein the optimization through semi-supervised learning on the labeled image training set and the unlabeled image training set using the teacher-student framework includes: performing the bounding box detection and object classification tasks on the unlabeled image training set by the teacher model to obtain inferred crop and weed detection results, the inferred crop and weed detection results having corresponding classification confidences by the teacher model The method comprises the steps of: performing the root coordinate regression task on the unlabeled image training set to obtain inferred weed root coordinates, wherein the inferred weed root coordinates have corresponding root location confidences; using the inferred crop and weed detection results whose classification confidences are higher than the classification threshold confidence, and the inferred weed root coordinates whose root location confidences are higher than the root location confidence threshold, as pseudo labels of the unlabeled image training set, and annotating the unlabeled image training set to obtain a pseudo-annotated image training set; and performing the multi-task learning on the pseudo-annotated image training set and the labeled image training set based on the comprehensive loss by the student model to obtain the second optimized object detection framework.

[0027] A method as described in any of the above items, wherein the labeled image training set is enhanced by a first image enhancement algorithm to obtain a first enhanced image training set, and the unlabeled image training set is enhanced by a second image enhancement algorithm to obtain a second enhanced image training set, the second image enhancement algorithm is different from the first image enhancement algorithm, wherein the teacher model is obtained by the target detection framework performing the multi-task learning on the first enhanced image training set based on the comprehensive loss L, and is configured to perform the bounding box detection and object classification tasks and the root coordinate regression task on the second enhanced image training set to output the pseudo-label of the second enhanced image training set, and wherein the pseudo-labeled image training set is obtained by annotating the second enhanced image training set with the pseudo-label, and the student model is configured to be trained on the pseudo-labeled image training set and the first enhanced image training set based on the comprehensive loss.

[0028] In a fourth aspect of the present invention, a machine-readable medium is provided, comprising machine-executable instructions stored thereon, wherein when the instructions are executed, the machine executes any of the methods described above.

[0029] In a fifth aspect of the present invention, a computer program product is provided, comprising computer instructions, which implement the method as described in any one of the above when executed. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Some implementations of the present invention will be described below by way of example only and with reference to the accompanying drawings. In the drawings, each identical or nearly identical component shown throughout the various drawings may be represented by the same reference numeral. For clarity, not every component is labeled in each figure. In the drawings:

[0031] Figure 1 A schematic diagram showing an object detection framework as an illustrative example;

[0032] Figure 2 A schematic diagram showing a target detection framework according to an embodiment of the present invention;

[0033] Figure 3 A schematic block diagram showing a computing device for crop detection and weed root location according to an embodiment of the present invention;

[0034] Figure 4 A schematic block diagram showing a weeding device according to an embodiment of the present invention;

[0035] Figure 5A schematic flow chart showing a method for crop detection and weed root location according to an embodiment of the present invention;

[0036] Figure 6 A schematic flow chart showing a method for crop detection and weed root location according to another embodiment of the present invention;

[0037] Figure 7 A schematic flow chart showing a process of training an object detection framework based on a teacher-student framework according to an embodiment of the present invention;

[0038] Figure 8 A schematic diagram showing the architecture of a teacher-student framework according to an embodiment of the present invention;

[0039] Fig. 9 FIG. 1 shows a comparison of weed root positioning errors of different target detection frameworks according to an embodiment of the present invention;

[0040] Fig.10 An exemplary application scenario of a solution for crop detection and weed root location according to an embodiment of the present invention is shown;

[0041] Figure 11-Figure 12 The implementation effects of the crop detection and weed root location solution according to the embodiment of the present invention are shown. DETAILED DESCRIPTION

[0042] Next, exemplary embodiments of the present invention will be described in detail with reference to the attached drawings, wherein, unless otherwise expressly stated, the same or similar reference numerals in different drawings represent the same or similar elements. In addition, it should be noted that the exemplary embodiments of the present invention may perform the steps of the corresponding method in a different order than that shown and described in the specification. In some embodiments of the present invention, the method may include more or fewer steps than those described in the specification and shown in the attached drawings. In addition, a single step used in some embodiments of this specification may be decomposed into multiple steps in other embodiments, or multiple steps used in some embodiments of this specification may be combined into a single step in other embodiments.

[0043] In the following description, numerous specific details are set forth. However, it should be understood that embodiments of the present invention may be practiced without these specific details. In other instances, well-known circuits, structures, and techniques are not shown in detail to avoid obscuring the understanding of this description.

[0044] References in the specification to "one embodiment," "an embodiment," "an example embodiment," etc. indicate that the described embodiment may include a particular feature, structure, or characteristic, but not every embodiment may include the particular feature, structure, or characteristic. Furthermore, such phrases do not necessarily refer to the same embodiment. Furthermore, when a particular feature, structure, or characteristic is described in conjunction with an embodiment, it is considered to be within the knowledge of those skilled in the art to affect such feature, structure, or characteristic in conjunction with other embodiments, whether or not explicitly described.

[0045] For purposes of this disclosure, the phrase "A and / or B" means (A), (B), or (A and B). For purposes of this disclosure, the phrase "A, B, and / or C" means (A), (B), (C), (A and B), (A and C), (B and C), or (A, B, and C).

[0046] In the following description and claims, the terms "coupled" and "connected" and their derivatives may be used. It should be understood that these terms are not intended to be synonymous with each other. On the contrary, in certain embodiments, "connected" may be used to indicate that two or more elements are in direct physical or electrical contact with each other. "Coupled" may mean that two or more elements are in direct physical or electrical contact. However, "coupled" may also mean that two or more elements are not in direct contact with each other, but still cooperate or interact with each other.

[0047] The embodiment of the present invention provides a computing device for crop detection and weed root location, which can be used to detect crops and weeds contained in a farmland and locate the weed roots. The crop detection and weed root location technical solution according to the embodiment of the present invention is based on a target detection framework.

[0048] Figure 1 An example of a target detection framework is schematically shown in FIG. Figure 1 As shown, basic components of the target detection framework 10 include a backbone network 102, a neck network 104 and a head network 106, which are usually implemented using neural networks.

[0049] In the object detection framework 10 , the backbone network 102 can generally be implemented as a convolutional neural network with pre-trained parameters that has been pre-trained on a large dataset, which is used to extract image features from an input image.

[0050] In the target detection framework 10, the neck network 104, as the middle part connecting the backbone network 102 and the head network 106, can be configured to process the feature map extracted by the backbone network 102 and combine the feature maps from different levels, thereby providing the target detection framework 10 with multi-scale target detection capabilities.

[0051] In the target detection framework 10, the head network 106 can receive the processed feature maps from the neck network 104, and perform target detection and positioning based on these processed feature maps. In general, the head networks of different frameworks may be different due to the specific requirements of their target detection tasks. As an exemplary implementation, in a single-stage target detection framework such as YOLO (You Only Look Once) and SSD (Single Shot MultiBox Detector), each grid unit of the head network 106 predicts multiple bounding boxes (e.g., anchor boxes of a specific size) based on the received feature map, and fits the bounding box to the actual target through a regression process based on a loss function, thereby directly outputting the position of the bounding box, the category of the object contained in the bounding box and its probability, and the corresponding confidence from the feature map.

[0052] According to an embodiment of the present invention, the object detection framework 10 can be trained to detect and locate weeds. As an example, the object detection framework 10 can be trained on a large number of training images that contain and / or do not contain weeds (e.g., one or more types of weeds) so that it can be used to detect and locate specific one or more categories of weeds (and / or crops).

[0053] However, as described above, the target detection framework 10 detects and locates one or more categories of weeds (and / or crops) based on the bounding box, through the bounding box regression and object classification process, so the weed position output by the target detection framework 10 is actually the position of the bounding box, that is, it can only roughly determine the range of the weeds, but cannot further accurately locate the root position of the weeds. When the weeding device weeds based on the weed positioning result of the target detection framework 10, in order to ensure the weeding effect, it needs to aim at a larger range to emit laser, which will result in greater energy consumption and may cause damage to nearby crops.

[0054] Figure 2 The target detection framework 20 according to an embodiment of the present invention is schematically shown. Figure 2 As shown, compared to Figure 1 The target detection framework 10 shown in FIG. 20 includes an additional head network component (e.g., Figure 2 ) and Figure 1 The bounding box regression and object classification processes performed by the head network 106 shown in FIG. Figure 2 The head network 208 shown in FIG. 2 can perform a point regression process that fits the predicted points to the true points.

[0055] According to an embodiment of the present invention, in the target detection framework 20, the head network 206 can receive the processed feature map from the neck network 204, and infer the position of the bounding box, the category of the object contained in the bounding box and its probability, etc. based on the bounding box regression and object classification process; the head network 208 can receive the processed feature map from the neck network 204, and predict the position of the point based on the point regression process.

[0056] According to an embodiment of the present invention, in the target detection framework 20, the bounding box regression and object classification processes performed by the head network 206 may be associated with the bounding box loss and the classification loss, wherein the bounding box loss indicates the difference between the position of the inferred bounding box and the actual position of the bounding box, and the classification loss indicates the difference between the category of the inferred object and the category of the actual object. According to an embodiment of the present invention, the point regression process performed by the head network 208 may be associated with the point position loss, wherein the point position loss indicates the distance between the position of the point predicted by the head network 208 and the position of the actual point. In an embodiment of the present invention, the target detection framework 20 may perform the bounding box regression and object classification processes and the point position regression process through the head network 206 and the head network 208 based on a loss function including at least the bounding box loss, the classification loss, and the point position loss.

[0057] According to some other embodiments of the present invention, the head network 208 may perform a point regression process based on the inference result of the head network 206. As an exemplary implementation, the head network 208 may perform a point regression process for image features corresponding to the object based on the object category and its probability inferred by the head network 206.

[0058] According to an embodiment of the present invention, the object detection framework 20 can be configured to perform crop detection and weed root location. As an exemplary implementation, the head network 206 of the object detection framework 20 can be configured to detect crops and / or weeds in the image by performing bounding box detection and object classification tasks, and the head network 208 can be configured to determine the root position of the weeds in the image by performing a root coordinate regression task.

[0059] According to an embodiment of the present invention, the head network 206 can be configured to perform bounding box detection and object classification tasks on the feature maps extracted by the backbone network 202 and the neck network 204 via the bounding box regression and object classification process described above to infer the categories of the various objects contained in the image and their probabilities (for example, the object contained in the inferred bounding box is (for example, a crop of a specific category) and the probability that the object is a crop of (for example, a specific category)), and the corresponding confidence. According to an embodiment of the present invention, the head network 206 can perform bounding box detection and object classification tasks with the bounding box detection loss L bboxand classification loss L cls Associated, where the bounding box detection loss L bbox Indicates the gap between the inferred bounding box and the actual bounding box (for example, it can be calculated based on the vertex coordinates of the inferred bounding box and the vertex coordinates of the actual bounding box), the classification loss L cls Indicates the distance between the inferred object category and the actual object category for the corresponding bounding box (for example, it can be calculated based on the inferred category probability distribution and the actual category probability distribution).

[0060] According to an embodiment of the present invention, the head network 208 may be configured to perform a root coordinate regression task on the feature maps extracted by the backbone network 202 and the neck network 204 via the root coordinate regression process described above to infer the root position of the weeds in the image. According to an embodiment of the present invention, the head network 208 may perform the root coordinate regression task with the root coordinate regression loss L reg As an exemplary implementation, the root coordinate regression loss L reg It can be calculated based on the Euclidean distance between the actual weed root coordinates and the inferred weed root coordinates: Where n indicates the number of input images, y i Indicates the actual weed root coordinates in the i-th input image, Indicates the inferred weed root coordinates in the i-th input image.

[0061] According to an embodiment of the present invention, the object detection framework 20 is an end-to-end model that receives an input image at a receiving end and outputs crop detection and weed root positioning results at an output end. As an example, the object detection framework 20 can receive an image captured from a scene containing crops and weeds, and output the category of each object in the scene and its probability (e.g., (crop, 0.8), indicating that the object in the scene is a crop, and the probability that the object is a crop is 0.8) and the root coordinates of the weeds.

[0062] When the target detection framework 20 described above is used to detect crops and locate weed roots, the weeding device can emit lasers centrally to the weed root positions output by the target detection framework 20, which can improve the weeding accuracy, avoid damage to nearby crops, reduce energy consumption, and improve the weeding effect.

[0063] According to an embodiment of the present invention, the head network 208 may be further configured to perform a root coordinate regression task based on the inference of the head network 206. As an exemplary implementation, the head network 206 may be configured to detect crops and weeds from the feature map, and the head network 208 may be configured to perform a root coordinate regression process based on image features corresponding to the weeds detected by the head network 206 to determine the weed root coordinates. As an alternative implementation, in the case where the head network 206 classifies the object in the detection frame as a weed by performing bounding box detection and object classification tasks on the feature map, since the head network 206 has inferred the position of the bounding box (e.g., the vertex coordinates of the bounding box), the head network 208 may obtain the position of the bounding box in which the weeds are detected from the head network 206 and perform the root coordinate regression task based on the positions of these bounding boxes.

[0064] On the one hand, compared with the root coordinate regression task directly on the feature map extracted by the backbone network 202 and the neck network 204, the root coordinate regression process performed on the basis of the weeds detected from the feature map by the head network 206 can further reduce the error of the determined weed root coordinates and improve the accuracy of determining the weed root coordinates. On the other hand, when the head network 206 classifies the object in the detection frame as a weed, the situation that there are crops and / or weeds are blocked by other objects (e.g., crops) in the detection frame can be effectively excluded (because in these cases, the head network 206 will not classify the object in the detection frame as a weed or the probability that the object in the detection frame is a weed is low). That is, in the embodiment of the present invention, the head network 208 further determines the root coordinate position of the object only when the head network 206 classifies the object in the detection frame as a weed and the probability that the object is a weed is high enough, which can save computing resources and improve the accuracy and safety of the weeding operation, thereby further reducing energy consumption and improving the weeding effect.

[0065] An embodiment of the present invention provides a computing device for crop detection and weed root location, which can detect crops and weeds and locate weed roots based on the target detection framework 20 provided by the present invention.

[0066] Figure 3 A schematic block diagram of a computing device 30 for crop detection and weed root location according to an embodiment of the present invention is shown.

[0067] like Figure 3 As shown, computing device 30 may include computing resources 302 and target detection framework 304. According to an embodiment of the present invention, computing resources 302 may include various hardware and software resources for performing data processing, storage, transmission, etc.

[0068] According to an embodiment of the present invention, the target detection framework 304 may be obtained by end-to-end training of the target detection framework 20 on a large number of training images that contain and / or do not contain (for example, specific categories of) crops, and the target detection framework 304 can synchronously output the detected crops and weeds, as well as the root positions of the weeds from the input images.

[0069] According to an embodiment of the present invention, the object detection framework 304 may call the computing resources 302 to perform crop detection and weed root location.

[0070] According to an embodiment of the present invention, the target detection framework 304 may be configured to receive images (e.g., field images) captured by an image acquisition device inside and / or outside the computing device 30 from a scene containing crops and / or weeds, and perform bounding box detection and object classification tasks on these images to detect crops and weeds, and perform root coordinate regression tasks on these images to determine the weed root coordinates. As an exemplary implementation, the target detection framework 304 may be configured to perform bounding box regression and object classification processes on image features based on a loss function to infer the categories of each object in the received image and its probability (e.g., the object contained in the bounding box is a crop and the probability that the object is a crop) and the corresponding confidence; and may be configured to perform a weed root coordinate regression process on image features based on a loss function to infer the root position of the weeds in the received image.

[0071] According to an embodiment of the present invention, the target detection framework 304 can perform bounding box detection and object classification tasks with the bounding box detection loss L bbox and classification loss L cls Associated, where the bounding box detection loss L bbox Indicates the gap between the inferred bounding box and the actual bounding box, the classification loss L cls Indicates the gap between the inferred object category and the actual object category for the corresponding bounding box. As an exemplary implementation, the bounding box detection loss L bbox It can be calculated as the Euclidean distance between the coordinates of the inferred bounding box (e.g., vertex coordinates, center point coordinates, feature point coordinates, etc.) and the actual bounding box coordinates; the classification loss can be calculated as the gap between the probability distribution of the inferred object category and the actual object category probability distribution for the corresponding bounding box (e.g., cross entropy).

[0072] According to an embodiment of the present invention, the target detection framework 304 can perform the weed root coordinate regression process with the root coordinate regression loss L reg As an exemplary implementation, the root coordinate regression loss L regIt can be calculated based on the Euclidean distance between the actual weed root coordinates and the inferred weed root coordinates: Where n indicates the number of input images, y i Indicates the actual weed root coordinates in the i-th input image, Indicates the inferred weed root coordinates in the i-th input image. According to an embodiment of the present invention, the loss function may be based at least on the bounding box detection loss L bbox , classification loss L cls , and the root coordinate regression loss L reg .

[0073] According to some other embodiments of the present invention, the object detection framework 304 may be configured to receive images captured from scenes containing crops and / or weeds, and perform bounding box detection and object classification tasks on these images to detect crops and weeds; and perform root coordinate regression tasks based on image features corresponding to weeds detected in these images to determine weed root coordinates. As an exemplary implementation, the object detection framework 304 may be configured to perform bounding box regression and object classification processes on all image features extracted from the received images based on a loss function to infer the categories of objects contained in these images and their probabilities, as well as corresponding confidences; in addition, the object detection framework 304 may be configured to perform a root coordinate regression process based on a loss function only on image features corresponding to objects determined to be weeds (e.g., inferring that the category of the object is a weed and the probability that the object is a weed is higher than a threshold) to infer the root position for the object.

[0074] Different from the existing weed detection and positioning solutions that focus on the overall detection and positioning of weeds, the computing device 30 according to the embodiment of the present invention can use the target detection framework 304 to accurately detect and distinguish crops and weeds in the farmland scene, and accurately locate the roots of the weeds. In addition, according to the embodiment of the present invention, the target detection framework 304 can also perform a root coordinate regression process based on the crop and weed detection it performs, which can further reduce the error of the determined weed root coordinates and improve the accuracy of determining the weed root coordinates.

[0075] According to an embodiment of the present invention, the target detection framework 304 can be implemented as the target detection framework 20 further fine-tuned through a multi-task learning process based on the comprehensive loss L. In other words, the target detection framework 304 can be configured to be optimized based on the comprehensive loss L through multi-task learning.

[0076] As an exemplary implementation, considering the losses of each head network included in the target detection framework 20, the comprehensive loss L can be defined as including the classification loss L cls , bounding box detection loss L bbox, and the root coordinate regression loss L reg , thus, optimizing the target detection framework 20 based on the comprehensive loss L can simultaneously improve the crop detection capability and weed root location capability of the target detection framework. As an exemplary implementation, the comprehensive loss L can be calculated as L = α·L cls +β·L bbox +γ·L reg , where the hyperparameters α, β, and γ indicate the classification loss L cls , bounding box detection loss L bbox , and the root coordinate regression loss L reg The weight in the comprehensive loss L can be adjusted dynamically when performing multi-task learning. As an exemplary implementation, the hyperparameter α can be set to be greater than the threshold weight, and the hyperparameters β and γ can be adjusted based on the setting of the hyperparameter α and multi-task learning performed via the target detection framework 20. For example, the sum of the hyperparameters α, β, and γ can be set to 1 and the hyperparameter α can be set to be greater than 0.5 (or another appropriate threshold weight value), and the values ​​of the hyperparameters α, β, and γ can be adjusted and traversed via multi-task learning performed via the target detection framework 20.

[0077] According to the embodiment of the present invention, through multi-task learning, the target detection framework 304 can be better adapted to crop detection and weed root positioning, and the accuracy of detection and positioning is further improved.

[0078] According to an embodiment of the present invention, the target detection framework 304 can be further configured to further adopt a teacher-student framework to optimize by performing semi-supervised learning on a labeled image training set and an unlabeled image training set based on multi-task learning. As an exemplary implementation, the target detection framework 20 fine-tuned by multi-task learning on a labeled image training set based on a comprehensive loss L can be used as a teacher model Teacher(·), and the student model Student(·) can be configured with reference to the model parameters of the teacher model Teacher(·).

[0079] As an exemplary implementation, a target detection framework 20 that has been fine-tuned through multi-task learning on a labeled image training set based on a comprehensive loss L can be used as a teacher model Teacher(·), and an untrained target detection framework 20 can be used as a student model Student(·). According to an embodiment of the present invention, the unlabeled image training set can be input into the teacher model Teacher(·), and the teacher model Teacher(·) performs bounding box detection and object classification tasks on the unlabeled image training set to infer the crops and weeds contained in these unlabeled training images. It should be noted that the difference between the unlabeled image training set and the labeled image training set referred to in this article is only whether they are labeled, that is, the labeled image training set is a version of the unlabeled image training set that is labeled based on the true labels. As mentioned above with reference to Figure 2 As described above, each inference for a crop or weed has a corresponding classification confidence. As an exemplary implementation, the classification confidence can be calculated as ConfScore=Max(Softmax(Teacher(E u ))), where E u Indicates the image features corresponding to the unlabeled training image. According to an embodiment of the present invention, the inference that the ConfScore is higher than the classification confidence threshold τ can be used as the pseudo label of the corresponding unlabeled training image, that is, the category of the corresponding object in the unlabeled training image is annotated based on the inference that the classification confidence is higher than the classification confidence threshold τ (for example, (weed, 0.86), which indicates that the object contained in the bounding box is a weed, and the probability that the object is a weed is 0.86).

[0080] According to an embodiment of the present invention, the teacher model Teacher(·) can also perform a root coordinate regression task on an unlabeled image training set to infer the root coordinates of weeds in these unlabeled training images. In an embodiment of the present invention, each inference of the weed root coordinates has a corresponding root position confidence SimScore, and the weed root position in the corresponding unlabeled training image can be annotated based on the inference whose SimScore is higher than the root position confidence threshold ξ. As an exemplary implementation, the root position confidence SimScore can be calculated as the weed image feature extracted by the teacher model Teacher(·) from the unlabeled training image Compared to the corresponding weed image features of the labeled training image corresponding to the unlabeled training image For example, the root position confidence SimScore is calculated as cosine similarity:

[0081] According to an embodiment of the present invention, the student model Student(·) can be trained on a labeled image training set and a pseudo-labeled image training set based on a comprehensive loss L. In an embodiment of the present invention, the student model Student(·) can perform supervised learning on a labeled image training set based on a comprehensive loss L, and perform unsupervised learning on a pseudo-labeled image training set based on a comprehensive loss L.

[0082] The following will refer to Figure 8 The overall architecture of optimizing the target detection framework by using a teacher-student framework for semi-supervised learning according to an embodiment of the present invention will be described in further detail.

[0083] According to an embodiment of the present invention, through semi-supervised learning, the object detection framework 304 is trained on a larger training set, so that it can learn more common image features and thus has higher accuracy, generalization ability and robustness.

[0084] According to an embodiment of the present invention, before the teacher model Teacher(·) and the student model Student(·) perform semi-supervised learning on the labeled image training set and the unlabeled image training set based on the teacher-student framework, image enhancement may be first performed on the labeled image training set and the unlabeled image training set.

[0085] According to an embodiment of the present invention, different image enhancement algorithms may be used to perform image enhancement on a labeled image training set and an unlabeled image training set. According to an embodiment of the present invention, a first image enhancement algorithm may be used to perform image enhancement on a labeled image training set. Specifically, the first image enhancement algorithm is used to process each labeled training image in the labeled image training set to obtain an enhanced labeled training image, thereby obtaining a first enhanced image training set, and the first enhanced image training set includes enhanced labeled training images. According to an embodiment of the present invention, a second image enhancement algorithm different from the first image enhancement algorithm may be used to perform image enhancement on an unlabeled image training set. Specifically, the second image enhancement algorithm is used to process each unlabeled training image in the unlabeled image training set to obtain an enhanced unlabeled training image, thereby obtaining a second enhanced image training set, and the second enhanced image training set includes enhanced unlabeled training images. As an exemplary implementation, the first image enhancement algorithm may include brightness adjustment, contrast adjustment, image cropping, and image flipping, and the second image enhancement algorithm may only include brightness adjustment and contrast adjustment.

[0086] According to an embodiment of the present invention, the teacher model Teacher(·) can be configured to adopt a target detection framework 20 fine-tuned by multi-task learning on a first enhanced image training set based on a comprehensive loss L, and can be configured to perform bounding box detection and object classification tasks and root coordinate regression tasks on a second enhanced image training set, and screen the corresponding inferences obtained based on confidence to obtain pseudo labels for the second enhanced image training set. Thus, according to an embodiment of the present invention, these pseudo labels can be used to annotate each enhanced unlabeled training image in the second enhanced image training set to obtain a pseudo-labeled image training set. For the student model Student(·), according to an embodiment of the present invention, it can be configured to train on such a pseudo-labeled image training set and the first enhanced image training set based on the comprehensive loss L.

[0087] According to an embodiment of the present invention, the generalization ability of the object detection framework can be further improved through image enhancement.

[0088] An embodiment of the present invention provides a weeding device, which can be used to remove weeds in farmland by detecting crops and weeds and locating weed roots based on the target detection framework provided by the present invention.

[0089] See below Figure 4 The weeding device 40 according to the embodiment of the present invention will be described in detail.

[0090] like Figure 4 As shown, the weeding device 40 may include a camera 402, as described above with reference to Figure 3 The computing device 30 and the laser emitting device 404 are described, and the computing device 30 can be coupled to the camera 402 and the laser emitting device 404. According to an embodiment of the present invention, the camera 402 can be configured to capture field images, which are provided to the computing device 30 for crop detection and weed root location. The computing device 30 performs bounding box detection and object classification tasks and root coordinate regression tasks to detect crops and weeds from these images and determine the weed root coordinates. Figure 3 Detailed description is given in the above description, which will not be repeated here to avoid blurring the understanding of the present invention. According to an embodiment of the present invention, the laser emitting device 404 can emit laser based on the detection of crops and weeds by the computing device 30 and the positioning of the weed roots. As an exemplary implementation, the laser emitting device 404 can receive the determined weed root coordinates from the computing device 30 and emit laser to the position indicated by the weed root coordinates.

[0091] like Figure 4As shown, in an embodiment of the present invention, the weeding device 40 may also optionally include a moving device 406 for moving the weeding device 40. As an exemplary implementation, the moving device 406 may be implemented by tracks, wheels, legs, etc., so that the weeding device 40 can be conveniently moved to any position in the field.

[0092] As described above, the computing device 30 can accurately identify crops and weeds in the farmland and precisely locate the roots of the weeds. On this basis, the weeding device 40 according to an embodiment of the present invention can accurately emit lasers to the roots of the weeds to remove the weeds, and can efficiently remove weeds in the field, effectively prevent damage to nearby crops and significantly reduce energy consumption.

[0093] Table 1 below shows the performance of a conventional weeding device based on a conventional target detection framework and a weeding device 40 according to an embodiment of the present invention.

[0094] Table 1 Comparison of performance of weeding equipment

[0095] Root distance error False positive rate Accuracy Energy costs Traditional weed control equipment 2.9770 10％ 75.37％ 1.55 Weeding equipment 40 2.4838 0 80.42％ 1.05

[0096] As shown in Table 1, under the same weeding operation conditions, compared with the conventional weeding equipment using a conventional target detection framework based on a bounding box to detect and locate weeds, the weeding equipment 40 according to the embodiment of the present invention reduces energy consumption by about 32.3%, and the accuracy of weeding is improved by 5.05%. The weeding equipment 40 according to the embodiment of the present invention uses the target detection framework 304 proposed by the present invention to perform weeding operations, which can significantly improve the accuracy of crop and / or weed detection and reduce the energy cost of weeding operations. In addition, as shown in Table 1, the target detection framework 304 according to the embodiment of the present invention can reduce the weed false detection rate to 0, which especially ensures that the weeding equipment 40 will not mistake crops as weeds in actual operation, and can improve the safety of weeding operations.

[0097] The embodiments of the present invention provide a method for detecting crops and locating weed roots, which can detect crops and weeds and locate weed roots based on the target detection framework 20 provided by the present invention.

[0098] Figure 5 A schematic flow chart of a method 50 for crop detection and weed root location according to an embodiment of the present invention is shown.

[0099] like Figure 5As shown, the method 50 may include: at block 502, receiving an image collected from a scene including crops and weeds by a target detection framework. As an exemplary implementation, the target detection framework may receive field images captured by the image acquisition device in real time, and / or receive field images stored in the image storage device.

[0100] like Figure 5 As shown, the method 50 may also include: at box 504, the target detection framework performs bounding box detection and object classification tasks to detect crops and weeds from the image. As an exemplary implementation, the target detection framework may perform the above-mentioned reference on the received image. Figure 2 The bounding box regression and object classification process described above is used to infer the categories and probabilities of the objects contained therein, and the inference has corresponding confidence levels. According to an embodiment of the present invention, the bounding box detection and object classification tasks performed by the target detection framework can be combined with the bounding box detection loss L bbox and classification loss L cls Associated, where the bounding box detection loss L bbox indicates the gap between the inferred bounding box and the actual bounding box (for example, it can be calculated as the Euclidean distance between the inferred bounding box position and the actual bounding box position), and the classification loss L cls Indicates the gap between the inferred object category and the actual object category for the corresponding bounding box (for example, it can be calculated as the cross entropy between the inferred object category probability distribution and the actual object category probability distribution).

[0101] like Figure 5 As shown, the method 50 may also include: at box 506, the target detection framework performs a root coordinate regression task to determine the weed root coordinates from the image. As an exemplary implementation, the target detection framework may perform the above-mentioned reference on the received image. Figure 2 The root coordinate regression process described above is used to determine the root coordinates of the weeds. According to an embodiment of the present invention, the root coordinate regression task performed by the target detection framework can be combined with the root coordinate regression loss L reg As an exemplary implementation, the root coordinate regression loss L reg It can be calculated based on the Euclidean distance between the actual weed root coordinates and the inferred weed root coordinates: Where n indicates the number of input images, y i Indicates the actual weed root coordinates in the i-th input image, Indicates the inferred weed root coordinates in the i-th input image.

[0102] Different from the existing weed detection and positioning schemes which focus on the overall detection and positioning of weeds, the method 50 according to the embodiment of the present invention can use the target detection framework proposed by the present invention to accurately detect and distinguish crops and weeds in the farmland scene, and can accurately locate the roots of the weeds.

[0103] Figure 6 A schematic flow chart of a method 60 for crop detection and weed root location according to some other embodiments of the present invention is shown.

[0104] like Figure 6 As shown, method 60 may include: at block 602, receiving, by an object detection framework, an image captured from a scene including crops and weeds.

[0105] like Figure 6 As shown, the method 60 may further include: at block 604 , performing bounding box detection and object classification tasks by a target detection framework to detect crops and weeds from the image.

[0106] like Figure 6 As shown, the method 60 may further include: at block 606, the object detection framework performs a root coordinate regression task based on image features corresponding to the weeds detected in the image to determine the weed root coordinates. For example, at block 604, the object detection framework detects that object 1 is a weed, then, at block 606, the object detection framework may perform a root coordinate regression task on the image features corresponding to object 1 to determine the weed root coordinates for object 1.

[0107] According to an embodiment of the present invention, in method 60, the object detection framework performs a root coordinate regression process based on the crop and weed detection it performs, which can further reduce the error of the determined weed root coordinates and improve the accuracy of determining the weed root coordinates.

[0108] According to an embodiment of the present invention, the above reference Figure 5 The method 50 described and referenced Figure 6 The object detection framework used in the method 60 described above may be as described with reference to Figure 2 The described object detection framework 20 is trained end-to-end on a large number of training images containing and / or not containing (eg, a specific category of) crops, and is capable of synchronously outputting detected crops and weeds, as well as the root locations of the weeds, from input images.

[0109] According to an embodiment of the present invention, the target detection framework can be implemented as a first optimized target detection framework obtained by further fine-tuning the target detection framework 20 through a multi-task learning process based on the comprehensive loss L.

[0110] As an exemplary implementation, considering the losses of each head network included in the target detection framework 20, the comprehensive loss L can be defined as including the classification loss L cls , bounding box detection loss L bbox , and the root coordinate regression loss L reg , thus, optimizing the target detection framework 20 based on the comprehensive loss L can simultaneously improve the crop detection capability and weed root location capability of the target detection framework. As an exemplary implementation, the comprehensive loss L can be calculated as L = α·L cls +β·L bbox +γ·L reg , where the hyperparameters α, β, and γ indicate the classification loss L cls , bounding box detection loss L bbox , and the root coordinate regression loss L reg The weight in the comprehensive loss L can be adjusted dynamically when performing multi-task learning. As an exemplary implementation, the hyperparameter α can be set to be greater than the threshold weight, and the hyperparameters β and γ can be adjusted based on the setting of the hyperparameter α and multi-task learning performed via the target detection framework 20. For example, the sum of the hyperparameters α, β, and γ can be set to 1 and the hyperparameter α can be set to be greater than 0.5 (or another appropriate threshold weight value), and the values ​​of the hyperparameters α, β, and γ can be adjusted and traversed via multi-task learning performed via the target detection framework 20.

[0111] According to the embodiment of the present invention, the first optimized target detection framework obtained by the target detection framework 20 through multi-task learning can be better adapted to crop detection and weed root positioning, and the detection and positioning accuracy is further improved.

[0112] According to an embodiment of the present invention, the target detection framework can also be implemented as a second optimized target detection framework obtained by further adopting a teacher-student framework on the basis of the multi-task learning described above and performing semi-supervised learning on a labeled image training set and an unlabeled image training set.

[0113] Figure 7The process 70 of obtaining a second optimized target detection framework by semi-supervised learning by a teacher model and a student model based on a teacher-student framework on a labeled image training set and an unlabeled image training set according to an embodiment of the present invention is schematically shown. It should be noted that the difference between the unlabeled image training set and the labeled image training set referred to in this article is only whether they are labeled, that is, the labeled image training set is a version of the unlabeled image training set that is labeled based on the true label. As an exemplary implementation method, a target detection framework 20 that is fine-tuned through multi-task learning on a labeled image training set based on a comprehensive loss L can be used as the teacher model Teacher(·), and the student model Student(·) can be configured with reference to the model parameters of the teacher model Teacher(·).

[0114] like Figure 7 As shown, process 70 may include: at box 702, the teacher model Teacher(·) performs bounding box detection and object classification tasks on the unlabeled image training set to infer the crops and weeds contained in these unlabeled training images. According to an embodiment of the present invention, each inference for crops or weeds has a corresponding classification confidence ConfScore. As an exemplary implementation, the classification confidence can be calculated as ConfScore=Max(Softmax(Teacher(E u ))), where E u Indicates the image features corresponding to the unlabeled training images. Further, if Figure 7 As shown, process 70 may further include: at block 704, determining whether the inferred classification confidence ConfScore for the crop or weed is higher than the classification confidence threshold τ. According to an embodiment of the present invention, the inference whose classification confidence ConfScore is higher than the classification confidence threshold τ may be retained as a pseudo label for the corresponding unlabeled training image. According to an embodiment of the present invention, for the inference whose classification confidence ConfScore is not higher than the classification confidence threshold τ, as Figure 7 As shown, at block 705, these inferences are discarded and not used as pseudo labels.

[0115] like Figure 7As shown, the process 70 may also include: at box 706, the teacher model Teacher(·) performs a root coordinate regression task on the unlabeled image training set to infer the root coordinates of the weeds in these unlabeled training images. According to an embodiment of the present invention, each inference of the weed root coordinates also has a corresponding root position confidence SimScore. As an exemplary implementation, the root position confidence SimScore can be calculated as the weed image features extracted by the teacher model Teacher(·) from the unlabeled training images Compared to the corresponding weed image features of the labeled training image corresponding to the unlabeled training image For example, the root position confidence SimScore is calculated as cosine similarity: Furthermore, if Figure 7 As shown, the process 70 may further include: at block 708, determining whether the inferred root position confidence SimScore for the weed root coordinates is higher than the root position confidence threshold ξ. According to an embodiment of the present invention, the inference that the root position confidence SimScore is higher than the root position confidence threshold ξ may be retained as a pseudo label for the corresponding unlabeled training image. According to an embodiment of the present invention, for the inference that the root position confidence SimScore is not higher than the root position confidence threshold ξ, as Figure 7 As shown, at block 709, these inferences are discarded and not used as pseudo labels.

[0116] like Figure 7 As shown, the process 70 may further include: at box 710, using the inference of the object category determined at box 706 that the classification confidence ConfScore is higher than the classification confidence threshold τ and the inference of the weed root coordinates determined at box 708 that the root position confidence SimScore is higher than the root position confidence threshold ξ as pseudo labels, annotating the unlabeled image training set to obtain a pseudo-annotated image training set. As an exemplary implementation, the teacher model Teacher(·) may annotate the category of the object in the corresponding unlabeled training image (for example, annotate the object as a crop or a weed) based on the inference of the object category determined at box 706 that the classification confidence ConfScore is higher than the classification confidence threshold, and may annotate the weed root position in the corresponding unlabeled training image based on the inference of the root position confidence SimScore determined at box 708 that is higher than the root position confidence threshold.

[0117] like Figure 7As shown, process 70 may also include: at box 712, the student model performs multi-task learning on the labeled image training set and the pseudo-labeled image training set based on the comprehensive loss L to obtain a second optimized target detection framework. In an embodiment of the present invention, the student model Student(·) can perform supervised learning based on the comprehensive loss L on the labeled image training set, and perform unsupervised learning based on the comprehensive loss L on the pseudo-labeled image training set to obtain a second optimized target detection framework.

[0118] According to an embodiment of the present invention, through semi-supervised learning, the object detection framework can be trained on a larger training set, so that it can learn more common image features and thus has higher accuracy, generalization ability and robustness.

[0119] According to an embodiment of the present invention, before the teacher model Teacher(·) and the student model Student(·) perform semi-supervised learning on the labeled image training set and the unlabeled image training set based on the teacher-student framework, image enhancement may be first performed on the labeled image training set and the unlabeled image training set.

[0120] According to an embodiment of the present invention, different image enhancement algorithms may be used to perform image enhancement on a labeled image training set and an unlabeled image training set. According to an embodiment of the present invention, a first image enhancement algorithm may be used to perform image enhancement on a labeled image training set. Specifically, the first image enhancement algorithm is used to process each labeled training image in the labeled image training set to obtain an enhanced labeled training image, thereby obtaining a first enhanced image training set, and the first enhanced image training set includes enhanced labeled training images. According to an embodiment of the present invention, a second image enhancement algorithm different from the first image enhancement algorithm may be used to perform image enhancement on an unlabeled image training set. Specifically, the second image enhancement algorithm is used to process each unlabeled training image in the unlabeled image training set to obtain an enhanced unlabeled training image, thereby obtaining a second enhanced image training set, and the second enhanced image training set includes enhanced unlabeled training images. As an exemplary implementation, the first image enhancement algorithm may include brightness adjustment, contrast adjustment, image cropping, and image flipping, and the second image enhancement algorithm may only include brightness adjustment and contrast adjustment.

[0121] According to an embodiment of the present invention, the teacher model Teacher(·) can be configured to adopt a target detection framework 20 fine-tuned by multi-task learning on a first enhanced image training set based on a comprehensive loss L, and can be configured to perform bounding box detection and object classification tasks and root coordinate regression tasks on a second enhanced image training set, and screen the corresponding inferences obtained based on confidence to obtain pseudo labels for the second enhanced image training set. Thus, according to an embodiment of the present invention, these pseudo labels can be used to annotate each enhanced unlabeled training image in the second enhanced image training set to obtain a pseudo-labeled image training set. For the student model Student(·), according to an embodiment of the present invention, it can be configured to train on such a pseudo-labeled image training set and the first enhanced image training set based on the comprehensive loss L.

[0122] According to an embodiment of the present invention, the generalization ability of the object detection framework can be further improved through image enhancement.

[0123] In addition, as mentioned above Figure 5-Figure 7 As described, the method for crop detection and weed root positioning provided by the present invention can accurately locate the root position of the weeds. On this basis, when performing weeding operations, a laser beam can be accurately emitted to the point indicated by the root coordinates, thereby reducing the energy cost of the weeding operation.

[0124] Figure 8 The architecture of the teacher-learning framework according to an embodiment of the present invention is shown as a whole.

[0125] like Figure 8 As shown, the teacher model 802 and the student model 804 are used in unlabeled data (e.g., Figure 3 , Figure 7 The unlabeled image training set described above) and labeled data (e.g., Figure 3 , Figure 7 The training is performed on a labeled image training set described in .

[0126] According to an embodiment of the present invention, labeled data may be input into the teacher model 802, and then, as shown in box 8022, the teacher model 802 extracts image features from the labeled data and for objects labeled as weeds in the labeled data, and stores the obtained weed feature vectors in the weed feature vector library 806.

[0127] According to an embodiment of the present invention, unlabeled data is also input into the teacher model 802, and then, as shown in box 8024, the teacher model 802 performs bounding box detection and object classification tasks on the unlabeled data, and performs bounding box screening based on the corresponding classification confidence ConfScore and retains inferences with classification confidence ConfScore higher than a threshold value τ (that is, ConfScore>τ) as pseudo-labels for the unlabeled data. For inferences that satisfy ConfScore>τ, as shown in box 8026, it is further determined whether the inferred object category is a weed. If the object is determined to be a weed at box 8026, then, as shown in box 8028, the teacher model 802 further performs a root coordinate regression task on the image features corresponding to the object, and performs weed root coordinate screening based on the corresponding root position confidence SimScore and retains inferences with root position confidence SimScore higher than a threshold value ξ (that is, SimScore>ξ) as pseudo-labels for the unlabeled data. Figure 8 As shown, a pre-stored weed feature vector can be obtained from the weed vector library and its similarity with the feature vector of the corresponding image feature of the corresponding object inferred as weed by the teacher model 802 is calculated as the root position confidence SimScore. The process of the teacher model 802 performing bounding box detection and object classification tasks and root coordinate regression tasks on unlabeled data to obtain and filter pseudo labels is similar to the above reference. Figure 3 , Figure 7 (e.g., blocks 702-710) correspond to the descriptions performed.

[0128] According to an embodiment of the present invention, Figure 8 As shown, labeled data is input into the student model 804, and the student model 804 performs supervised learning based on the real labels of the labeled data; and unlabeled data is input into the student model 804, and the student model 804 performs unsupervised learning based on the pseudo labels of the unlabeled data. The training process of the student model described here on labeled data and unlabeled data is the same as that described above. Figure 3 , Figure 7 (e.g., block 712).

[0129] In addition, the inventors found that compared with multi-stage detection models such as Faster-RCNN and SSD, single-stage models, especially YOLO series models, perform better in tasks that require high-speed reasoning and maintain competitive accuracy. Figure 2 The object detection framework described in 20, reference Figure 3 The object detection framework 304 described herein and reference Figure 5-Figure 6The target detection framework used in the described methods 50 and 60 can preferably be based on a single-stage target detection framework, such as YOLOv7, YOLOv8, and YOLOv10. The inventors also found that the target detection framework without anchor boxes (such as YOLOv8 and YOLOv10) may drift in root prediction, while the target detection framework with anchor boxes (such as YOLOv7) effectively prevents the occurrence of drift by limiting the loss calculation within the respective grids. Therefore, the target detection framework proposed in the present invention can be further preferably based on a single-stage target detection framework with anchor boxes (such as YOLOv7). On this basis, the target detection framework can be optimized via semi-supervised learning, thereby further improving the model performance.

[0130] Fig. 9 FIG. 5 shows a comparison of the root distance errors of the weed root positions inferred by the target detection framework based on YOLOv7 and the target detection framework based on YOLOv8-M according to an embodiment of the present invention compared with the actual weed root positions, and further shows that the two target detection frameworks are compared through semi-supervised learning (such as Fig. 9 ) and without semi-supervised learning (as shown in the shaded bars in Fig. 9 Comparison of weed root distance errors in different cases (shown in the blank bar graph).

[0131] like Fig. 9 As shown in Figure 2, compared with the target detection framework based on YOLOv8-M, the weed root distance error of the target detection framework based on YOLOv7 is significantly lower, and as shown in Figure 2 Fig. 9 As shown in the figure, the object detection framework used, whether based on YOLOv7 or YOLOv8-M, shows lower weed root distance error after semi-supervised learning. For example, combined with Table 1 above, without semi-supervised learning, the error of the weed root position inferred by the object detection framework based on YOLOv7 for crop detection and weed root location is 2.4838 compared to the actual weed root position. After semi-supervised learning, the root distance error is reduced to 2.1485 (as shown in Table 1 above). Fig. 9 As shown in Figure 2). It can be seen that semi-supervised learning can effectively improve the performance of the target detection framework.

[0132] Fig.10 An exemplary application scenario 1000 of a crop detection and weed root location solution according to an embodiment of the present invention is schematically shown. Figure 3 The computing device 300 described for crop detection and weed root location, Figure 5-Figure 6 The methods 500 and 600 described for crop detection and weed root location can be found in reference Fig.10The described application scenario 1000 is implemented.

[0133] like Fig.10 As shown, in the sub-scene 1000A, the automatic robot 1002 takes an image, thereby obtaining a field image containing crops and / or weeds. According to an embodiment of the present invention, the automatic robot 1002 can be implemented as described above with reference to Figure 4 According to an embodiment of the present invention, in sub-scene 1000B, the image sensor 1004 collects the field image taken by the automatic robot 1002 to extract image data (such as Fig.10 As shown in FIG. 1 ). Those skilled in the art should understand that any image sensor suitable for image acquisition can be used to implement the image sensor 1004 according to the embodiment of the present invention. For the image data acquired by the image sensor 1004, as shown in FIG. Fig.10 As shown, in sub-scene 1000C, the image data is processed by neural network 1006 to perform crop detection and weed root location. Figure 2 The object detection framework described in 20, reference Figure 3 The object detection framework 304 described herein and reference Figure 5-Figure 6 The object detection framework used in the described methods 50 and 60 can be implemented using the neural network 1006 described herein. According to an embodiment of the present invention, the neural network 1006 can be configured to have a head network (e.g., such as Fig.10 10062) and a head network having a weed root location function (e.g., Fig.10 (as shown in middle frame 10064). Fig.10 The result output of the neural network 1006 performing crop detection and weed root location is shown in sub-scene 1000C. As shown, the neural network 1006 can output the category and probability of each object in the image data, and the root position of the object classified as a weed.

[0134] To further illustrate the beneficial effects of the crop detection and weed root location solution proposed in the present invention, Figure 11-Figure 12 To further compare the results of crop detection and weed root positioning using a conventional target detection framework with the results of crop detection and weed root positioning using the crop detection and weed root positioning scheme proposed in the present invention.

[0135] exist Fig.11In the figure, the first row shows the results of crop detection and weed location obtained by processing different field image data using a conventional target detection framework; the second row shows the results of crop detection and weed location obtained by processing different field image data using a crop detection and weed root location solution according to an embodiment of the present invention; the third row shows the corresponding true labels indicating the category of the object and the root location of the weed. Fig.11 It can be seen from the vertical comparison of the columns in that, compared with the conventional target detection framework, the object categories and weed root positions inferred by using the crop detection and weed root positioning scheme according to the embodiment of the present invention for crop detection and weed root positioning can well match the object categories and weed root positions indicated by the true labels in the third row. As an example, as shown in the sixth column, the inference results of the conventional target detection framework are (weed, 0.19), (weed, 0.21), (weed, 0.17), (weed, 0.24), (weed, 0.22), (weed, 0.32) and the positions of the corresponding bounding boxes; the inference results of the crop detection and weed root positioning scheme according to the embodiment of the present invention are (weed, 0.73), (weed, 0.24) and the corresponding root positions; the true label indicates two objects classified as weeds and their root positions; it can be seen that the inference results (weed, 0.73), (weed, 0.24) and the corresponding root positions of the crop detection and weed root positioning scheme according to the embodiment of the present invention significantly better match the true label. It can be seen that the crop detection and weed root positioning scheme proposed in the present invention can accurately detect and distinguish crops and weeds in farmland scenes.

[0136] Fig.12 The positioning result is shown in more detail, wherein the diamond points indicate the inferred positions and the circular points indicate the actual weed root positions. Fig.12 In FIG. 1 , the first row shows the weed positions inferred by using a conventional target detection framework, and the second row shows the weed root positions inferred by using a crop detection and weed root location solution according to an embodiment of the present invention. Fig.12 As shown, the inferred position shown in the first row deviates from the actual weed root position, while the inferred position shown in the second row can be well matched with the actual weed root position. It can be seen that the crop detection and weed root positioning scheme proposed in the present invention can accurately locate the roots of weeds.

[0137] On this basis, when performing weeding operations, the weeding scheme according to the embodiment of the present invention can accurately emit lasers to the roots of weeds, thereby efficiently removing weeds in the field, effectively preventing damage to nearby crops, and significantly reducing energy consumption.

[0138] One or more aspects of at least one embodiment of the present invention may be implemented as a computer program product, wherein the computer program product includes computer instructions. Program code is applied to input instructions to perform the functions described herein and generate output information. Output information can be applied to one or more output devices in a known manner. For the purpose of this application, the present invention also provides a computing device, which includes any system with a processor, such as, for example, a digital signal processor (DSP), a microcontroller, an application specific integrated circuit (ASIC) or a microprocessor. The computing device also includes a memory, which is coupled to the processor for storing program code. The program code can be implemented with a high-level process programming language or an object-oriented programming language.

[0139] One or more aspects of at least one embodiment of the invention may be implemented by representative instructions stored on a machine-readable medium that represent various logic within a processor, which when read by a machine causes the machine to fabricate logic for performing the techniques described herein.

[0140] The technical solution for crop detection and weed root positioning provided by the present invention performs crop and / or weed detection and weed root positioning based on the target detection framework proposed by the present invention. In the present invention, the target detection framework can perform both object detection and classification tasks and root coordinate regression tasks, so it can accurately detect and distinguish crops and weeds in farmland scenes, and accurately locate the roots of weeds. Further, in the present invention, it is also proposed to further optimize the target framework proposed by the present invention based on multi-task learning and further adopting a teacher-student framework for semi-supervised learning on this basis, so that the target detection framework proposed by the present invention can be better adapted to crop detection and weed root positioning, and can learn more general image features, so it has higher detection accuracy, generalization ability, and robustness to new data. On this basis, when weeding, it is possible to accurately emit lasers to the roots of weeds, so that weeds in the field can be efficiently removed, effectively prevent damage to nearby crops, and significantly reduce energy consumption.

[0141] The preferred embodiments of the present invention are described in detail above. However, it should be understood that the present invention can be implemented in various ways and variations without departing from its broad spirit and scope. A person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should fall within the scope of protection determined by the claims of the present invention.

Claims

1. A computing device for crop detection and weed root location, comprising: Computing resources; as well as The target detection framework calls the computing resources to perform crop detection and weed root location, wherein: The target detection framework is configured as follows: receiving an image collected from a scene including crops and weeds; performing bounding box detection and object classification tasks to detect crops and weeds from the image; as well as A root coordinate regression task is performed to determine the weed root coordinates from the image.

2. The computing device of claim 1, wherein: The target detection framework is configured as follows: The root coordinate regression task is performed based on image features corresponding to the weeds detected in the image to determine the weed root coordinates.

3. The computing device of claim 2, wherein: The execution of the root coordinate regression task by the target detection framework and the root coordinate regression loss L reg Associated, the root coordinate regression loss L reg Based on the Euclidean distance between the true weed root coordinates and the determined weed root coordinates.

4. The computing device of claim 2, wherein: The target detection framework is also configured to: Optimization is performed through multi-task learning based on a comprehensive loss L, where the comprehensive loss L is based on the classification loss L cls , bounding box detection loss L bbox , and the root coordinate regression loss L reg , the object detection framework performs the bounding box detection and object classification tasks with the bounding box detection loss L bbox and the classification loss L cls Associated.

5. The computing device of claim 4, wherein: The comprehensive loss L is calculated as: L=α·L cls +β·L bbox +γ·L reg , Among them, the hyperparameters α, β, and γ indicate the classification loss L cls , the bounding box detection loss L bbox , and the root coordinate regression loss L reg The weight in the composite loss.

6. The computing device of claim 5, wherein: The hyperparameter α is set to be greater than or equal to a threshold weight, and the hyperparameters β and γ are adjusted based on the setting of the hyperparameter α by performing multi-task learning via the target detection framework.

7. The computing device of claim 4, wherein: The target detection framework is also configured to: A teacher-student framework is further adopted to optimize the teacher model and the student model through semi-supervised learning on the labeled image training set and the unlabeled image training set.

8. The computing device of claim 7, wherein: The teacher model is obtained by the target detection framework performing the multi-task learning on the labeled image training set based on the comprehensive loss L, and wherein the teacher model is configured as follows: performing the bounding box detection and object classification tasks on the unlabeled image training set to obtain inferred crop and weed detection results, the inferred crop and weed detection results having corresponding classification confidences; performing the root coordinate regression task on the unlabeled image training set to obtain inferred weed root coordinates, the inferred weed root coordinates having corresponding root location confidence; and The inferred crop and weed detection results with classification confidence higher than a classification threshold confidence and the inferred weed root coordinates with root location confidence higher than a root location confidence threshold are output as pseudo labels for the unlabeled image training set.

9. The computing device of claim 8, wherein: The student model is obtained by configuring the target detection framework with reference to the model parameters of the teacher model, and wherein the student model is configured as follows: Training is performed on a pseudo-annotated image training set and the annotated image training set based on the comprehensive loss, wherein the pseudo-annotated image training set is obtained by annotating the unannotated image training set with the pseudo label.

10. The computing device of claim 9, in, The labeled image training set is enhanced by a first image enhancement algorithm to obtain a first enhanced image training set, and the unlabeled image training set is enhanced by a second image enhancement algorithm to obtain a second enhanced image training set, wherein the second image enhancement algorithm is different from the first image enhancement algorithm, The teacher model is obtained by the target detection framework performing the multi-task learning on the first enhanced image training set based on the comprehensive loss L, and is configured to perform the bounding box detection and object classification tasks and the root coordinate regression task on the second enhanced image training set to output the pseudo label of the second enhanced image training set, And wherein, the pseudo-annotated image training set is obtained by annotating the second enhanced image training set using the pseudo label, and the student model is configured to be trained on the pseudo-annotated image training set and the first enhanced image training set based on the comprehensive loss.

11. The computing device of any one of claims 1 to 10, wherein: The object detection framework is based on a single-stage object detection framework with anchor boxes.

12. A weeding device comprising: A computing device as claimed in any one of claims 1 to 11; a camera for capturing the image; as well as The laser emitting device is used to emit a laser beam to the roots of weeds based on the determined coordinates of the roots of weeds.

13. The weeding device of claim 12, further comprising: A moving device is used to move the weeding equipment.

14. A method for crop detection and weed root location, comprising: receiving, by an object detection framework, an image collected from a scene including crops and weeds; performing bounding box detection and object classification tasks by the object detection framework to detect crops and weeds from the image; as well as A root coordinate regression task is performed by the object detection framework to determine the weed root coordinates from the image.

15. The method of claim 14, wherein the target detection framework performs a root coordinate regression task comprising: The root coordinate regression task is performed by the object detection framework based on image features corresponding to the weeds detected in the image to determine the weed root coordinates.

16. The method of claim 15, wherein: The target detection framework performs the root coordinate regression task and the root coordinate regression loss L reg Associated, the root coordinate regression loss L reg Based on the Euclidean distance between the true weed root coordinates and the determined weed root coordinates.

17. The method of claim 15, wherein: The target detection framework is a first optimized target detection framework obtained by performing multi-task learning on the target detection framework based on a comprehensive loss L, wherein the comprehensive loss L is based on a classification loss L cls , bounding box detection loss L bbox , and the root coordinate regression loss L reg , and the object detection framework performs the bounding box detection and object classification tasks with the bounding box detection loss L bbox and the classification loss L cls Associated.

18. The method of claim 17, wherein: The comprehensive loss L is calculated as: L=α·L cls +β·L bbox +γ·L reg Among them, the hyperparameters α, β, and γ indicate the classification loss L cls , the bounding box detection loss L bbox , and the root coordinate regression loss L reg The weight in the composite loss.

19. The method of claim 18, wherein: The hyperparameter α is set to be greater than or equal to a threshold weight, and the hyperparameters β and γ are adjusted based on the setting of the hyperparameter α by performing multi-task learning via the target detection framework.

20. The method of claim 17, wherein: The target detection framework is a second optimized target detection framework, which is obtained by further adopting a teacher-student framework in the target detection framework to perform semi-supervised learning on a labeled image training set and an unlabeled image training set through a teacher model and a student model.

21. The method of claim 20, in, The teacher model is obtained by the target detection framework performing the multi-task learning on the labeled image training set based on the comprehensive loss L, and the student model is obtained by configuring the target detection framework with reference to the model parameters of the teacher model, And wherein, the optimization is performed by semi-supervised learning on the labeled image training set and the unlabeled image training set using a teacher-student framework, including: performing the bounding box detection and object classification tasks on the unlabeled image training set by the teacher model to obtain inferred crop and weed detection results, wherein the inferred crop and weed detection results have corresponding classification confidences; The teacher model performs the root coordinate regression task on the unlabeled image training set to obtain inferred weed root coordinates, wherein the inferred weed root coordinates have corresponding root location reliability; taking the inferred crop and weed detection results with classification confidences higher than a classification threshold confidence, and the inferred weed root coordinates with root location confidences higher than a root location confidence threshold, as pseudo labels of the unlabeled image training set, and labeling the unlabeled image training set to obtain a pseudo-labeled image training set; and The multi-task learning is performed by the student model on the pseudo-annotated image training set and the annotated image training set based on the comprehensive loss to obtain the second optimized object detection framework.

22. The method of claim 21, in, The labeled image training set is enhanced by a first image enhancement algorithm to obtain a first enhanced image training set, and the unlabeled image training set is enhanced by a second image enhancement algorithm to obtain a second enhanced image training set, wherein the second image enhancement algorithm is different from the first image enhancement algorithm, The teacher model is obtained by the target detection framework performing the multi-task learning on the first enhanced image training set based on the comprehensive loss L, and is configured to perform the bounding box detection and object classification tasks and the root coordinate regression task on the second enhanced image training set to output the pseudo label of the second enhanced image training set, And wherein, the pseudo-annotated image training set is obtained by annotating the second enhanced image training set using the pseudo label, and the student model is configured to be trained on the pseudo-annotated image training set and the first enhanced image training set based on the comprehensive loss.

23. A machine-readable medium comprising machine-executable instructions stored thereon, which when executed cause a machine to perform the method of any one of claims 14 to 22.

24. A computer program product comprising computer instructions which, when executed, implement the method of any one of claims 14 to 22.

Images