Method, device and equipment for detecting and positioning weeds in rice field based on unmanned aerial vehicle, and medium

Abstract

The invention relates to a rice field weed detection and positioning method, device and equipment based on an unmanned aerial vehicle and a medium, and the method comprises the steps: updating a backbone network in an original YOLOv5 model to a GhostNet network, updating a feature pyramid network in a feature fusion network and convolution in a path aggregation network to Ghost convolution, and introducing a spatial pyramid pooling module, constructing a field weed detection model; inputting the to-be-detected rice image into a field weed detection model trained to be in a convergence state to determine a prediction frame corresponding to field weeds of each grid unit in the to-be-detected rice image, and determining pixel coordinates of the field weeds in an image coordinate system according to the prediction frames corresponding to the field weeds; and determining absolute position coordinates of the field weeds according to the pixel coordinates of the field weeds in the image coordinate system, and removing the weeds in the rice field based on the absolute position coordinates. According to the invention, the recognition and positioning precision of field weeds is improved.

Description

Technical Field

[0001] The present application relates to the field of agriculture, and in particular to a method for detecting and locating weeds in a rice field based on an unmanned aerial vehicle, a corresponding device, an electronic device, and a computer-readable storage medium. Background Art

[0002] In the process of rice production, the rapid reproduction of weeds not only poses a threat to crops, but also provides a breeding ground for the spread of pests and diseases, which is extremely detrimental to the healthy development of rice. Therefore, accurate identification of weeds in rice fields is the key to reducing the use of pesticides, reducing costs and improving the efficiency of mechanical weeding. With the rapid development of low-altitude remote sensing technology of unmanned aerial vehicles, this technology has become an important means of obtaining remote sensing images of rice field plants. However, due to the complexity of remote sensing images, traditional image processing methods often cannot meet the requirements of efficient processing and accurate detection of large-scale data. At the same time, the high-dimensional and complex features of plant remote sensing images also bring many challenges.

[0003] In the prior art for removing weeds in rice fields, the efficiency of obtaining weed images in rice fields is low, and weeds are difficult to distinguish from crop seedlings, resulting in difficulties in early weed identification and positioning. Considering the low efficiency and time delay of manual identification of weeds in rice fields, the inventors have conducted in-depth research and exploration, aiming to propose an effective solution.

[0004] In summary, the applicant has made corresponding explorations to address the problems of low efficiency in acquiring images of weeds in rice fields and difficulty in distinguishing weeds from crop seedlings, which leads to difficulties in early weed identification and positioning. Summary of the invention

[0005] The purpose of the present application is to solve the above-mentioned problem and to provide a method for detecting and locating weeds in rice fields based on an unmanned aerial vehicle, a corresponding device, an electronic device and a computer-readable storage medium.

[0006] In order to meet the various objectives of this application, this application adopts the following technical solutions:

[0007] A method for detecting and locating weeds in a rice field based on an unmanned aerial vehicle is proposed to meet one of the purposes of this application, comprising:

[0008] The UAV responds to the command to detect and locate weeds in the rice field, and obtains an image of rice to be detected in the rice field containing weeds in the field;

[0009] The backbone network in the original YOLOv5 model is updated to the GhostNet network, the feature pyramid network in the feature fusion network and the convolution in the path aggregation network are updated to Ghost convolution and a spatial pyramid pooling module is introduced, and a channel attention module is introduced before the feature pyramid network to build a field weed detection model;

[0010] Inputting the rice image to be detected into a field weed detection model that has been trained to a convergent state to determine a prediction frame corresponding to the field weeds in each grid unit in the rice image to be detected, and determining the pixel coordinates of the field weeds in the image coordinate system according to the prediction frame corresponding to the field weeds;

[0011] The pixel coordinates of the field weeds in the image coordinate system are converted into the weed geographic coordinates and weed plane coordinates of the field weeds, the weed geographic coordinates and weed plane coordinates are used as the absolute position coordinates of the field weeds, and the weeds in the rice field are removed based on the absolute position coordinates to complete the detection and positioning of the weeds in the rice field.

[0012] Optionally, the step of converting the pixel coordinates of the field weeds in the image coordinate system into the weed geographic coordinates and the weed plane coordinates of the field weeds includes:

[0013] Acquire the center coordinates of the prediction box, the index of the grid unit in the horizontal direction, the index of the grid unit in the vertical direction, the number of grid units, and the width and height of the rice image to be detected, wherein the center coordinates of the prediction box represent the position of the center point of the prediction box relative to the current grid unit, and the center coordinates of the prediction box include the center abscissa and the center ordinate of the prediction box;

[0014] Calculate and determine a first sum value between the central horizontal coordinate of the prediction frame and the index of the grid unit in the horizontal direction, calculate and determine a first ratio between the first sum value and the number of grid units, and calculate and determine a first product between the first ratio and the width of the rice image to be detected, so as to determine the pixel horizontal coordinate of the field weeds in the image coordinate system;

[0015] Calculate and determine a second sum value between the central ordinate of the prediction frame and the index of the grid unit in the vertical direction, calculate and determine a second ratio between the second sum value and the number of grid units, and calculate and determine a second product between the second ratio and the height of the rice image to be detected, so as to determine the pixel ordinate coordinate of the field weeds in the image coordinate system;

[0016] The pixel coordinates of the field weeds in the image coordinate system are determined according to the pixel abscissas of the field weeds in the image coordinate system and the pixel ordinates of the field weeds in the image coordinate system.

[0017] Optionally, the step of converting the pixel coordinates of the field weeds in the image coordinate system into the weed geographic coordinates of the field weeds includes:

[0018] Get the longitude and latitude of the first pixel in the upper left corner of the rice image to be detected, and the aerial survey Figure 1 The unit longitude scale and the unit latitude scale corresponding to each pixel point, the pixel abscissa of the field weeds in the image coordinate system, and the pixel ordinate of the field weeds in the image coordinate system, wherein the rice image to be detected is an RGB image;

[0019] Calculate and determine the aerial survey Figure 1 a third product between the unit longitude scale corresponding to each pixel point and the pixel horizontal coordinate of the field weeds in the image coordinate system, and calculating and determining a third sum between the third product and the longitude of the first pixel point in the upper left corner of the rice image to be detected, so as to determine the longitude of the center point of the weeds in the geographic coordinates;

[0020] Calculate and determine the aerial survey Figure 1 a fourth product between the unit latitude scale corresponding to each pixel point and the pixel ordinate of the field weeds in the image coordinate system, and calculating and determining a fourth sum between the fourth product and the latitude of the first pixel point in the upper left corner of the rice image to be detected, so as to determine the latitude of the center point of the weeds in the geographic coordinates;

[0021] The weed geographical coordinates of the field weeds under the geographical coordinates are determined according to the longitude of the center point of the weeds under the geographical coordinates and the latitude of the center point of the weeds under the geographical coordinates.

[0022] Optionally, the step of converting the pixel coordinates of the field weeds in the image coordinate system into the weed plane coordinates of the field weeds includes:

[0023] The preset Gaussian direct calculation formula is used to project the weed geographic coordinates of the field weeds under the geographic coordinates to the local plane coordinate system according to the longitude of the center point of the weeds under the geographic coordinates and the latitude of the center point of the weeds under the geographic coordinates, so as to obtain the plane rectangular coordinate value of each field weed target and determine the weed plane coordinates of the field weeds.

[0024] Optionally, the GhostNet network includes a Ghost Bottleneck structure, and the Ghost Bottleneck structure includes two Ghost Module modules and a deep convolutional layer.

[0025] Optionally, the step of inputting the rice image to be detected into a field weed detection model that has been trained to a convergent state to determine a prediction box corresponding to the field weeds in each grid unit in the rice image to be detected comprises:

[0026] Inputting the rice image to be detected into a field weed detection model that has been trained to a convergence state, and converting input feature maps of different sizes into feature vectors of the same dimension in a spatial pyramid pooling module;

[0027] In the feature pyramid network, feature maps from different levels are fused to enhance the model's ability to recognize objects of different scales;

[0028] In the path aggregation network, the spatial positioning accuracy of the feature map is improved through bottom-up aggregation;

[0029] The channel attention module is located before the feature pyramid network performs feature information fusion, and performs channel weighting on the input feature map to enhance the model's attention to important channel features.

[0030] Optionally, the field weeds include one or any multiple of Lycopodiella cuneata, Lycopodiella chinensis, Alternanthera zeylanifolia or Glechoma longituba; and the basic network architecture of the field weed detection model is an improved YOLOv5 model.

[0031] A rice field weed detection and positioning device based on an unmanned aerial vehicle is provided to meet another purpose of the present application, comprising:

[0032] An image acquisition module is configured to cause the UAV to respond to an instruction to detect and locate weeds in a rice field, and acquire an image of rice to be detected in the rice field containing weeds in the field;

[0033] A detection model building module is configured to update the backbone network in the preset YOLOv5 model to a GhostNet network, update the feature fusion network to a spatial pyramid pooling, a feature pyramid network, and a path aggregation network, and introduce a channel attention module before the feature pyramid network to build a field weed detection model;

[0034] a field weed detection module, configured to input the rice image to be detected into a field weed detection model that has been trained to a convergence state to determine a prediction frame corresponding to the field weeds in each grid unit in the rice image to be detected, and determine the pixel coordinates of the field weeds in the image coordinate system according to the prediction frame corresponding to the field weeds;

[0035] The field weed positioning module is configured to convert the pixel coordinates of the field weeds in the image coordinate system into the weed geographic coordinates and weed plane coordinates of the field weeds, use the weed geographic coordinates and weed plane coordinates as the absolute position coordinates of the field weeds, and remove the weeds in the rice field based on the absolute position coordinates to complete the detection and positioning of the weeds in the rice field.

[0036] An electronic device provided to meet another purpose of the present application includes a central processing unit and a memory, wherein the central processing unit is used to call and run a computer program stored in the memory to execute the steps of the drone-based rice field weed detection and positioning method described in the present application.

[0037] A computer-readable storage medium is provided to meet another purpose of the present application, which stores a computer program implemented according to the method for detecting and locating weeds in rice fields based on drones in the form of computer-readable instructions. When the computer program is called and executed by a computer, the steps included in the corresponding method are executed.

[0038] Compared with the prior art, the present application aims to solve the problems of low efficiency in acquiring images of weeds in rice fields and difficulty in distinguishing weeds from crop seedlings, which leads to difficulties in early weed identification and positioning. The present application includes but is not limited to the following beneficial effects:

[0039] First, based on the UAV remote sensing image acquisition, the weed detection and positioning model in the present invention is used for a large number of high-resolution rice field weed images in different environments acquired by UAV remote sensing, which solves the problem that traditional artificial or ground machine vision is difficult to cover a large area and the generalization ability of the detection model is small;

[0040] Second, the DarkNet53 network in the original YOLOv5 model was replaced with the Ghostnet network, and a channel attention mechanism was added to greatly reduce the amount of computation while ensuring recognition accuracy, thus improving the recognition performance of weeds in the complex natural environment of rice fields.

[0041] Third, in order to lay the basis for precise operations, the type and specific location of weeds in rice fields are accurately identified and located through multi-level conversion from pixel level to geographic coordinate level, providing a precise location basis for the implementation of subsequent mechanical weeding operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0043] Figure 1 Schematic diagram of the process of detecting and locating weeds in rice fields based on drones in an embodiment of the present application;

[0044] Figure 2 A structural diagram of the improved YOLOv5 target detection model in the embodiment of the present application;

[0045] Figure 3 Schematic diagram of the Ghost Bottleneck structure in the embodiment of the present application;

[0046] Figure 4 A schematic diagram of a rice image to be detected containing field weeds in the embodiment of the present application;

[0047] Figure 5 Schematic diagram of weed detection and positioning effect in the embodiment of the present application;

[0048] Figure 6 This is a principle block diagram of a rice field weed detection and positioning device based on a drone in an embodiment of the present application;

[0049] Figure 7 It is a schematic diagram of the structure of the computer device in the embodiment of the present application. DETAILED DESCRIPTION

[0050] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be interpreted as limiting the present application.

[0051] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0052] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with those in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.

[0053] It will be understood by those skilled in the art that the "client", "terminal" and "terminal device" used herein include both devices with wireless signal receivers, which are devices with only wireless signal receivers without transmission capabilities, and devices with receiving and transmitting hardware, which are devices with receiving and transmitting hardware capable of two-way communication on a two-way communication link. Such devices may include: cellular or other communication devices such as personal computers, tablet computers, which have single-line displays or multi-line displays or cellular or other communication devices without multi-line displays; PCS (Personal Communications Service, personal communication system), which can combine voice, data processing, fax and / or data communication capabilities; PDA (Personal Digital Assistant, personal digital assistant), which may include a radio frequency receiver, pager, Internet / intranet access, web browser, notepad, calendar and / or GPS (Global Positioning System, global positioning system) receiver; conventional laptop and / or palmtop computers or other devices, which have and / or include a conventional laptop and / or palmtop computer or other device with and / or including a radio frequency receiver. The "client", "terminal" and "terminal device" used herein may be portable, transportable, installed in a vehicle (air, sea and / or land), or suitable for and / or configured to run locally, and / or in a distributed form, at any other location on the earth and / or in space. The "client", "terminal" and "terminal device" used herein may also be a communication terminal, an Internet terminal, a music / video playing terminal, for example, a PDA, a MID (Mobile Internet Device) and / or a mobile phone with a music / video playing function, or a smart TV, a set-top box and other devices.

[0054] The hardware referred to by the names such as "server", "client", and "service node" in this application is essentially an electronic device with the equivalent capabilities of a personal computer. It is a hardware device with the necessary components revealed by the von Neumann principle, such as a central processing unit (including an arithmetic unit and a controller), a memory, an input device, and an output device. The computer program is stored in its memory, and the central processing unit calls the program stored in the external memory into the internal memory for execution, executes the instructions in the program, and interacts with the input and output devices to complete specific functions.

[0055] It should be pointed out that the concept of "server" referred to in this application can also be extended to the case of server clusters. According to the network deployment principle understood by those skilled in the art, the servers should be logically divided. In physical space, these servers can be independent of each other but can be called through interfaces, or integrated into a physical computer or a set of computer clusters. Those skilled in the art should understand this flexibility, and should not use it to restrict the implementation of the network deployment method of this application.

[0056] Unless expressly specified, one or more technical features of the present application can be deployed on a server for implementation and accessed by a client through a remote call to obtain an online service interface provided by the server, or can be directly deployed and run on a client for access.

[0057] The neural network models referenced or may be referenced in this application, unless expressly specified, can be deployed on a remote server and remotely called on the client, or can be deployed and directly called on a client with sufficient device capabilities. In some embodiments, when it runs on the client, its corresponding intelligence can be obtained through transfer learning to reduce the requirements for the client's hardware operating resources and avoid excessive occupation of the client's hardware operating resources.

[0058] Unless explicitly specified, the various data involved in this application can be stored remotely on a server or on a local terminal device, as long as it is suitable for being called by the technical solution of this application.

[0059] Those skilled in the art should be aware that, although the various methods of the present application are described based on the same concept and thus present commonality to each other, unless otherwise specified, these methods can be independently executed. Similarly, for each embodiment disclosed in the present application, they are all proposed based on the same inventive concept, therefore, concepts with the same expression, and concepts that are appropriately changed for convenience despite different expressions, should be understood as equivalent.

[0060] Unless the mutually exclusive relationship between the embodiments to be disclosed in this application is explicitly stated, the relevant technical features involved in each embodiment can be cross-combined to flexibly construct a new embodiment, as long as such combination does not deviate from the creative spirit of this application and can meet the needs of the prior art or solve certain deficiencies in the prior art. Those skilled in the art should be aware of this flexibility.

[0061] See also Figures 1 to 5 In one embodiment, the method for detecting and locating weeds in rice fields based on drones of the present application includes:

[0062] Step S10: The UAV responds to the instruction to detect and locate weeds in the rice field, and obtains an image of rice to be detected in the rice field containing weeds in the field;

[0063] The drone can respond to an instruction to remove weeds in a rice field and obtain an image of rice to be detected in the rice field containing field weeds; the field weeds include one or any multiple of Herba Lycopodii, Herba Lycopodii, Herba Alternariae, or Herba Corydalis;

[0064] In some embodiments, all different types of weeds in the rice field are classified as weeds, and orthophotos are taken by drone equipment at 7 meters and 10 meters above the weeds to construct an original data set; specifically, images of rice to be detected are taken in the rice fields of a certain province and city, and the time of taking the images is the morning, noon and evening 14 days after the seedlings are transplanted. The weather for taking the images includes sunny days and cloudy days.

[0065] In a further embodiment, the image annotation tool LabelImg is used to annotate the rice field weeds in the original image, and all types of weeds such as Lycopodiella cuneata, Lycopodiella chinensis, Alternanthera zeylanifolia, and Glechoma longituba are uniformly labeled as weeds; after the annotation, the obtained data set is divided into a training set and a verification set of the field weed detection model according to a preset ratio.

[0066] Step S20, updating the backbone network in the original YOLOv5 model to a GhostNet network, updating the feature pyramid network in the feature fusion network and the convolution in the path aggregation network to Ghost convolution and introducing a spatial pyramid pooling module, and introducing a channel attention module before the feature pyramid network to construct a field weed detection model;

[0067] After obtaining a rice image to be detected containing field weeds in a rice field, the backbone network in the original YOLOv5 model is updated to a GhostNet network, the feature pyramid network in the feature fusion network and the convolution in the path aggregation network are updated to Ghost convolution and a spatial pyramid pooling module is introduced, and a channel attention module is introduced before the feature pyramid network to construct a field weed detection model; wherein the basic network architecture of the field weed detection model is an improved YOLOv5 model.

[0068] In some embodiments, the GhostNet network includes a Ghost Bottleneck structure, and the Ghost Bottleneck structure includes two Ghost Module modules and a deep convolutional layer.

[0069] In a specific embodiment, based on the original YOLOv5 target detection model, the GhostNet network is used to replace the original DarkNet53 network in its backbone network (Backbone), and the improved YOLOv5 target detection model is composed of three parts: the GhostNet network, the feature fusion network (Neck), and the head network (Head); wherein the GhostNet network generates a feature map through the Ghost module, through two steps, specifically including:

[0070] First, the Ghost module uses a 1*1 convolution to reduce the number of channels of the input feature to generate a real feature map; then, each channel of the real feature layer is processed by a depth-separable convolution to obtain a Ghost feature map; finally, the real feature map and the Ghost feature map are spliced ​​in the channel dimension to form the final output feature map. The GhostNet network in this application consists of multiple Ghost modules and is divided into five stages. The feature map output of the first stage is 320*320, the feature map output of the second stage is 160*160, the feature map output of the third stage is 80*80 (C3), the feature output of the fourth stage is 40*40 (C4), and the fifth stage is 20*20 (C5).

[0071] The Ghost Bottleneck structure is a bottleneck structure built by two Ghost Module modules and a deep convolution layer. The first Ghost Module module is used to expand the number of channels, specifically using a 1*1 convolution operation to increase the depth of the feature map. The second Ghost Module module is used to reduce the number of channels to match the number of channels of the shortcut branch. Specifically, the number of channels increased by a 1*1 convolution operation is reduced back. Between the two Ghost Modules, a depth-separable convolution with a stride of 2 is used, each input is independently convolved, and then a 1*1 convolution is applied to combine these channels to achieve feature fusion.

[0072] The feature fusion network (Neck) of the field weed detection model of the present application is composed of spatial pyramid pooling (SPP), feature pyramid network (FPN) and path aggregation network (PAN); wherein spatial pyramid pooling (SPP) is a spatial pooling pyramid, which ensures that input feature maps of different sizes can be converted into feature vectors with the same feature size after the spatial pyramid pooling (SPP) module. The SPP module contains three pooling kernels of different sizes (5*5, 9*9, 13*13), corresponding to different paddin (2, 4, 6) and step sizes (all 1).

[0073] By adopting the spatial pyramid pooling (SPP) module to implement pooling at different levels, the robustness to changes in target position and size is increased.

[0074] The feature pyramid network (FPN) performs information fusion on the high-resolution shallow features and low-resolution deep features generated by the GhostNet network in the backbone feature extraction network in a top-down manner; the feature pyramid network (FPN) horizontally links the output of the spatial pyramid pooling (SPP) module with the 40*40 feature layer (C4) of the GhostNet network, specifically, performs a 1*1 convolution on the 20*20 feature layer to adjust the number of channels, then upsamples it twice to a 40*40 feature map and performs feature fusion with C4 through Ghost convolution to form a feature map P5;

[0075] Then, feature P5 is convolved to reduce the number of channels and upsampled to 80*80 to obtain the next layer of feature maps. This feature map is horizontally linked with C3, and through the Ghost convolution operation, the final feature prediction P4 is formed and input into the path aggregation network (PAN).

[0076] The path aggregation network (PAN) enhances the positioning information of the feature layer by fusing the feature information input by the feature pyramid network (FPN) from bottom to top. Starting from the bottom-level feature map 80*80 of the feature pyramid network (FPN), it is downsampled to a size of 40*40, and then horizontally connected with the feature map P5, and feature fusion is performed through Ghost convolution to output a 40*40 feature layer; then, the feature is subjected to a 3*3 convolution operation, and the number of channels is reduced and then downsampled to a size of 20*20. The downsampled feature layer is horizontally connected to the 20*20 feature layer in the feature pyramid network (FPN), and then feature fusion is performed through Ghost convolution to output a 20*20 feature layer.

[0077] The channel attention module is introduced. This module is placed before the feature information fusion of the feature pyramid network (FPN). The purpose is to perform channel weighting on the input feature layer to enhance the focus on important channel features. The specific implementation method is: first, perform a global average pooling operation on the input feature layer; then, through the processing of two fully connected layers, and finally apply the sigmoid function to obtain the weight value of each channel; multiply the obtained channel weight with the original input feature layer to achieve the weighting of the input feature channel. By introducing the channel attention module, the feature information loss that may be caused by the use of a lightweight backbone network can be effectively compensated, thereby improving the detection accuracy of weed targets. Figure 4 Schematic diagram showing the structure of the channel attention mechanism.

[0078] The head network (Head) uses a network-based prediction method to divide the input P3, P4, and P5 feature maps into S*S small squares, and each network unit predicts the weed target in the area. A certain number of anchor boxes are preset as the prior box of the target bounding box; for each grid unit, the head network (Head) output contains the following two vectors:

[0079] (1) A one-dimensional vector used to predict whether there is a weed target in the network unit;

[0080] (2) A vector of four values, representing the predicted x and y center coordinates of the target bounding box and the length adjustment parameter.

[0081] Furthermore, the output of the above prediction vector is brought into the loss function and compared with the annotated true value to calculate the loss between the two. Further, the output prediction box is non-maximum suppressed according to the confidence level to remove the bounding box of repeated detection. Then the confidence level and size threshold are set to remove invalid prediction boxes with low confidence level or abnormal box size. Finally, the confidence level of each detected weed target and the precise position and scale of the target on the image are output.

[0082] Furthermore, the parameters of the improved YOLOv5 target detection model were set and trained, and the best model was saved. The specific training parameters were as follows: the optimizer was selected as SGD, the batch size was set to 18, the total number of iterations was 300 epochs, the learning rate adopted the cosine annealing algorithm, the maximum learning rate was set to 1*e-3, and the weight decay value was 0.05; after the training reached the set number of iterations, the training was stopped, and the best performing model was selected, and its structure and weights were saved.

[0083] Furthermore, the rice image to be detected is input into the optimal model for inference prediction, and the weeds in the rice image to be detected are obtained after non-maximum suppression post-processing, and the pixel coordinates of the weeds on the image are obtained according to the bounding box information. The specific implementation steps are: using an image acquisition drone to obtain the rice image to be detected containing field weeds, and inputting the rice image to be detected into a pre-saved model for forward reasoning to generate N candidate results; during the detection process, multiple candidate results may overlap the same target, so the non-maximum suppression (NMS) method is used for processing, and then the final target and its bounding box information are output. After experimental determination, the selection threshold of non-maximum suppression is set to 0.4 to achieve the best recognition effect.

[0084] Step S30, inputting the rice image to be detected into a field weed detection model that has been trained to a convergent state to determine a prediction frame corresponding to the field weeds of each grid unit in the rice image to be detected, and determining the pixel coordinates of the field weeds in the image coordinate system according to the prediction frame corresponding to the field weeds;

[0085] The backbone network in the original YOLOv5 model is updated to a GhostNet network, the feature pyramid network in the feature fusion network and the convolution in the path aggregation network are updated to Ghost convolution and a spatial pyramid pooling module is introduced, and a channel attention module is introduced before the feature pyramid network to construct a field weed detection model, and then the rice image to be detected is input into the field weed detection model that has been trained to a convergent state to determine the prediction box corresponding to the field weeds of each grid unit in the rice image to be detected, and the pixel coordinates of the field weeds in the image coordinate system are determined according to the prediction box corresponding to the field weeds;

[0086] In some embodiments, the step of inputting the rice image to be detected into a field weed detection model that has been trained to a convergent state to determine a prediction box corresponding to the field weeds in each grid unit in the rice image to be detected includes:

[0087] Step S301, inputting the rice image to be detected into a field weed detection model that has been trained to a convergence state, and converting input feature maps of different sizes into feature vectors of the same dimension in a spatial pyramid pooling module;

[0088] Step S302: In the feature pyramid network, feature maps from different levels are fused to enhance the model's ability to recognize objects of different scales;

[0089] Step S303: In the path aggregation network, the spatial positioning accuracy of the feature map is improved through bottom-up aggregation;

[0090] Step S304: The channel attention module is located before the feature pyramid network performs feature information fusion, and performs channel weighting on the input feature map to enhance the model's attention to important channel features.

[0091] Step S40: converting the pixel coordinates of the field weeds in the image coordinate system into the weed geographic coordinates and the weed plane coordinates of the field weeds, using the weed geographic coordinates and the weed plane coordinates as the absolute position coordinates of the field weeds, and removing the weeds in the rice field based on the absolute position coordinates to complete the detection and positioning of the weeds in the rice field.

[0092] The rice image to be detected is input into a field weed detection model that has been trained to a convergent state to determine a prediction frame corresponding to the field weeds of each grid unit in the rice image to be detected, and after the pixel coordinates of the field weeds in an image coordinate system are determined according to the prediction frame corresponding to the field weeds, the pixel coordinates of the field weeds in the image coordinate system are converted into weed geographic coordinates and weed plane coordinates of the field weeds, the weed geographic coordinates and weed plane coordinates are used as absolute position coordinates of the field weeds, and the weeds in the rice field are removed based on the absolute position coordinates to complete the detection and positioning of the weeds in the rice field.

[0093] In some embodiments, the step of converting the pixel coordinates of the field weeds in the image coordinate system into the weed geographic coordinates and the weed plane coordinates of the field weeds includes:

[0094] Step S401, obtaining the center coordinates of the prediction box, the index of the grid unit in the horizontal direction, the index of the grid unit in the vertical direction, the number of grid units, and the width and height of the rice image to be detected, wherein the center coordinates of the prediction box represent the position of the center point of the prediction box relative to the current grid unit, and the center coordinates of the prediction box include the center abscissa and the center ordinate of the prediction box;

[0095] Step S402: calculating and determining a first sum value between the central horizontal coordinate of the prediction frame and the index of the grid unit in the horizontal direction, calculating and determining a first ratio between the first sum value and the number of grid units, and calculating and determining a first product between the first ratio and the width of the rice image to be detected, so as to determine the pixel horizontal coordinate of the field weeds in the image coordinate system;

[0096] Step S403, calculating and determining a second sum value between the central ordinate of the prediction frame and the index of the grid unit in the vertical direction, calculating and determining a second ratio between the second sum value and the number of grid units, and calculating and determining a second product between the second ratio and the height of the rice image to be detected, so as to determine the pixel ordinate coordinate of the field weeds in the image coordinate system;

[0097] Step S404: determining the pixel coordinates of the field weeds in the image coordinate system according to the pixel abscissas of the field weeds in the image coordinate system and the pixel ordinates of the field weeds in the image coordinate system.

[0098] The step of converting the pixel coordinates of the field weeds in the image coordinate system into the weed geographic coordinates and weed plane coordinates of the field weeds comprises:

[0099] The center coordinates of the prediction frame, the index of the grid unit in the horizontal direction, the index of the grid unit in the vertical direction, the number of grid units, and the width and height of the rice image to be detected are obtained, wherein the center coordinates of the prediction frame represent the position of the center point of the prediction frame relative to the current grid unit, and the center coordinates of the prediction frame include the center horizontal coordinate and the center vertical coordinate of the prediction frame. The pixel coordinates of the field weeds in the image coordinate system are determined according to the above parameters, and the calculation formula is expressed as follows:

[0100]

[0101] Among them, x img represents the pixel horizontal coordinate of the field weeds in the image coordinate system, y img represents the pixel ordinate of the field weeds in the image coordinate system; pred Indicates the central horizontal coordinate of the prediction box, y pred represents the central ordinate of the prediction box, grid_x represents the horizontal index of the grid unit, grid_y represents the vertical index of the grid unit, S is the number of grid units, and W and H are the width and height of the original image of the rice image to be detected, respectively.

[0102] In a further embodiment, the step of converting the pixel coordinates of the field weeds in the image coordinate system into the weed geographic coordinates of the field weeds includes:

[0103] Step S4001, obtain the longitude and latitude of the first pixel point in the upper left corner of the rice image to be detected, and the aerial survey Figure 1 The unit longitude scale and the unit latitude scale corresponding to each pixel point, the pixel abscissa of the field weeds in the image coordinate system, and the pixel ordinate of the field weeds in the image coordinate system, wherein the rice image to be detected is an RGB image;

[0104] Step S4002: Calculate and determine the aerial survey Figure 1 a third product between the unit longitude scale corresponding to each pixel point and the pixel horizontal coordinate of the field weeds in the image coordinate system, and calculating and determining a third sum between the third product and the longitude of the first pixel point in the upper left corner of the rice image to be detected, so as to determine the longitude of the center point of the weeds in the geographic coordinates;

[0105] Step S4003: Calculate and determine the aerial survey Figure 1 a fourth product between the unit latitude scale corresponding to each pixel point and the pixel ordinate of the field weeds in the image coordinate system, and calculating and determining a fourth sum between the fourth product and the latitude of the first pixel point in the upper left corner of the rice image to be detected, so as to determine the latitude of the center point of the weeds in the geographic coordinates;

[0106] Step S4004: determining the geographical coordinates of the weeds in the field under the geographical coordinates according to the longitude of the center point of the weeds under the geographical coordinates and the latitude of the center point of the weeds under the geographical coordinates.

[0107] The aerial survey map of the rice image to be detected is reconstructed by DJI Zhitu software to obtain the pixel information and geographic coordinate information of the RGB image of each rice image to be detected; the pixel coordinates (x img ,y img ) is converted to the geographical coordinates of the weeds in geographical coordinates (L i ,B i ), i.e., longitude and latitude coordinates. The present invention is based on the WGS84 geographic coordinate system, and the pixel coordinates are converted into the weed geographic coordinates of the field weeds under the geographic coordinates by the following conversion formula, and the calculation formula is expressed as follows:

[0108]

[0109] Among them, L i Indicates the longitude of the center point of the weed in geographic coordinates, B i Indicates the latitude of the center point of the weed in geographic coordinates, X 0 Indicates the longitude of the first pixel in the upper left corner of the RGB image of the rice image to be detected, Y 0Indicates the latitude of the first pixel in the upper left corner of the RGB image of the rice image to be detected, U 0 and V 0 Aerial survey of rice images to be detected Figure 1 The unit longitude scale and unit latitude scale corresponding to each pixel point,

[0110] (x img ,y img ) is the pixel coordinate of the field weeds in the RGB image in the image coordinate system.

[0111] In a further embodiment, the step of converting the pixel coordinates of the field weeds in the image coordinate system into the weed plane coordinates of the field weeds includes:

[0112] The preset Gaussian direct calculation formula is used to project the weed geographic coordinates of the field weeds under the geographic coordinates to the local plane coordinate system according to the longitude of the center point of the weeds under the geographic coordinates and the latitude of the center point of the weeds under the geographic coordinates, so as to obtain the plane rectangular coordinate value of each field weed target and determine the weed plane coordinates of the field weeds.

[0113] Specifically, the geographic coordinates are projected into the local plane coordinate system to obtain the plane rectangular coordinate value of each weed target, and the (X i ,Y i ), that is, the weed plane coordinates of the field weeds, the Gaussian forward calculation formula includes:

[0114]

[0115] Among them, l is the arc corresponding to the longitude, B is the arc corresponding to the latitude, a is the major semi-axis of the WGS84 ellipsoid, e is the first eccentricity 0.081819790992, e' is the second eccentricity 0.082095040121, X is the meridian arc length measured from the equator, N is the radius of curvature of the meridian circle, and to avoid negative coordinates, Y calculated in my country is i 500 kilometers need to be added to finally obtain the weed plane coordinates of the field weeds.

[0116] It can be seen from the above embodiments that, compared with the prior art, the present application aims to solve the problems of low efficiency in acquiring images of weeds in rice fields, difficulty in distinguishing weeds from crop seedlings, and difficulty in early weed identification and positioning. The present application includes but is not limited to the following beneficial effects:

[0117] First, based on the UAV remote sensing image acquisition, the weed detection and positioning model in the present invention is used for a large number of high-resolution rice field weed images in different environments acquired by UAV remote sensing, which solves the problem that traditional artificial or ground machine vision is difficult to cover a large area and the generalization ability of the detection model is small;

[0118] Second, the DarkNet53 network in the original YOLOv5 model was replaced with the Ghostnet network, and a channel attention mechanism was added to greatly reduce the amount of computation while ensuring recognition accuracy, thus improving the recognition performance of weeds in the complex natural environment of rice fields.

[0119] Third, in order to lay the basis for precise operations, the type and specific location of weeds in rice fields are accurately identified and located through multi-level conversion from pixel level to geographic coordinate level, providing a precise location basis for the implementation of subsequent mechanical weeding operations.

[0120] See also Figure 6 , a rice field weed detection and positioning device based on drone is provided to meet one of the purposes of the present application, including an image acquisition module 1100, a detection model construction module 1200, a field weed detection module 1300 and a field weed positioning module 1400. Among them, the image acquisition module 1100 is configured so that the drone responds to the instruction to detect and locate the rice field weeds, and obtains the rice image to be detected in the rice field containing the field weeds; the detection model construction module 1200 is configured to update the backbone network in the preset YOLOv5 model to the GhostNet network, update the feature fusion network to the spatial pyramid pooling, the feature pyramid network and the path aggregation network, and introduce a channel attention module before the feature pyramid network to construct a field weed detection model; the field weed detection module 1300 is configured to input the rice image to be detected into the trained The field weed detection model in a converged state is used to determine the prediction frame corresponding to the field weeds in each grid unit in the rice image to be detected, and the pixel coordinates of the field weeds in the image coordinate system are determined according to the prediction frame corresponding to the field weeds; the field weed positioning module 1400 is configured to convert the pixel coordinates of the field weeds in the image coordinate system into the weed geographic coordinates and weed plane coordinates of the field weeds, use the weed geographic coordinates and weed plane coordinates as the absolute position coordinates of the field weeds, and remove the weeds in the rice field based on the absolute position coordinates to complete the detection and positioning of the weeds in the rice field.

[0121] Based on any embodiment of this application, please refer to Figure 7 Another embodiment of the present application further provides an electronic device, which can be implemented by a computer device, such as Figure 7As shown, a schematic diagram of the internal structure of a computer device. The computer device includes a processor, a computer-readable storage medium, a memory, and a network interface connected via a system bus. Among them, the computer-readable storage medium of the computer device stores an operating system, a database, and computer-readable instructions. The database may store a control information sequence. When the computer-readable instructions are executed by the processor, the processor can implement a method for detecting and locating weeds in rice fields based on drones. The processor of the computer device is used to provide computing and control capabilities to support the operation of the entire computer device. The memory of the computer device may store computer-readable instructions. When the computer-readable instructions are executed by the processor, the processor can execute the method for detecting and locating weeds in rice fields based on drones of the present application. The network interface of the computer device is used to connect and communicate with a terminal. Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0122] In this embodiment, the processor is used to execute Figure 6 The memory stores the program code and various data required to execute the above modules or submodules. The network interface is used to transmit data between user terminals or servers. The memory in this embodiment stores the program code and data required to execute all modules / submodules in the unmanned aerial vehicle-based rice field weed detection and positioning device of the present application, and the server can call the program code and data of the server to execute the functions of all submodules.

[0123] The present application also provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the method for detecting and locating weeds in rice fields based on a drone as described in any embodiment of the present application.

[0124] The present application also provides a computer program product, including a computer program / instruction, which, when executed by one or more processors, implements the steps of the method for detecting and locating weeds in rice fields based on a drone as described in any embodiment of the present application.

[0125] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments of the present application can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a computer-readable storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0126] The above description is only a partial implementation method of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for detecting and locating weeds in rice fields based on drones, characterized in that: include: The UAV responds to the command to detect and locate weeds in the rice field, and obtains an image of rice to be detected in the rice field containing weeds in the field; The backbone network in the original YOLOv5 model is updated to the GhostNet network, the feature pyramid network in the feature fusion network and the convolution in the path aggregation network are updated to Ghost convolution and a spatial pyramid pooling module is introduced, and a channel attention module is introduced before the feature pyramid network to build a field weed detection model; Inputting the rice image to be detected into a field weed detection model that has been trained to a convergent state to determine a prediction frame corresponding to the field weeds in each grid unit in the rice image to be detected, and determining the pixel coordinates of the field weeds in the image coordinate system according to the prediction frame corresponding to the field weeds; The pixel coordinates of the field weeds in the image coordinate system are converted into the weed geographic coordinates and weed plane coordinates of the field weeds, the weed geographic coordinates and weed plane coordinates are used as the absolute position coordinates of the field weeds, and the weeds in the rice field are removed based on the absolute position coordinates to complete the detection and positioning of the weeds in the rice field.

2. The method for detecting and locating weeds in rice fields based on an unmanned aerial vehicle according to claim 1, characterized in that: The step of converting the pixel coordinates of the field weeds in the image coordinate system into the weed geographic coordinates and weed plane coordinates of the field weeds comprises: Acquire the center coordinates of the prediction box, the index of the grid unit in the horizontal direction, the index of the grid unit in the vertical direction, the number of grid units, and the width and height of the rice image to be detected, wherein the center coordinates of the prediction box represent the position of the center point of the prediction box relative to the current grid unit, and the center coordinates of the prediction box include the center abscissa and the center ordinate of the prediction box; Calculate and determine a first sum value between the central horizontal coordinate of the prediction frame and the index of the grid unit in the horizontal direction, calculate and determine a first ratio between the first sum value and the number of grid units, and calculate and determine a first product between the first ratio and the width of the rice image to be detected, so as to determine the pixel horizontal coordinate of the field weeds in the image coordinate system; Calculate and determine a second sum value between the central ordinate of the prediction frame and the index of the grid unit in the vertical direction, calculate and determine a second ratio between the second sum value and the number of grid units, and calculate and determine a second product between the second ratio and the height of the rice image to be detected, so as to determine the pixel ordinate coordinate of the field weeds in the image coordinate system; The pixel coordinates of the field weeds in the image coordinate system are determined according to the pixel abscissas of the field weeds in the image coordinate system and the pixel ordinates of the field weeds in the image coordinate system.

3. The method for detecting and locating weeds in rice fields based on an unmanned aerial vehicle according to claim 2, characterized in that: The step of converting the pixel coordinates of the field weeds in the image coordinate system into the weed geographic coordinates of the field weeds comprises: Obtaining the longitude and latitude of the first pixel point in the upper left corner of the rice image to be detected, the unit longitude scale and the unit latitude scale corresponding to a pixel point of the aerial survey map, the pixel horizontal coordinate of the field weeds in the image coordinate system, and the pixel vertical coordinate of the field weeds in the image coordinate system, wherein the rice image to be detected is an RGB image; Calculate and determine the third product between the unit longitude scale corresponding to a pixel point of the aerial survey map and the pixel horizontal coordinate of the field weeds in the image coordinate system, and calculate and determine the third sum value between the third product and the longitude of the first pixel point in the upper left corner of the rice image to be detected, so as to determine the longitude of the center point of the weeds in the geographic coordinates; Calculate and determine a fourth product between a unit latitude scale corresponding to a pixel point of the aerial survey map and a pixel ordinate of the field weeds in the image coordinate system, and calculate and determine a fourth sum value between the fourth product and the latitude of the first pixel point in the upper left corner of the rice image to be detected, so as to determine the latitude of the center point of the weeds in the geographic coordinates; The weed geographical coordinates of the field weeds under the geographical coordinates are determined according to the longitude of the center point of the weeds under the geographical coordinates and the latitude of the center point of the weeds under the geographical coordinates.

4. The method for detecting and locating weeds in rice fields based on an unmanned aerial vehicle according to claim 3, characterized in that: The step of converting the pixel coordinates of the field weeds in the image coordinate system into the weed plane coordinates of the field weeds comprises: The preset Gaussian direct calculation formula is used to project the weed geographic coordinates of the field weeds under the geographic coordinates to the local plane coordinate system according to the longitude of the center point of the weeds under the geographic coordinates and the latitude of the center point of the weeds under the geographic coordinates, so as to obtain the plane rectangular coordinate value of each field weed target and determine the weed plane coordinates of the field weeds.

5. The method for detecting and locating weeds in rice fields based on an unmanned aerial vehicle according to claim 1, characterized in that: The GhostNet network includes a Ghost Bottl eneck structure, which includes two GhostModule modules and a deep convolutional layer.

6. The method for detecting and locating weeds in rice fields based on an unmanned aerial vehicle according to claim 1, characterized in that: The step of inputting the rice image to be detected into a field weed detection model that has been trained to a convergent state to determine a prediction frame corresponding to the field weeds in each grid unit in the rice image to be detected comprises: Inputting the rice image to be detected into a field weed detection model that has been trained to a convergence state, and converting input feature maps of different sizes into feature vectors of the same dimension in a spatial pyramid pooling module; In the feature pyramid network, feature maps from different levels are fused to enhance the model's ability to recognize objects of different scales; In the path aggregation network, the spatial positioning accuracy of the feature map is improved through bottom-up aggregation; The channel attention module is located before the feature pyramid network performs feature information fusion, and performs channel weighting on the input feature map to enhance the model's attention to important channel features.

7. The method for detecting and locating weeds in rice fields based on an unmanned aerial vehicle according to any one of claims 1 to 6, characterized in that: The field weeds include one or any multiple of Herba Lycopodii, Herba Lycopodii, Herba Alternariae, or Herba Corydalis; and the basic network architecture of the field weed detection model is an improved YOLOv5 model.

8. A rice field weed detection and positioning device based on an unmanned aerial vehicle, characterized in that: include: An image acquisition module is configured to cause the UAV to respond to an instruction to detect and locate weeds in a rice field, and acquire an image of rice to be detected in the rice field containing weeds in the field; A detection model building module is configured to update the backbone network in the preset YOLOv5 model to a GhostNet network, update the feature fusion network to a spatial pyramid pooling, a feature pyramid network, and a path aggregation network, and introduce a channel attention module before the feature pyramid network to build a field weed detection model; a field weed detection module, configured to input the rice image to be detected into a field weed detection model that has been trained to a convergence state to determine a prediction frame corresponding to the field weeds in each grid unit in the rice image to be detected, and determine the pixel coordinates of the field weeds in the image coordinate system according to the prediction frame corresponding to the field weeds; The field weed positioning module is configured to convert the pixel coordinates of the field weeds in the image coordinate system into the weed geographic coordinates and weed plane coordinates of the field weeds, use the weed geographic coordinates and weed plane coordinates as the absolute position coordinates of the field weeds, and remove the weeds in the rice field based on the absolute position coordinates to complete the detection and positioning of the weeds in the rice field.

9. An electronic device, comprising a central processing unit and a memory, characterized in that: The central processing unit is used to call and run the computer program stored in the memory to execute the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: It stores a computer program implemented according to the method described in any one of claims 1 to 7 in the form of computer-readable instructions, and when the computer program is called and executed by a computer, the steps included in the corresponding method are executed.

Images