Weed area identification method and device based on fusion network, and electronic equipment

Abstract

The invention discloses a weed area identification method and device based on a fusion network, and electronic equipment. The method comprises the steps that an image to be recognized is acquired, and the image to be recognized comprises crops and weeds; a first model in the fusion model is adopted to recognize a first bounding box corresponding to crops in the to-be-recognized image, a second model in the fusion model is adopted to recognize a second bounding box corresponding to plants in the to-be-recognized image, the second bounding box comprises the first bounding box, and the plants comprise the crops and weeds; and according to the second bounding box and the first bounding box, determining a target area where the weeds are located in the to-be-identified image. The technical problems of high cost and low accuracy due to the fact that a large number of weed data sets need to be constructed for model training when weeds are recognized in the prior art and the recognition process is sensitive to illumination variation are solved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a weed region recognition method, device and electronic equipment based on a fusion network. Background Art

[0002] The weed identification methods used in related technologies usually use features such as color, shape and texture to distinguish weeds from crops. They are sensitive to changes in lighting and have poor robustness. When using deep learning methods for weed identification, a large amount of weed data sets are required to train the model to identify all types of weeds, which is costly and difficult to achieve.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiments of the present application provide a weed area recognition method, device and electronic device based on a fusion network, so as to at least solve the technical problems that in the related art, a large amount of weed data sets need to be constructed for model training when identifying weeds, and the recognition process is sensitive to changes in illumination, resulting in high cost and low accuracy.

[0005] According to one aspect of an embodiment of the present application, a weed region identification method based on a fusion network is provided, comprising: acquiring an image to be identified, wherein the image to be identified includes crops and weeds; using a first model in a fusion model to identify a first bounding box corresponding to the crops in the image to be identified, and using a second model in the fusion model to identify a second bounding box corresponding to the plants in the image to be identified, wherein the second bounding box includes the first bounding box, and the plants include crops and weeds; determining a target region where the weeds in the image to be identified are located based on the second bounding box and the first bounding box.

[0006] In some embodiments of the present application, determining a target area where weeds are located in a to-be-identified graphic based on a second bounding box and a first bounding box includes: determining a first matching value between each second bounding box and each first bounding box, wherein the first matching value is used to represent a degree of overlap between the second bounding box and the first bounding box; determining a target matching value among the first matching values ​​that is greater than or equal to a preset threshold; and determining an area where the second bounding box corresponding to the target matching value is located as the target area where the weeds are located.

[0007] In some embodiments of the present application, determining a first matching value between each second bounding box and each first bounding box includes: obtaining a first area of ​​a region where each second bounding box is located; determining an area of ​​an overlapping region between a region where each second bounding box is located and a region where each first bounding box is located as a second area; weighting the second area to obtain a third area; and determining a first matching value based on the first area and the third area.

[0008] In some embodiments of the present application, weighting the second area to obtain a third area includes: determining a weight coefficient corresponding to the second area, wherein the weight coefficient includes: a plant probability value of an area where the second bounding box is located and a crop probability value of an area where the first bounding box overlapping with the second bounding box is located; and weighting the second area using the weight coefficient to obtain the third area.

[0009] In some embodiments of the present application, the crop probability value of the area where each first bounding box is located can be determined in the following manner: obtain the first crop probability corresponding to each pixel in the first bounding box and the number of pixels in the first bounding box; determine the second crop probability corresponding to all pixels in the first bounding box, wherein the second crop probability includes the sum of the first crop probability corresponding to each pixel; and determine the crop probability value of the area where the first bounding box is located based on the second crop probability and the number of pixels.

[0010] In some embodiments of the present application, the fusion model is trained in the following manner: obtaining an image data set, wherein the image data set includes a first real area where crops are located and a second real area where plants are located; inputting the image data set into the first initial model and the second initial model in the initial fusion model, respectively, to obtain a first predicted area corresponding to the crops in the image data set and a second predicted area corresponding to the plants in the image data set, respectively; determining a target loss function value corresponding to the image data set based on the first real area, the second real area, the first predicted area, and the second predicted area, wherein the target loss function value at least includes a first loss function value corresponding to the first real area and the first predicted area, and a second loss function value corresponding to the second real area and the second predicted area; using the target loss function value to adjust the parameters of the initial fusion model, and stopping the adjustment until the updated target loss function value obtained by the adjusted initial fusion model meets a preset condition, thereby obtaining a fusion model.

[0011] In some embodiments of the present application, determining a target loss function value corresponding to an image data set includes: determining a second matching value between a second prediction region and a first prediction region, wherein the second matching value is at least used to represent a degree of overlap between the second prediction region and the first prediction region; determining a third loss function value based on the second matching value; and determining a target loss function value based on the first loss function value, the second loss function value, and the third loss function value.

[0012] According to another aspect of the embodiment of the present application, a weed region identification device based on a fusion network is also provided, including: an acquisition module, used to acquire an image to be identified, wherein the image to be identified includes crops and weeds; an identification module, used to use a first model in the fusion model to identify a first bounding box corresponding to the crops in the image to be identified, and use a second model in the fusion model to identify a second bounding box corresponding to the plants in the image to be identified, wherein the second bounding box includes the first bounding box, and the plants include crops and weeds; a determination module, used to determine a target area where the weeds in the image to be identified are located based on the second bounding box and the first bounding box.

[0013] According to another aspect of the embodiments of the present application, an electronic device is provided, including: a memory and a processor, the memory being used to store program instructions; the processor being connected to the memory and being used to execute the above-mentioned weed area identification method based on the fusion network.

[0014] According to another aspect of the embodiments of the present application, a non-volatile storage medium is provided, which includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the above-mentioned weed area identification method based on the fusion network by running the computer program.

[0015] According to another aspect of the embodiments of the present application, a computer program product is provided, including computer instructions, which implement the above-mentioned weed area identification method based on the fusion network when executed by a processor.

[0016] In the embodiment of the present application, a fusion model is used for weed identification. The two branches of the fusion model are used to identify crops and plants in the image to be identified, respectively. The identified crops and plants are used to determine the area where the weeds are located, so that crops can be accurately identified by constructing only a small amount of crop data sets for model training. In addition, the plant identification model only needs to distinguish between soil background and plants, reducing the workload of data collection and labeling, thereby achieving the technical effect of accurately identifying the weed area, and further solving the technical problems that the related technology needs to construct a large amount of weed data sets for model training when identifying weeds, and the identification process is sensitive to light changes, resulting in high cost and low accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0018] Figure 1 It is a hardware structure block diagram of a computer terminal of a weed region identification method based on a fusion network according to an embodiment of the present application;

[0019] Figure 2 is a flow chart of a weed region identification method based on a fusion network according to an embodiment of the present application;

[0020] Figure 3 is a schematic diagram of an image to be identified in a weed region identification method based on a fusion network according to an embodiment of the present application;

[0021] Figure 4 is a schematic diagram of crop identification results of a weed region identification method based on a fusion network according to an embodiment of the present application;

[0022] Figure 5 is a schematic diagram of plant recognition results of a weed region recognition method based on a fusion network according to an embodiment of the present application;

[0023] Figure 6 It is a structural schematic diagram of a weed region identification device based on a fusion network according to an embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0025] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0026] In order to better understand the embodiments of the present application, the technical terms involved in the embodiments of the present application are explained as follows:

[0027] Semantic Segmentation (SemSeg for short): It is used to classify each pixel of the input image and mark it as belonging to a predefined category, such as crops, weeds, etc.

[0028] Object Detection (OD): It is used to identify the location and category of objects in an image, and usually outputs the bounding box and category label of the object.

[0029] Convolutional Neural Network (CNN): It is used to process input data with a grid structure, such as images, and automatically learns the feature representation of images through structures such as convolutional layers, pooling layers, and fully connected layers.

[0030] In modern agricultural production, resource competition between weeds and crops is one of the main factors leading to reduced crop yields. Weeds compete for water, nutrients and light, breed pests and diseases, and seriously affect the growth and development of crops. The development of intelligent weeding equipment is restricted by weed recognition technology, but weeds are of various types and shapes, and have a high degree of similarity with the appearance characteristics of crops, which brings challenges to computer vision recognition. The weed recognition methods used in related technologies mainly rely on:

[0031] (1) Image processing technology: Use features such as color, shape, and texture to distinguish weeds from crops. However, this method is sensitive to changes in lighting, has poor robustness, and is difficult to handle complex field environments.

[0032] (2) Deep learning methods: mainly include target detection networks and semantic segmentation networks, which are used to identify and locate weeds. However, directly training the model to identify all types of weeds requires a large data set, which is costly and difficult to achieve. In addition, the similarity between weeds and crops limits the recognition accuracy of the model, resulting in high false detection and missed detection rates.

[0033] In order to solve the above technical problems, the embodiments of the present application provide corresponding solutions, which are described in detail below.

[0034] The weed area identification method embodiment based on fusion network provided in the embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 The hardware structure block diagram of a computer terminal for implementing a weed area identification method based on a fusion network is shown. Figure 1As shown, the computer terminal 10 may include one or more (102a, 102b, ..., 102n are used to illustrate) processors (the processor may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission module 106 for communication functions connected via a wired and / or wireless network. In addition, it may also include: a display, a keyboard, a cursor control device, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, and a BUS bus. Those skilled in the art can understand that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0035] It should be noted that the one or more processors and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10. As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0036] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the weed region identification method based on the fusion network in the embodiment of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, the above-mentioned weed region identification method based on the fusion network is realized. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0037] The transmission module 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission module 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission module 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0038] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 .

[0039] It should be noted that, in some optional embodiments, the above Figure 1 The computer terminal shown may include hardware elements (including circuits), software elements (including computer code stored on a computer-readable medium), or a combination of hardware elements and software elements. It should be noted that Figure 1 This is merely one example of a particular embodiment and is intended to illustrate the types of components that may be present in the computer terminal described above.

[0040] In the above operating environment, the embodiment of the present application provides an embodiment of a weed area identification method based on a fusion network. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0041] Figure 2 is a flow chart of a weed region identification method based on a fusion network according to an embodiment of the present application, such as Figure 2 As shown, the method comprises the following steps:

[0042] Step S202: acquiring an image to be identified, wherein the image to be identified includes crops and weeds.

[0043] In the above step S202, the image to be identified refers to an original image collected from a field or agricultural scene, and the image to be identified includes crops and weeds, and serves as an input of the fusion model.

[0044] In some embodiments of the present application, a high-definition camera carried by a drone can be used to shoot over the farmland to collect field images containing crops and weeds, or a fixed camera can be installed in the farmland to regularly shoot images of the crop growth area. The captured images to be identified can be used to record the appearance characteristics of crops and weeds in different planting areas, different growth stages, and different lighting conditions.

[0045] Step S204, using the first model in the fusion model to identify a first bounding box corresponding to the crops in the image to be identified, and using the second model in the fusion model to identify a second bounding box corresponding to the plants in the image to be identified, wherein the second bounding box includes the first bounding box, and the plants include crops and weeds.

[0046] In the above step S204, the fusion model is a multimodal deep learning fusion model, which combines the outputs of the crop recognition model (focusing on the detection or segmentation of crops) and the plant recognition model (distinguishing plants from non-plant backgrounds), and uses the accuracy of the crop model and the extensiveness of the plant model to jointly identify weeds. In some embodiments of the present application, the fusion model can be divided into two stages, the first stage identifies specific targets (such as crops), and the second stage identifies broader targets (such as all plants), and the final target (such as weeds) is more accurately located and identified by fusing the recognition results of the two stages.

[0047] The first model (i.e., crop recognition model) is used to recognize crops. It learns the characteristics of crops through training, so that it can accurately recognize and locate crops in the image to be recognized, and output the first bounding box corresponding to the crops. The first model may include a semantic segmentation model (such as U-Net), a target detection model (such as Faster Region-based Convolutional Neural Network, referred to as Faster R-CNN), etc. When the first model is a semantic segmentation model, the first bounding box refers to the minimum circumscribed rectangle or boundary area calculated from the crop region mask output by the segmentation model. The first bounding box accurately encloses the crop region identified by the segmentation model, including the main growing part of the crop; when the first model is a target detection model, the first bounding box directly refers to the rectangular area output by the detection model for clearly enclosing the position of the crop. At this time, the output of the first model may also include crop category information and the confidence that the crop object identified in the first bounding box is a real crop.

[0048] The second model (i.e., plant recognition model) is used to identify all plants (including crops and weeds), that is, to distinguish plants from non-plant backgrounds, but not to specifically distinguish the category of each plant, and output a second bounding box corresponding to the plant. The second model can use a target detection network, such as YOLOv5 (You Only Look Once version 5). The second bounding box output by the second model covers the range of all plants in the image to be identified, and is used to provide distribution information of all plants for comparison and analysis with the output of the first model, thereby identifying the location of the weeds.

[0049] In some embodiments of the present application, the fusion model can be trained in the following manner: obtaining an image data set, wherein the image data set includes a first real area where crops are located and a second real area where plants are located; inputting the image data set into the first initial model and the second initial model in the initial fusion model respectively, and obtaining a first predicted area corresponding to the crops in the image data set and a second predicted area corresponding to the plants in the image data set respectively; determining a target loss function value corresponding to the image data set based on the first real area, the second real area, the first predicted area, and the second predicted area, wherein the target loss function value at least includes a first loss function value corresponding to the first real area and the first predicted area, and a second loss function value corresponding to the second real area and the second predicted area; using the target loss function value to adjust the parameters of the initial fusion model, and stopping the adjustment until the updated target loss function value obtained by the adjusted initial fusion model meets the preset conditions, thereby obtaining a fusion model.

[0050] The image dataset consists of a preset number of annotated field images, each of which is labeled with the true location of the crops and the true location of all plants. In some embodiments of the present application, images can be collected on the spot, and bounding boxes or segmentation masks can be annotated for the crops and plant areas in each image using manual or machine-assisted annotation tools (such as LabelImg, VGG Image Annotator, Roboflow).

[0051] The first initial model is used to identify crops, and the second initial model is used to identify all plants. Both models are not fully trained at the beginning of training and will be gradually optimized during the training process to improve the accuracy of the prediction. The first prediction area and the second prediction area refer to the areas obtained after the first initial model and the second initial model predict the input image, respectively. In some embodiments of the present application, U-Net can be used to perform semantic segmentation on the image to predict the pixel-level mask of the crops, which is converted into the first prediction area, that is, the bounding box of the crops; YOLOv5 is used to perform target detection on the image, and the bounding boxes of all plants, including crops and weeds, are output to form the second prediction area.

[0052] The target loss function value refers to the difference measure between the model prediction result (i.e., the first prediction area and the second prediction area) and the true result (i.e., the first true area and the second true area), and can be composed of the first loss function value and the second loss function value. In some embodiments of the present application, the accuracy of the crop segmentation model (first initial model) can be measured by using the intersection over union (IoU) loss or the Dice coefficient loss, and the performance of the plant detection model (second initial model) can be evaluated by using the loss functions of the YOLO series, including classification loss, bounding box prediction loss, and confidence loss.

[0053] The preset conditions refer to the criteria used to determine whether the model training has achieved the expected goals, for example, setting the loss function threshold, model accuracy requirements or the number of training rounds, etc. In some embodiments of the present application, a gradient descent algorithm, such as stochastic gradient descent (SGD) or Adam, can be used to minimize the loss function value and optimize the model parameters. For example, at the end of each round of training, the overall loss (the weighted sum of the first loss function value and the second loss function value) is calculated, and the gradient of the model parameters is calculated based on this loss backpropagation, and then the model parameters are updated.

[0054] In order to more comprehensively evaluate the performance of the model and improve the training effect of the model, the target loss function value can also be determined by comprehensively considering the matching degree between the predicted area and the real area of ​​crops and plants. Specifically: determine the second matching value between the second prediction area and the first prediction area, wherein the second matching value is at least used to represent the overlap between the second prediction area and the first prediction area; determine the third loss function value based on the second matching value; determine the target loss function value based on the first loss function value, the second loss function value and the third loss function value.

[0055] The second matching value represents the degree of overlap between the plant region (second predicted region) output by the second initial model and the crop region (first predicted region) output by the first initial model, and is used to evaluate the spatial consistency between the bounding box output by the plant detection model (second initial model) and the crop region (such as a bounding box or a pixel-level probability map of the crop region) output by the crop segmentation model (first initial model). In some embodiments of the present application, if the crop segmentation model uses a pixel-level mask output, the intersection-over-union (IoU) between the mask and each plant bounding box can be calculated. The higher the IoU value, the higher the overlap between the plant bounding box and the crop region. The IoU value can be used as a direct measure of the second matching value.

[0056] In order to quantitatively evaluate the performance of the plant detection model (second initial model) in the crop area, and the consistency of the results of the plant detection model and the crop segmentation model (first initial model), the second matching value can also be determined in the following way:

[0057] (1) Crop segmentation model prediction (first initial model): The image is processed using the crop segmentation model to obtain the probability value of each pixel in the image belonging to a crop. The probability values ​​corresponding to all pixels in the image are combined to form a pixel-level probability distribution map of the crop area.

[0058] (2) Determination of crop region bounding box: From the crop probability distribution map, a threshold is set, and all pixels above the threshold are regarded as crop regions. The minimum enclosing rectangle of the crop region is calculated to obtain the crop bounding box.

[0059] (3) Calculation of crop area probability mean: For each crop boundary box, the average crop probability of all pixels inside it is calculated to reflect the average existence probability of crops in the boundary box.

[0060] (4) Plant detection model prediction (second initial model): Use the plant detection model to independently identify all plants in the image and output a series of plant bounding boxes and the average probability of a plant within each bounding box.

[0061] (5) Weighted overlapping area calculation: For each bounding box output by the plant detection model, the overlapping area between it and the crop bounding box is calculated, and the weighted overlapping area is obtained according to the average probability of the crop area and the plant area. The weighted overlapping area describes the degree of consistency between the plant bounding box and the crop bounding box.

[0062] (6) Calculation of crop weight matching value: Calculate the ratio of the weighted overlapping area to the total area of ​​the plant boundary box to obtain a second matching value.

[0063] The third loss function value is calculated based on the second matching value and is used to penalize the consistency deviation between the crop identification result and the plant identification result. In some embodiments of the present application, a predefined penalty function can be used. For example, when the second matching value is lower than a preset threshold, a penalty term is added to the loss function of the plant detection model. The penalty term can be the square of the difference between the second matching value and the threshold, thereby forming the third loss function value; or, the calculation formula of the third loss function value can be:

[0064] L consistency =1-W j

[0065] Among them, L consistency Represents the third loss function value, Wj Indicates the second matching value.

[0066] When calculating the overall loss, the loss of the crop segmentation model (the first loss function value), the loss of the plant detection model (the second loss function value), and the consistency loss (the third loss function value) can be weighted and summed according to a certain weight coefficient to obtain the target loss function value. The calculation formula can be:

[0067] L total =αL crop +βL plant +γL consistency

[0068] Among them, L total Represents the target loss function value, L crop Represents the first loss function value, L plant represents the second loss function value, α represents the weight coefficient corresponding to the first loss function value, β represents the weight coefficient corresponding to the second loss function value, and γ represents the weight coefficient corresponding to the third loss function value.

[0069] The setting of the weight coefficient corresponding to the loss function value can be adjusted according to the requirements of the specific task. For example, if the accuracy of crop segmentation is more important, the weight coefficient of the first loss function value can be increased; if you want the consistency loss to have a greater impact on model training, you can increase the weight coefficient of the third loss function value.

[0070] In some embodiments of the present application, the image data set can also be processed by a shared feature extractor to achieve efficient extraction of common features of the image, promote collaborative learning of the two tasks of crop recognition and plant recognition, and enhance the recognition ability and generalization ability of the model. Specifically: the first four residual blocks of ResNet-50 (50-layer Deep Residual Network) are selected as shared feature extractors, and this part of the network includes multiple convolutional layers, batch normalization layers and activation layers, which can extract multi-level feature representations from the image; the pre-processed image data is input into the shared feature extractor, the convolutional layer of the convolutional neural network slides on the image through the convolution kernel (filter) to extract local features of the image, the batch normalization layer is used to stabilize the training process, and the activation layer (such as ReLU) is used to increase the nonlinear expression ability of the model. After these layers, the output feature map includes a high-dimensional feature representation of the image, which can be used as the basis for crop recognition and plant recognition tasks; the feature map will be sent to two branches (crop recognition branch and plant recognition branch) for specific recognition.

[0071] Step S206: determining a target area where weeds are located in the image to be identified according to the second bounding box and the first bounding box.

[0072] In the above step S206, the target area can be determined in the following manner: for each second bounding box of the plant detection model, check whether there is an overlapping area with the first bounding box of the crop segmentation model; if the overlapping area between the second bounding box and the first bounding box is less than a first preset threshold, and the probability of crops in the crop segmentation model in the area is lower than a second preset threshold, it is determined that the second bounding box is mainly weeds. In some embodiments of the present application, whether there is an overlapping area can be determined in the following manner:

[0073] (1) Coordinate alignment: Ensure that the output coordinate systems of the crop segmentation model and the plant detection model are consistent, for example, operating at the same image resolution and coordinate space.

[0074] (2) Mask creation: Create a mask of the crop area from the output of the crop segmentation model. The mask is a binary image in which the pixel value of the crop area is 1 and the pixel value of other areas is 0.

[0075] (3) Bounding box coordinate extraction: The coordinates of each plant bounding box are extracted from the output of the plant detection model. The coordinates may include the coordinates of the upper left corner and lower right corner of the bounding box, or the coordinates of the center point and the width and height of the bounding box.

[0076] (4) Calculation of overlapping area: For each plant bounding box, check the mask value within its coordinate range and count the number of pixels that overlap with the crop area mask, that is, calculate the intersection size of the plant bounding box and the crop area mask. The number of overlapping pixels can be used as a measure of the overlapping area, or converted into area units to obtain the actual overlapping area value.

[0077] In order to effectively filter out areas with low overlap with crops and reduce misidentification, a threshold can be set to determine the target area where weeds are located in the to-be-identified graphic. Specifically, a first matching value of each second bounding box and each first bounding box is determined, wherein the first matching value is used to represent the overlap between the second bounding box and the first bounding box; a target matching value greater than or equal to a preset threshold is determined in the first matching values; and the area where the second bounding box corresponding to the target matching value is located is determined as the target area where the weeds are located.

[0078] The first matching value is used to measure the degree of overlap between the second bounding box and the first bounding box. The first matching value may be the ratio of the overlapping area to the total area of ​​the second bounding box, or other indicators for measuring the degree of overlap. In some embodiments of the present application, in order to more accurately reflect the importance of the overlapping area and avoid inaccurate recognition due to differences in area size, the first matching value may be determined by the following steps: obtaining the first area of ​​the area where each second bounding box is located; determining the area of ​​the overlapping area between the area where each second bounding box is located and the area where each first bounding box is located as the second area; weighting the second area to obtain the third area; and determining the first matching value based on the first area and the third area.

[0079] For each second bounding box output by the plant detection model, the area of ​​the entire region enclosed by each second bounding box is calculated to obtain the first area. In some embodiments of the present application, the first area can be obtained by calculating the four corner coordinates of the second bounding box, using the four corner coordinates to calculate the width and height of the bounding box, and multiplying the width and height.

[0080] For each first bounding box output by the second bounding box and the crop segmentation model, the area of ​​the overlapping region between the first bounding box and the second bounding box is determined to obtain the second area. In some embodiments of the present application, each pixel of the second bounding box can be traversed, and if the pixel is also in the first bounding box, the pixel is marked as a pixel of the overlapping area, and finally the number of overlapping pixels is calculated to be the second area.

[0081] When weighting the second area, the probability value can be used as a weight to more accurately evaluate the category of the overlapping area and improve the accuracy of weed recognition. Specifically, a weight coefficient corresponding to the second area is determined, wherein the weight coefficient includes: a plant probability value of the area where the second bounding box is located and a crop probability value of the area where the first bounding box overlapping with the second bounding box is located; and the second area is weighted using the weight coefficient to obtain a third area.

[0082] The weight coefficient refers to a coefficient used to weight the second area when determining the third area, and is used to quantify the importance of the overlapping area. The weight coefficient can comprehensively consider the plant probability value of the area where the second bounding box is located and the crop probability value of the area where the first bounding box overlaps with it.

[0083] The plant probability value of the area where the second bounding box is located can be obtained by the output of the plant detection model, and the plant probability value of each area where the second bounding box is located is obtained. The higher the plant probability value, the greater the possibility of the existence of plants in the bounding box. The crop probability value of the area where the first bounding box is located can be the crop probability value of the area where the minimum circumscribed rectangle is located obtained from the output of the crop segmentation model. The crop probability value reflects the density and possibility of crops in the area.

[0084] In order to more carefully analyze the crop distribution within each first bounding box and improve the recognition accuracy, the crop probability value of the area where the first bounding box is located can be determined in the following manner: obtain the first crop probability corresponding to each pixel in the first bounding box and the number of pixels in the first bounding box; determine the second crop probability corresponding to all pixels in the first bounding box, wherein the second crop probability includes the sum of the first crop probability corresponding to each pixel; and determine the crop probability value of the area where the first bounding box is located based on the second crop probability and the number of pixels.

[0085] The first crop probability refers to the probability value of each pixel point output by the crop segmentation model belonging to a crop. It can be a value between 0 and 1, which is used to reflect the uncertainty of whether each pixel point in the image belongs to a crop. The higher the value, the more likely the pixel point belongs to a crop.

[0086] The second crop probability includes the sum of the first crop probabilities of all pixels in the first bounding box, which is a numerical value used to represent the comprehensive crop probability of the entire crop area.

[0087] In some embodiments of the present application, the crop probability value of the area where the first bounding box is located can be calculated by the following formula:

[0088]

[0089] in, represents the crop probability value within the first bounding box, C j represents the first bounding box (i.e., the minimum bounding rectangle), (x, y) represents the pixel coordinates within the first bounding box, S j represents the set of all pixel coordinates within the first bounding box (i.e., the pixel area segmented by the crop segmentation model), P crop (x,y) represents the probability of the first crop, N i Indicates the number of pixels within the first bounding box.

[0090] In some embodiments of the present application, the weight coefficient can be the product of two probability values, indicating the importance of the overlapping part of the plant and crop area in the second bounding box segmentation model; it can also be other combinations, such as weighted average, etc., depending on the specific application scenario and requirements.

[0091] The third area can be calculated by the following formula:

[0092]

[0093] Among them, A weighted (B i ) represents the third area, B irepresents the second bounding box, C j represents the first bounding box, A overlap (B i ,C j ) represents the second area (i.e., the area of ​​the overlapping area where the second bounding box is located and the area where the first bounding box is located), represents the crop probability value within the first bounding box, Represents the probability value of vegetation within the second bounding box.

[0094] After determining the third area, the first matching value can be calculated by the following formula:

[0095]

[0096] Among them, W i Indicates the first matching value, A weighted (B i ) represents the third area, A(B i ) represents the first area (that is, the area of ​​the region where the second bounding box is located).

[0097] In some embodiments of the present application, the first matching value can be compared with a preset threshold value to screen out a target matching value, and the target area where the weeds are located can be determined by the target matching value. For example, when the first matching value is greater than or equal to 0.8 (the first preset threshold value), it is determined that the second bounding box mainly contains crops; when the first matching value is greater than or equal to 0.2 (the second preset threshold value), and the first matching value is less than 0.8 (the first preset threshold value), it is determined that the second bounding box contains both crops and weeds; when the first matching value is less than 0.2 (the second preset threshold value), it is determined that the second bounding box mainly contains weeds.

[0098] Through the above steps S202 to S206, a fusion model is used to identify weeds. The two branches of the fusion model are used to identify crops and plants in the image to be identified, respectively. The identified crops and plants are used to determine the area where the weeds are located, so that crops can be accurately identified by constructing only a small amount of crop data sets for model training. In addition, the plant identification model only needs to distinguish between soil background and plants, reducing the workload of data collection and labeling, thereby achieving the technical effect of accurately identifying the weed area, and further solving the technical problems that the related technology needs to construct a large amount of weed data sets for model training when identifying weeds, and the identification process is sensitive to light changes, resulting in high cost and low accuracy.

[0099] In order to facilitate understanding of the above process, an explanation is given below in conjunction with a specific embodiment:

[0100] Step 1: Build and train a crop recognition model.

[0101] 1) Dataset construction: Collect crop images in different field environments and lighting conditions. If a segmentation model is used, manually annotate the pixel-level contours of the crops to establish a semantic segmentation dataset of the crops; if a detection model is used, manually annotate the bounding boxes of the crops to establish a target detection dataset of the crops.

[0102] 2) Model training: Select a suitable semantic segmentation network (such as U-Net, DeepLab, etc.) or target detection model (such as the YOLO series), and train it with the semantic segmentation dataset of crops or the target detection dataset until the model converges or reaches the preset accuracy requirement.

[0103] 3) Model validation: Use an independent validation set to evaluate the performance of the model to ensure that the model can accurately segment the crop area.

[0104] Step 2: Build and train a plant target detection model.

[0105] 1) Dataset construction: Manually annotate the bounding boxes of all plants (crops and weeds) in the image and build a plant object detection dataset.

[0106] 2) Model training: Select a suitable target detection network (such as the YOLO series) and train it with the above dataset until the model converges or reaches the preset accuracy requirement.

[0107] 3) Model validation: Use an independent validation set to evaluate the classification accuracy of the model to ensure that the model can effectively distinguish between plants and background.

[0108] Step 3: Fusion of two models to identify weeds.

[0109] Image acquisition: Use a camera or the vision system of smart weeding equipment to collect original images of the field. Figure 3 is a schematic diagram of an image to be identified according to a weed region identification method according to an embodiment of the present application, such as Figure 3 As shown, the image contains mixed areas of crops and weeds. The distribution of crops and weeds in the image is irregular, which requires the recognition model to have high precision and robustness to accurately distinguish between the two.

[0110] Crop region segmentation: When the crop recognition model is a semantic segmentation model (the first model), the image to be recognized is input into the trained semantic segmentation model to obtain the segmentation result of the crop, generate a binary mask (the crop region is 1 and other regions are 0), and record the coordinate information of the crop region in the image for subsequent processing. Figure 4 FIG. 1 is a schematic diagram of crop identification results of a weed region identification method according to an embodiment of the present application. Figure 4As shown, the crop recognition model has successfully identified the location of the crops (the area where the dotted line is located, that is, the first bounding box).

[0111] The object detection model detects all plants: Figure 5 1 is a schematic diagram of a plant recognition result of a weed region recognition method according to an embodiment of the present application, that is, the recognition result of the second model. The second model not only recognizes crops, but also recognizes all plant regions containing crops and weeds, represented by a second bounding box. The second bounding box covers the first bounding box, which means that all crop regions are included, and additional regions are recognized, which contain weeds.

[0112] For all detected second bounding boxes, the weight proportion in the semantic segmentation model of each second bounding box is analyzed one by one. If it exceeds 80%, it is determined that the target contained in the second bounding box is a crop. If it is 80%-20%, it is determined that weeds and crops exist in the second bounding box at the same time. If it is less than 20%, it is determined that the second bounding box contains weeds. Figure 5 As shown, the second bounding box ①, ⑨, ⑤, ⑩ has a pixel ratio of 100%, which is a crop target; the second bounding box ②, ③, ④, ⑧ has a pixel ratio of 0%, which is a weed target; the second bounding box ⑥, ⑦ has a pixel ratio between 20%-80%, and the bounding box contains both weeds and crop targets.

[0113] Since this application uses a deep learning semantic segmentation and target detection model, the training process includes image data collected under a variety of lighting conditions, which makes the model more robust to light changes and can stably identify crops and weeds in different lighting environments.

[0114] In addition, since the types of crops are single or limited, when training the first model (semantic segmentation model or target detection model), only images of a single crop need to be collected and labeled, and there is no need to specially build a wide variety of weed data sets, which greatly reduces the workload of data collection and labeling, reduces the cost of training set construction, and ensures the training quality of the model. When training the second model (target detection model), there is no need to collect and label all types of weed images, nor is there any need to distinguish whether the plants in the training set are weeds or crops, because plants have obvious characteristics relative to the soil background, and the model only needs to learn to distinguish between plants and background, thereby further reducing the workload of data collection and labeling, reducing costs, and improving the generalization ability and recognition rate of the model.

[0115] Both models (semantic segmentation model or target detection model) of the present application do not need to directly distinguish between weeds and crops. The first model (semantic segmentation or target detection model) focuses on identifying crop areas, and the second model (target detection model) only needs to distinguish between plants and backgrounds, thereby reducing the complexity of weed identification and improving the recognition accuracy and reliability of the model.

[0116] Figure 6 is a structural diagram of a weed region identification device based on a fusion network according to an embodiment of the present application, such as Figure 6 As shown, the device comprises:

[0117] An acquisition module 602 is used to acquire an image to be identified, wherein the image to be identified includes crops and weeds;

[0118] The recognition module 604 is used to use the first model in the fusion model to recognize a first bounding box corresponding to the crops in the image to be recognized, and use the second model in the fusion model to recognize a second bounding box corresponding to the plants in the image to be recognized, wherein the second bounding box includes the first bounding box, and the plants include crops and weeds;

[0119] The determination module 606 is used to determine the target area where the weeds in the image to be identified are located according to the second bounding box and the first bounding box.

[0120] In the above-mentioned weed area recognition device based on fusion network, the determination module is also used to determine the first matching value of each second bounding box and each first bounding box, wherein the first matching value is used to represent the overlap degree between the second bounding box and the first bounding box; determine a target matching value greater than or equal to a preset threshold in the first matching values; and determine the area where the second bounding box corresponding to the target matching value is located as the target area where the weeds are located.

[0121] In the above-mentioned weed area recognition device based on fusion network, the determination module is also used to obtain a first area of ​​the area where each second bounding box is located; determine the area of ​​the overlapping area between the area where each second bounding box is located and the area where each first bounding box is located as the second area; weight the second area to obtain a third area; and determine a first matching value based on the first area and the third area.

[0122] In the above-mentioned weed area identification device based on fusion network, the determination module is also used to determine the weight coefficient corresponding to the second area, wherein the weight coefficient includes: the plant probability value of the area where the second bounding box is located and the crop probability value of the area where the first bounding box overlapping with the second bounding box is located; the second area is weighted by using the weight coefficient to obtain the third area.

[0123] In the above-mentioned weed area identification device based on fusion network, the determination module is also used to obtain the first crop probability corresponding to each pixel in the first bounding box and the number of pixels in the first bounding box; determine the second crop probability corresponding to all pixels in the first bounding box, wherein the second crop probability includes the sum of the first crop probability corresponding to each pixel; determine the crop probability value of the area where the first bounding box is located based on the second crop probability and the number of pixels.

[0124] In the above-mentioned weed area recognition device based on fusion network, the recognition module is also used to obtain an image data set, wherein the image data set includes a first real area where crops are located and a second real area where plants are located; the image data set is respectively input into the first initial model and the second initial model in the initial fusion model to obtain a first predicted area corresponding to the crops in the image data set and a second predicted area corresponding to the plants in the image data set; according to the first real area, the second real area, the first predicted area and the second predicted area, a target loss function value corresponding to the image data set is determined, wherein the target loss function value at least includes a first loss function value corresponding to the first real area and the first predicted area, and a second loss function value corresponding to the second real area and the second predicted area; the parameters of the initial fusion model are adjusted using the target loss function value, and the adjustment is stopped until the updated target loss function value obtained by the adjusted initial fusion model meets the preset conditions, thereby obtaining a fusion model.

[0125] In the above-mentioned weed area identification device based on fusion network, the identification module is also used to determine a second matching value between the second prediction area and the first prediction area, wherein the second matching value is at least used to indicate the degree of overlap between the second prediction area and the first prediction area; determine a third loss function value based on the second matching value; and determine a target loss function value based on the first loss function value, the second loss function value and the third loss function value.

[0126] It should be noted that Figure 6 The weed area recognition device based on the fusion network is used to perform Figure 2 The weed area recognition method based on fusion network is shown. Figure 2 The relevant explanations in the weed area recognition method based on fusion network in are also applicable to Figure 6 The weed area identification device based on the fusion network shown will not be described in detail here.

[0127] An embodiment of the present application further provides an electronic device, which includes a memory and a processor, wherein the memory is used to store program instructions; the processor is connected to the memory and is used to execute the steps of the weed area identification method based on the fusion network in each embodiment of the present application.

[0128] For example, the processor performs the following functions by executing program instructions stored in the memory:

[0129] An image to be identified is obtained, wherein the image to be identified includes crops and weeds; a first model in a fusion model is used to identify a first bounding box corresponding to the crops in the image to be identified, and a second model in the fusion model is used to identify a second bounding box corresponding to the plants in the image to be identified, wherein the second bounding box includes the first bounding box, and the plants include crops and weeds; a target area where the weeds in the image to be identified are located is determined based on the second bounding box and the first bounding box.

[0130] An embodiment of the present application further provides a non-volatile storage medium, which includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the steps of the weed area identification method based on the fusion network in each embodiment of the present application by running the computer program.

[0131] The embodiment of the present application further provides a computer program product, including computer instructions, which, when executed by a processor, implement the steps of the weed area identification method based on a fusion network in each embodiment of the present application.

[0132] The embodiment of the present application further provides a computer program, which, when executed by a processor, implements the steps of the weed area identification method based on a fusion network in each embodiment of the present application.

[0133] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0134] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0135] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0136] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0137] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0138] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0139] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A weed area recognition method based on fusion network, characterized in that: include: Acquire an image to be identified, wherein the image to be identified includes crops and weeds; Using a first model in the fusion model to identify a first bounding box corresponding to the crop in the image to be identified, and using a second model in the fusion model to identify a second bounding box corresponding to the plant in the image to be identified, wherein the second bounding box includes the first bounding box, and the plant includes the crop and the weed; A target area where the weeds in the image to be identified are located is determined according to the second bounding box and the first bounding box.

2. The method according to claim 1, characterized in that Determining a target area where the weeds in the to-be-identified graphic are located according to the second bounding box and the first bounding box includes: Determine a first matching value between each second bounding box and each first bounding box, wherein the first matching value is used to indicate a degree of overlap between the second bounding box and the first bounding box; Determine a target matching value among the first matching values ​​that is greater than or equal to a preset threshold; The area where the second bounding box corresponding to the target matching value is located is determined as the target area where the weeds are located.

3. The method according to claim 2, characterized in that Determining a first matching value between each second bounding box and each first bounding box includes: Get the first area of ​​the region where each second bounding box is located; Determine the area of ​​an overlapping region between the region where each second bounding box is located and the region where each first bounding box is located as a second area; weighting the second area to obtain a third area; The first matching value is determined according to the first area and the third area.

4. The method according to claim 3, characterized in that The second area is weighted to obtain a third area, including: Determine a weight coefficient corresponding to the second area, wherein the weight coefficient includes: a plant probability value of an area where the second bounding box is located and a crop probability value of an area where a first bounding box overlapping with the second bounding box is located; The second area is weighted by using the weight coefficient to obtain the third area.

5. The method according to claim 4, characterized in that The crop probability value of each first bounding box region can be determined as follows: Obtaining a first crop probability corresponding to each pixel in the first bounding box and the number of pixels in the first bounding box; Determine a second crop probability corresponding to all pixels within the first bounding box, wherein the second crop probability comprises the sum of the first crop probability corresponding to each pixel; A crop probability value of the area where the first bounding box is located is determined according to the second crop probability and the number of pixels.

6. The method according to claim 1, characterized in that The fusion model is trained in the following way: Acquire an image data set, wherein the image data set includes a first real area where the crops are located and a second real area where the plants are located; Inputting the image data set into a first initial model and a second initial model in an initial fusion model respectively, and obtaining a first prediction region corresponding to the crops in the image data set and a second prediction region corresponding to the plants in the image data set respectively; Determine a target loss function value corresponding to the image data set according to the first real area, the second real area, the first predicted area, and the second predicted area, wherein the target loss function value at least includes a first loss function value corresponding to the first real area and the first predicted area, and a second loss function value corresponding to the second real area and the second predicted area; The parameters of the initial fusion model are adjusted using the target loss function value until the updated target loss function value obtained by the adjusted initial fusion model meets a preset condition and the adjustment is stopped to obtain the fusion model.

7. The method according to claim 6, characterized in that Determining a target loss function value corresponding to the image data set includes: determining a second matching value between the second prediction area and the first prediction area, wherein the second matching value is at least used to indicate a degree of overlap between the second prediction area and the first prediction area; Determine a third loss function value according to the second matching value; The target loss function value is determined according to the first loss function value, the second loss function value and the third loss function value.

8. A weed area identification device based on a fusion network, characterized in that: include: An acquisition module, used for acquiring an image to be identified, wherein the image to be identified includes crops and weeds; a recognition module, configured to use a first model in a fusion model to recognize a first bounding box corresponding to the crop in the image to be recognized, and use a second model in the fusion model to recognize a second bounding box corresponding to the plant in the image to be recognized, wherein the second bounding box includes the first bounding box, and the plant includes the crop and the weed; A determination module is used to determine a target area where the weeds in the image to be identified are located according to the second bounding box and the first bounding box.

9. An electronic device, characterized in that: include: A memory and a processor, wherein the memory is used to store program instructions; the processor is connected to the memory and is used to execute the weed area identification method based on a fusion network as described in any one of claims 1 to 7.

10. A non-volatile storage medium, characterized in that: The non-volatile storage medium includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the weed area identification method based on a fusion network according to any one of claims 1 to 7 by running the computer program.

11. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by a processor, the weed area identification method based on a fusion network described in any one of claims 1 to 7 is implemented.

Images