Method for localizing and / or classifying weeds in an input image, computer program, storage medium, weed searching system and weed control system

Abstract

The invention relates to a method for localizing and / or classifying weeds (7) in an input image (12), wherein the input image (12) is fed as the basis for an input matrix to a network arrangement (10), wherein the network arrangement (10) comprises a feature map device (14), wherein the feature map device (14) generates a feature map (15) on the basis of the input matrix, wherein the network arrangement (10) comprises a detection device (16), wherein the detection device (16) detects weeds (7) on the basis of the feature map (15), wherein the network arrangement (10) carries out and / or outputs a localization and / or a classification of the weeds (7) on the basis of the detected weeds (7) in the input matrix, characterized in that the feature map device (14) comprises a data technology circuit of convolutional (22) and transformer blocks (23) for generating the feature map (15).

Description

[0001] Description

[0002] title

[0003] Method for locating and / or classifying weeds in an input image, computer program, storage medium, weed search system and weed control system

[0004] State of the art

[0005] In the past, weed control in fields often involved the use of pesticides that were applied over large areas. With increased awareness of the potential risks of pesticides to the environment and humans, efforts have been made to reduce the amount of pesticides used.

[0006] Recent developments enable the selective spraying of pesticides on weeds in row crops, thus saving pesticides and resulting in a reduction in environmental impact. For selective spraying of pesticides, the weeds must be detected in real time while a suitable agricultural vehicle drives across the field. It is known that such weed detection can be achieved using cameras and image processing algorithms.

[0007] For example, document EP 3815529 A1, which constitutes the closest prior art, discloses a computer system comprising image receiving logic configured to receive image data indicative of an image of a field, soil identification logic configured to identify a first image portion of the image representing the soil in the field, image segmentation logic configured to identify a remaining image portion, omit the first image portion from the image, and crop classification logic configured to apply a crop classifier to the remaining image portion and identify a second image portion of the image representing locations of crop plants in the field. The computer system also comprises weed identification logic configured toIdentify locations of weeds in the field based on the identification of the first and second image portions, and control signal generation logic configured to generate a machine control signal based on the identified locations of the weeds.

[0008] Disclosure of the invention

[0009] The subject matter of the invention is a method for locating and / or classifying weeds in an input image having the features of claim 1, a computer program having the features of claim 11, a storage medium having the features of claim 12, a weed search system having the features of claim 13, a weed control system having the features of claim 14 and a method for creating the network arrangement having the features of claim 15.

[0010] Preferred or advantageous embodiments of the invention emerge from the subclaims, the following description and the attached figures.

[0011] The invention relates to a method for locating and / or classifying at least one or exactly one weed in an input image. The input image originates, in particular, from a camera, which can be a grayscale camera or a color camera. In particular, the input image is configured as a two-dimensional matrix, with image information, such as grayscale information or color value information, arranged at the matrix positions.

[0012] The input image can depict weeds and, optionally, crops. The distinction between weeds and crops should be understood as a definition. A weed is defined as a plant that disrupts a crop in a field or grows there unintentionally. Optionally, the crop is also detected. This has the advantage that the areas of the crop can be protected and / or excluded during weed control.

[0013] The input image is fed to a network device. In other words, the input image is provided to the network device or received / read by the network device.

[0014] The input image forms the basis for an input matrix for a network arrangement. The input image can be identical to the input matrix. However, it is optionally possible that preprocessing steps were performed on the input image to create the input matrix. For example, it is conceivable that image processing steps were performed to improve contrast in the input image or that certain contiguous image areas in the input image are already segmented and / or colored. Consequently, an input matrix of a network arrangement is determined based on the input image.

[0015] A network arrangement is understood, in particular, to be a data-based arrangement and / or interconnection of networks, particularly in the form of a cluster. In this case, the input image and / or the input matrix constitutes input information into the network arrangement.

[0016] The network arrangement comprises a feature map device for generating a feature map based on the input matrix and / or which generates the feature map based on the input matrix. Thus, the input matrix forms input information to the feature map device, and the feature map forms output information from the feature map device. In particular, the feature map device is configured as a "backbone." The feature map device refers to a feature extraction network, feature extractor, and / or feature extractor, which can optionally be used within a learning model architecture. This feature extractor is used to encode the input of the network arrangement into a specific feature representation. The learning model framework "wraps" functionalities around this feature extractor.This allows the feature extractor to be replaced with a feature mapper, and a feature mapper can be selected that meets the specific task in terms of accuracy, efficiency, etc. Specifically, the backbone corresponds to a classifier (object detection without localization). The class information is then assigned a probability, as usual.

[0017] The network arrangement comprises a detection device for detecting the weed based on the feature map, which detects the weed based on the feature map. Optionally, the detection device is configured to classify the detected weed and / or localize the detected weed based on the feature map. Based on the detected weed, a localization and / or classification of the weed is output in the input matrix. The network arrangement thus outputs the position and existence, and optionally additionally the weed type, as information. In the simplest case, the detection device can be formed from a pooling layer and a fully linked layer, as is known, for example, from convolutional neural networks (CNNs).

[0018] Within the scope of the invention, it is proposed that the feature map device generates the feature map using a data circuit of convolution blocks and transformer blocks, or has a data circuit of convolution blocks and transformer blocks for generating the feature map. Thus, a data circuit is formed that converts an information flow from the input matrix to the feature map, with convolution blocks and transformer blocks being arranged in the information flow.

[0019] A convolution block is understood, in particular, to be a functional block that implements at least one convolution operation. In particular, the convolution block has the structure of a classic convolutional neural network consisting of one or more convolutional layers followed by a pooling layer. This unit can, in principle, be repeated any number of times; with sufficient repetitions, it is referred to as a deep convolutional neural network. In particular, an intermediate convolutional input matrix is ​​analyzed with a filter, whereby the analyzed intermediate convolutional input matrix forms an intermediate convolutional output matrix of the convolution block. Convolution operations can also be performed sequentially. In particular, one or more convolutional layers are arranged one after the other and / or networked within the convolution block.

[0020] A transformer block for the function block is understood in particular to be one that divides an intermediate transformer input matrix into several patches (image sections) and calculates the correlations / causalities between the patches. In particular, the transformer block consists of series-connected encoders and series-connected decoders. Correlations are diverse, but the transformer can learn the crucial causalities. Convolutional networks, on the other hand, recognize plant characteristics comparatively well, but the global correlations / causalities comparatively less well. It has been shown that the multitude of implicit and explicit correlations / causalities of characteristics or patches, or the characteristics contained therein, in a sequence of patches of a crop image with weeds, can be mapped particularly preferably using transformer models.

[0021] All information, and thus also causalities, are represented in the feature map. Object detection is based on this. The feature map ultimately represents the global features (causal relationships from the transformer block) and the local features (regarding weed detection and localization from the convolution block).

[0022] The invention is based on the idea that CNNs are faster than transformers, while transformers are more accurate and more robust against disturbances.

[0023] Instead of using only CNNs, transformer and convolutional blocks are combined for weed (species) detection and ultimately localization.

[0024] The advantage of this combination lies in a hardware and software resource-efficient network with lower latency for mobile implementations with computer vision tasks such as weed (species) detection on a sprayer. In a preferred embodiment of the invention, the data circuit has at least one parallel section, with at least one of the convolution blocks and at least one of the transformer blocks being arranged in parallel data processing.

[0025] By connecting transformer and convolutional blocks in parallel, the small features of the weeds are not lost when using the convolutional block (better feature maps).

[0026] The linking of the outputs of a parallel connection (merge) of blocks (in particular the intermediate convolution output matrices and the intermediate transformer output matrices) can be done, for example, by means of vector or matrix addition or the outputs are stored sequentially in vectors or matrices

[0027] In a possible development of the invention, the feature map device has at least one series section, wherein at least one of the convolution blocks and at least one of the transformer blocks are arranged in series. Thus, an output from the preceding block forms an input into the subsequent block of the series section. On the other hand, the series connection of transformer and convolutional / convolution blocks can save parameters, which ultimately saves hardware and software resources, i.e. implementations on existing controllers become possible in the first place and runtimes are shortened, so that the real-time requirements in the possible chain of effects from camera - controller - to sprayer nozzles are met due to sprayer travel speeds.

[0028] Particularly preferably, the convolution block is arranged before the transformer block in the series section. The transformer block can access the detailed features of the convolution block. In this case, the required runtime is shorter, but the detection accuracy is lower, in each case compared to the subsequent alternative. Alternatively, the transformer block is arranged before the convolution block in the series section. In this case, an intermediate transformer output matrix, which already contains global features, is passed to the convolution block. In this case, the required runtime is longer, but the detection accuracy is better, in each case compared to the previous alternative.

[0029] Preferably, the data circuit comprises a combination of series and parallel sections with convolution and transformer blocks. One or more series connections of convolution and transformer blocks are followed by one or more parallel connections of convolution and transformer blocks, or vice versa (e.g., R + P + P + P or R + R + P + R + P, etc.).

[0030] Preferably, the convolution block generates an intermediate convolution output matrix from an intermediate convolution input matrix, and / or the transformer block generates an intermediate transformer output matrix from an intermediate transformer input matrix. Preferably, the matrices each have the same number of columns and rows at an interface and / or a junction point in the data circuit. In this way, the information flow in the network arrangement can be implemented without adjusting the dimensions.

[0031] For example, it is provided that in the series section the intermediate convolution output matrix, the intermediate transformer input matrix and / or the intermediate transformer output matrix, the

[0032] In the example of a series circuit, the intermediate convolution output matrix is ​​equal to the

[0033] Intermediate transformer input matrix, so only one matrix.

[0034] In the parallel section, however, it is preferably provided that the intermediate convolution output matrix and the intermediate transformer output matrix are connected to one another, in particular merged.

[0035] One or more convolution operations can be performed in the convolution block, although the number of columns and rows in the intermediate convolution input matrix can correspond to the number of columns and rows of the intermediate convolution output matrix. In particular, the dimension with respect to the columns and rows is not changed in the convolution block. Preferably, in the transformer block, the intermediate transformer input matrix is ​​decomposed into several patches and / or matrix sections. A transformer vector is formed for each patch or matrix section, and the transformer vector is mapped into the intermediate transformer output matrix. The number of columns and rows of the intermediate transformer input matrix can correspond to the number of intermediate transformer output matrix.

[0036] For processing and / or for later searching and combining series and parallel circuits (hybrid...) it is advantageous if the number of columns and rows of the matrices are all the same.

[0037] In a preferred development of the invention, the detection device is designed to determine and / or output a bounding box as a localization for the detected weed. In particular, the bounding box is calculated in the input matrix and / or for the input image. The bounding box is passed on as data so that the localization of the detected weed in relation to the input image can subsequently be used for further purposes, such as weed control. The bounding box can be used for visualization. For further processing, in particular, a data set comprising a classification of the weed, a position of the weed, and optionally a probability / confidence level can be used.

[0038] In particular, the feature map facility forms a “backbone”.

[0039] It is preferred that the detection device comprises a head device (“head, detector head”), wherein the head device is designed to carry out the classification of the detected weeds and the localization of the detected weeds based on the feature map. The actual task of detecting, segmenting, etc., weeds (object) is carried out by applying a head device as a detection head to (optionally) the neck according to the feature map. For object detection, two output types are preferably used: Classification optionally with

[0040] Recognition probability and localization, for example, using a bounding box. The head device can be implemented as two different, decoupled heads or as a single head that supports both outputs.

[0041] Optionally, the detection device can have a neck device (C.neck), whereby the neck device is arranged between the feature map device and the head device in terms of data technology. The neck device processes the feature map, whereby the then processed feature map is passed on to the head device as a feature map. The neck device is arranged after the hybrid backbone as a feature map device as a feature extractor, which supplies the input for a feature map representation (feature map). This can, for example, improve the detection of both small and large features simultaneously by aggregating and fusing both small and large features, by normalization, by recentering and / or rescaling, in particular of the feature map, etc.

[0042] A further subject matter of the invention is a computer program, wherein the computer program is designed and / or configured to carry out, apply and / or implement the method according to the invention and / or the steps of the method when executed.

[0043] A further subject of the invention is a machine-readable storage medium, wherein the computer program is stored on the storage medium.

[0044] A further subject matter of the invention is a weed search system with a data processing device, wherein the data processing device is designed to carry out the previously described method and / or to implement the computer program and / or to have the network arrangement. The data processing device is designed, for example, as a computer or as a cloud. The weed search system comprises an input interface for accepting the input image, wherein the input interface is connected to the data processing device for data technology purposes and transfers the input image to the data processing device. Furthermore, the weed search system comprises an output interface for outputting the localization and / or classification of the weed, wherein the output interface is connected to the data processing device for data technology purposes in order to accept and output the said data. The data can, for example,X, y, width and height of the bounding box and the object class with detection probability for each object in the image that is to be detected or has been trained (e.g. weeds, crops, ...).

[0045] Optionally, the weed detection system comprises a camera for capturing the input image, wherein the camera is connected to the input interface via data transmission. Alternatively or additionally, the weed detection system comprises an output device, such as a screen, a display, or the like, wherein the output device is connected to the output interface via data transmission and is configured to output the localization and / or classification of the weed. For example, the input image can be displayed on the output device with a drawn boundary frame and / or information for classifying the weed.

[0046] Preferably, the weed detection system is designed as a real-time system, wherein the runtime between the acquisition and / or recording of the input image and the output of said data is less than 1 s, preferably less than 0.1 s.

[0047] A further subject matter of the invention is a weed control system, wherein the weed control system comprises the weed search system and the camera. Furthermore, the weed control system has a weed control arrangement, wherein the weed control arrangement is designed to carry out weed control on the basis of the localization and / or classification of the weeds. For example, the weed control arrangement has a sprayer for dispensing plant protection agent, wherein the weed control arrangement is designed to monitor, in particular to regulate and / or control, a dispensing of the plant protection agent on the basis of the said data, in particular the localization and / or classification of the weeds, and / or to monitor, in particular to regulate and / or control, a selection of the plant protection agent, and / or to monitor, in particular to regulate and / or control, a dispensing direction of the plant protection agent.Alternatively or additionally, the weed control system is designed to dispense the plant protection agent in a location-selective manner, with the location-selective dispensing being carried out based on the location, and / or to select the plant protection agent in a weed-selective manner, with the weed-selective selection of the plant protection agent being based on the weed classification. Optionally, the quantity of plant protection agent to be dispensed can also be based on the weed classification and / or the weed location. Plant protection agents are understood to mean, in particular, a pesticide and / or a weed killer.

[0048] Further features, advantages, and effects of the invention will become apparent from the following description of a preferred embodiment of the invention and the accompanying figures. These show:

[0049] Figure 1 is a schematic block diagram of a weed control system as an embodiment of the invention;

[0050] Figure 2 is a schematic block diagram of a network arrangement of the weed control system in Figure 1;

[0051] Figure 3 is a schematic block diagram of a convolution block from the network arrangement in Figure 2;

[0052] Figure 4 is a schematic block diagram of a transformer block from the network arrangement in Figure 2.

[0053] Figure 1 shows a schematic block diagram of a weed control system 1, wherein the weed control system 1 comprises a weed detection system 2, a camera 3, and a weed control arrangement 4. The weed control system 1 has the function of capturing input images from a field 5 via the camera 3 and making them available to the weed detection system 2 via an input interface 6.

[0054] The weed detection system 2 is designed to implement a method for locating and / or classifying weeds 7 in the input image. The method is executed in the weed detection system 2 via a computer program 8, which runs on a data processing device 9. The weed detection system 2 has a network arrangement 10, wherein the network arrangement 10 is designed to output a localization and / or classification of the weed 7 based on the detected weed 7 in the input matrix and / or in the input image to the weed control arrangement 4 via an output interface 11 from the input image as the basis for an input matrix.

[0055] Weed control system 4 is configured to control weeds 7 based on the output data. For example, weed control system 4 has at least one sprayer for dispensing plant protection agent, wherein the weed control system 4 is configured to selectively select the plant protection agent for weeds and / or to measure the quantity and / or to dispense it in a location-selective manner in order to control the detected weeds 7.

[0056] Figure 2 shows a schematic block diagram of the network arrangement 10 of the weed detection system 2.

[0057] The input image 12 is fed to the network arrangement 10 via the input interface 6. In this exemplary embodiment, the detection results are output via the output interface 11, for example in the form of an object list or matrix and optionally additionally in an output image 13, wherein the weed 7 is localized in the output image 13 via bounding boxes and the classification of the weed 7 is represented via flags.

[0058] Roughly speaking, the input image 12 is converted, if necessary, into an input matrix and passed on to a feature map device 14. The feature map device 14 creates a feature map 15. This feature map 15 is made available to a detection device 16, wherein the detection device 16 implements the classification and / or localization of the weed 7 in the input image 12 and / or in the input matrix. The localization and / or classification of the weed 7 is then output via the output interface 11. The detection device 16 has a neck device 17 and a head device 18. The neck device 17 compresses the information from the feature map 15, and the head device 17 then determines the localization and / or classification. The neck device 17 can be an NN / CNN or a transformer that carries out feature aggregation (e.g. feature pyramid). The head device 18 implements the detection orSegmentation and localization etc. of weeds by applying the "detection head" to the "neck" and the "feature map".

[0059] The feature map device 14 forms a backbone, the neck device 16 a neck and the head device 17 a detection head.

[0060] The feature card device 14 has a digital data circuit 19, wherein the data circuit 19 and / or the feature card device 14 has an interconnection of series sections 20 and parallel sections 21.

[0061] The series sections 20 and / or the parallel sections 21 each have a convolution block 21 and a transformer block 22. In the series section 20, the convolution block 22 and the transformer block 23 are connected in series and / or in series for data purposes. In the parallel section 21, the convolution block 22 and the transformer block 23 are connected in parallel, with the outputs of the convolution block 22 and the transformer block 23 being merged, i.e., combined for data purposes. For example, element-wise addition of matrices, for example, requires the dimensions of the intermediate convolution output matrix and the intermediate transformer output matrix to be the same, or the dimension of the merged matrix is ​​simply expanded / summed.

[0062] Figure 3 shows a schematic block diagram of such a convolution block 22, wherein the convolution block 22 receives an intermediate convolution input matrix 24 at the input and outputs an intermediate convolution output matrix 25 at the output. The dimensions, in particular the number of rows and columns of the intermediate convolution input matrix 24 and the intermediate convolution output matrix 25, can be identical; the depth can be different. One or more filter layers and / or one or more pooling layers can be arranged in the convolution block 22. In particular, the convolutional block is shown as a convolution block 22 with one or more convolution operations; the multiple kernels for the multiple convolution operations are only indicated here with individually shown kernels or must be imagined in the area of ​​the arrows.Figure 4 shows a schematic block diagram of such a transformer block 23, wherein the transformer block 23 receives an intermediate transformer input matrix 26 at the input and outputs an intermediate transformer output matrix 27 at the output. The dimensions, in particular the number of rows and columns of the intermediate transformer input matrix 26 and the intermediate transformer output matrix 27 can be identical, but the depth can be different. The transformer block as transformer block 23 has a transformer encoder 28, wherein the transformer encoder 28, for example,the Vision Transformer (ViT) as described in the scientific article “AN IMAGE IS WORTH 16X16 WORDS: TRANSFORMERS FOR IMAGE RECOGNITION AT SCALE” by Alexey Dosovitskiy*, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, Neil Houlsby, Published as a conference paper at ICLR 2021 , available at https: / / arxiv.org / pdf / 2010.11929.pdf, the disclosure of which is incorporated by reference into the present application.

[0063] The illustrated “flatten” block 29 is used to map / transform the patches into a vector (no convolution) and the “unfold” block 30 is used to map / transform vectors into the feature map 15 with respect to the patches, ie all vectors are summarized in a feature map 15.

[0064] The parallel connection of the convolution and transformer blocks or the convolution block 22 and transformer block 23 from Figure 2 has similarities with the Mobile ViT block (green box) of the Mobile ViT Figure 1: Visual transformers vs. MobileViT, (b) MobileViT from the scientific article “MOBILEVIT: LIGHT-WEIGHT, GENERAL-PURPOSE, AND MOBILE-FRIENDLY VISION TRANSFORMER” by Sachin Mehta Apple, Mohammad Rastegari Apple, published as a conference paper at ICLR 2022; available at https: / / arxiv.org / pdf / 2110.02178.pdf the disclosure of which is incorporated into the present disclosure by way of reference.

[0065] The series connection of convolution and transformer block or convolution block

[0066] 22 and Transformer block 23 from Figure 2 has similarities here with the Stage 2 block in Figure 2, whereby the Transformer is shown in more detail in the SDTA Encoder block, according to the scientific article: “EdgeNeXt: Efficiently Amalgamated CNN-Transformer Architecture for Mobile Vision Applications” by Muhammad Maaz, Abdelrahman Shaker, Hisham Cholakkall Salman Khan, Syed Waqas Zamir, Rao Muhammad Answer, Fahad Shahbaz Khan (arXiv:2206.10589v3 [cs.CV] 22 Oct 2022), available at whose disclosure is incorporated into the present disclosure via reference.

[0067] In the case of serial linking, it should be mentioned that the output is equal to the input. In the case of parallel linking, it should be mentioned that the outputs must first be linked (C.merge) e.g. by means of vector or matrix addition, the result of which is the input of the next block.

[0068] By connecting blocks in parallel, the global features in particular are preserved (better feature maps, where “better” here means, for example, that not only individual features of a single plant are learned, but also row information of plants such as the planting distance in the row or the distance between rows, etc.), with convolution and transformer blocks having their own feature maps

[0069] By connecting blocks in series, we can save parameters (CNNs require comparatively fewer parameters than transformers), which ultimately saves hardware and software resources, making implementations on existing controllers possible and shortening runtimes so that real-time requirements can be met. By combining convolution and transformer blocks in series and in parallel, depending on the application, these two advantages can be combined.

[0070] Method for searching for an application-specific hybrid backbone architecture with a to-be-determined combination of series and parallel circuits of convolution and transformer blocks, or convolution block 22 and transformer block 23, as shown in Figure 2: The specifications for the search method are the runtime (e.g., less than 50 msec) to meet the real-time requirements of the sprayer, and accuracy (e.g., a false negative detection rate of less than 5%). A false negative detection rate (detection as a crop, but in reality a weed) is more critical in the weed detection application than a false positive detection rate (detection as a weed, but in reality a crop), since weed eradication has priority. The specification of the % false negative detection rate is possible because we have the "ground truth" / training data available. Depending on what is more important to us (here, false negative), the parallel and series circuit structure to be searched for is influenced.

[0071] Furthermore, hardware restrictions and limitations can be specified, e.g. the number of all serial and parallel connections each with its convolutional block + transformer block or convolutional block 22 and transformer block 23)

[0072] The task of the search procedure is now to find a combination of series and parallel circuits of convolution and transformer blocks or convolution blocks 22 and transformer blocks 23 that fulfills the requirements.

[0073] Methods for searching for an architecture are well known (DEtNas, EdgeNext, etc.)

[0074] We use DetNAS as our pipeline. This is a neural architecture search algorithm for designing better backbones for object detection. It is based on the one-shot supernet technique, which contains all possible networks in the search space. The supernet is trained according to the typical detector training plan: ImageNet pre-training and detection fine-tuning. Then, the architecture search is performed on the trained supernet, using the detection task as a guide. DetNAS uses evolutionary search, as opposed to reinforcement learning-based or gradient-based methods.

[0075] The search procedure can now execute all possible or random combinations of the specified number of serial and parallel connections and terminate when the specified runtime and accuracy are met. Training and evaluation are performed, for example, with NVIDIA, and deployment is carried out with embedded hardware, such as an i.MX8 CPU. Since there are differences in the specifications and restrictions for the search procedure between NVIDIA and i.MX8, certain factors are used beforehand to translate the specifications and restrictions for i.MX8 to NVIDIA. In this process, other restrictions such as processing bandwidths and formats can also be transferred from i.MX8 to NVIDIA.

[0076] Finally, after completing the search process to find the basic architecture, fine-tuning is carried out by trying out different CNN and / or transformer variants and their combinations in the found architecture in order to achieve further improvements.

[0077] Global context information, correlations and / or causalities are the keys for the use of transformer blocks 23 in addition to convolution blocks 22, as will be plausibly explained below:

[0078] Definition of context: In linguistics and communication theory, this refers to all elements of a communication situation that influence the understanding of an utterance, i.e. the “context of the use of a word” that exists for the respective speaking situation.

[0079] Definition of correlation: Describes a relationship between two or more characteristics, states, or functions. The relationship does not necessarily have to be causal: Some elements of a system do not influence each other, or there may be a stochastic, randomly influenced, relationship between them.

[0080] Essentially, the transformer model in the transformer block 23 consists of series-connected encoders and decoders with self-attention modules. There are several

[0081] Self-attention layers are implemented. Using the attention mechanism, different parts of an input can be assigned different importance for the transformation of a sequence. Input data is essentially processed in the extended context of the surrounding data. In language models, the context can extend over many thousands of words and is easily scalable.

[0082] In our application example of weed detection and localization in images of a crop, we should perhaps no longer speak of context (e.g. of words in a word sequence), but rather correlations / causalities of patches, or the features contained therein, in a sequence of patches of an image.

[0083] The transformer model in the transformer block 23 divides an image or the inter-transformer input matrix 26 into a sequence of patches and then learns inter-patch representations using multi-headed self-attention in transformers.

[0084] The attention mechanism provides information about which parts of a patch sequence should be given more attention when forwarding the information. This creates a kind of link and shortcut between the compressed (context) vector and the complete original input sequence.

[0085] The Transformer architecture achieves better sequence transformation results with less effort and shorter training time than recurrent models such as CNN.

[0086] What, for example, are global correlations / causalities in the images of weeds in row crops? This refers to correlations / causalities regarding the occurrence and location of specific weeds, and the visible manifestations of weed characteristics in row crops.

[0087] Explanations of correlations / causalities:

[0088] The fact that certain weeds often only occur in connection with certain crops is related to many factors:

[0089] - Same requirements for general climatic conditions in the region and microclimatic conditions for weeds and crops, so that optimal growth conditions are present (heat, light, water) - Same requirements for generally / principally available nutrients of the soil, through fertilization and irrigation, so that optimal growth conditions are present for weeds and crops

[0090] - Same requirements for soil type, texture, compactness

[0091] - Depending on the field processes taking place for the crops (soil cultivation, seedbed preparation, sowing / planting, fertilization, irrigation, plant growth protection, weed control, harvesting, hay, ...), favorable or less favorable germination and growth conditions for weed seeds arise

[0092] - Certain crop rotations focusing on summer crops preferentially sow weeds known as spring germinators; crop rotations focusing on winter crops sow weeds known as autumn germinators.

[0093] - Depending on the processing depth, light or dark germinators are promoted more.

[0094] It can be visually proven in mulch sowing that, for example, the weed Conyza prefers to grow between the rows of plants in the mulch, but the weed Digitaria prefers to grow in the row of crops... a possible explanation:

[0095] - Moisture can be retained better in the mulch, but there is less light... in the crop row, it is comparatively the other way round (light or water requirements of the weed species may be different)

[0096] - During the sowing process, the mulch is pushed between the rows of plants and only in the sowing row the soil is worked / loosened / aerated / possibly fertilized together with the seed

[0097] - i.e. weed seeds in the crop row can be locally modified, come sufficiently close to the soil surface (air, light, heat ...) which can improve germination and growth conditions (light germinators)

[0098] It has been shown that the multitude of implicit and explicit correlations / causalities of these features or patches, or the features contained therein in a sequence of patches of a crop image with weeds, can be particularly advantageously represented by transformer models. When selecting image data for training the transformer model or the hybrid backbone with transformer and CNN, it is particularly appropriate to acquire and select image data with the implicit knowledge of various historical information such as crop rotation, weather, climate, sequence of certain field processes, etc., in order to avoid or reduce overfitting, i.e., ultimately non-causal correlations in the detection.

[0099] In addition, we also need image data that does not imply false causalities. For example, if I only have images with straw and weeds, this cannot necessarily be causal. For example, I also need training images with weeds without straw. This requires domain expertise for training data selection (straw plays a role, but not the only one).

[0100] Transformers learn the correlations / causalities, unlike convolutional networks. Correlations are diverse, but the transformer can learn the crucial causalities. Convolutional networks, on the other hand, only recognize the plant's characteristics. Convolutional networks don't see the global correlations / causalities.

[0101] All information, and thus also causalities, are mapped in the feature map. Object detection is based on this.

Claims

Claims 1 . Method for localizing and / or classifying weeds (7) in an input image (12), wherein the input image (12) is fed to a network arrangement (10) as a basis for an input matrix, wherein the network arrangement (10) has a feature map device (14), wherein the feature map device (14) generates a feature map (15) on the basis of the input matrix, wherein the network arrangement (10) has a detection device (16), wherein the detection device (16) detects weeds (7) on the basis of the feature map (15), wherein the network arrangement (10) carries out and / or outputs a localization and / or a classification of the weed (7) on the basis of the detected weed (7) in the input matrix, characterized in that the feature map device (14) generates the feature map (15) by means of a data circuit of convolution (22) and transformer blocks (23).

2. Method according to claim 1, characterized in that the feature map device (14) for generating the feature map (15) has at least one parallel section (21), wherein at least one of the convolution blocks (22) and at least one of the transformer blocks (23) are arranged in parallel data-technically.

3. Method according to claim 1 or 2, characterized in that the feature map device (14) for generating the feature map (15) has at least one series section (20), wherein at least one of the convolution blocks (22) and at least one of the transformer blocks (23) are arranged in series.

4. Method according to claim 3, characterized in that in the row section (20) the folding block (22) is arranged in front of the transformer block (23).

5. Method according to one of the preceding claims, characterized in that the convolution block (22) generates an intermediate convolution output matrix (25) from an intermediate convolution input matrix (24) and / or that the transformer block (23) generates an intermediate transformer output matrix (27) from an intermediate transformer input matrix (26) 6. The method according to claim 5, characterized in that one or more convolution operations are carried out in the convolution block (22), the intermediate convolution input matrix (24) corresponding in the number of columns and rows to the intermediate convolution output matrix (25).

7. Method according to one of the preceding claims 5 or 6, characterized in that in the transformer block (23) the intermediate transformer input matrix (26) is broken down into a plurality of patches and / or matrix sections and a transformer vector is formed for each patch, the transformer vector being mapped into the intermediate transformer output matrix (27), the intermediate transformer input matrix (26) corresponding to the number of columns and rows of the intermediate transformer output matrix (27).

8. Method according to one of the preceding claims, characterized in that the detection device (16) determines and outputs the classification and a boundary frame as a localization for the detected weed (7).

9. Method according to one of the preceding claims, characterized in that the detection device (16) has a head device (18), wherein the head device (18) carries out the classification of the detected weed (7) and / or the localization of the detected weed (7) on the basis of the feature map (15).

10. The method according to claim 9, characterized in that the detection device (16) has a neck device (17), wherein the neck device (17) is arranged in terms of data technology between the feature card device (14) and the head device (18), wherein the neck device processes the feature card (15).

11. A computer program, wherein the computer program is designed and / or configured to execute, apply and / or implement the method according to one of claims 1 to 10 when executed.

12. A machine-readable storage medium, wherein the computer program according to claim 11 is stored on the storage medium.

13. Weed search system (2) for locating / or classifying weeds, comprising a data processing device (9), wherein the data processing device (9) is particularly designed to carry out the method according to one of claims 1 to 10, an input interface (6) for accepting an input image (12) and for transferring the input image (12) to the data processing device (9), an output interface (11) for accepting the localization and / or classification of the weed from the data processing device (9) and for outputting the localization and / or classification of the weed (7), wherein the data processing device (9) comprises a network arrangement (10), wherein the network arrangement (10) has a feature map device (14) for generating a feature map (15) based on an input matrix, wherein the input matrix is created based on the input image (2),wherein the network arrangement (10) has a detection device (16) for detecting the weeds (7) on the basis of the feature map (15), wherein the network arrangement (10) is designed to output the localization and / or the classification of the weed (7) on the basis of the detected weed (7) in the input matrix, characterized in that the feature map device (14) has a data circuit of convolution blocks (22) and transformer blocks (23) for generating the feature map (15).

14. Weed control system (1) comprising a camera (3) for generating an input image, the weed search system (2) according to claim 13, wherein the camera is data-technically connected to the input interface (6), a weed control arrangement (4), wherein the weed control arrangement (4) is data-technically connected to the output interface (11) and wherein the weed control arrangement (4) is designed to carry out weed control on the basis of the localization and / or classification of the weed (7).

Images