Mobile analysis and processing device

Abstract

The invention relates to a mobile analysis and processing device (14) for agriculture for processing the soil and / or manipulating flora and fauna. The device (14) comprises at least one sensor (62), a tool unit (52) having at least one motor-driven tool (58), an actuator (48) for moving at least the tool (58) of the tool unit (52), a motor (50) for driving the tool (58) and / or the actuator (48), a database (78), a first communication unit (60) having an interface (70a), and a first computer (46) for controlling the sensor (62), the tool unit (52) and the actuator (48) by means of generated control commands. The data captured by means of the sensor (62) are continually compared with the data stored in the database (78) in order to generate corresponding control signals for the actuator (48), the tool unit (52) and / or the motor (50). By means of said device, mobility and flexibility are created, in accordance with which flexibility the device (14) forms a unit by means of which all data can be processed in real time, control signals can be generated for the actuator (48), the tool unit (52) and / or the motor (50) and immediate operation in accordance with the control signals is possible. Thus, combination with, for example, different carriers (12), which move the device (14) over the field if necessary, is possible.

Description

[0001] Mobile analysis and processing device

[0002] The invention relates to a mobile analysis and processing device for agriculture for soil cultivation and / or for manipulating flora and fauna according to claim 1, and a method for real-time control of soil cultivation and / or manipulation of flora and fauna by the device according to claim 15.

[0003] Weed control in agriculture is a very labor-intensive task, especially in organic farming, which prohibits or restricts the use of chemicals. Depending on the crop being cultivated, weed control is necessary in the immediate vicinity of the crop. This control generally takes place in the early stages of growth. Both the crop and the weeds are still very small and close together at this stage. To avoid damaging the crop, it is advisable to use selective methods. In organic farming, for example with carrots, this is achieved through a labor-intensive,

[0004] Physically damaging manual labor using so-called "weeding machines". Seasonal workers lie on their stomachs on a cot and remove the weeds.

[0005] For specialty crops with wider plant spacing, such as sugar beets or lettuce, tractor-mounted implements are available that can recognize individual crops and control the appropriate tools so that the area of ​​the crop remains untouched. Selectivity is not necessary for this task. This means that these systems do not check the areas to be worked, but rather select the tool "blindly" based on the known plant characteristics.

[0006] Controlling crop position. Generally, the distance to the crop defines the

[0007] Accuracy requirements. From DE 40 39 797 A1, a device for weed control is known, wherein an actuator for destroying the weeds runs continuously and is only briefly interrupted when a sensor detects a cultivated plant.

[0008] From DE 10 2015 209 879 A1, a device for damaging weeds is known, which includes a processing tool. This processing tool serves to damage the weeds. In addition, a classification unit is provided which either contains the positional data of the weeds or recognizes the weeds and determines the positional data.

[0009] The localization unit determines a relative position between the cultivation tool and the weed. A manipulator unit in the form of a carriage positions the cultivation tool accordingly, depending on the determined relative positions.

[0010] A corresponding device with a pressure delivery unit and a liquid dispensing unit is known from DE 10 2015 209 891 A1. In this embodiment, the weeds are destroyed by spraying liquid under pressure.

[0011] From DE 10 2015 209 888 A1 it is known to apply the liquid to the weeds in an impulse manner and thus damage them.

[0012] German patent application DE 10 2013 222 776 A1 discloses a stamp stored in a wagon, which is arranged in a guide device for guiding the stamp. The stamp is placed on the weeds and pressure is applied. The weeds are destroyed by this pressure.

[0013] New paths are currently being forged with agricultural robots and harvesting machines that are automated and equipped with telematics to provide technical support for agriculture. Many technical principles and insights from space travel, remote sensing, and robotics can be applied to agricultural challenges; however, these must be adapted to the specific needs of the agricultural sector.

[0014] Adapting tasks in agriculture in a targeted manner requires new equipment and procedures.

[0015] For example, the aforementioned existing automated agricultural robots are inherently designed to only process one row of plants at a time. Interventions are limited to the flora and are carried out sequentially. A subsequent inspection is typically performed by a qualified person, for example, through a walk-through inspection.

[0016] A disadvantage of the known devices is that they are all special constructions of wagons, which can only transport the crops in rows and are relatively inflexible in their use.

[0017] The invention is based on the objective of providing a mobile analysis and processing device for agriculture for soil cultivation and / or the manipulation of flora and fauna, as well as a method for the device that enables real-time controlled, qualified removal of the detected flora and / or fauna and also a parallel analysis of flora and fauna. Preferably, the device should be connectable to different carriers that move the device to and across the deployment site.

[0018] This problem is solved for the mobile analysis and processing device by the features of claim 1 and for the method by the features of claim 15.

[0019] The dependent claims constitute advantageous further developments of the invention.

[0020] The invention is based on the insight that by creating a carrier

[0021] independent mobile device, which has all the necessary components for analysis and processing, significantly increases the operational flexibility and the resulting possibilities.

[0022] The invention therefore relates to a mobile analysis and processing device for agriculture for soil cultivation and / or for manipulating flora and fauna. The device comprises at least one sensor, a tool unit with at least one motor-driven

[0023] tool, an actuator for moving at least the tool of the tool unit, a motor for driving the tool unit and / or the actuator, a database, a first

[0024] A communication unit with an interface and a primary computer for controlling the sensor, tool unit, and / or actuator based on generated control commands. The data acquired by the sensor is continuously compared with the data stored in the database to generate corresponding control signals for the sensor, tool unit, and / or actuator. This device provides mobility and flexibility, forming a single unit that processes all data in real time, generates control signals for the sensor, tool unit, and / or actuator, and executes them immediately. This allows for combinations with various carriers, enabling the device to be moved across the field as needed.

[0025] Preferably, the data acquired by the sensor is compared with the database in real time, in particular with verification and classification of the data acquired by the sensor. This increases the responsiveness of the device.

[0026] According to one embodiment of the invention, the sensor is a visual detection unit with a camera. The data to be processed is therefore image data, which can easily be compared with a database.

[0027] In order to be able to connect the device to a carrier which moves the device as needed, appropriate means for connecting it to the carrier are provided.

[0028] To allow for easy replacement of individual components and thus reduce setup times, the device is designed in two parts. The first unit contains the sensor, which

[0029] The first unit contains the tool unit, the motor for driving the tool unit's tool and / or the actuator, the actuator itself, the first computer, and the first communication unit with an interface. The second unit contains the database, a second computer, and a second communication unit with an interface. The first and second units can be connected via the interface for data exchange. Furthermore, this two-part design allows the two units to be positioned separately. This is advantageous, for example, when the weight of the moving parts of the device needs to be minimized. In this case, the second unit could be fixed centrally, while the first unit could be moved around the field.

[0030] It is advantageous if the first unit comprises a first housing and the second unit a second housing, which protects the components provided in the units from external influences.

[0031] The first and second housings can be detachably connected to each other via a plug connection. This allows for the modular assembly of the two units, as well as easy replacement in the event of a unit failure. According to one embodiment of the invention, the first and second housings have receptacles for connecting to the carrier as needed. These receptacles correspond to the carrier's holding elements, allowing the device to be gripped and moved by the carrier. Alternatively or additionally, the first and second housings can have coupling elements for connecting to the carrier as needed. These coupling elements correspond to the carrier's holding elements, allowing the device to be connected to and moved by the carrier. This enables simple and quick connection to a carrier for transporting the device.

[0032] The tool unit preferably comprises at least one feed unit and one rotation unit, which interacts with the motor. This easily expands the tool's range of applications without requiring the device to be moved.

[0033] Preferably, the rotating unit is equipped at a distal end with at least one tool, in particular a milling cutter or a cutting unit. For example, small insects or weeds can be selectively destroyed by rotating the cutting unit.

[0034] To further reduce the weight of the device, a power connection for an external power supply is provided. This power connection can be located on the first unit. When the first and second units are assembled, the second unit can supply power to both the first and second units via this power connection. Preferably, the power source of a support structure is used for this purpose.

[0035] Power supply used.

[0036] To enable data exchange between a carrier and the device, an additional communication interface is provided on the device for the carrier.

[0037] The additional communication interface can be located in either the first or the second unit. Preferably, it is located in the second unit.

[0038] The aforementioned task is also solved by a method for real-time control of soil processing and / or manipulation of flora and fauna by the device of the type just mentioned, wherein the method comprises the following steps: continuous temporal recording of data-technically defined voxels and / or pixels and / or images by the sensor;

[0039] Transmission of the recording data to the database;

[0040] Storage of the recording data in the database;

[0041] qualitative data comparison of the recording data with the data stored in the database, preferably with the execution of segmentation, data reduction and / or verification of the recording data by the computer;

[0042] Evaluation of the compared data with existing defined datasets in the

[0043] Database through a classifier in conjunction with the computer;

[0044] Processing and implementation of the evaluation by the computer into control and / or regulation data for the motor, the actuator, the tool unit and / or an associated carrier.

[0045] Preferably, once the control and / or regulation data is available, the motor, actuator, tool unit and / or an associated carrier is started up for soil processing and / or manipulation of flora and fauna.

[0046] According to a preferred method of the invention, the evaluation is performed in a computer interacting with the classifier, in particular in the second computer, and the

[0047] The processing and conversion of the evaluation into control and / or automation data is carried out on another computer, specifically on the first computer, for which purpose the evaluation is transmitted from one computer to the other. This reduces the computing time, as the computers can work in parallel. Furthermore, it allows the two computers to be located separately. For example, the second computer with the second unit can be located remotely from the first unit with the first computer.

[0048] The storage, qualitative data comparison of the recording data with data stored in the database, and / or the evaluation by the classifier are preferably supported by artificial intelligence. This allows for the creation of a virtually autonomous system.

[0049] The device, in particular the first and second units, can be modular, allowing them to be connected to each other and to other units of a complete system. Real-time refers to the ability to perform analysis and processing operations in a single step.

[0050] To be able to carry out the work step in situ.

[0051] In the context of the invention, a voxel is understood to be a spatial data set that is generated discretely or continuously over time via the sensor or a visual detection unit in an imaging process.

[0052] The actuator preferably has a mechanism, in particular a rotary unit, which is located in a holder in the housing of the first unit.

[0053] Further advantages, features and applications of the present invention will become apparent from the following description in conjunction with the drawings.

[0054] Examples of implementation.

[0055] The description, claims, and drawing use the terms and associated reference numerals listed below. In the drawing, this means:

[0056] Fig. 1 shows a schematic representation of a support system with spatially separated housings of a mobile device according to a first embodiment of the invention;

[0057] Fig. 2 shows a schematic representation of a support system with spatially separated housings of a mobile device, which are connected to each other via a plug connection, according to a second embodiment of the invention;

[0058] Fig. 3 shows a side view of the support system according to the first embodiment of the invention with the mobile device attached to a flying drone;

[0059] Fig. 4 shows a flowchart illustrating the steps of a process using the support system.

[0060] illustrated;

[0061] Fig. 5 is a flowchart illustrating the steps of the procedure for determining the necessary measures; Fig. 6 is an image captured by the visual detection unit;

[0062] Fig. 7 shows a schematic structure of a convolutional neural network based on the image in Fig. 6;

[0063] Fig. 8 shows a flowchart illustrating a method of the segmentation and data reduction unit;

[0064] Fig. 9 shows an intermediate image created from the segmentation and data reduction unit;

[0065] Fig. 10 shows a partial section of the intermediate image with three different cases for the

[0066] Classifier;

[0067] Fig. 1 shows two basic representations of further pixel fields for evaluation by the

[0068] Classifier;

[0069] Fig. 12 shows two basic representations of further pixel fields for evaluation by the

[0070] Classifier;

[0071] Fig. 13 shows an image created and evaluated by the classifier, and

[0072] Fig. 14 shows the basic representation of how the classifier works.

[0073] Figure 1 schematically depicts a carrier system 10, consisting of a carrier in the form of a drone 12 and a mobile device 14 for soil cultivation and manipulation of flora and fauna in agriculture. The drone 12 includes a drive system 16 comprising four electric motors 18 and propellers 20 driven by these motors (see Figure 3). The drone 12 is also equipped with four feet 22 below the electric motors 18.

[0074] According to the first embodiment of the invention, the drone 12 comprises an energy source in the form of batteries 24, which provides the energy supply for the drive 16 as well as for the other components of the drone 12 and the mobile device 14. For this purpose, a

[0075] A voltage interface 26a is provided on the drone 12, and a corresponding voltage interface 26b is provided on the mobile device 14. These are connected to each other via a detachable connector 28. A communication unit 30 with an antenna 32 and a GPS unit 34 are also provided. These continuously determine the location of the drone 12 and transmit this location data to the mobile device 14 for correlation with the data acquired by the mobile device 14, as well as to a remote central processing unit, which is not shown in detail here. Telemetry can be performed using the GPS unit 34, the communication unit 30, and the mobile device 14. A control unit 12b is also provided, which controls the drive 16.

[0076] The communication unit 30 of the flying drone 12 includes, in addition to the antenna 32, a further interface 36a, which is assigned to an associated interface 36b of the mobile device 14 and are connected to each other via a detachable plug connection 38 for data exchange.

[0077] The mobile device 14 consists of two units 14a and 14b: a first unit 14a with a first housing 40 and a second unit 14b with a second housing 42. The first housing 40 and the second housing 42 are detachably connected to each other via a plug connection 44 to form the mobile device 14. There is a set of different first units 14a on one side and a set of different second units 14b on the other, which can be individually configured and adapted to specific needs by simply connecting them together.

[0078] The first housing 40 contains a first computer 46, an actuator in the form of a movable, motor-driven arm 48, a motor 50 interacting with the arm 48, and a tool unit 52 arranged on the arm 48, which comprises a feed unit 54 and a rotary unit 56. A milling cutter 58 is provided as a tool at the distal end of the rotary unit 56. The motor 50 drives the arm 48, the feed unit 54, the rotary unit 56, and thus also the milling cutter 58. The arm 48 can be multi-part and have various joints, which are not shown individually, as such motor-driven kinematics are known.The tool unit 52 is moved relative to the drone 12 to its operating area via the arm 48, so that the tool unit 52, together with the feed unit 54 and the rotation unit 56, can use the milling cutter 58 for processing plants, for example removing weeds, and / or for cultivating the soil. Furthermore, the first unit 14a includes a communication unit 60 and a visual detection unit 62. The visual detection unit 62 comprises a camera 64 for capturing images, a segmentation and data reduction unit 66, and a classifier 68, which is generated based on data from the segmentation and data reduction unit 66.

[0079] The classification of several pixel fields, consisting of pixels, is performed using intermediate images or intermediate data, as described in more detail below. The visual detection unit 62 is connected to the communication unit 60.

[0080] The first unit 14a has an interface 70a, which is assigned to an interface 70b of the second unit 14b. Communication unit 60 is connected via a communication link 72 to interface 70a and then to a communication unit 74 in the second unit 14b. Communication unit 74 of the second unit 14b is connected via interface 36b and connector 38 to interface 36a and communication unit 30 of the drone 12.

[0081] In the second unit 14b, a second computer 76 and a database 78 are also provided.

[0082] Figure 2 shows a further embodiment of the carrier system 10, wherein the flying drone 12 is identical to the first embodiment. Only the mobile device 14 differs by a plug connection 80 between the first unit 14a and the second unit 14b, which also connects the communication unit 60 of the first unit 14a to the

[0083] Communication unit 74 connects detachably. They can be easily connected by simply plugging them together.

[0084] different first units 14a are combined with different second units 14b and joined together to form a mobile unit 14.

[0085] Figure 3 shows a side view of the drone 12, revealing only two of the four electric motors 18 with their associated propellers 20. The feet 22 are located below each of the electric motors 18. Two gripping arms 82a and 82b are positioned between the feet 22. These arms grasp and lift the mobile device 14 as needed, and then lower and set it down again. The mobile device 14 consists of two units, 14a and 14b, which are detachably connected via a plug connector 80. The camera 64, part of the visual detection unit 62, and the milling cutter 58 at the distal end of the rotary unit 56 are visible in the first unit, 14a. The mobile device 14 can also be equipped with several different tool units 52, which are connected to a common arm 48 and, for example, a tool turret that moves the required tool unit 52 into the activation position.It is also conceivable that the different tool units each have their own actuator.

[0086] Figure 4 shows in a flowchart the steps that are carried out successively to use the carrier system 10 to cultivate the soil and manipulate flora and fauna in agriculture.

[0087] In a first step 84, the necessary measures on the assigned agricultural area are determined using the carrier system 10. For example, the carrier system 10 is transported to an agricultural area to be worked on, such as a field, or flown there directly from a central location. The drone 12 with the mobile device 14 takes off and flies over the agricultural field. The carrier system 10 receives the necessary data about the agricultural field to be assessed via a stationary central processing unit. This central processing unit can also be a smartphone. The agricultural field is captured in images using the visual detection unit 62 with the camera 64 of the mobile device 14. The images are evaluated, and the necessary measures for this agricultural field are ultimately determined by comparing them with the database 78.

[0088] In a next step 86, based on the measures identified for the agricultural field or for parts of the agricultural field, the mobile unit 14 suitable for the necessary measure is assembled from a set of first units 14a and a set of different second units 14b and the two units 14a, 14b are connected together.

[0089] In a next step 88, the mobile device 14 is grasped laterally by the gripper arms 82a and 82b, respectively, and moved upwards towards the drone 12 into a receptacle 12a of the drone 12. During this process, the voltage interfaces 26a and 26b are connected via connector 28, and the interfaces 36a and 36b are connected via connector 38. This supplies the mobile device 14 with power from the battery 24 of the drone 12 and enables data exchange via the antenna 32 of the communication unit 30 of the drone 12 with the communication units 60 and 74 of the mobile device 14 on one side and a central processing unit on the other. As described above, the central processing unit, which is independent of the carrier system 10, can also be a smartphone.

[0090] In the next step, the identified measures are implemented in the agricultural field using the carrier system. For example, the drone 12 flies to the

[0091] The arm 48 with the tool unit 52 moves to the weed to be removed. The feed unit 54 moves the milling cutter 58 to the weed so that it is milled away when the rotary unit 56 is activated.

[0092] In a fifth step 92, the drone 12 then flies back and exchanges the mobile device 14 for another mobile device 14, which is optimized for a different measure, for example with an application device for pesticides or fertilizer.

[0093] Alternatively, steps 86 and 88 can be omitted if the drone 12 is already fully equipped for the task to be carried out.

[0094] The determination of the necessary measures by the carrier system 10, in particular by the mobile device 14, is now explained in detail with reference to Figure 5.

[0095] In a first step 94, the visual detection unit 62 of the mobile device 14 continuously records data of technically defined voxels and / or pixels and / or images. The voxels, pixels and images form recording data which are continuously transmitted to the database 78 - second step 96.

[0096] In a third step, 98, the recording data is stored.

[0097] In a fourth step, a qualitative data comparison of the recording data with the data stored in database 78 is carried out. This involves performing a segmentation and

[0098] Data reduction of the acquisition data by the segmentation and data reduction unit 66. In particular, verification of the acquisition data can also be carried out by the second computer 76.

[0099] In a fifth step 102, the evaluation is carried out by the classifier 68 in conjunction with the second computer 76, supported by artificial intelligence, as will be explained in detail below. In a sixth step 104, the evaluation is finally processed and implemented by the first computer 46 into control and regulation data for the motor 50, the arm 48, the tool unit 52, and the drone 12.

[0100] In a seventh step 106, the motor 50, the arm 48 and the tool unit 52 are finally started up for working the soil or for manipulating flora and fauna.

[0101] When this application refers to artificial intelligence, it is referring, among other things, to the use of a classic convolutional neural network (CNN) consisting of one or more convolutional layers, followed by a pooling layer. This sequence of convolutional and pooling layers can, in principle, be repeated any number of times. Typically, the input is a two- or three-dimensional matrix, such as the pixels of a grayscale or color image. The neurons in the convolutional layer are arranged accordingly.

[0102] The activity of each neuron is calculated via discrete convolution. Intuitively, this involves incrementally moving a relatively small convolutional matrix (filter kernel) over the input. The input of a neuron in the convolutional layer is calculated as the inner product of the filter kernel and the currently underlying image area. Accordingly, neighboring neurons in the convolutional layer respond to overlapping areas.

[0103] A neuron in this layer responds only to stimuli in a local area of ​​the previous layer. This follows the biological model of the receptive field. Furthermore, the weights for all neurons in a convolutional layer are identical (shared weights). This means that, for example, each neuron in the first convolutional layer encodes the intensity of an edge in a specific local area of ​​the input. Edge detection, as the first step in image recognition, has high biological plausibility. From the shared weights, it follows directly that translational invariance is an inherent property of CNNs.

[0104] The input of each neuron, determined by discrete convolution, is now processed by a

[0105] The activation function, usually called the Rectified Linear Unit (ReLu) (f(x) = max(0, x) in CNNs, is transformed into the output that is intended to model the relative firing frequency of a real neuron.

[0106] Since backpropagation requires the calculation of gradients, in practice a differentiable approximation of ReLu is used: f(x) = ln(1 + e x ). Analogous to the visual cortex, in deeper convolutional layers both the size of the receptive fields and the complexity of the recognized features increase.

[0107] In the next step, pooling, superfluous information is discarded. For object recognition in images, for example, the exact position of an edge in the image is negligible—the approximate localization of a feature is sufficient. There are different types of pooling. By far the most common is max pooling, where, from each 2 x 2 square of neurons in the convolutional layer, only the activity of the most active (hence "max") neuron is used for the other neurons.

[0108] The computational steps are retained; the activity of the remaining neurons is discarded. Despite the data reduction (75% in the example), the network's performance is generally not reduced by pooling.

[0109] The use of the Convolutional Neural Network and segmentation and

[0110] Data reduction device 66 is explained in more detail below with reference to Figures 6 to 14.

[0111] Several approaches exist for classifying all objects in an image using classifier 68. Many approaches begin by first finding the individual objects in the image and then classifying them. However, this is not always possible. As an example, the classification of plants 108 in a field will be used. An example image 108 is shown in Fig. 6.

[0112] Figure 6 shows various plants 108, which are to be classified by the classifier 68 in real time. In this case, real time is defined as the camera rate of 10 frames per second. Since, as in this example, it is not easy to distinguish exactly where a plant 1 10 ends, a different approach must be used, as the computation time is insufficient to first distinguish the plants 1 10 themselves and then classify them.

[0113] Image 108 from Fig. 6 consists of pixels, and each pixel can logically only contain exactly one class. Therefore, a trivial approach would be to classify the entire image 108 pixel by pixel. That is, each pixel is assigned to a class one after the other.

[0114] However, since a single pixel does not contain the necessary information to make a statement about the

[0115] To determine class membership, a surrounding area must be used for classification. This area can then be classified using a Convolutional Neural Network (CNN), as described above. The network can exhibit a sequence like Fig. 7. The input image 1 10 is the image from Fig. 6. The following are then applied to this input image 1 10:

[0116] Elements of the CNN are applied. In this example, these would be the convolution 112 with the features, a subsequent pooling 114, another convolution with further features, another pooling, and a summarization in the dense layer 116. The output of the network then outputs the class membership of the middle pixel of the input image 110, or of a pixel of the image 110 from Fig. 6.

[0117] A new image section, usually shifted by one pixel, is then selected and reclassified using a CNN. This approach reduces the number of calculations required by the convolutional neural network for the number of...

[0118] The classifying pixels must be repeated. This is time-consuming. Image 1 10 of Fig. 6 has a resolution of 2000 x 1000 pixels. The CNN would therefore have to be calculated two million times. However, the initial problem is only the classification of the plants 108 itself. On average, such an image contains about 5% plant pixels, which corresponds to only about 100,000 pixels.

[0119] Through simple segmentation and data reduction using the segmentation and

[0120] Data reduction unit 66 can determine whether a pixel represents part of a plant 108 or background 118. This segmentation is not as computationally complex as a CNN and is therefore faster. The segmentation and

[0121] Data reduction by the segmentation and data reduction unit 66 is carried out analogously to Fig. 8. The individual steps are shown in Fig. 8.

[0122] In a first step 120, each image transmitted to the database 78, consisting of several pixels, is converted into the RGB (Red, Green, Blue) color model.

[0123] In the next step, 122, each pixel of the transmitted image is transferred to an HSV (hue, saturation, value) color model based on the RGB color model.

[0124] In the next step, 124, this HSV color model will be evaluated.

[0125] Each pixel based on the HSV color model is evaluated with respect to color saturation using a threshold value, whereby if the color saturation value exceeds a threshold, the pixel is assigned the binary value 1, and if the color saturation value falls below a threshold, the pixel is assigned the binary value 0.

[0126] In parallel, each pixel is evaluated based on the HSV color model with regard to its hue within a predetermined range, whereby if the hue is within the

[0127] If the color angle lies within a predetermined area, the pixel is assigned the binary value 1, and if the color angle lies outside the area, the pixel is assigned the binary value 0.

[0128] In the next step, 126, the binary information of the color angle and the

[0129] Color saturation creates an intermediate image which contains significantly less data than the image 108 produced by the camera.

[0130] The segmentation shown in Fig. 8 yields the following formula, which must be applied to each pixel. The segmented image S(x,y) is divided into its three components, red, green, and blue, by partitioning the RGB image y{c,g}. A pixel of the segmented image is then set to 1 if the minimum value of red, green, or blue in that pixel, divided by the value of the green pixel, is less than or equal to a threshold value (THs). The threshold value is set to the 8-bit space of the image by scaling using 255. If the threshold value is not reached, the pixel of the segmented image is set to 0, as in Equation 1.

[0131] Ci)

[0132]

[0133] This results in the first optimization: Before the entire image 108 is divided into two million images, the segmentation, as shown in Fig. 8, is applied. This means that the entire image 108 is processed, and a decision is made using the formula given above as to whether it is a plant pixel or not. Firstly, the image 108 is segmented, meaning the background 118 is set to black (0), as shown in Fig. 9. Secondly, if it is a

[0134] The system deals with plant pixels, whose coordinates are written to a list. Subsequently, only the coordinates that are also in this list are fed into the CNN. The unnecessary pixels of the Earth, i.e., the background 1 18, are omitted. This reduces the number of times the CNN is called approximately 20. Through segmentation, the background 1 18 is set to the value 0. The image elements that the CNN considers now also have segmented images. Normally, on a convolution layer, the feature calculation would be applied to each pixel of the image element. However, this results in three cases 128, 130, 132 for the calculation, which are shown in Fig. 10, each for a feature 134 of size 5x5.

[0135] Red Case 128 shows a feature calculation where the feature lies entirely on the background 1 18. Here, each element is multiplied by 0, resulting in the entire calculation yielding 0, or the bias value. The result of this calculation is therefore known before the calculation. Even if the background 1 18 were not zero, i.e., if it contained soil, this calculation would contain no information about the plant 1 10; therefore, the result can simply be a constant, fictitious value.

[0136] In the yellow case 130, the middle feature value does not lie on a plant 1 10. This means that part of it is also a multiplication by zero. This case consumes the plant 1 10 in the border area and makes this size in the feature map.

[0137] In the blue case 132, at least the middle pixel of the feature lies on a plant.

[0138] After considering these three cases 128, 130, and 132, only the yellow and blue cases 130 and 132 need to be calculated. These are cases 130 and 132 in which the feature has at least one non-zero input value. The results of all other feature calculations are known before the calculation; they are either zero or only contain the bias value. The coordinates where the blue case 132 occurs are known. These are the coordinates stored during segmentation. The yellow case 130, in turn, requires a calculation to determine whether it has occurred. This would require checking every plant pixel found in the segmentation. Since this check is too resource-intensive and the yellow case 130 only occurs at the edge of plant 110, this case will be ignored.

[0139] Therefore, the calculation can be optimized so that the feature calculation and all other elements of the CNN are only applied to the plant pixels found.

[0140] Figure 11 schematically illustrates how two adjacent plant pixels 136 and 138 differ. On the left is plant pixel 136, and on the right is its neighboring plant pixel 138. The blue / purple area 140 and 142 could represent different plants that need to be classified. The red / purple area 144 and 142 represents the image element that the CNN considers to classify the orange pixel 146.

[0141] Upon closer inspection, it can be seen that the area under consideration (red / purple) 144, 142 overlaps significantly. This, in turn, means that both image elements 136, 138 contain largely the same values. If the CNN now calculates the feature in the convolution layer, the same values ​​would also be calculated from the feature calculation.

[0142] Figure 12 schematically depicts a 5 x 5 pixel feature 148 in green. This feature 148 is located at the same coordinates within the entire image, but it is shifted within the image element (red / purple) 144, 142 considered by the CNN. Since the location is the same throughout the entire image, the calculation for the middle black box 150 would yield the same value in both the left image 136 and the right image 138. This principle applies to all elements of a CNN. Therefore, if the edge region is ignored, the individual feature calculation can first be applied to the entire image. Theoretically, the decomposition of the input image 108 only becomes crucial with the addition of the dense layer 116. However, the dense layer 116 can be calculated in the same way as a convolution 112. The feature size results from the interplay between the input image size and the available pooling layers in the network.This allows for further optimization of the classification; the CNN elements are now applied only to the identified plant pixels. The feature map calculated from the last convolution represents the classification result, as shown in Fig. 13. Here, all carrot plants (152) are classified pixel-wise in green, and all weeds (154) are classified pixel-wise in red.

[0143] However, these optimizations also change the classification result. The pooling layers have the greatest impact here. With each pooling step, information is removed from the network. Because the individual image elements are no longer considered, the pooling loses its spatial reference. Figure 14 illustrates this problem.

[0144] In Fig. 14, each image element 156 is represented by a red frame 158. Before optimization, each image element would be individually processed by the CNN to classify its central pixel. The right image element 160 is shifted one pixel further to the right. The four colors—purple, blue, yellow, and green—indicate the individual applications of the pooling. As can be seen, they can yield different results because the pooling always starts at the edge and moves forward by one pooling element (here, two fields). This results in two adjacent image elements 156,

[0145] 160 two different pooling elements. This means that if this is to be considered in the optimization, each pooling would result in two new branches for further calculation. This is because the pooling would have to be applied once to the entire image with the starting point in the top left corner, and then again with the starting point in the top left corner plus one pixel. In the subsequent calculation, both pooling results would then have to be processed separately. With a further second pooling, two new paths would again result, so four separate results would have to be calculated. The final result is then composed of the four results, rotating pixel by pixel through them. If only one path is considered after the pooling, it would be...

[0146] The original image is smaller after two pooling processes. The length and width of the original image would then each be only 14 times larger than the original image. Considering all paths, approximately...

[0147] The resulting input image size will be...

[0148] Another difference lies in the missing boundary areas of the plants. Since the features are not applied to all elements by having any overlap with the plant, there are differences in the calculations. This, too, can alter the classification result compared to the conventional calculation.

[0149] The lack of calculation of feature values ​​outside the plant can cause different values, as the result is displayed as zero, which is actually the bias value.

[0150] Although these three factors influence the results, it turns out that the CNN is very robust and therefore the results still meet a very high accuracy standard.

[0151] The next step would be to train the network directly with these modifications so that the network can better adapt to its new calculation and thus compensate for any errors directly in the calculation.

[0152] The segmentation and data reduction unit assigns positional coordinates to the pixels relating to weed 154. Reference symbol list.

[0153] 10 Carrier system

[0154] 12 flying drones

[0155] 12a Recording, recording space of the drone 12 12b Control device of the drone

[0156] 14 mobile devices

[0157] 14a first unit

[0158] 14b second unit

[0159] 14c Mounting for gripper arm on the mobile device 14

[0160] 16 Drive

[0161] 18 Electric motor

[0162] 20 propellers

[0163] 22 feet

[0164] 24 battery

[0165] 26a Voltage interface on the drone 12

[0166] 26b Voltage interface on the mobile device 14

[0167] 28 connectors

[0168] 30 communication units

[0169] 32 Antenna

[0170] 34 GPS Unit

[0171] 36a interface on the drone 12

[0172] 36b Interface on the mobile device 14

[0173] 38 connector

[0174] 40 first housing of the first unit 14a

[0175] 42 second housing of the second unit 14b

[0176] 44 Connector 46 First computer

[0177] 48 Arm as actuator

[0178] 50 engine

[0179] 52 tool units

[0180] 54 Feed unit

[0181] 56 rotation units

[0182] 58 milling machine

[0183] 60 first communication unit of the first unit 14a

[0184] 62 visual detection units

[0185] 64 camera

[0186] 66 Segmentation and data reduction device

[0187] 68 Classifier

[0188] 70a Interface of the first unit 14a

[0189] 70b Interface of the second unit 14b

[0190] 72 Communication connection

[0191] 74 second communication unit of the second unit 14b

[0192] 76 second computer

[0193] 78 database

[0194] 80 connector

[0195] 82a Gripper arm, left

[0196] 82b Gripper arm, right

[0197] 84 First step: Determining the necessary measures

[0198] 86 Second step: Selecting the mobile device 14 from the available mobile devices 14

[0199] 88 Third step: Connecting the device to the drone 12

[0200] 90 Fourth step: Implementing the identified measures

[0201] 92 fifth step: replacing the mobile device 14 with another mobile device 14 and carrying out a further measure

[0202] 94 First step: continuous intake

[0203] 96 Second step: Data transmission

[0204] 98 Third step: Storing the data

[0205] 100 Fourth step: Data comparison

[0206] 102 fifth step: Evaluation by the classifier 68

[0207] 104 sixth step: Implementation in control engineering data 106 seventh step: Starting up the units

[0208] 108 Example image, entrance image

[0209] 110 plants

[0210] 112 Convolution

[0211] 114 Pooling

[0212] 116 Summary in a Dense Layer

[0213] 118 Background

[0214] 120 First step: Convert to an RGB color model

[0215] 122 Second step: Transfer to an HSV color model

[0216] 124 Third step: Evaluation of the HSV image

[0217] 126 Fourth step: Generating an intermediate image

[0218] 128 first case, red

[0219] 130 second case, yellow

[0220] 132 third case, blue

[0221] 134 Features

[0222] 136 plant pixels, left

[0223] 138 plant pixels, right

[0224] 140 blue area

[0225] 142 purple area

[0226] 144 red zone

[0227] 146 orange area

[0228] 148 Feature, green

[0229] 150 medium black box

[0230] 152 Carrot plant

[0231] 154 Weeds, unwanted plants

[0232] 156 Image element, left

[0233] 158 red frame

[0234] 160 image element, right

Claims

Patent claims 1. Mobile analysis and processing device (14) for agriculture for soil cultivation and / or for manipulating flora and fauna, comprising at least one sensor (62), a tool unit (52) with at least one motor-driven tool (58), an actuator (48) for moving at least the tool (58) of the tool unit (52), a motor (50) for driving the tool unit (52) and / or the actuator (48), a database (78), a first communication unit (60) with interface (70a) and a first computer (46) for controlling the sensor (62), the tool unit (52) and the actuator (48) based on generated control commands, wherein the data acquired via the sensor (62) are continuously compared with the data stored in the database (78) in order to to generate appropriate control signals for the actuator (48), the tool unit (52) and / or the motor (50).

2. Analysis and processing device according to claim 1, characterized in that the data comparison of the data determined by the sensor (62) with the database (78) takes place in real time, in particular with a verification and classification of the data determined by the sensor (62).

3. Analysis and processing device according to claim 1 or 2, characterized in that the sensor is a visual detection unit (62) with a camera (64).

4. Analysis and processing device according to one of the preceding claims, characterized in that means (82a, 82b) are provided for connecting to a carrier (12) as required, which moves the device (14).

5. Analysis and processing device according to one of the preceding claims, characterized by a two-part design, wherein in a first unit (14a) the sensor (62), the tool unit (52), the motor (50) for driving the tool (58), the tool unit (52) and / or the actuator (48), the actuator (48), the first computer (46) and the first communication unit (60) with interface (70a) are provided, and in the second unit (14b) the database (78), a second computer (76) and a second Communication unit (74) with interface (36b, 70b) are provided, wherein the first unit (14a) and the second unit (14b) are connected via the interfaces (70a, 70b) to the Data exchange is possible.

6. Analysis and processing device according to claim 5, characterized in that the first unit (14a) comprises a first housing (40) and the second unit (14b) comprises a second housing (42).

7. Analysis and processing device according to claim 6, characterized in that the first and the second housing (40, 42) are detachably connected to each other via a plug connection (44, 80).

8. Analysis and processing device according to claim 6 or 7, characterized in that the first and the second housing (40, 42) have receptacles (14c) as means for connecting to the carrier (12) as required, which are assigned to corresponding holding means (82a, 82b) of the carrier (12), by means of which the device (14) can be grasped and moved by the carrier (12).

9. Analysis and processing device according to one of claims 6 to 8, wherein characterized in that the first and second housings (40, 42) have coupling means (26b, 28, 36b, 38) as means for connecting to the carrier (12) as required, which are assigned to corresponding coupling means (26a, 28, 36a, 38) of the carrier (12), via which the device (14) can be connected to the carrier (12) and moved.

10. Analysis and processing device according to one of the preceding claims, characterized in that the tool unit (52) comprises at least one feed unit (54) and has a rotating unit (56) which interacts with the motor (50).

11. Analysis and processing device according to claim 10, characterized in that the rotation unit (56) has at least the tool (58) at a distal end, in particular a milling cutter (58) or a knife unit.

12. Analysis and processing device according to one of the preceding claims, characterized by a voltage connection (26b) for an external Power supply.

13. Analysis and processing device according to claim 12 and one of claims 4 to 1 1 , characterized in that the voltage connection (26b) is provided on the first unit (14a) and the first unit (14a) as well as the second unit (14b) are supplied with voltage via this.

14. Analysis and processing device according to one of claims 4 to 13, characterized by a further communication interface (36b) for a carrier (12).

15. Analysis and processing device according to claim 14, characterized in that the further communication interface (36b) is arranged in the second unit (14b).

16. Method for real-time control of soil cultivation and / or manipulation of flora and fauna by the device according to one of the preceding claims comprising the following steps: a. continuous recording of data-defined voxels and / or pixels and / or images by the sensor over time; b. Transmission of the recording data to the database (78); c. Storage of the recording data in the database (78); d. qualitative data comparison of the recording data with those in the database (78) stored data, preferably with the performance of segmentation, data reduction and / or verification of the recording data by the computer (46, 76); e. Evaluation of the compared recording data with existing defined data records in the database (78) by a classifier (68) in conjunction with the computer (46, 76); f. Processing and implementation of the evaluation by the computer (46, 76) into control and / or regulation data for the motor, the actuator, the tool unit and / or an associated carrier.

17. Method according to claim 16, characterized in that, after the availability of the control and / or regulation data, the motor (50), the actuator (48), the tool unit (52) and / or the carrier (12) for processing the soil and / or an associated carrier (12) for manipulating flora and fauna is started.

18. A method according to claim 16 or 17, characterized in that the evaluation is performed in a computer (46, 76) cooperating with the classifier (68), in particular in the second computer (76), and the processing and conversion of the evaluation into control and / or automation data is performed in another computer (46, 76), in particular in the first computer (46), for which purpose the evaluation is transmitted from one computer (46, 76) to the other computer (46, 76).

19. A method according to claim 16 or 17, characterized in that the storage, the qualitative data comparison of the recording data with data stored in the database (78), and / or the evaluation by the classifier (68) are supported by artificial intelligence.

Images