Method for the computer-aided learning of an artificial neural network for the recognition of structural features of objects

Description

[0001] The invention relates to a method for computer-aided learning of an artificial neural network for detecting suitable cutting positions for cloning a plant according to claim 1. Furthermore, the invention relates to an artificial neural network according to claim 16.

[0002] In automation technology, robotics, autonomous driving, and generally for pattern or person recognition, etc., systems based on artificial intelligence (AI) or artificial neural networks (ANN) are used for fully or partially automated control. These AI systems or ANNs are based on trained data and are intended to take the robot's environment into account during control, at least in semi-autonomous operation of a robot, for example. To this end, appropriate reactions are initiated for events or situations that occur in the robot's environment. Using the prominent example of autonomous driving, for example, a vehicle must be controlled in such a way that collisions with obstacles and / or other road users are avoided, or that the vehicle follows the course of the road. This ANN is trained using training data sets in order to gradually teach the AI system roadworthy autonomous driving.

[0003] Due to the high complexity of environmental recognition processes, the use of ANNs is not only suitable for autonomous vehicle driving. For example, ANNs use image data captured by a camera of (any) object to classify the objects or structural features recognizable in the image data. This allows the objects to be identified automatically.

[0004] The patent application US 2018 / 220589 A1 discloses a method in which suitable cutting positions for pruning, harvesting, spraying and / or caring for plants are determined by means of an artificial neural network.

[0005] In addition to autonomous driving or person or speech recognition, the full automation of complex production processes represents another area of application for AI systems. For example, it is known that ornamental and useful plants can be propagated or multiplied almost fully automatically. Due to the steadily increasing demand for ornamental and useful plants in recent years, the entire propagation process has been industrialized. This industrialization of plant propagation includes the automated separation of a plant component (clone) from a mother plant and the automated feeding of the separated plant component to a nutrient medium. This vegetative propagation process, or plant cloning, almost completely eliminates the need for cost-intensive personnel.

[0006] Identifying a suitable cutting position on the mother plant and then separating or cutting it has proven particularly difficult and therefore critical in the fully automated plant propagation process. The success of the propagation and the reproduction or growth rate of the separated plant part or clone depend crucially on the position on the mother plant at which the cut is made. Every plant has areas that are particularly likely to continue growing or where roots are particularly likely to form after a cut. In addition to the varying suitability of areas or parts of a plant for division or cloning, these areas vary from plant to plant and from species to species. Areas on an orchid, for example, that are particularly suitable for a separating cut may be less suitable for a corresponding cut on a hemp plant.Therefore, it is crucial to identify an optimal cutting area not only for each individual plant, but also for different plant species intended for propagation.

[0007] This plant-specific and plant-species-specific recognition of preferred cutting areas poses a major challenge for industrialized plant propagation. While it is not difficult for trained personnel to select an optimal cutting area for each specific plant, this selection of a preferred cutting area is impossible with conventional image recognition systems. Even known AI systems or ANNs do not provide reliable results for the recognition of structural features on individual objects, such as plants, that are suitable for specifically cutting plants. In particular, when training an ANN for image recognition, known methods tend to ignore structural details of objects. However, when these details are crucial for object classification or feature recognition, the known ANNs prove unsuitable.

[0008] The invention is therefore based on the object of creating a method for computer-aided learning of an artificial neural network for the recognition of structural features on objects and an artificial neural network by means of which identified structural features on objects can be recognized quickly and reliably.

[0009] A solution to this problem is described by the measures of claim 1. Accordingly, it is provided that for training an artificial neural network for feature recognition on objects, a convolutional neural network (CNN), in particular a regional convolutional neural network (R-CNN), with a large number of neurons. This network has a large number of convolutional and / or pooling Layers ( layers) to extract information from images of individual objects. This serves to recognize the structural features of the objects for a classification of the features by further layers of the network. The images of the objects are scaled or upscaled and / or downscaled from layer to layer of the ANN. When scaling the images from layer to layer, information about the structural features of the objects is retained, regardless of the scaling of the images. Scaling by a well-known bilinear interpolation always leads to a blurring effect, since intensities of pixels in the images are calculated as a weighted sum of neighboring pixels and thus details or finer structures are lost. The advantage of the method described here compared to the so-called Nearest neighbor interpolation(bilinear interpolation) consists precisely in the fact that in a reduced image, finer structures are initially present, even if their pixel intensity is often considerably reduced.

[0010] To learn what a possible cut through a plant, especially through a meristem of a plant or through another plant part, looks like, the ANN needs to know which pixels belong to the possible cut. This is determined by scanning each pixel of a cut during a Labelings is marked as a white pixel. If these white pixels or pixel intensity are dimmed or erased during the required scaling steps in the learning process, the ANN will learn exactly this, which will negatively impact the results.

[0011] To solve this problem, the invention stores image information or coordinates of the relevant points or features of a possible intersection pattern or intersection curve instead of the pixel intensities. These coordinates are scaled to the new image size during scaling. The intersection curve is then drawn on the newly scaled image so that each pixel intensity of each curve pixel is white, as it should be. This allows the ANN to learn all the details of the intersection curve or structural features.

[0012] Furthermore, according to the invention, several, in particular 2, 3, 4, 5 or 6, images of the same object are created from different perspectives to recognize the structural features or the intersection curve and simultaneously transferred to the ANN, whereby computer-aided operations for recognizing the structural features of the object in the images are carried out in parallel on several GPUs. This enables very fast recognition of the features compared to processing the individual images. Several cameras are available to capture the object from different perspectives. These cameras, in particular 2, 3, 4, 5 or 6, are arranged around the object, preferably the plant, or the object is fed to the cameras. The cameras are triggered by hardware in a predefined sequence to ensure optimal image quality with regard to image exposure and the absence of cross-fading from the lights in front of a camera.The cameras are located at the corners of a polygon, particularly a hexagon, and image the object in increments of, for example, 60 degrees. If further increments or a different increment are required, a robotic arm can rotate the object by 30 degrees or the appropriate number of degrees and restart the imaging process. Afterward, for example, 12 images in 30-degree increments can be available. This process can be repeated until the desired increment is reached (18 images in 20-degree increments, 36 images in 10-degree increments, etc.).

[0013] The AI system or ANN must evaluate all images captured of an object to determine the best angle (in the case of plants, the angle or orientation between the object and the cutting tool that is most suitable for cutting). More than one suitable angle may also exist. If the images are passed to the ANN individually, the process takes a very long time, as the ANN must be reinitialized after each image. If, instead, six or more images are combined and presented simultaneously, the processing time is reduced by a factor of almost the number of images presented (six times faster for six images, etc.). This is, of course, only possible as long as the scaled or reduced-size and combined images still contain enough information for the ANN to achieve good performance. In particular, the invention further provides that the multiple images of the same object are scaled before being passed to the neural network, in particular. rescaled, and stitched become.

[0014] Furthermore, it can be provided according to the invention that a Labeling, or a naming, marking, labeling of the features of the objects in the images is carried out semi-automatically in preparation for the learning process of the ANN, wherein preferably first an isolation of the features and / or the objects from a background of the images takes place and in particular subsequently a Labeling of the characteristics by a person. This minimizes the necessary person-related interaction and therefore leads to a faster and more accurate Labeling or labeling. This step is crucial for the AI system to learn how to process new objects.

[0015] Typically, image labeling requires that the person who LabelingThe person carrying out the labeling or labeling precisely identifies the objects of interest in order to separate them from other objects and from the background. This process is very time-consuming and requires a high level of attention. To improve and speed up this process, the ANN separates the objects from each other and from the background using color segmentation. When cutting up the plant, the person labeling only has to place the cuts. All other contours of the object or plant are then automatically detected or extracted. The same process is also used to label or label the plant. labeling Person at Labeling of plants on the conveyor belt. The plant contour is automatically extracted, and the labeler only needs to mark the point where the robot should detect the plant and its orientation. This significantly accelerates the entire ANN learning process.

[0016] Furthermore, it may be provided that a Labeling of the characteristics of the objects is suggested by computer and / or that before Labeling the images used are pre-sorted, whereby only images for the Labeling whose objects and / or features of the objects differ from objects and / or features of the objects of other images.

[0017] A further embodiment may provide that the images are Labelingare grouped into groups of images with objects that have few structural features, many structural features, and / or complex structural features. To achieve this, the ANN is trained to measure the similarity between images and thus determine how likely it is that two different images were taken of the same object and / or from the same perspective. This similarity measure can be used to determine whether both images should be labeled ( labeled ) if they add new information to the ANN, or whether one of them is sufficient, since using the second image would not provide any significant new information.

[0018] The ANN is also trained to determine the complexity of an object or object class by determining similarities between different objects of the same class, species, or genus. Simpler objects, i.e., objects with few structural details, exhibit higher similarity between different instances. More complex objects, on the other hand, exhibit lower similarity. Based on this similarity measurement, it can be estimated how many different images of objects of a particular class need to be labeled and used to train the ANN.

[0019] Preferably, it is further provided that, for the learning of the ANN, the images with the objects whose features are to be recognized are fed to only some layers, in particular the upper layers, and only the weightings of these layers are adjusted, whereby the remaining, fixed layers are not adjusted for the network's learning process, in particular their weightings remain unchanged for all images. To accelerate the training process, the output of these fixed layers can be pre-calculated and cached. During training, these cached values are then used for each image instead of being recalculated again and again. Reading a cached value is at least 10 to 100 times faster than recalculating the values, which leads to a significant acceleration of the learning process.

[0020] The individual processing steps performed in an ANN can be viewed as a series of mathematical operations that are then applied to the input data. Each layer represents a different operation that is applied to that layer's input data. Suppose the input data is called i. The example ANN has five layers, labeled I1, I2, I3, I4, and I5. The processing proceeds as follows: I1 applies its operation, or transformation, to i, producing the output o1 = I1(i). In the same way, when the second layer is applied to the output of layer 1, o2 = I2(o1) results. The same applies to I3, I4, and I5. If I1, I2, and I3 are not trained, o3 for i is always the same, no matter how many times it is calculated. When I4 and I5 are trained, their output changes during the training process, so o4 and o5 are different after each iteration of the learning process.However, instead of calculating o1, o2, and o3 each time for input i, one can store o3 and use it each time i is used as input. Therefore, for each input ik and output o3, k is stored and passed to I4. This saves processing time for the first three layers because loading the output of layer 3 is significantly faster than calculating the output of the first three layers. The speedup is greater the more layers a network has and the less data is present at the output of the last fixed layer.

[0021] A particularly preferred embodiment of the invention can provide that only the weightings of the upper layers are adapted for the learning process and the remaining layers are not adapted for the learning process, wherein in particular the weightings of the upper layers are adapted for each image and for the remaining layers the weightings are not adapted for all images.

[0022] In particular, it is also conceivable for the individual process steps to be executed in parallel on multiple computing units simultaneously, with the necessary operations being distributed across all computing units in such a way that optimal utilization of computing capacity is achieved. Furthermore, the software architecture is developed in the form of a pipeline: All units can run in parallel and are interconnected by the use of buffers, which transfer the outputs to the next station in the pipeline. In contrast to sequential processing, where the cycle time is given by the sum of all computing units, the cycle time for pipeline processing is independent of the number of computing units and is given by the slowest computing unit, including reading / writing the buffer.The processing pipeline for the hardware and software of the device used is designed similarly to that of a traditional assembly line. In the application example of plant propagation, specifically a meristem of a plant or another plant component, this assembly line or device is assigned several stations. A first station removes a plant from a cup. In a second station, images of the plant are taken from all sides. In the third station, the plant is cut up. In the fourth station, the plant is inspected on the conveyor belt. There it is determined whether further cuts are necessary. In the fifth station, the plant is identified on the conveyor belt and grasped in order to place the cut part of the plant into a new cup. A sixth station determines the optimal use of space for the plants in the output cups.

[0023] Such a pipeline can only operate efficiently if all its components can operate independently and in parallel. To achieve this, both the process and the computation are parallelized. The process is parallelized by having two robotic arms serve the first three stations. A third arm is used for the last two stations. All image processing modules are also operated in parallel by distributing the computational load across multiple computers.

[0024] Furthermore, it is conceivable that the Labelingis performed on a 3-dimensional image or object, whereby the 3-dimensional image or object is projected onto two dimensions and fed to the neural network for learning, and then converted back into a 3-dimensional image or object. It is planned to incorporate the additional dimension into the neural network by using recurrent neural networks that treat image series as part of a sequence. In this way, information is generated from individual views that can be passed on to the other perspectives to obtain a complete evaluation of a 3D image of the object or plant. The labeling ( Labeling ) is performed on a 3D model of the object. This label / model is later projected back onto 2D images. These are then evaluated by the ANN and combined again into a 3D model.

[0025] Previously, images captured by the cameras were processed and evaluated individually by the ANN, regardless of whether they were of the same object or not. However, this additional information can be very useful, especially when the same part of the object was captured from more than one perspective.

[0026] In this context, it may further be provided that for the processing of the third dimension of the image or object a recurrent A neural network is used that processes series of images of an object as part of an image sequence, thereby transferring information from one perspective of an object to other perspectives of the same object. To obtain meaningful labels for all images of the same object, 3D reconstruction of the object and labeling of the 3D model are recommended.

[0027] A further embodiment of the invention can provide for the outputs of the neurons to be fed back to the AINN for self-learning, with output errors being detected by the network and / or a person and marked as such. Thus, according to the invention, a mixture of supervised and unsupervised learning can take place. With the help of supervised learning, the machine can decide where to cut. Furthermore, cut plants on the conveyor belt can also be assessed. Based on this second assessment of the plant or cut, a cutting unit can align or configure itself in a timely manner to reduce the errors detected by the AI system. This approach can be generalized for all steps in which a manufactured part is assessed multiple times. The individual image processing stations are first trained using supervised learning, and new Labeling dataare automatically generated online. The automatic generation of new Labels or labels is done by re-evaluating the output of a particular vision station with the help of another vision station that can see the object at a later time.

[0028] For plants, the ANN determines at which positions the plants should be cut, specifically at which position a meristem of the plant or at which position another plant component. The images of the plants cut according to the specifications are then saved and used to automatically retrain the ANN (self-supervised learning). Another possibility for self-learning is to directly quantify the cutting quality of the plants and feed this information back online to then directly update the ANN ( Reinforcement learning ) .In particular, it is further conceivable that the person provides the network with information on how it recognizes how the feature of the object is to be treated, in particular how it is to be cut and / or grasped.

[0029] Preferably, the detected structural features of the objects are used to calibrate a laser for treating the object and / or to control a laser such that it cuts the object in a targeted manner. One of the core elements of the method according to the invention is laser camera calibration, which enables the so-called "what you see is what you cut" principle. This laser camera calibration enables a cut previously determined in a plant image by the ANN to be converted into laser coordinates, so that the laser cuts precisely what the ANN has suggested.

[0030] Unlike lasers, which are used to scan a surface, the laser is used in the plant pruning process. Therefore, conventional methods used for laser camera calibration of scanning lasers cannot be used here. A conventional method would use a calibration object captured by both the camera and the laser, and would calculate the transformation between the laser and camera coordinate systems based on the different views on the same object. Since the laser is not used for scanning in the process described here, this approach cannot be used, so a novel approach must be invented. After intrinsic and extrinsic camera calibration of the camera used, especially stereo cameras, the camera can reconstruct any 3D point in its view.To properly align the laser with the camera, a sheet of paper is placed approximately in the laser's main focal plane. The laser is then controlled to mark a grid of known size on the paper. In the next step, a new sheet of paper is placed a few centimeters closer to the laser, and the marking is repeated using the same coordinates as in the previous step. The two sheets are then imaged by the stereo camera immediately after laser marking. All intersection points of the grid are then automatically extracted. Due to the previous camera calibration, their 3D coordinates can be calculated. If the grid consists of m rows and n columns, a view will contain a total of mxn points detected by the cameras, so a total of 2 xmxn points are available from the two laser markers.These points are used to estimate the position of the laser's origin transformation with 6 degrees of freedom with respect to the camera coordinate system. After this transformation is estimated, each 3D point can be converted from camera coordinates to laser coordinates. Therefore, each of the 3D cutting lines determined by the ANN can be converted to laser coordinates, allowing the laser to cut precisely along the lines suggested by the ANN.

[0031] A neural network for solving the above-mentioned problem has the features of claim 16. Accordingly, it is provided that a neural network comprises a plurality of neurons, wherein the network is designed such that it is trained using a method according to one of the preceding claims 1 to 15.

[0032] Preferred embodiments in connection with the invention are described in more detail below with reference to the drawings, in which: Fig. 1 shows a representation of a first embodiment of a device, Fig. 2 shows a representation of a second embodiment of the device, Fig. 3 shows a representation of a further embodiment of the device, Fig. 4 shows a representation of a first step of image recognition, Fig. 5 shows a representation of a second step of image recognition, Fig. 6 shows a representation of a third step of image recognition, Fig. 7 shows a representation of a further embodiment of image recognition, Fig. 8a shows a representation of a first cutting line, Fig. 8b shows a representation of a second cutting line, and Fig. 8c shows a representation of a third cutting line.

[0033] In the Fig. 1An exemplary embodiment of a device is shown in highly schematic form. The method according to the invention and the ANN according to the invention are applied in connection with this exemplary device. However, it should be emphasized that the method according to the invention and the ANN according to the invention are not limited to use in connection with this device, but rather can also be used in connection with other problems.

[0034] Essentially, the method and device serve for the automated propagation of plants. The device presented here and the method according to the invention improve the propagation rate and growth rate of plants. Fig. 1In the embodiment shown, a plant 10 or a component of a plant 10 is first fed in a preferably sterile container 11 on a conveyor 12 to a work area 13. This work area 13 can also be kept sterile if necessary. From the conveyor 12, which can be designed as a lock, for example, the container 11 together with the plant 10 is fed to a first conveyor 14. This first conveyor 14 can be, in addition to the Fig. 1 The embodiment shown as a rotating disc can also be a conveyor belt or a tray.

[0035] On this first conveyor 14, the plant 10 is captured by an image recognition device having two cameras 15, 16. Based on the information about the plant 10 thus obtained, the control unit determines positions at which the plant 10 can be particularly preferably captured by the first gripping means 18. The first gripping means 18, which is designed here as a robotic arm, has tweezers 19 with which the plant 10 is removed from the container 11. If the plant 10 is firmly rooted in the container 11, the plant 10 can be pulled out of the container 11 or separated from the roots, for example by a random cut using a cutting means. The then empty container 11 is removed from the work area 13 via another conveyor 20.

[0036] The plant 10 hanging from the tweezers 19 is then fed to a further image recognition device with two further cameras 21, 22. These cameras 21, 22 take images of the hanging plant 10 from different perspectives. The information about the plant 10 thus obtained is used by the control unit 17 to recognize plant-specific characteristics of the plant. These plant-specific characteristics can be, for example, the type of plant and properties of leaves, stems or branches. It is also conceivable for the control unit 17 to recognize the type of plant. However, it is equally conceivable for an operator to have previously entered the type of plant to be propagated into the control unit 17 via an input device. In the control unit 17, the ANN according to the invention then determines an ideal cutting position or cutting speed based on the recognized plant-specific characteristics.an ideal cutting pattern is determined. For this determination, the ANN uses not only information about the current plant 10, but also information about previous plants and data previously provided to the neural network by an operator.

[0037] Using the ANN and the method according to the invention, not only the ideal cutting line can be determined, but also the type and / or physical properties of a cutting tool for an optimal cut can be determined. Fig. 1In the illustrated embodiment, the cutting means is a laser 23. This laser 23 can be modulated depending on the recognized plant-specific characteristics in such a way that a cross-sectional image particularly preferred for the rooting of the clone is generated. It has been shown that rooting and thus the growth of the clone can be stimulated by selecting the laser properties. For an ideal cross-sectional image or cutting pattern, the intensity of the laser, the wavelength and the focus or focal length are changed in such a way that the plant is cut without contact and thus without crushing. Through this targeted deposition of energy, the plant tissue is virtually unaffected; in fact, further growth is even stimulated.

[0038] The separated component 24 or the clone then falls onto a second conveyor 25. It can be provided that a second gripping means 26 grasps the clone 24 from this second conveyor 25 and feeds it to a container 27 with a nutrient medium 28. A camera 29, which is also connected to the control unit 17 for determining an optimal gripping position, is also used for preferential recording of the clone 24 by the second gripping means 26. The containers 27 thus filled are then discharged from the work area 13 via a third conveyor 30 and a conveyor 31. With the Fig. 1 The arrows shown symbolize the directions of movement of the individual components.

[0039] Through the Fig. 1Using the image recognition shown, plant-specific characteristics of the plant 10 are thus recognized, and the ANN determines cutting lines along which the plant is then cut with a suitable cutting means. It should be noted that the invention is not limited to the illustrated number of gripping means 18, 26, the number of conveyors 14, 25, 30, and the number of cameras 15, 21, 22, 29. Rather, it can be provided that a plurality of cameras are assigned to the image recognition device. Likewise, it is conceivable that the device has only one conveyor for conveying a clone.

[0040] In Fig. 22 shows a further exemplary embodiment of a device that can be used in conjunction with the method and the ANN according to the invention. Here, the cameras 33, 34 of the image recognition device and the ANN determine a position directly on the plant 32 that is particularly well suited for cutting a clone. Here, too, the cameras 33, 34 are connected to a control unit 35. With the help of the ANN, the control unit 35 determines not only an optimal cutting position or an optimal cutting line, but also optimal cutting conditions. The neural network thus determines how the laser 36 used here is to be modulated or controlled in order to create an ideal cutting image. In addition to the modulation, the focal length is also adapted to the position to be cut.As soon as a corresponding position for cutting the plant 32 has been determined by the control unit 35, the plant component is grasped by the gripping means 37 using tweezers 38 and the second robot arm holding the laser 36 is moved such that the plant is cut according to the determined cutting pattern. The severed component 40 or the clone of the plant 32 can then be placed on a conveyor 41 by the gripping means 37. The conveyor 41 then transports the severed component 40 to a further processing station. Here, the severed component 40 can, for example, be processed according to the method shown in FIG. Fig. 1 The procedures presented will be further discussed.

[0041] In the Fig. 3A further embodiment of the invention is shown. In this embodiment, a plant 42 or a component of a plant 42 lies on a conveyor 43 and is fed in the direction of arrow 44 to an image processing device consisting of two cameras 45 and 46. Independently of the embodiment shown here, the image recognition device can also have further cameras. Just as before in the Fig. 2As described in the example shown, here too, an optimal cutting image as well as preferred physical properties of the cutting means or the laser are determined by the cameras 45, 46 and by a control unit 47 and the ANN. As soon as this information has been generated by the control unit 47, a corresponding cut is made by the laser 48. It is also conceivable that the plant 42 is cut several times by the laser 48. Here, too, the laser 48 is assigned to a robot arm 49 for optimal mobility. Fig. 3 In the embodiment shown, the gripping means 50 serves only to either place the plant 42 onto the conveyor 43, to remove remaining components of the plant 42 from the conveyor 43, or to feed the separated component 51 to a further processing station. However, it is also conceivable that in the embodiment shown in the Fig. 3illustrated embodiment of the invention, the gripping means 50 is completely omitted.

[0042] Depending on the type of plant and the requirements for division, it may be advantageous to use different cutting patterns for cutting. Fig. 8aa U-shaped cut is shown which separates two leaves 53 as well as part of the stem 54 from the plant 55. Thus, three components of the plant 55 are severed by a single cut using a laser, a water jet, or a plasma jet. With conventional manual cutting, three individual cuts would be necessary. This U-cut 52 can be made with the plant 55 hanging from a gripping device or lying on a conveyor. It is also conceivable for such a cut to be made using a punching device made of metal or ceramic. In particular, a punching iron can be kept germ-free by applying heat or electric current. However, a particularly preferred contactless and crush-free cut can only be achieved by using a laser beam, a water jet, or a plasma jet.

[0043] In addition to the Fig. 8a In addition to the U-cut 52 shown, it is also possible to make a V-cut 56 with the laser ( Fig. 8b ). Here, too, two leaves 53 and a part of the trunk 54 are cut off. However, the cut in the trunk 54 differs from the straight cut from the Fig. 8a This V-cut 56 can be particularly advantageous for the rooting of the cut edge, at least for some plant species.

[0044] Furthermore, it may be that the V-cut 56 from the Fig. 8b is placed slightly next to the stem 54, whereby two leaves 53 are still separated from the plant 55 by one cut and, moreover, a cutting surface on the stem 54 is maximized. Due to this particularly large cutting surface, the severed part of the plant 55 can absorb a particularly large number of nutrients, which leads to a particularly preferred growth rate or rooting.

[0045] In addition to the Fig. 8a to 8c In addition to the examples of cross-sectional images presented, a variety of other cross-sectional images are conceivable, which efficiently cut several components of a plant simultaneously and stimulate the rooting of the severed plant. The type and position of the cuts are also determined by the ANN.

[0046] In the Fig. 4 to 6An embodiment of the image recognition of a plant 57 is shown in a highly schematic manner. To capture a spatial image of the plant 57, the plant is suspended in a ring-like image recognition device 58 by a gripping means (not shown). The plant 57 is preferably positioned centrally in the image recognition device 58. The ring can have a diameter of a few decimeters, i.e. 20 cm to 30 cm or 30 cm to 40 cm or 40 cm to 50 cm, and a wall height of 5 cm to 10 cm to 10 cm to 20 cm or 20 cm to 30 cm. However, it is also conceivable that the image recognition device 58 shown here has different dimensions. In the Fig. 4 to 6In the illustrated embodiment of the image recognition device 58, the ring has straight sections. A camera 59 and a light source 60 are alternately assigned to each of these twelve sections. This means that the illustrated embodiment of the image recognition device 58 is assigned six cameras 59 and six light sources 60.

[0047] In a first step of image recognition, two cameras 59, preferably located next to each other, are activated. At the same time, lamps 60 located near the cameras 59 are triggered, which sufficiently illuminate the plant 57 ( Fig. 4 ). The lamps 60 are to be switched in such a way that they do not dazzle the activated cameras 59. The remaining cameras 59 or lamps 60 are not activated. In a Fig. 5In the second step shown, a subsequent pair of cameras 59 is activated and the two adjacent lamps 60 are also switched on. In the following step of image recognition according to Fig. 6a next pair of cameras 59 and the corresponding lamps 60 are activated. This process is continued until the plant 57 has been captured from all directions over a solid angle of 360° by the camera pairs, i.e., in the example mentioned, twelve images have been taken. If the cameras 59 are named K1, K2, K3, K4, K5, K6 and the lamps 60 are named L1, L2, L3, L4, L5, L6, the three steps result: Step 1: L1, K1, L2, K2, L3; Step 2: L3, K3, L4, K4, L5; Step 3: L5, K5, L6, K6, L1. The successive control of the cameras 59 and the lamps 60 is carried out by a control device assigned to the image recognition device 58. It is of course also conceivable that fewer or more cameras 59 are used, which then, if necessary, successively image different angular sections of the plant 57 in pairs.

[0048] The images thus captured are evaluated by the previously discussed control unit or ANN. This evaluation involves identifying plant-specific characteristics along which the plant can be preferentially divided by a cutting tool. This image acquisition, or this sequence of individual images, takes a few hundred milliseconds.

[0049] Furthermore, it can be provided that the plant 57 is cut directly in the ring-like image recognition device 58, also called a theater, by a cutting means. The severed part of the plant 57 can either be grasped by another gripping means or conveyed onto a conveyor positioned below the image recognition device 58.

[0050] The Fig. 4 to 6 The cameras activated are highlighted by a schematically represented recording cone. Fig. 4 to 6 Each of the activated lamps 60 is assigned a light bulb.

[0051] In the Fig. 7 A further embodiment of an image recognition device 61 is shown. This embodiment is also similar to the one shown in the Fig. 4 to 6illustrated embodiment of an image recognition device 58 is ring-shaped. Here, however, the ring has only six straight side surfaces. Accordingly, only two opposing cameras 62 and four lighting devices 63 are assigned to the inner walls of the image recognition device 61. Exactly as described above, one camera 62 is activated together with two adjacent lighting devices 63 in order to create an image of the plant 64. The correspondingly opposite camera 62 and the opposite lighting devices 63 are then activated in order to create a second image of the plant 64. In order to compensate for the smaller number of cameras, the plant 64 is rotated by a certain angular range after each image is taken, so that several images are taken one after the other from different positions of the plant 64. From the totality of all images thus obtained, the control unit orthe ANN can recognize plant-specific characteristics of the plant 64.

[0052] In addition to the exemplary embodiments of the image recognition devices 58, 61 shown here, further geometries with more or fewer cameras are conceivable. These image recognition devices 58, 61 can be used in accordance with the exemplary embodiments of the invention. Fig. 1 to 3 This combination of image recognition and the specified cutting method allows for a particularly efficient method for propagating plants. List of reference symbols

[0053] 10 plant 43 Sponsors 11 container 44 Arrow direction 12 Funding 45 camera 13 Workspace 46 camera 14 first sponsor 47 Control unit 15 camera 48 Laser 16 camera 49 robot arm 17 Control unit 50 Gripping devices 18 first gripping device 51 ingredient 19 tweezers 52 U-cut 20 Funding 53 Sheet 21 camera 54 tribe 22 camera 55 plant 23 Laser 56 V-cut 24 ingredient 57 plant 25 second sponsor 58 Image recognition device 26 second gripping device 59 camera 27 container 60 Light bulbs 28 Culture medium 61 Image recognition device 29 camera 62 camera 30 third sponsor 63 Light bulbs 31 Funding 64 plant 32 plant 33 camera 34 camera 35 Control unit 36 Laser 37 Gripping devices 38 tweezers 39 robot arm 40 ingredient 41 Sponsors 42 plant

Claims

1. A method for the computer-aided training of an artificial neural network for recognizing structural features on objects, namely on plants or on plant constituents for the cloning of these plants, wherein the structural features represent particularly suitable cutting positions on the plant, wherein the network used is a convolutional neural network (CNN), in particular a regional convolutional neural network (R-CNN), having a multiplicity of neurons, said network comprising a multiplicity of convolutional and / or pooling layers for the extraction of information from images of the objects having the structural features to be recognized for a classification of the features by further layers, wherein the images are scaled up and down from layer to layer and wherein during the scaling of the images from layer to layer information about the structural features of the objects is obtained, specifically independently of the scaling of the images, wherein a plurality of images from different perspectives of the same object for recognizing the structural features are transferred simultaneously to the neural network, and wherein computer-aided operations for recognizing the structural features of the object are carried out on the images in parallel on a plurality of GPUs.

2. The method for the computer-aided training of an artificial neural network for recognizing structural features on objects as claimed in claim 1, characterized in that the plurality of images of the same object are scaled, in particular rescaled, and stitched prior to transfer to the neural network.

3. The method for the computer-aided training of an artificial neural network for recognizing structural features on objects as claimed in the preceding claims, characterized in that labelling or designation of the features of the objects is carried out semi-automatically on the images in preparation for the training process of the neural network, wherein preferably firstly the features and / or the objects are isolated from a background of the images and in particular afterward labelling of the features is carried out by a person.

4. The method for the computer-aided training of an artificial neural network for recognizing structural features on objects as claimed in claim 3, characterized in that labelling of the features of the objects in a computer-aided manner is proposed.

5. The method for the computer-aided training of an artificial neural network for recognizing structural features on objects as claimed in claim 3 or 4, characterized in that the images used are pre-sorted prior to the labelling, wherein only images whose objects and / or features of the objects differ from objects and / or features of the objects of other images are used for the labelling.

6. The method for the computer-aided training of an artificial neural network for recognizing structural features on objects as claimed in any of claims 3 to 5, characterized in that after the labelling the images are grouped into groups of images having objects which have few structural features, many structural features and / or complex structural features.

7. The method for the computer-aided training of an artificial neural network for recognizing structural features on objects as claimed in the preceding claims, characterized in that for the training of the network the images having the objects whose features are to be recognized are fed only to a few layers, in particular to the upper layers, and only the weightings of these layers are adapted, wherein the rest of the layers are not adapted for the training process of the network, in particular their weightings remain unchanged for all of the images.

8. The method for the computer-aided training of an artificial neural network for recognizing structural features on objects as claimed in the preceding claims, characterized in that for the training process only the weightings of the upper layers are adapted and the rest of the layers are not adapted for the training process, wherein in particular the weightings of the upper layers are adapted for each image and for the rest of the layers the weightings are not adapted for all of the images.

9. The method for the computer-aided training of an artificial neural network for recognizing structural features on objects as claimed in the preceding claims, characterized in that the individual method steps are carried out simultaneously in parallel on a plurality of computer units, wherein the necessary operations are distributed among all the computer units in such a way that an optimum utilization of the computer capacity is attained.

10. The method for the computer-aided training of an artificial neural network for recognizing structural features on objects as claimed in the preceding claims, characterized in that the labelling is carried out on a 3-dimensional image or object, wherein the 3-dimensional image or object is projected on two dimensions and is fed to the neural network for training purposes and is subsequently converted back into a 3-dimensional image or object.

11. The method for the computer-aided training of an artificial neural network for recognizing structural features on objects as claimed in claim 10, characterized in that a recurrent neural network is used for the processing of the third dimension of the image or object, said network processing series of images of an object as a part of an image sequence, whereby information of an object from one perspective is transferred to other perspectives of the same object.

12. The method for the computer-aided training of an artificial neural network for recognizing structural features on objects as claimed in the preceding claims, characterized in that the outputs of the neurons are fed again to the neural network for self-training purposes, wherein output errors are recognized by the network and / or a person and are marked as such.

13. The method for the computer-aided training of an artificial neural network for recognizing structural features on objects as claimed in claim 12, characterized in that the person makes available to the network the information regarding how said network recognizes how the feature of the object is to be treated, in particular cut and / or grasped.

14. The method for the computer-aided training of an artificial neural network for recognizing structural features on objects as claimed in the preceding claims, characterized in that the recognized structural features of the objects are used in order to calibrate a laser for a treatment of the object and / or in order to control a laser in such a way that the latter cuts the object in a targeted manner.

15. An artificial neural network comprising a multiplicity of neurons, wherein the network is configured in such a way that it is trained by a method as claimed in any of the preceding claims 1 to 14.