Method for generating training image and method for training machine learning algorithm

Abstract

The present invention relates to a method of generating a training image using an enhanced machine learning algorithm, comprising: providing input data for the enhanced machine learning algorithm, comprising a first data set comprising an image of an environment in which a mobile device should be moved and a tag for an image of an object, and a second data set comprising a tag for the image of the object; wherein one or more objects that should be considered when navigating the mobile device in the environment are shown at least in some images of the environment, and wherein the second data set comprises images of objects that should be considered when navigating the mobile device in the environment and tags for the images of objects; enhancement by means of an enhancement machine learning algorithm comprises: inserting objects in images of the second dataset into images of the first dataset and assigning labels of the objects in the images of the second dataset to related images of the first dataset in order to obtain an enhanced first dataset; and providing the enhanced first data set as output data of the enhanced machine learning algorithm.

Description

Technical Field

[0001] The present invention relates to a method for generating training images while using an enhanced machine learning algorithm, a method for training an enhanced machine learning algorithm, and a computing unit and a computer program for executing the method, wherein the training images can be used for training an applied machine learning algorithm. Background Art

[0002] Mobile devices or processing devices such as mowing robots, vacuum or mopping robots or other household robots typically move in the environment to be processed, such as in a garden or in a house. In this case, the task is especially the navigation of the mobile device, which should move in the environment, for example on a specific path within certain boundaries (working area). Summary of the invention

[0003] According to the invention, a method for generating training images, a method for training a machine learning algorithm, and a computing unit and a computer program for carrying out the method are proposed with the features of the independent patent claims. Advantageous embodiments are the subject matter of the dependent claims and the following description.

[0004] The present invention generally discusses mobile devices that move in particular automatically, such as preferably robots, for example mowing robots. Although the present invention should be explained below mainly with respect to mowing robots, other mobile devices or robots may also be considered. Examples of such mobile devices (or mobile working devices) are generally, for example, robots and / or drones and / or vehicles that move partially or (completely) automatically (on land, in water or in the air). As robots, for example, household robots such as suction and / or wiping robots, ground or road cleaning equipment or mowing robots may be considered, as well as other so-called service robots as vehicles that move at least partially automatically, such as passenger vehicles or freight vehicles (also land transport vehicles, for example in warehouses) and aircraft, such as so-called drones or ships. Automatic driving or autonomous driving passenger or trucks may also be considered as mobile devices.

[0005] For such a mobile device, navigation information can be determined, based on which the mobile device can be moved. Such navigation information can, for example, include a specific movement path or trajectory along or according to which the mobile device should move. For example, in the case of a lawnmower robot, a movement path can be determined, which allows the lawn (i.e. the working area in the environment) to be traversed and mowed as quickly and efficiently as possible. For this purpose, control information for moving the mobile device can be determined based on the navigation information, based on which the mobile device can be moved. Such a mobile device typically has a drive system and a control or regulation unit for operating according to such control information.

[0006] For example, it is usually desirable for a lawnmower robot to stay within the boundaries of a garden or lawn, since there may be areas that are not suitable for optimal operation, such as ponds, mud, gravel roads, flower beds, etc. For this purpose, wires or cables that can be detected or detected by means of sensors in the robot can be laid in the garden or in the lawn, especially at the edge of the lawn or where the lawnmower robot is to mow. In this context, it is also possible to speak of lawn edges, where such wires are laid. Although this allows the movement of the robot to be limited as desired, it requires a high level of effort due to the laying of wires or cables.

[0007] Another possibility for the (mobile) device to be able to recognize specific features in the environment, for example, the edge of a lawn in the case of a lawn mower robot, is to use an environment detection device or one or more sensors, such as a camera or video camera, a thermal imager, a radar sensor, a lidar sensor, a laser rangefinder, an ultrasonic sensor, an inertial sensor and / or an odometer. In particular, within the scope of the present invention, detection devices or sensors that enable an image representation of the environment should be considered, i.e. preferably a camera or video camera. However, it is also possible to obtain depth information, which can be represented in an image or as a (digital) image, for example, with the aid of other sensors, such as a lidar or ultrasonic sensor. Such an image of the environment can then be analyzed in order to recognize or identify specific things or objects in the environment, such as the lawn edge just mentioned in the case of a lawn mower robot.

[0008] In particular, in this case, many different objects or object types in the environment can be recognized or identified, which can be taken into account when determining navigation information. Such objects can be, for example, houses, walls, trees, stumps, stones, rocks, roads, lawns, flowers, flower beds, as well as, for example, people, animals, leaves or the sky or clouds in the sky.

[0009] In particular, so-called semantic segmentation can be used here. In this case, each image point (pixel) in the image is assigned one of a number of classes, i.e. the image points are labeled or classified. For this purpose, firstly features of the image points are typically determined or extracted, and then the classes are assigned based on these features. Such features can be, for example, shape, color, context, pattern, light changes and / or image context. Thus, for example, a picture point can be assigned the color green, and the surrounding picture points can also all be green. This would indicate a lawn.

[0010] Different features can be considered as categories here. In the example of a lawnmower robot, for example, two categories can be used in a simple case, namely "lawn" and "non-lawn", i.e. for each image point in the image it is determined whether the image point indicates a lawn. It goes without saying that more than two categories can also be used, in order to, for example, also recognize paths, roads, buildings, vehicles or people or other objects mentioned, and assign them to image points if necessary. Based on the category "non-lawn" - the category "background" is also conceivable, to which everything that is not assigned to another category is assigned. Compared to object recognition that purely (only) recognizes objects in the image, more information is added by all these labelings.

[0011] In principle, the classification does not have to be performed for each individual pixel, but can also be performed for segments or parts of the image, each of which includes a plurality of pixels. Such segments can be fixedly predetermined, but can also be formed only within the scope of semantic segmentation, optionally with different sizes within the image. For this purpose, one or more artificial neural networks or generally artificial intelligence-based pattern recognition methods are preferably used, which are also referred to below as machine learning algorithms, and which will be discussed in more detail below.

[0012] In this way, a classified result image (output image), i.e. a kind of map, can be generated from an image (input image), which indicates where specific objects or general features are located. In a simple example with two categories "lawn" and "non-lawn", the edge of the lawn can be recognized or determined by the boundary between the areas of the image assigned either to "lawn" or "non-lawn" (the associated result image can be, for example, pure black / white). The result image can then be used in particular for controlling the device, i.e., for example, the device can be stopped or turned before or when the edge of the lawn is reached. It is also conceivable to distinguish, for example, between uncut lawns and already cut lawns as categories.

[0013] At this point it should be mentioned that, although a camera or video camera at or in such a mobile device can preferably be considered for detecting such (digital) images, within the scope of the present invention, as already mentioned, a representation of the environment detected by means of other sensors such as the mentioned lidar or ultrasonic sensors or thermal imagers or, if necessary, obtained by further processing should also be understood as an image; semantic segmentation can then also be applied there.

[0014] By carrying out this procedure repeatedly or continuously, in particular for the respectively current image of the environment, i.e. in particular when the (mobile) device is moving, the mobile device can be operated or controlled based on the respectively current result image; thus, for example, a lawnmower robot can be controlled so that it only drives as far as the identified lawn edge (and mows the lawn there), but then, for example, continues to drive along the lawn edge without exceeding it.

[0015] For such semantic segmentation and in general for the navigation of such mobile devices, machine learning algorithms can be used. The following should also be mentioned here about the application of machine learning algorithms. As now shown, although it is advantageous in principle when as many different objects or object types as possible are identified or classified so as to determine navigation information based on this. However, some objects appear more frequently than other objects in reality. In an environmental image detected, for example, by a mobile device or its environmental detection device, for example, vegetation, lawns or the mobile device itself can be seen very often or a very high proportion of vegetation, lawns or the mobile device itself can be seen, and the proportion of the sky or static objects is also mostly large. And other objects, such as animals, paths, roads, bicycles or general dynamic objects, are seen less frequently or very rarely.

[0016] This results in that when training machine learning algorithms for navigation and in particular object recognition, i.e., for example, when applying the machine learning algorithms mentioned above, there is an imbalance or imbalance in the images available as training data (training images) with regard to possible objects suitable for recognition. However, objects that are present less frequently in the images are important for navigation, but a machine learning algorithm trained with the mentioned imbalanced images will have difficulties in recognizing such objects.

[0017] In this context, a possibility is proposed to generate improved training images for training such an applied machine learning algorithm, namely using a (further) machine learning algorithm, also referred to below as a boosted machine learning algorithm.

[0018] In this case, input data are provided for enhancing the machine learning algorithm, wherein the input data comprises a first data set and a second data set. The first data set comprises images of the environment in which the mobile device, in particular a vehicle or a robot that moves at least partially automatically, should be moved. Here, at least in some of the images of the environment one or more objects that should be taken into account when the mobile device navigates in the environment are shown. Thus, for example, this can be an image detected during normal operation of such a mobile device, such as the mentioned lawn mowing robot. As mentioned, these images are then usually unbalanced. In addition, the first data set comprises labels for the images of the environment. For example, a label can be understood as a description or annotation (so-called annotation) or classification, which explains what can be seen in the image or its parts or segments, and which objects can be seen therein. These labels (the above-mentioned classification) can be obtained, for example, during the semantic segmentation of such an image, or even during regular operation.

[0019] The second data set includes images of objects that should be taken into account when the mobile device navigates in the environment and labels for the images of the objects. The second data set therefore includes in particular images of objects that are important or relevant for the navigation of the mobile device, i.e. in particular, objects that occur less frequently, such as animals, paths, roads, bicycles or generally dynamic objects, as mentioned above. The second data set does not have to be detected by the mobile device or its environment detection device; on the contrary, it can be images of objects from any data source, such as from the Internet. In this case, in particular, the objects themselves are relevant, while any environment or background is not relevant and therefore does not have to be included in the images in the second data set.

[0020] The enhancement is then performed with the aid of an enhancement machine learning algorithm. In this case, objects from the images of the second data set are inserted into the images of the first data set, and the labels of the objects from the images of the second data set are added to the associated images of the first data set. In other words, the labels of the objects are thus added to the existing labels of the images of the environment. This can be done, for example, with the aid of so-called "inpainting". An enhanced first data set is thereby obtained. This enhanced first data set is then provided as output data of the enhancement machine learning algorithm. The images of the enhanced first data set are here provided as training images for training the applied machine learning algorithm.

[0021] In this way, a significantly balanced data set of images of the environment can be obtained, in which objects which occur less frequently in reality are then present more frequently.

[0022] The images of the objects from the second data set are preferably inserted into the images of the first data set such that the insertion increases the relative frequency of one or more predetermined types (or categories, e.g. animals, children) in the images of the first data set, e.g. at least to such an extent that reasonable learning is possible. Alternatively or additionally, it is preferred if the insertion causes the relative frequencies of a plurality of objects of a predetermined type (or categories, e.g. animals, children on the one hand and trees, walls on the other hand) in the images of the first data set to be adapted to one another. In this case, not all must be present with the same frequency, but at least to such an extent that reasonable learning is possible. Accordingly, a corresponding and corresponding plurality of images or objects can be selected from the second data set and inserted.

[0023] In one embodiment, the first data set further comprises spatial information about the image of the environment, in particular depth information. Objects from the image of the second data set are then inserted into the image of the first data set taking into account the spatial information. In particular, the spatial information can include at least one of the following: depth information, the size of the object, the orientation of the object, the position of the object. In this case, the spatial information can be intrinsic information of the image of the environment, i.e., for example, by features that can be recognized in the image, which indicate that one object is located in front of another object or the like. Alternatively or additionally, however, the spatial information can also be separate spatial information, i.e., such spatial information that has been detected by a lidar sensor, for example; here, for example, different distances of different objects in the image can then be determined directly. A depth map for the image contained in the depth information is also conceivable. For example, the depth information can also be obtained based on an image pair of the environment obtained, for example, by means of a stereo camera (if such a stereo camera is used).

[0024] The use of spatial information allows inserting objects from the second dataset into the images of the first dataset, so that the obtained enhanced image (of the enhanced first dataset) is also realistic; for example, this may mean that a dog inserted in the image is smaller than the person, but not the same size or even larger, if it can be seen at approximately the same distance as the person (already present in the image earlier). Training images obtained in this way allow better training of applied machine learning algorithms and thus better and more accurate object recognition and navigation of mobile devices.

[0025] In one embodiment, the enhancement comprises determining one or more selected sets of spatial information of an image of an environment from a first data set into which an object from a second data set has been inserted. For example, the set may comprise the position and orientation of two objects in the image, e.g. a person in front, a large dog in front. The set relates in particular to the inserted object from the second data set, i.e. for example a dog if exactly a dog is inserted as an object. Furthermore, a measure is determined or provided (e.g. may have been determined in advance) for a (same) set or a comparable set of spatial information (comparable may mean that similar or more or less identical objects can be seen), i.e. determined or provided from a plurality of images of the environment from the first data set before the object from the second data set has been inserted.

[0026] Only the images of the environment in the first data set into which the objects from the second data set have been inserted (enhanced images) are then used for the enhanced first data set, i.e., only if the metric is within the assigned tolerance range, it is added to the enhanced first data set. The metric can be, in particular, a probability of occurrence (or frequency), and for example indicates how frequently a particular set, i.e., the position and / or orientation and / or size ratio of the object, is in the relevant image. For example, an upright person will have, for example, a probability of 90% (i.e., if a person is present in the image, the person will almost always be upright), while an inverted person will have a very low probability; this applies accordingly to other combinations or sets that do not occur or occur rarely in reality. Such an enhanced image can then simply be discarded, since it cannot be real. The tolerance range can be appropriately selected, for example, set to greater than 80% (i.e., 80% to 100%), but can also be set lower, for example, greater than 50%. In this way, a particularly realistic enhanced first data set can be obtained, which in turn leads to a particularly good trained applied machine learning algorithm.

[0027] Preferably, the first data set also includes photographic information about the image of the environment, in particular exposure information. Objects from the image of the second data set are then inserted into the image of the first data set taking into account the photographic information. This also makes it possible to obtain a particularly realistic enhanced first data set.

[0028] A particular advantage of the proposed approach is that semantic information and in particular also spatial information is taken into account during the enhancement. Most other methods for enhancement modify the image only at the pixel level, but do not take into account the larger context of the image and in particular also the context of the label. The proposed approach modifies the image itself and the label. The training images obtained here therefore allow significantly better training of the applied machine learning algorithm, because the applied machine learning algorithm then provides better results. (Apart from the use of the generated training images or the enhanced first data set) there is also no mention of the specific type of applied machine learning algorithm or its training.

[0029] The present invention also relates to a method for training a machine learning algorithm, namely training the above-mentioned enhanced machine learning algorithm. The enhanced machine learning algorithm is in turn configured to generate training images, which are used to train the applied machine learning algorithm, in particular to improve object recognition of the applied machine learning algorithm. In this case, training data are provided, wherein the training data comprises a first training data set and a second training data set.

[0030] The first training data set comprises images of an environment in which the mobile device, in particular an at least partially autonomous vehicle or a robot, is to be moved. The first training data set also comprises labels for the images of the environment; since the images comprise objects, the labels also apply to the objects. The second training data set comprises images of objects that should be taken into account when the mobile device is navigating in the environment and further comprises labels for the images of the objects.

[0031] Based on the training data, the boosted machine learning algorithm is then adapted (trained, in particular the weights of the boosted machine learning algorithm are adapted) so that objects from the images of the second training data set are inserted into the images of the first training data set and the labels of the objects from the images of the second training data set are assigned to the associated images of the first training data set. Typically, the weights of the boosted machine learning algorithm are adapted in this case. In the case of an artificial neural network as a boosted machine learning algorithm, so-called neurons can also be mentioned as weights. The boosted machine learning algorithm (or a corresponding model) trained in this way can then be provided and used, for example, for navigating a mobile device.

[0032] In this case, the first and second training data sets may be similar or comparable, for example, to the first and second data sets when applying the enhanced machine learning algorithm described above.

[0033] However, the training data particularly preferably also include a third training data set. The third training data set includes images of the environment in which the mobile device, in particular an at least partially autonomous vehicle or a robot, is to be moved. In each of the images of the environment one or more objects are shown which are to be taken into account when the mobile device is navigating in the environment. Furthermore, the third training data set includes labels for the images of the environment. In this case, the third training data set can be comparable to the first training data set - or even identical if necessary; it may also be expedient if more relevant objects are visible in the images of the third training data set than in the images of the first training data set.

[0034] Adapting the enhanced machine learning algorithm then includes comparing an image of the environment of the first training data set into which the object from the second training data set has been inserted with one or more images of the environment of the third training data set. In other words, it is thus checked, for example, to what extent the type of the image of the first training data set after the insertion of the object corresponds to the type of the image of the third training data set.

[0035] In one exemplary embodiment, the first training data set and the third training data set each furthermore include spatial information about the image of the environment, in particular depth information. The spatial information may in particular be the spatial information already explained above.

[0036] The comparison then includes determining one or more selected sets of spatial information of an image of an environment of the first training data set into which an object from the second data set has been inserted. This set is particularly related to the object from the second training data set that has been inserted. In addition, before inserting the object from the second data set, a metric is determined for this set or a comparable set of spatial information of multiple images from the environment of the third training data set. The explanation already mentioned above is applicable to the metric (i.e., for example, the probability of occurrence). Then, the image from the environment of the first training data set into which an object from the second data set has been inserted is adapted to such an image from the environment of the third training data set for which the metric is within a predetermined tolerance range in relation to the set for spatial information. Therefore, for example, it is adapted to an image in which the probability of occurrence is, for example, greater than 80%. In this way, it is therefore possible to achieve an enhanced machine learning algorithm that produces an image that is enhanced as much as possible, in which the spatial information is comparable to the spatial information in the image that is actually detected. Since these metrics for spatial information can also be independent of the specific object in the image, the first training data set can also be used as the third training data set, as long as at least some objects are displayed in the image there (but this is actually always the case).

[0037] Since labels can be generated for the images of the first training data set as explained above for the first data set in the application, for example directly by detecting the mobile device, and the individual object images of the second training data set actually only require simple labels, the training proposed here can also be performed in an unsupervised manner, i.e. it can be so-called "unsupervised learning".

[0038] The invention also relates to a method for determining navigation information for a mobile device, in particular a vehicle or a robot that moves at least partially automatically. For this purpose, an image of an environment is provided in which the mobile device moves. The navigation information is determined based on the image of the environment, i.e. using an applied machine learning algorithm that is trained using training images that have been obtained as described above. The navigation information can then be provided and used, in particular, to determine control information in order to move the mobile device.

[0039] A computing unit according to the invention, for example a control unit or a control unit of a mobile device or a server or another computer is configured, in particular in terms of programming, to carry out the method according to the invention.

[0040] The mobile device can be configured to receive the control information already determined as described above, or it can have a computing unit according to the invention. The mobile device also has a drive system and a control or regulating unit for controlling according to the control information. The mobile device can in particular be a mobile device as described above, i.e., for example, a lawn mower robot.

[0041] It is also advantageous to implement the method according to the invention in the form of a computer program or computer program product with a program code for executing all method steps, because this causes particularly low costs, especially if the execution control device is also used for other tasks and therefore originally exists. Finally, a machine-readable storage medium is provided, with a computer program as described above stored thereon. Suitable storage media or data carriers for providing computer programs are especially magnetic, optical and electrical memories, such as hard disks, flash memories, EEPROMs, DVDs, etc. It is also possible to download programs via computer networks (the Internet, intranets, etc.). Such downloading can be done wired or cabled or wirelessly (e.g., via WLAN networks, 3G, 4G, 5G or 6G connections, etc.).

[0042] Further advantages and embodiments of the invention are apparent from the description and the drawings.

[0043] The invention is schematically illustrated in the drawings on the basis of exemplary embodiments and is described below with reference to the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A mobile device is schematically shown in the context of explaining the present invention.

[0045] Figure 2 An image detected by an environment detection device of a mobile device is shown.

[0046] Figure 3 In the context of the method according to the invention, in a preferred embodiment, the Figure 2 The resulting image is generated from the image.

[0047] Figure 4 The sequence of the method according to the invention is schematically shown in a preferred embodiment.

[0048] Figure 5 The sequence of the method according to the invention in a further preferred embodiment is schematically shown.

[0049] Figure 6 The corresponding Figure 3 The resulting image is an image of

[0050] Figure 7 The corresponding Figure 3 Another image of the result image.

[0051] Figure 8 The sequence of the method according to the invention is schematically shown in a preferred embodiment.

[0052] Fig. 9 The sequence of the method according to the invention is schematically shown in a preferred embodiment. DETAILED DESCRIPTION

[0053] exist Figure 1 1 shows schematically and by way of example a mobile device 100 in an environment 120, in particular a work area, for explaining the invention. The mobile device 100 is, by way of example, a lawnmower robot having a control or regulating unit 102 and a drive unit 104 (with wheels) for moving the lawnmower robot 100, for example, along a trajectory 130. The lawnmower robot 100 also has, by way of example, a sensor 106 designed as a camera with a field of view (indicated by dashed lines). For better illustration, the field of view is selected to be relatively small here; in practice, however, the field of view can also be up to 360° (for example, but at least 180° or at least 270°).

[0054] The mowing robot 100 also has a computing unit 108, for example a control device, by means of which data can be exchanged with a superordinate system 110, for example via the indicated radio connection. For example, a trajectory (or navigation information in general) can be determined in the system 110 and then transmitted to the system 108 in the mowing robot 100, which is then to follow the trajectory. However, it can also be provided that the trajectory (or navigation information in general) is determined in the system 108 itself or is obtained there in another way. Instead of the trajectory, however, the system 108 can also obtain control information, for example, which is determined based on the trajectory and based on which the control or regulating unit 102 can move the mowing robot 100 via the drive unit 104 in order to follow the trajectory. The trajectory 130 is indicated here only as an example.

[0055] The mowing robot 100 is intended to move independently on the ground or in the environment 120 or to navigate there, for example, and in this case to mow the grass (not explicitly shown here). In addition, by way of example, a plurality of different objects are shown in the environment, namely a house 140, a person 142 and a tree 144. As already mentioned, these objects can be detected by the mobile device 100 or its camera 106 (ie, they are visible in the images recorded by means of the camera) and then taken into account during navigation, i.e., when determining navigation information.

[0056] exist Figure 2 The figure shows the environment detection device, such as Figure 1 250 detected by the camera 106 of the mowing robot 100 in FIG. Various objects, a house 140, a person 142, and a tree 144 can be seen therein (from the perspective of the mowing robot or its camera).

[0057] exist Figure 3 For example, semantic segmentation from Figure 2 A result image 350 is generated from the image 250 of FIG. 250 . In this case, features are extracted from the image 250 , for example, for each image point (pixel), and then, as a function of the features, one of a plurality of classes or labels is assigned to each image point, for example. By way of example, the classes or labels “house” C1, “person” C2 and “tree” C3 should be used. In the result image, these classes are shown with hatching; in practice, this can be black / white or, in the case of more than two classes, in different colors; various other features that can be processed digitally can also be used. The result image 350 shown here can also be understood in particular as a combination of the image 250 with the assigned classes.

[0058] At this point it should be mentioned that, for the sake of clarity, only the relevant (and shown) objects themselves are shown according to the classification. However, depending on the implementation, other areas of the image 250 can also be classified accordingly.

[0059] Images of this type, i.e. images such as image 250, together with labels (i.e. result image 350), can be used to train applied machine learning algorithms. As already mentioned, different types of objects appear with different frequencies in such images, especially when these images are acquired in actual use using a mobile device.

[0060] exist Figure 4 The frequency distribution of different types of objects in an exemplary dataset of images of an environment is shown in FIG. Here, a number N is plotted on the class C, where N is the number of objects of the corresponding class C that can be assigned to the image description in the exemplary dataset. These classes (or labels or types of objects) can be, for example, as follows from left to right, i.e. from frequent to less frequent: vegetation, grass, mobile device (mowing robot), sky, (other) static objects, houses, fences, walls (or stumps), stumps, gardening tools, paths, leaves, soil, people, ground, vehicles, objects outside the area of ​​interest, (other) buildings, roads, docking stations, stones, animals, bicycles, (other) dynamic objects.

[0061] As mentioned, such a dataset allows training of applied machine learning algorithms for later navigation of mobile devices, but especially object recognition will have difficulties in the case of objects that appear infrequently or almost never in the image. However, it is precisely these objects that may be highly relevant for navigation.

[0062] It is therefore within the scope of the invention that objects relevant for navigation or object recognition, in particular such objects which hardly appear in such images, should be added to images such as Figure 2 An image such as image 250 in or a corresponding result image 350 .

[0063] exist Figure 5 , an image 550 of an object 546 (here an animal) is shown by way of example in FIG. Furthermore, a label or class "animal" C4 is assigned to the image 550. Images of objects such as the image 550 can be obtained from various data sources, for example from the Internet.

[0064] exist Figure 6 An image 650 is shown in FIG. 650 , which corresponds to the image from Figure 3 The result image 350, that is Figure 2 The image 250 in FIG. 6 is also provided with the label or category to which it belongs. However, in the image 650 , there is additionally Figure 4Animal 546 in image 550 of has been inserted. In addition, the associated label C4 is added.

[0065] exist Figure 7 An image 750 is shown in FIG. 750 , which corresponds to the image from Figure 3 The result image 350, that is Figure 2 250 in the image 750 together with the associated label or category. However, in the image 750 - as in the image 650 - there is additionally Figure 4 Animal 546 in image 550 of has been inserted. In addition, the associated label C4 is added.

[0066] The difference between image 650 and image 750 is that animal 546 is shown significantly larger in image 750 than in image 650. This can occur when objects such as animal 546 are (more or less) inserted arbitrarily into images such as image 250 or result image 350. However, within the scope of the present invention, it is in particular possible to sort images such as image 750 in which the size ratios are not true. For this purpose, spatial information about the image can be used, as has already been explained and as will be explained again below.

[0067] exist Figure 8 Schematically, the sequence of the method according to the invention in a preferred embodiment is shown in FIG. 4 , more precisely in the form of a flow chart.

[0068] In step 800, input data are provided. The input data include a first data set 802 and a second data set 804. The first data set includes an image of an environment in which a mobile device, in particular an at least partially autonomous vehicle or a robot, is to be moved, i.e., for example, Figure 2 In this case, at least in some images of the environment one or more objects are shown, which should be taken into account when navigating the mobile device in the environment. In addition, the first data set also includes labels for the images of the environment, i.e., categories or their assignments, such as in the images or result images 350. Preferably, the first data set 802 also includes spatial information 806 about the images, such as depth information.

[0069] The second data set 804 includes images of objects that should be considered when navigating the mobile device in the environment, i.e., such as Figure 5 In addition, the second data set 804 also includes labels for the images of the objects, that is, for example, corresponding categories or their assignments, such as using image 550 in Figure 5 Also shown in .

[0070] The input data, i.e. the first and second data sets, are then fed to an enhanced machine learning algorithm 810. There, enhancement is then performed (step 812). In this case, objects from the images of the second data set 804 are inserted into the images of the first data set 802, and furthermore the labels of the objects from the images of the second data set are assigned to the relevant images of the first data set, e.g. Figures 5 to 7 Explained.

[0071] Here, step 814, a selected set 816 of spatial information of an image of the environment from the first data set 802 into which an object from the second data set has been inserted is determined. In step 818, a metric 820 is determined or provided for the same or comparable set of spatial information of a plurality of images of the environment from the first data set prior to inserting the object from the second data set.

[0072] In step 822, labels of objects from images of the second data set 804 are also assigned to the relevant images of the first data set 802 in order to obtain an enhanced first data set 824. However, if the measure is within the assigned tolerance range, only such images of the environment of the first data set into which objects from the second data set have been inserted are used for the enhanced first data set here.

[0073] Then, the enhanced first data set 824 is provided as output data of the enhanced machine learning algorithm, step 826. Here, the images of the enhanced first data set are set as training images for training the applied machine learning algorithm.

[0074] exist Fig. 9 Schematically, the process of the method according to the invention in a preferred embodiment is shown in , more precisely in the form of a flow chart. Here, the training using the machine learning algorithm 9 is shown.

[0075] For this purpose, training data 502 are provided in step 900. These training data include a first training data set 902 and a second training data set 904, and in particular a third training data set 902', which can however also correspond to the first training data set (in this context, the method can also be performed only with the first and second training data sets).

[0076] The first training data set 902 comprises images of an environment in which the mobile device, in particular a vehicle or a robot that moves at least partially autonomously, is to move, i.e. images such as from Figure 2 In addition, the first training data set 902 includes labels of images of the environment, ie, for example, categories or their assignments such as in the image or result image 350. The first data set 902 preferably also includes spatial information 906 about the image, such as depth information.

[0077] The second training data set 904 includes images of objects that should be considered when navigating the mobile device in the environment, i.e., such as Figure 5 In addition, the second training data set 904 also includes labels of the images of the objects, that is, for example, corresponding categories or their assignments, such as using Figure 5 Image 550 is also shown.

[0078] In step 910, the enhanced machine learning algorithm to be trained is then adapted based on the training data so that objects in images from the second training data set are inserted into images of the first training data set and labels of objects from images of the second training data set are assigned to relevant objects of the first training data set.

[0079] In this case, an image from the environment of the first training data set into which objects from the second training data set have been inserted is compared with one or more images from the environment of the third training data set. This includes, in step 912, determining one or more selected sets 914 of spatial information of the image from the environment of the first training data set into which objects from the second training data set have been inserted. In step 918, a metric is determined 920 for the same or comparable set of spatial information of multiple images from the environment of the third training data set. When adapting the enhanced machine learning algorithm, an effort is made or performed to adapt the enhanced image to such an image of the third training data set in which the metric is within a predetermined tolerance range. In step 922, a trained enhanced machine learning algorithm 924 is then provided. The trained enhanced machine learning algorithm can then be used for navigating the mobile device or for determining navigation information for this purpose.

[0080] A reinforcement machine learning algorithm—which may, for example, include or be based on an artificial neural network—may be trained, for example, in a similar manner as in the case of so-called “Generative Adversarial Networks” (GANs).

[0081] Generative adversarial networks typically consist of two artificial neural networks that perform a zero-sum game. One of them creates candidates (the generator) and the second neural network evaluates the candidates (the so-called discriminator). Typically, the generator maps from a vector of latent variables to the desired result space. The goal of the generator is to learn to produce results according to a certain distribution. The discriminator is trained to distinguish the results of the generator from data from the real pre-given distribution. The objective function of the generator is then to produce results that the discriminator cannot distinguish. This gradually adapts the generated distribution to the real distribution.

[0082] For this reason, a sufficiently large data set consisting of real data is usually required to train such a model. The discriminator is trained with the data set until it reaches an acceptable accuracy. During subsequent training, a random selection is given to the generator, which is selected from a previously defined range of latent variables. From the random selection, the generator tries to generate a new distribution. This distribution is then presented to the discriminator, which tries to distinguish it from the true distribution. The weights of the two models are improved independently of each other by backpropagation, so that the generator can create a better distribution and the discriminator can better recognize the distribution. Through this game, the two models continuously improve each other, which, with sufficient training time, leads to a generated distribution that cannot be distinguished from the true distribution.

[0083] However, this is much simpler within the scope of the present invention, since only the labeled image (see, for example, image 350 ) must be used, into which the labeled object (see image 550 ) is inserted.

Claims

1. A method for generating training images when using an enhanced machine learning algorithm, wherein the training images can be used for training the applied machine learning algorithm, in particular in order to improve the object recognition of the applied machine learning algorithm, the method comprising: providing (800) input data to the enhanced machine learning algorithm, wherein the input data comprises a first data set (802) and a second data set (804), wherein the first data set (802) comprises images (250) of an environment in which a mobile device, in particular an at least partially autonomous vehicle or a robot, is to be moved, wherein at least in some of the images of the environment one or more objects (140, 142, 144) are shown which are to be taken into account when navigating the mobile device in the environment, and wherein the first data set (802) further comprises labels (C1, C2, C3) for the images of the environment, wherein the second data set (804) comprises images of objects (546) that should be considered when navigating the mobile device in the environment, and wherein the second data set (804) further comprises labels (C4) for the images of the objects; Performing enhancement (812) by means of an enhancement machine learning algorithm (810), comprising: inserting objects from the images of the second data set into the images of the first data set, and assigning labels of the objects from the images of the second data set to the relevant images of the first data set, so as to obtain an enhanced first data set; and An enhanced first data set (822) is provided (824) as output data of the enhanced machine learning algorithm, wherein images of the enhanced first data set are set as training images for training the applied machine learning algorithm.

2. The method according to claim 1, wherein the first data set further comprises spatial information about the image of the environment, in particular depth information, and wherein objects from the image of the second data set are inserted into the image of the first data set taking into account the spatial information.

3. The method according to claim 2, wherein the spatial information comprises at least one of the following: depth information, size of the object, orientation of the object, position of the object, and wherein preferably the spatial information is intrinsic information of the image of the environment and / or separate spatial information.

4. The method according to claim 2 or 3, wherein the enhancing further comprises: determining (814) one or more selected sets of spatial information of an image from an environment of the first data set into which an object from the second data set has been inserted (816), wherein the sets relate in particular to the object from the second data set into which the object has been inserted, providing (818) or determining a metric (820), in particular a probability of occurrence, for the set or comparable set of spatial information of a plurality of images of the environment from the first data set before inserting the object from the second data set, Therein, an image from the environment of the first data set into which the object from the second data set has been inserted is used for the enhanced first data set only if the metric is within an assigned tolerance range.

5. A method according to any of the preceding claims, wherein the first data set further comprises photographic information about the image of the environment, in particular exposure information, and wherein objects from the image of the second data set are inserted into the image of the first data set taking into account the photographic information.

6. A method according to any of the preceding claims, wherein images of objects from the second data set are inserted into images of the first data set so that the relative frequency of one or more objects of a predetermined type in the images of the first data set is increased by the insertion.

7. A method according to any of the preceding claims, wherein images of objects from the second data set are inserted into images of the first data set so that the relative frequencies of a plurality of predefined types of objects in the images of the first data set are adapted to one another by the insertion. 8 . The method according to claim 1 , wherein at least a part of the plurality of images of the first data set is detected, in particular by means of an image sensor of a mobile device in the range in which the mobile device is moved in an environment.

9. A method for training an enhanced machine learning algorithm, the enhanced machine learning algorithm being configured to generate training images which can be used for training an applied machine learning algorithm, in particular in order to improve object recognition of the applied machine learning algorithm, comprising: providing (900) training data, wherein the training data comprises a first training data set (902) and a second training data set (904), wherein the first training data set (902) comprises images of an environment in which the mobile device, in particular an at least partially autonomous vehicle or a robot, is to move, and wherein the first training data set further comprises labels for the images of the environment, wherein the second training data set (904) comprises images of objects that should be considered when navigating the mobile device in the environment, and wherein the second training data set further comprises labels for the images of the objects, Adapting (910) the enhanced machine learning algorithm based on the training data so that objects in images from the second training data set are inserted into images of the first training data set and labels of objects from images of the second training data set are assigned to associated images of the first training data set; and The trained enhanced machine learning algorithm (922) is provided (924).

10. The method according to claim 9, wherein the training data further comprises a third training data set, wherein the third training data set (902') comprises images of an environment in which the mobile device, in particular an at least partially autonomous vehicle or a robot, is to be moved, wherein one or more objects are respectively shown in the images of the environment that are to be taken into account when the mobile device is navigating in the environment, and wherein the third training data set further comprises labels of the images of the environment, The adapting of the enhanced machine learning algorithm includes comparing an image of an environment from the first training dataset into which an object from the second training dataset has been inserted with one or more images of an environment from a third training dataset.

11. The method according to claim 10, wherein the first training data set and the third training data set each furthermore comprise spatial information about the image of the environment, in particular depth information, and wherein the comparison comprises: determining (914) a set (916) of one or more selected spatial information of an image of an environment from a first training data set into which an object from the second training data set has been inserted, wherein the set particularly relates to an object from the second training data set into which the object has been inserted, providing or determining (916) a metric (920), in particular a probability of occurrence, for the set or a comparable set of spatial information of a plurality of images of the environment from a third training data set, The enhanced machine learning algorithm is adapted in this case by adapting images of an environment from a first training data set into which objects from a second data set have been inserted to images of an environment from a third training data set for which the metric is within a predefined tolerance range.

12. A method for determining navigation information for a mobile device, in particular a vehicle or a robot that moves at least partially autonomously, the method comprising: providing an image of an environment in which the mobile device moves, determining navigation information based on images of the environment using an applied machine learning algorithm, the applied machine learning algorithm being trained using training images obtained by means of a method according to any one of claims 1 to 8, and Provide navigation information, and In particular determining control information for moving the mobile device based on the navigation information, and Control information is provided and / or a mobile device is moved based on the control information.

13. A computing unit (108) comprising a processor configured such that the processor performs the method according to any one of the preceding claims.

14. A computer program comprising instructions which, when executed by a computer, in particular a computing unit according to claim 13, cause the computer to perform the method according to any one of claims 1 to 12.

15. A computer-readable data carrier having stored thereon a computer program according to claim 14.

Images