Determining data set for use in training, apparatus and computer-implemented method for training and operating machine learning system

Abstract

Apparatus and computer-implemented method of determining a data set for use in machine learning system training, combining a first data set and a second data set into a data set for use in training, the first data set comprising a first digital image, and the second data set comprising a second digital image. The first data set is marked by at least one tag from the first set of tags, in particular by one-hot coding of one tag or by multi-hot coding of a plurality of tags, and the second data set comprises a second digital image, the first digital image is marked by at least one tag from a second set of tags, in particular by means of one-hot coding of one tag or by means of multi-hot coding of a plurality of tags, at least one tag of the two sets of tags differing, a first code is determined from the two sets of tags for marking the first digital image by means of tags from the two sets of tags, and a second code is determined from the two sets of tags, the first coding is mapped to a first representation in a state space for the data set in training comprising the first digital image and the first representation. Computer-implemented methods and apparatus for training and operating the system.

Description

Technical Field

[0001] The present invention is based on a device and a computer-implemented method for determining a data set for use in training a machine learning system, for training a machine learning system, and for running a machine learning system. Summary of the invention

[0002] A computer-implemented method for determining a data set for use in training a machine learning system provides for combining at least two data sets into a data set for use in training, wherein the at least two data sets include a first data set and a second data set, wherein the first data set includes a first digital image, which is labeled with at least one label from a first set of labels, in particular by one-hot encoding with one label or by multi-hot encoding with multiple labels, and the second data set includes a second digital image, which is labeled with at least one label from a second set of labels, in particular by one-hot encoding with one label or by multi-hot encoding with multiple labels, wherein at least one of the two sets of labels is different, wherein a first encoding is determined based on the two sets of labels for labeling the first digital image with labels from the two sets of labels, wherein the first encoding is mapped to a first representation in a state space The data set for use in training includes a first digital image and the first representation. The representation is: a continuous representation of at least one label from a union of the first image based on which the label is based. The data set for use in training the machine learning system is particularly suitable for training semantic image synthesis based on the first image and the first representation in the state space. For example, semantic image synthesis is performed using a generative model designed to map the first representation onto a synthetic digital image.

[0003] It may be provided that a second encoding is determined based on the two sets of labels for labeling a second digital image with labels from the two sets of labels, wherein the second encoding is mapped to a second representation in the state space, wherein the data set for use in training comprises the second digital image and the second representation. Thus, the two images and the representations respectively assigned to them can be used to train a machine learning system, in particular for semantic image synthesis.

[0004] A computer-implemented method for training a machine learning system provides for determining a data set for use in training by a method for determining a data set for use in training a machine learning system, wherein the system comprises a model designed to map at least one label, in particular a one-hot encoding of a label or a multi-hot encoding of a plurality of labels, to a representation of at least one label in a state space provided in the data set for use in training, and to map the representation from the state space to a synthetic digital image, wherein the model is trained with a second digital image and a second representation from the data set and / or a first digital image and a first representation to map the representation from the state space to the synthetic digital image; or wherein the model is designed to map a digital image to a representation of at least one label in a state space provided in the data set for use in training, and to determine the at least one label for the digital image based on the representation from the state space, in particular a one-hot encoding of a label or a multi-hot encoding of a plurality of labels, wherein the model is trained with a second digital image and a second representation from the data set and / or a first digital image and a first representation to map the digital image to a representation in the state space. This means that the model is trained with a representation of at least one combined label, for example for semantic image synthesis.

[0005] A computer-implemented method for operating a machine learning system provides that the machine learning system comprises a model, which is designed to map at least one label for a synthetic digital image, in particular a one-hot encoding of a label or a multi-hot encoding of multiple labels, to a representation of at least one label in a state space provided in a data set for use in training, and to map the representation from the state space to the synthetic digital image, or to determine the at least one label, in particular a one-hot encoding of a label or a multi-hot encoding of multiple labels, for a digital image, wherein the model is trained using a method for training, wherein the at least one label is mapped using the model to a representation of at least one label in the state space, and the representation in the state space is mapped to the synthetic digital image, or wherein at least one label for the digital image is determined using the model. This means that the model is trained with a representation of at least one combined label, for example for semantic image synthesis. The model can be trained based on the synthetic digital image generated by the model and / or based on these labels for labeling the synthetic digital image, in particular for classification or semantic segmentation.

[0006] It may be provided that the machine learning system comprises a technical system which is designed to operate at least partially autonomously as a function of at least one label for a digital image, in particular a one-hot encoding of a label or a multi-hot encoding of a plurality of labels, wherein in the method for operating a machine learning system a digital image is detected, wherein at least one label for the digital image is determined using the model, and wherein the technical system operates at least partially autonomously as a function of the at least one label for the digital image. This means that the system with the trained model reacts autonomously to previously unknown digital images.

[0007] A device for determining a data set for use in training a machine learning system has advantages corresponding to the advantages of the method for determining the data set, wherein the device comprises at least one processor and at least one memory, wherein the at least one memory comprises instructions executable by the at least one processor, and when the instructions are executed by the at least one processor, the device performs the method for determining the data set, wherein the at least one processor is designed to execute these instructions.

[0008] A device for training a machine learning system has advantages corresponding to the advantages of the method for training a machine learning system, the device comprising at least one processor and at least one memory, wherein the at least one memory comprises instructions executable by the at least one processor, and when the instructions are executed by the at least one processor, the device performs the method for training a machine learning system, wherein the at least one processor is designed to execute these instructions.

[0009] A device for running a machine learning system has advantages corresponding to the advantages of the method for running a machine learning system, wherein the device includes at least one processor and at least one memory, wherein the at least one memory includes instructions executable by the at least one processor, and when the instructions are executed by the at least one processor, the device performs the method for running a machine learning system, wherein the at least one processor is designed to execute these instructions.

[0010] A computer program has advantages corresponding to those of the method, the computer program comprising instructions executable by a computer, which when executed by the computer executes the method. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Further advantageous embodiments can be found in the following description and in the accompanying drawings. In these drawings:

[0012] Figure 1 A schematic diagram showing an apparatus for determining a data set,

[0013] Figure 2 A schematic diagram showing the dataset,

[0014] Figure 3 A flowchart showing the steps of a computer-implemented method for determining a data set,

[0015] Figure 4 a schematic diagram showing an apparatus for training a machine learning system,

[0016] Figure 5 A flowchart showing the steps of a computer-implemented method for training a machine learning system,

[0017] Figure 6 a schematic diagram showing an apparatus for running a machine learning system,

[0018] Figure 7 A flow chart with steps for a computer-implemented method for running a machine learning system is shown. DETAILED DESCRIPTION

[0019] Figure 1 A schematic diagram of a first device 100 is shown in FIG. The first device is designed to determine a data set 102 for use in training.

[0020] The first device 100 includes at least one first processor 104 and at least one first memory 106 .

[0021] The at least one first memory 106 includes instructions executable by the at least one first processor 104. When the instructions are executed by the at least one first processor 104, the first device 100 performs a method for determining a data set 102 for use in training. The at least one first processor 104 is designed to execute these instructions. In this example, the data set 102 for use in training is stored in the at least one first memory 106.

[0022] Figure 2 1 shows an architecture for determining a data set 102 for use in training. The data set 102 for use in training is determined based on at least two data sets. This is described by taking the example of determining the data set 102 for use in training based on the first data set 202 and the second data set 204.

[0023] An example of the first dataset 202 is Berkeley Deep Drive, or BDD. An example of the second dataset 204 is Cityscapes.

[0024] The first data set 202 includes digital images, each of which is assigned at least one label, which labels the corresponding digital image. In this example, a first digital image 206 labeled by at least one label 208 is provided. In this example, the first image 206 is labeled by using a unique hot encoding of one label or by using a multi-hot encoding of multiple labels. A first set of labels is provided for the first image 206, which includes the one or more labels for the first image.

[0025] These labels from the first set of N labels each mark a class. In the case of one-hot encoding, a class is mapped to the vector {0, 1} N , the vector contains exactly 1 and 0. For example, the elements of the vector representing a specific category are assigned to a category respectively, where the element is 1, which is assigned to the specific category represented by the vector. In the case of multi-hot encoding, multiple elements of the vector are allowed to be 1, that is, the vector can represent multiple categories.

[0026] Examples of the first set of labels are two-wheeled vehicles, cars, and background people. The exemplary first digital image 206 represents a two-wheeled vehicle. The exemplary first label 208 is a two-wheeled vehicle. An example of a one-hot encoding of the exemplary first digital image 206 is a vector [1, 0, 0, 0], which represents the exemplary first label 208 "two-wheeled vehicle".

[0027] The second data set 204 includes digital images, each of which is assigned at least one label, which labels the corresponding digital image. In this example, a second digital image 210 labeled by at least one label 212 is provided. In this example, the second digital image 210 is labeled by using a unique hot encoding of one label or by using a multi-hot encoding of multiple labels. A second set of labels is provided for the second image 210, which includes the one or more labels for the second digital image 210.

[0028] These labels from the second set of M labels each mark a category. In the case of one-hot encoding, a category is mapped to the vector {0, 1} N , the vector contains exactly 1 and 0. For example, the elements of the vector representing a specific category are assigned to a category respectively, where the element is 1, it is assigned to the specific category represented by the vector. In the case of multi-hot encoding, multiple elements of the vector are allowed to be 1, that is, the vector can represent multiple categories.

[0029] Examples of the second set of labels are sky, vegetation, people, vehicles. The exemplary second digital image 210 represents a person. The exemplary second label 212 is a person. An example of a one-hot encoding of the exemplary second digital image 210 is a vector [0,0,1,0], which represents the exemplary second label 212 "person".

[0030] In this example, at least one label in the two sets of labels is different. In this example, N=M=4 labels are provided in both sets of labels. The number of labels in the two sets may also be different.

[0031] Each digital image from the first data set 202 is assigned a code for marking the respective digital image, which codes the labels from the two groups (Satz). The first digital image 206 is assigned a first code 214 for marking the first digital image 206. The first code 214 codes the labels from the two groups. This is described by way of example for the first digital image 206.

[0032] The tags contained in the two groups are combined in a first encoding 214. In this example, the two groups of tags included in the first encoding 214 are different from each other. In this example, the order in which the tags are contained in the first encoding 214 corresponds to the order in which the tags in the first group of tags are arranged, the first group of tags is followed by tags from the second group of tags, wherein the second group of tags is different from the tags in the first group of tags, wherein the second group of tags is arranged in the order in which the tags in the second group of tags are different from the tags in the first group of tags. Other orders are also possible.

[0033] The first encoding 214 of the exemplary first digital image 206 is a vector [1,0,0,0,0,0,0], whose elements are assigned to the labels: two-wheeler, car, background, person, sky, vegetation, and vehicle.

[0034] Each of the digital images from the second data set 204 is assigned a code for labeling the respective digital image, which codes the labels from both groups. This is described by way of example for the second digital image 210 .

[0035] The second digital image 210 is assigned a second code 218 for labeling the second digital image 210. The second code 218 encodes the labels from both groups.

[0036] The labels contained in both groups are combined in a second encoding 218, just as described for the first encoding 214. The second encoding 218 of the exemplary second digital image 210 is the vector [0,0,0,1,0,0,0].

[0037] It may be provided that the first encoding 214 is generated based on labels from the two sets of labels. For example, a semantic assignment of the labels in the two sets is predetermined. The semantic assignment is determined, for example, by an expert or using a model that is designed to assign labels to one another. An example of such a model is WordNet as described in “YOLO9000: Better, Faster, Stronger (arxiv1612.08242)” by Redmon et al.

[0038] For example, the label "two-wheeler" and the label "vehicle" are assigned to each other in the semantic assignment. An exemplary first encoding 214 for the first digital image 206 is a multi-hot encoding, ie, a vector [1, 0, 0, 0, 0, 1].

[0039] First encoding 214 is represented by first representation 216 in state space 218 .

[0040] The first representation 216 of the first encoding 214 for the exemplary first digital image 206 is a first vector [0.0232, 0.1, 0.43] in the state space 218 .

[0041] Second encoding 220 is represented by second representation 222 in state space 218 .

[0042] The second representation 222 of the second encoding 216 for the exemplary second digital image 210 is a second vector [0.73, −0.21, −0.113] in the state space 218 .

[0043] In this example, encodings for other digital images are represented by respective representations of corresponding encodings in state space 218 .

[0044] In this example, the data set 102 for use in training includes a first digital image 206 and a first representation 216 and a second digital image 210 and a second representation 222. The first digital image 206 and the first representation 216 are assigned to each other as a first training data point 224. The second digital image 210 and the second representation 222 are assigned to each other as a second training data point 226.

[0045] The first data set 202 and the second data set 204 are combined using a computer-implemented method for determining a data set 102 for use in training.

[0046] Figure 3 2 shows a flow chart with the steps of a first method implemented by a computer. The first method is configured to determine a data set 102 for use in training. The first method is described using a first digital image 206 and a second digital image 210 as examples. The first method is performed accordingly for other digital images.

[0047] The first method includes step 302 .

[0048] In step 302 , a first data set 202 and a second data set 204 are provided.

[0049] Then execute step 304.

[0050] In step 304, a first code 214 is determined based on the two sets of tags, which is used to label the first digital image 206 with tags from the two sets of tags.

[0051] In step 304, a second code 218 is optionally determined based on the two sets of tags, which is used to label the second digital image 210 with tags from the two sets of tags.

[0052] Then execute step 306.

[0053] In step 306 , first encoding 214 is mapped to first representation 216 in state space 218 .

[0054] In step 306 , second encoding 220 is optionally mapped to second representation 222 in state space 218 .

[0055] In step 308 , the first data set 202 and the second data set 204 are combined into the data set 102 for use in training.

[0056] The dataset 102 for use in training includes a first digital image 206 and a first representation 216 .

[0057] Optionally, the dataset 102 for use in training includes a second digital image 210 and a second representation 222 .

[0058] Figure 4 A schematic diagram of a second device 400 is shown in FIG. The second device 400 is designed to train a machine learning system 402 .

[0059] The second device 400 includes at least one second processor 404 and at least one second memory 406 .

[0060] The at least one second memory 406 includes instructions executable by the at least one second processor 404, and when the instructions are executed by the at least one second processor 404, the second device 400 performs a method for training. The at least one second processor 404 is designed to execute these instructions. In this example, the second device 400 includes a machine learning system 402. In this example, the at least one second memory 406 includes a part of the machine learning system 402.

[0061] The machine learning system 402 includes a model. In this example, the model includes a generative adversarial network, i.e., a GAN, 410, which is designed for semantic image synthesis. The GAN 410 is designed to map at least one label 412 for a synthetic or real digital image 414 to a representation 416 of at least one label 412 in a state space 218. The at least one label 412 for the synthetic or real digital image 414 is, for example, a unique hot encoding of a label or a multi-hot encoding of multiple labels. The term "real image" refers to a digital image that reproduces a view that actually exists.

[0062] GAN 410 is designed, for example, to determine the corresponding representation in state space 218 by linear projection of the corresponding encoding. GAN 410 includes, for example, an encoder for state space 218 or uses knowledge of an already trained language model such as Vord2Vec.

[0063] GAN 410 is designed to map a representation 416 from state space 218 to a synthetic digital image 414 .

[0064] For example, GAN 410 includes an already trained neural network.

[0065] It is conceivable that the model includes a classifier 418. The classifier 418 is designed to determine at least one label 420, in particular a one-hot encoding of a label or a multi-hot encoding of multiple labels, for the synthetic digital image 414 or for the digital image 422. The classifier includes, for example, a trained neural network.

[0066] It may be provided that machine learning system 402 includes a technical system 424 which is designed to operate at least partially autonomously based on at least one label 420 of digital image 422 .

[0067] The technical system 424 is, for example, a physical system. The technical system 424 is, for example, a robot, in particular an autonomous vehicle or a household appliance or a tool or a production machine or a personal assistance system or an access control system.

[0068] Technical system 424 includes, for example, a sensor 426, which is designed to detect digital image 422. Sensor 426 is, for example, a camera, a radar sensor, a LiDAR sensor, an ultrasonic sensor, an infrared sensor, a motion sensor, or a thermal imaging sensor.

[0069] Figure 5 A flow chart with steps of a second computer-implemented method is shown in . The second method is configured to train a machine learning system 402 .

[0070] The training may include jointly training the GAN 410 and the classifier 418 using the dataset 102 for use in the training, training the GAN 410 alone, or training the classifier 418 alone.

[0071] The second method is described by taking the first digital image 206 and the second digital image 210 as examples. The second method is performed accordingly for other digital images.

[0072] The second method for training system 402 includes step 502 .

[0073] In step 502, a data set 102 for use in training is determined using a first method.

[0074] Then execute step 504.

[0075] In step 504, the GAN 410 is trained using the first representation 216 and the first digital image 206 from the dataset 102 for use in training.

[0076] The GAN 410 is trained to map a representation 416 from the state space 218 onto a synthetic digital image 414 .

[0077] Optionally, the GAN 410 is trained using the second representation 222 and the second digital image 210 from the dataset 102 for use in training.

[0078] This means that in this example, the GAN 410 is trained solely using the dataset 102 for use in training.

[0079] Figure 6 A schematic diagram of a third device 500 is shown in FIG. The third device 500 is designed to run the machine learning system 302 .

[0080] The third device 600 includes at least one third processor 602 and at least one third memory 604 .

[0081] The at least one third memory 604 includes instructions executable by the at least one third processor 602, and when the instructions are executed by the at least one third processor 602, the third device 600 performs a method for machine learning. The at least one third processor 602 is designed to execute these instructions.

[0082] Figure 7 A flow chart with steps of a third method implemented by a computer is shown in The third method is configured for machine learning, in particular for running a machine learning system 402 .

[0083] The third method includes step 702 .

[0084] In step 702, the GAN 410 is trained using the second method.

[0085] Then execute step 704.

[0086] In step 704 , at least one label 412 is predefined for the synthetic digital image 414 to train the classifier 418 .

[0087] Then execute step 706.

[0088] In step 706 , a synthesized digital image 414 is determined using the GAN 410 based on the at least one label 412 of the synthesized digital image 414 .

[0089] Then execute step 708.

[0090] In step 708 , a classifier 418 is trained based on the composite digital image 414 to determine at least one label 412 for the composite digital image 414 .

[0091] It can be provided that the classifier 418 or the GAN 410 is already trained. It can be provided that only the classifier 418 is trained alone or only the GAN 410 is trained alone.

[0092] Optionally, step 710 is then performed.

[0093] In step 710, digital image 422 is detected.

[0094] Optionally, step 712 is then performed.

[0095] In step 712 , at least one label 420 is determined for the digital image 422 using the classifier 418 .

[0096] Optionally, step 714 is then performed.

[0097] In step 714 , technical system 424 operates at least partially autonomously based on at least one tag 420 for digital image 422 .

[0098] For example, each of these methods is implemented as a computer program, which includes instructions that can be executed by a computer. When the instructions are executed by the computer, the corresponding method is executed.

Claims

1. A computer-implemented method for determining a data set (102) for use in training a machine learning system (402), characterized in that At least two data sets are combined (308) into a data set (102) for use in training, wherein the at least two data sets include a first data set (202) and a second data set (204), wherein the first data set (202) includes (302) a first digital image (206) labeled with at least one label (208) from a first set of labels, in particular by one-hot encoding with one label or by multi-hot encoding with a plurality of labels, and the second data set (204) includes (302) a second digital image (210) labeled with at least one label (208) from a second set of labels. At least one label (212), in particular labeled by using a one-hot encoding of one label or by using a multi-hot encoding of multiple labels, wherein at least one of the two sets of labels is different, wherein a first encoding (214) is determined (304) based on the two sets of labels for labeling the first digital image (206) with labels from the two sets of labels, wherein the first encoding (214) is mapped (306) to a first representation (216) in a state space (218), wherein the data set (102) for use in training includes the first digital image (206) and the first representation (216).

2. The method according to claim 1, characterized in that A second encoding (218) is determined (204) based on the two sets of labels for labeling the second digital image (210) with labels from the two sets of labels, wherein the second encoding (220) is mapped to a second representation (222) in the state space (218), wherein the data set (102) for use in training includes (308) the second digital image (210) and the second representation (222).

3. A computer-implemented method for training a machine learning system (402), characterized in that A data set (102) for use in training is determined (502) by a method according to any of the preceding claims, wherein the system (402) comprises a model (410) designed to map at least one label (412), in particular a one-hot encoding of a label or a multi-hot encoding of a plurality of labels, to a representation (416) of the at least one label (412) in a state space (218) provided in the data set for use in training, and to map the representation (416) from the state space (218) to a synthetic digital image (414), wherein the model (410) is trained (504) using the second digital image (210) and the second representation (222) and / or the first digital image (206) and the first representation (216) from the data set to map the representation (416) from the state space (218) to a synthetic digital image (414). (218) is mapped to a synthetic digital image; or wherein the model (418) is designed to map a digital image (422) to a representation (416) of at least one label (420) in a state space (218) set in a data set used in training, and the at least one label (420) is determined for the digital image (422) based on the representation (416) from the state space (218), in particular a one-hot encoding of one label or a multi-hot encoding of multiple labels, wherein the model (418) is trained (504) using the second digital image (210) and the second representation (222) from the data set and / or the first digital image (206) and the first representation (216) to map the digital image (422) to the representation (416) in the state space (218).

4. A computer-implemented method for operating a machine learning system (402), characterized in that The machine learning system (402) comprises a model (410, 418) designed to map at least one label (412), in particular a one-hot encoding of a label or a multi-hot encoding of a plurality of labels, for a synthetic digital image (414) to a representation (416) of at least one label (412) in a state space (218) provided in a data set used in training, and to map the representation (416) from the state space (218) to the synthetic digital image (416), or to determine at least one label (420) for a digital image (422), in particular In particular, a one-hot encoding of one label or a multi-hot encoding of multiple labels, wherein the model (410, 418) is trained (702) using the method according to claim 3, wherein the at least one label (412) is mapped to a representation (416) of the at least one label (412) in the state space (218) using the model (410), and the representation (416) in the state space is mapped to the synthetic digital image (414), or wherein at least one label (420) for the digital image (422) is determined using the model (418).

5. The method according to claim 4, characterized in that The machine learning system (402) includes a technical system (424) designed to operate at least partially autonomously based on at least one label (420) for a digital image (422), in particular a one-hot encoding of one label or a multi-hot encoding of multiple labels, wherein the digital image (422) is detected (710), wherein at least one label (420) for the digital image (422) is determined (712) using the model (418), and wherein the technical system (424) operates at least partially autonomously based on the at least one label (420) for the digital image (422) (714).

6. An apparatus (100) for determining a data set (102) for use in training a machine learning system (402), characterized in that The device comprises at least one processor (104) and at least one memory (106), wherein the at least one memory (106) comprises instructions executable by the at least one processor (104), and when the instructions are executed by the at least one processor (104), the device (100) performs the method according to any one of claims 1 or 2, wherein the at least one processor (104) is designed to execute the instructions.

7. An apparatus (400) for training a machine learning system (402), characterized in that The device (300) comprises at least one processor (404) and at least one memory (406), wherein the at least one memory (406) comprises instructions executable by the at least one processor (304), and when the instructions are executed by the at least one processor (404), the device performs the method according to claim 3, wherein the at least one processor (404) is designed to execute the instructions.

8. An apparatus (600) for running a machine learning system (402), characterized in that The device (600) comprises at least one processor (602) and at least one memory (604), wherein the at least one memory (604) comprises instructions executable by the at least one processor (602), and when the instructions are executed by the at least one processor (602), the device (600) performs the method according to any one of claims 4 or 5, wherein the at least one processor (602) is designed to execute the instructions.

9. A computer program, characterized in that The computer program includes instructions executable by a computer, and when the instructions are executed by the computer, the method according to any one of claims 1 to 5 is performed.

Images