Methods for operating an artificial intelligence system and systems with an artificial intelligence system

Abstract

The invention relates to a method for operating an artificial intelligence system (300), wherein the artificial intelligence system (300) is trained - in a training step - starting from synthetically generated image data (105), wherein the synthetically generated image data (105) are abstracted by means of a first abstraction function (106) to obtain synthetically generated, abstracted image data (107) and the artificial intelligence system is trained on the basis of the synthetically generated, abstracted image data (107) to obtain a trained artificial intelligence system (300); wherein - in an application step (200) following the training step (100) - real image data (201) are provided, which are generated by taking a real image, wherein the real image data (201) are abstracted by means of a second abstraction function (202) in order to obtain real, abstracted image data (203) and the real, abstracted image data (203) are supplied to the trained artificial intelligence system (300).

Description

[0001] The invention relates to a method for operating an artificial intelligence system. Furthermore, the invention relates to a system comprising an artificial intelligence system. The invention further relates to a computer program product.

[0002] Artificial intelligence systems (AI systems) that operate on the basis of image data typically require a large amount of training data. Such artificial intelligence systems are usually trained using labeled real image data generated by capturing a real image. The systematic provision and labeling of this real image data is expensive, time-consuming, and error-prone, as real objects must be provided and image data of these real objects must be captured. Labeling the real image data is typically performed at least partially manually and therefore requires a certain amount of effort. Finally, this approach assumes that the objects in question already exist and correspond to the scenario to be trained.

[0003] Therefore, image data is increasingly being synthesized to obtain training data for artificial intelligence systems. This approach is accompanied by the fundamental problem that there is a discrepancy between the synthetic image data used for training and the real image data consumed by the trained artificial intelligence system during the application phase, also known as the inference phase. This discrepancy is referred to as the "domain gap" and can lead to the performance of the artificial intelligence system lagging behind the training performance.

[0004] In order to keep the domain gap small, photorealistic synthetic image data is typically generated, which can be very complex because, in addition to the purely geometric information, aspects such as lighting, material appearance (color and other properties such as surface texture, etc.), environmental properties (haze, pollution, etc.) and sensor characteristics (chromatic aberration, distortion effects, etc.) must be physically correctly represented.

[0005] Against this background, the task arises of reducing the effort required to train artificial intelligence systems with synthetic image data without increasing the domain gap.

[0006] To solve this problem, the invention proposes a method for operating an artificial intelligence system, wherein the artificial intelligence system is trained - in a training step - based on synthetically generated image data, wherein the synthetically generated image data are abstracted by means of a first abstraction function in order to obtain synthetically generated, abstracted image data and the artificial intelligence system is trained based on the synthetically generated, abstracted image data in order to obtain a trained artificial intelligence system; wherein - in an application step following the training step - real image data are provided which are generated by recording a real image, wherein the real image data are abstracted by means of a second abstraction function in order to obtain real, abstracted image data and the real, abstracted image data are fed to the trained artificial intelligence system.

[0007] The object is further achieved by a system with an artificial intelligence system, wherein the system is configured to carry out a method described above.

[0008] Furthermore, the invention relates to a computer program product comprising instructions which, when executed by a computer, cause the computer to carry out such a method.

[0009] In the method according to the invention, the underlying image data is abstracted both in the training step and in the application step using a suitable abstraction function. Since the underlying image data is different - synthetic during training and real in the application - two different abstraction functions are used. This procedure allows the artificial intelligence system to be provided with image data both in training and in application that is abstracted in a similar way, i.e. that has similar features at the level of abstraction. The previously explained "domain gap" is thereby reduced without it being necessary to generate particularly realistic synthetic image data. Consequently, the effort required to train artificial intelligence systems with synthetic image data can be reduced without increasing the "domain gap".

[0010] The method according to the invention can be used in particular for machine visual inspection, for machine support of visual inspections, packaging personnel and / or for cobots, for example for pose detection.

[0011] Preferably, one or more of the following functions are used as the first and / or second abstraction function: an edge filter; an engraving filter; a pen filter; a brush filter; a neural network that reduces complexity without losing relevant, in particular geometric, information content; a reconstruction of depth from a video stream that is in particular smoother than the measurement result of a depth sensor.

[0012] The synthetically generated, abstracted image data is preferably annotated in the training step to obtain synthetically generated, abstracted, and labeled image data. The synthetically generated, abstracted, and labeled image data is preferably used for training the artificial intelligence system.

[0013] According to an advantageous embodiment of the invention, the trained artificial intelligence system is used—in the application step—to test and / or evaluate a test object, wherein the real image data contains a recording of the test object, in particular wherein the recording of the real test object is part of a video stream. A video stream allows a large number of individual images to be obtained in a short time, resulting in a particularly cost- and time-efficient method.

[0014] According to an advantageous embodiment of the invention, the synthetic, abstracted image data is evaluated to obtain a first evaluation result and the real, abstracted image data is evaluated to obtain a second evaluation result, wherein the first and second evaluation results are compared. It is fundamentally desirable for the first and second abstraction functions to generate abstracted image data that is as similar as possible. The image data is therefore evaluated to obtain a measure of the abstraction of the image data, and the evaluation results are compared to check whether the result of the abstraction functions is similar. This check or comparison of the evaluation results thus provides a measure of the remaining "domain gap." The first and second evaluation results can be a statistical indicator, for example a noise indicator, a mean value, or a distribution.A particularly advantageous embodiment of the method is one in which the first and second evaluation results are generated by a discriminator network. Alternatively, the activation behavior of a non-specifically pre-trained object detector can be used as the first and second evaluation results.

[0015] In this context, a preferred embodiment is one in which the first abstraction function and / or the second abstraction function are modified depending on the comparison of the first and second evaluation results. The modification is preferably carried out in such a way that the domain gap becomes smaller. For example, to increase the degree of abstraction, it is conceivable that the respective abstraction function is modified in such a way that it is filled with noise from random patterns and / or that it is overlaid with other abstracted image data that do not include any objects relevant to the application of the artificial intelligence system.

[0016] Particularly advantageous is an embodiment in which a post-training step is performed using the modified first abstraction function, in which the artificial intelligence system is trained based on synthetically generated image data. The synthetically generated image data are abstracted using the modified first abstraction function to obtain synthetically generated, abstracted image data, and the artificial intelligence system is trained based on the synthetically generated, abstracted image data. In this way, the domain gap of the artificial intelligence system can be reduced.

[0017] According to an advantageous embodiment of the invention, it is provided that – in the training step – synthetic image data is generated by a synthesis function, and the synthesis function is modified depending on the comparison of the first and second evaluation results. Preferably, the modification is carried out in such a way that the domain gap is reduced.

[0018] In this context, it is advantageous to perform a post-training step with the modified synthesis function, in which synthetic image data is generated using the modified synthesis function. The synthetically generated image data is abstracted using the first abstraction function to obtain synthetically generated, abstracted image data, and the artificial intelligence system is trained based on the synthetically generated, abstracted image data. In this way, the domain gap of the artificial intelligence system can be reduced.

[0019] Further details and advantages of the invention will be explained below with reference to the exemplary embodiment illustrated in the drawings. Herein: Fig. 1 an embodiment of a method according to the invention for operating an artificial intelligence system in a flowchart; and Fig. 2 a flowchart explaining the evaluation and comparison of the abstracted synthetic and real image data.

[0020] In the Fig. Figure 1 shows a flowchart of an embodiment of the method according to the invention for operating an artificial intelligence system 300, for example, an artificial neural network. The method comprises a training step 100, in which the artificial intelligence system 300 is trained based on synthetically generated image data, and an application step 200, in which a task, for example, a visual inspection or pose detection, is solved with the trained artificial intelligence system 300 based on real image data. In this respect, the trained artificial intelligence system 300 can be used in the application step 200 to test and / or evaluate a test object.

[0021] In the training step 100, starting from predetermined resources 101 for a synthetic scene, a scene composition 102 is carried out to obtain a synthetic scene 103. This synthetic scene 103 can, for example, be in the form of a grid model or 3D model. The scene composition 102 particularly comprises the construction of the synthetic scene 103 using the provided resources 101, for example, a background scene. The scene can be expanded by selecting a random number of (target) objects and clutter objects in various configurations. The scene does not necessarily have to contain a (target or clutter) object. The generation of an empty training image is also possible. Furthermore, the scene composition 102 comprises the placement of selected objects on previously determined storage areas. The placed objects can be freely displayed or can overlap one another.In particular, it is possible for physics-based placement to occur, so that objects, for example, lie on top of each other or are realistically tipped over.

[0022] Starting from the synthetic scene 103, a rendering 104 is performed to obtain synthetic image data 105. This synthetic image data 105 can be present, for example, as RGB(D) image data.

[0023] According to the present invention, in the training step 100, the synthetically generated image data 105 are abstracted using a first abstraction function 106 to obtain synthetically generated, abstracted image data 107. The synthetically generated, abstracted image data 107 are annotated to obtain synthetically generated, abstracted, and labeled image data 109. The explained composing 102 of the synthetic scene 103, rendering 104, abstraction 106, and annotation 108 are repeated several times (reference numeral 111) to obtain a set of synthetic image data that is used in its entirety as training data for the artificial intelligence system 300. This training is described in Fig. 1 is designated by reference number 110.

[0024] In an application step 200 following the training step 100, real image data 201 is provided, for example by a capture device such as a camera. This real image data 201 is generated by capturing a real image and can be present, for example, as RGB(D) image data. The real image data 201 can be a recording of a test object, wherein the recording is in particular part of a video stream. In the application step, the real image data 201 is abstracted using a second abstraction function 202 to obtain real, abstracted image data 203. This real, abstracted image data 203 is fed to the trained artificial intelligence system 300 as input data 210, on the basis of which the artificial intelligence system 300 performs its task, for example a visual inspection or pose detection.

[0025] The result 301 obtained by executing this task is output (reference numeral 400). Based on this output 400, further validation and adaptive resampling 401 can be performed.

[0026] The method according to the invention provides both synthetic, abstracted image data 107 and real abstracted image data 203 by using the first and second abstraction functions 106, 202. In order to keep the domain gap as small as possible, the abstracted image data 107, 203 should be as similar as possible. The representation in Fig.2 shows a flowchart of a portion of the method according to the exemplary embodiment, which includes an evaluation and comparison of the abstracted synthetic image data and abstracted real image data 203 in order to adapt the method such that abstracted image data 107, 203 that are as similar as possible are obtained both from the synthetic image data 105 and from the real image data 201. Here, the synthetic, abstracted image data 107 is evaluated (reference numeral 502) to obtain a first evaluation result 504, and the real, abstracted image data 203 is evaluated (reference numeral 501) to obtain a second evaluation result 503, wherein the first and second evaluation results 503, 504 are compared in a comparison step 505. Based on this comparison 505, an adaptation 506 then takes place.Within the scope of this adaptation 506, the first abstraction function 106 and / or the second abstraction function 202 can be modified depending on the comparison 505 of the first and second evaluation results 503, 504. Using the modified first abstraction function 106, a retraining step is then performed, in which the artificial intelligence system 300 is trained based on synthetically generated image data 105. The synthetically generated image data 105 is abstracted using the modified first abstraction function 106 to obtain synthetically generated, abstracted image data 107, and the artificial intelligence system 300 is trained based on the synthetically generated, abstracted image data 107.

[0027] Alternatively or additionally, the synthesis function 104 can be modified depending on the comparison 505 of the first and second evaluation results 503, 504. Using the modified synthesis function 104, a post-training step can then be performed, in which synthetic image data 105 is generated using the modified synthesis function 104. The synthetically generated image data 105 is abstracted using the first abstraction function 106 to obtain synthetically generated, abstracted image data 107, and the artificial intelligence system 300 is trained based on the synthetically generated, abstracted image data 107.

[0028] In summary, the embodiment relates to a method for operating an artificial intelligence system 300, wherein the artificial intelligence system 300 is trained - in a training step - based on synthetically generated image data 105, wherein the synthetically generated image data 105 are abstracted by means of a first abstraction function 106 in order to obtain synthetically generated, abstracted image data 107 and the artificial intelligence system is trained based on the synthetically generated, abstracted image data 107 in order to obtain a trained artificial intelligence system 300;wherein - in an application step 200 following the training step 100 - real image data 201 are provided, which are generated by capturing a real image, wherein the real image data 201 are abstracted using a second abstraction function 202 to obtain real, abstracted image data 203, and the real, abstracted image data 203 are fed to the trained artificial intelligence system 300. This makes it possible to reduce the effort required to train artificial intelligence systems with synthetic image data without increasing the "domain gap." In particular, the invention presents a procedure with which scenarios can be considered that do not work with purely depth-image-based observation: e.g., when the depth sensor cannot resolve geometric features (engravings, small parts, etc.).

Claims

[1] Method for operating an artificial intelligence system (300), wherein the artificial intelligence system (300) is trained - in a training step - based on synthetically generated image data (105), wherein the synthetically generated image data (105) are abstracted by means of a first abstraction function (106) in order to obtain synthetically generated, abstracted image data (107), and the artificial intelligence system is trained based on the synthetically generated, abstracted image data (107) in order to obtain a trained artificial intelligence system (300); wherein - in an application step (200) following the training step (100) - real image data (201) are provided which are generated by recording a real image, wherein the real image data (201) are abstracted by means of a second abstraction function (202) in order to obtain real, abstracted image data (203) and the real, abstracted image data (203) are fed to the trained artificial intelligence system (300). [2] Method according to claim 1, characterized by that the trained artificial intelligence system (300) - in the application step - is used to test and / or evaluate a test object, wherein the real image data (201) contain a recording of the test object, in particular wherein the recording of the real test object is part of a video stream. [3] Method according to one of the preceding claims, characterized bythat the synthetic, abstracted image data (107) are evaluated to obtain a first evaluation result (503) and the real, abstracted image data (203) are evaluated to obtain a second evaluation result (504), wherein the first and second evaluation results (503, 504) are compared. [4] Method according to claim 3, characterized by that the first abstraction function (106) and / or the second abstraction function (202) is changed depending on the comparison (505) of the first and second evaluation results (503, 504). [5] Method according to claim 4, characterized bythat a post-training step is carried out with the modified first abstraction function (106), in which the artificial intelligence system (300) is trained on the basis of synthetically generated image data (105), wherein the synthetically generated image data (105) are abstracted by means of the modified first abstraction function (106) in order to obtain synthetically generated, abstracted image data (107) and the artificial intelligence system (300) is trained based on the synthetically generated, abstracted image data (107). [6] Method according to claim 3 of 4, characterized by that - in the training step (100) - synthetic image data (105) are generated by a synthesis function (104) and the synthesis function (104) is changed depending on the comparison (505) of the first and second evaluation results (503, 504). [7] Method according to claim 6, characterized bythat a post-training step is carried out with the modified synthesis function (104), in which synthetic image data (105) are generated by means of the modified synthesis function (104), wherein the synthetically generated image data (105) are abstracted by means of the first abstraction function (106) in order to obtain synthetically generated, abstracted image data (107) and the artificial intelligence system (300) is trained based on the synthetically generated, abstracted image data (107). [8] System comprising an artificial intelligence system (300), the system being configured to carry out a method according to any one of the preceding claims. [9] A computer program product comprising instructions which, when executed by a computer, cause the computer to carry out a method according to any one of claims 1 to 7.

Images