Computer-implemented method, computer program product, and computer-readable medium

Abstract

The invention relates to a method for generating a test data set (12) for an algorithm (1), comprising the following steps: 1.1 providing a trained artificial intelligence (8), 1.2 stimulating the artificial intelligence (8), in particular with random signals (11) and / or with quasi-linear properties for visual concepts, and 1.3 the artificial intelligence (8) generating at least one test data set (14), the test data set comprising image data (15) and activity ranges (16) assigned to the image data (1g) and / or sensor data and activity ranges assigned to the sensor data. The invention also relates to a method for operating a system for automated image-related control of devices and apparatuses for carrying out the above method. Finally, the invention relates to a control system for an apparatus, the control system comprising such an apparatus, as well as to a computer program product, a computer-readable medium, the generation of a data carrier and the use of the artificial intelligence (8).

Description

Technical Field

[0001] The invention relates to a method for generating a test data set for an algorithm, and a method for testing such an algorithm. In addition, the invention relates to a method for operating a system for automatically image-dependently controlling a device and a device for carrying out the above method. Finally, the invention relates to: a control system for a device, the control system comprising such a device; a computer program product; a computer-readable medium, the generation of a data carrier and the use of artificial intelligence. Background Art

[0002] In the field of control software development, testing of the software is a significant part of the development effort. This can be attributed to the complex design of quality tests and the difficulty and uncommonness of error detection. This applies in particular: the more complex the algorithm to be tested, the more complex it is to test. Firstly, it is also difficult to perform "comprehensive" testing, i.e. to achieve the required test coverage.

[0003] Algorithms for image processing are among the most complex algorithms in computer systems. The applicant is aware that algorithms used, for example, in the field of Advanced Driver Assistance (ADAS) or in the field of autonomous driving control are tested with the aid of test scenarios, which are generated from recorded test data sets or carried out in "Real-Life Tests" on a test field or in a real environment.

[0004] It can be checked against predefined "best case actions" in real video sequences, which are recorded beforehand and generated with a high degree of human intervention. Each new test case is manually generated and does not allow any room for randomness in the test scenario.

[0005] The known methods have in common that both test data and expected behavior are recorded through close interaction with human participants, and are accompanied by difficulties in integration into automated test settings. Scaling of the tests is complex and difficult.

[0006] The applicant is also aware that in some manufacturers the testing phase even extends to a period after product delivery and that test data are collected during the monitored autonomous driving process in order to improve the algorithm based on these test data. Summary of the invention

[0007] Based on the above, it is an object of the present invention to provide improved possibilities for testing complex image processing algorithms.

[0008] The object is achieved by a method for generating at least one test data set for an algorithm, the algorithm receiving image data as input and outputting an activity range associated with the image data as output, and / or the algorithm receiving sensor data as input and outputting an activity range associated with the sensor data as output, the method comprising the following steps:

[0009] 1.1 Provide trained artificial intelligence,

[0010] 1.2 Motivate AI, and

[0011] 1.3 The stimulated artificial intelligence generates at least one test data set, which includes image data and activity ranges assigned to the image data and / or sensor data and activity ranges assigned to the sensor data.

[0012] In step 1.1, artificial intelligence is preferably provided, which includes a long short-term memory network, or the artificial intelligence is provided by the long short-term memory network. In addition, preferably, artificial intelligence is provided in step 1.1, which includes at least one generative network and / or at least one discriminative network, or the artificial intelligence is provided by the generative network and / or the discriminative network. Then, particularly preferably, in step 1.2, at least one generative network of the artificial intelligence is stimulated, and preferably in step 1.3, at least one stimulated generative network of the artificial intelligence outputs a test data set. The generated adversarial network, in particular at least one generative network and / or at least one discriminative network, is conveniently an artificial neural network.

[0013] It should be noted that instead of the term “network”, in principle the terms “system” or “model” or “net” can also be used synonymously.

[0014] Preferably, in step 1.1, a trained artificial intelligence is provided, which is adjusted using a learning data set. Particularly preferably, a learning data set is used, which includes real image data recorded by means of at least one image acquisition device, activity ranges associated with the real image data and / or real sensor data recorded by means of at least one sensor device and activity ranges associated with the real sensor data.

[0015] Particularly preferably, the following steps are performed to obtain the trained artificial intelligence provided in step 1.1:

[0016] 2.1 providing a learning data set, which includes real image data recorded by means of at least one image acquisition device, activity ranges associated with the real image data and / or real sensor data recorded by means of at least one sensor device and activity ranges associated with the real sensor data,

[0017] 2.2 Provide artificial intelligence, and

[0018] 2.3 Tune the AI ​​using the provided learning dataset.

[0019] It also preferably applies to the artificial intelligence provided in step 1.1 and / or in step 2.2 that the artificial intelligence comprises an adversarial network, in particular a generative adversarial network, or is provided by an adversarial network.

[0020] The invention is based on the consideration that artificial intelligence is used to generate image data and / or sensor data and the associated activity range from an initial data set, with the use of whose algorithms the performance of the artificial intelligence or its behavior can be checked in particular within the scope of image-related, in particular image sequence-related or sensor value-related control. In this case, according to the invention, a trained artificial intelligence is obtained.

[0021] Preferably, the artificial intelligence is adjusted, i.e. trained, using a learning data set, in particular a learning data set obtained by observing or recording real situations and through human interaction, or preferably, the artificial intelligence is adjusted / trained using a learning data set, in particular a learning data set obtained by observing or recording real situations and through human interaction, and is then prepared for generating test data.

[0022] The data set generated with the aid of the trained artificial intelligence represents "synthetically" obtained test data which is in particular identical to the provided learning data set in terms of its type and suitability and is close to reality or indistinguishable from real data. Artificial intelligence methods are used in particular to generate image sequences and actions adapted thereto, preferably by means of generative adversarial networks.

[0023] In particular, high-dimensional data, for example in the form of image sequences, and low-dimensional data in the form of activity profiles or action corridors are processed in order to train a test case generator for classifying the performance or behavior of, in particular, image-related or sensor-related control algorithms, wherein the high-dimensional data and the low-dimensional data represent particularly reliable control parameters. Preferably, the adjustment is based on image data and / or sensor data, which are supplemented with activity profiles as vector sequences.

[0024] The algorithm for determining the test data set and the image data received as input and the image data-dependent activity range output as output is particularly preferably a control algorithm. The control algorithm can in particular be an algorithm for controlling a device, such as a vehicle or a robot.

[0025] Conveniently, the activity range includes at least one permissible action, preferably includes a plurality of permissible actions or is provided thereby. The activity range is in particular a (target) range with actions, for example a control process that is permissible for or based on associated image data or sensor data.

[0026] In a preferred embodiment, the ranges of motion represent permissible control parameters. They represent or include particularly preferred permissible control parameters of the system for the automated, image-assisted control of the device or are formed by these parameters.

[0027] The activity range comprises, for example, at least one control parameter or control value or is formed or defined by, for example, at least one control parameter or control value. The at least one control parameter or control value can be a (control) action that is permitted as a function of the image data or sensor data assigned to it. It is preferably a (multiple) control value or control parameter of a device for sensor / image-related control.

[0028] Typically, for a given initial situation detectable by image data and / or sensor data, not only are control parameters or control values ​​feasible / permissible, but there is also an associated target range that includes a plurality of actions or control values ​​or parameters representing the actions. The range of action can accordingly be defined or formed by a plurality of control parameters or control values ​​that are within the target range or extend the target range.

[0029] The range of motion particularly represents a parameter or a value, preferably a control parameter or a control value, which is permissible depending on the image data or sensor data respectively assigned to it. The range of motion can be defined by a plurality of parameters or values, in particular control parameters or control values / by a set of parameters or values, in particular control parameters or control values, which results in a movement within the (or permissible) range of motion, in particular with a device (e.g. a vehicle or a robot) controlled via an algorithm.

[0030] By supplementing the activity profile of the image data and / or sensor data, an annotated data set with test information is obtained, which is necessary for generating further test scenarios. Preferably, the individual networks, in particular the generative and adversarial, preferably discriminative networks, are adjusted by alternately generating by a generative network and (adversarially) checking the generated products, i.e. the artificial image data and / or sensor data and the associated activity profiles, by an adversarial, in particular discriminative network.

[0031] Here, the scheme on which the present invention for test data generation is based is preferably based on the work of Ian Goodfellow et al., published in the article “Generative Adversarial Nets Generative Adversarial Networks” (GANs) on January 10, 2014, ArXiv:1406.2661[statML]661v1; and the work of Alec Redfort et al., excerpted from the article “Unsupervised Representation Learning With Deep Convolutional Generative Adversarial Networks Unsupervised Representation Learning With Deep Convolutional Generative Adversarial Networks”, ArXiv:1511.06434V2[cs.LG], published on January 7, 2016; in particular the work of Sepp Hochreiter and Jürgen Schmidhuber, excerpted from the article “Long Short Term Memory” (LSTM) Neural Computation 9(8):1735-1780, 1997.

[0032] The trained artificial intelligence is stimulated. The stimulation can be performed in the following manner: input, for example at least one input file, is fed to the artificial intelligence. As a reaction to the stimulation, in particular to the feeding of the input, the artificial intelligence generates an output.

[0033] In principle, the stimulation of the trained artificial intelligence can be carried out in different ways, for example using at least one random signal as input. The random signal can be provided, for example, by an image or an image file which only displays noise or includes noise.

[0034] The stimulation can also be carried out with at least one stimulation image, preferably with two or more stimulation images or stimulation image sequences and / or with at least one stimulation video, which preferably shows a specific content or scene or situation, for example a driving situation or a driving scene.

[0035] The use of motivational images (or sequences thereof) and / or motivational videos, which in particular display real content, generally offers higher control possibilities compared to the use of random signals.

[0036] In particular, a GAN scheme is used to generate data sets from artificial intelligence applications, preferably based on stimulation by a random signal (English: random signal) or based on the application of quasi-linear properties for visual design, wherein the nonlinear properties are derived in the aforementioned document by Alec Redfort et al. in the article "Unsupervised Representation Learning With Deep Convolutional Generative Adversarial Networks" published on January 7, 2016, especially in Figure 7 on page 10 of the document. The generative network or generative algorithm here is opposite to the usual method, which means that instead of a generative network or algorithm, which allows patterns to be recognized and classified, if the process is reversed and the system is stimulated and a new "pattern" is generated by a "classification input", such as a "classification input" of a random signal. Goodfellow and Redford pointed out in their works in 2014 and 2016 that the generation of new images is almost a linear operation in vector space. This means that it is easier to adjust the algorithm to generate new sequences.

[0037] In particular, for the case of at least one image and / or video (or corresponding file) being fed to the artificial intelligence for stimulation,

[0038] In the case where artificial intelligence delivers at least one image and / or video (or corresponding file) for stimulation for stimulation, the artificial intelligence is able to generate a synthetic image or image sequence using or through quasi-linear properties for visual design, wherein the image and / or video shows (real) content or scene or situation.

[0039] In this context, linearity is to be understood in particular as the superposition principle (image_A+parameter*image_B=image_C). Quasi is to be understood in particular as the generation of a new image or stimulus video from existing individual images (function_a(image_A)+parameter*function_b(image_B)=image_C, wherein function_a and function_b are nonlinear functions).

[0040] It is proposed purely by way of example that, taking into account the quasi-linear properties of the visual design, a (synthesized, generated) image is obtained from an image showing a street in good weather and an image showing rain, which image shows a street with rain or in the rain. If a stimulus image is supplied to the artificial intelligence, for example, with an image of a street in good weather and an image with rain, a synthetic image (with an associated range of motion) can be obtained, which shows the street in the rain. Thus, new scenes can also be derived from the existing images, which scenes were not originally shown in the existing images. It should be emphasized that the combination of street and rain is to be understood purely by way of example, and that other scenes or situations can obviously also be supplied to the artificial intelligence as stimulus images, and naturally more than two images can also be supplied.

[0041] It should be noted that the principle of the above-mentioned article “Unsupervised Representation Learning With Deep Convolutional Generative Adversarial Networks” is exemplarily shown in FIG. 7 based on an image with a face.

[0042] The motivational image and / or motivational video can be, for example, an image or video recorded with a camera and showing a real scene or situation, such as a real driving scene or driving situation. However, the use of alternatively or additionally constructed or synthesized images and / or videos for stimulating artificial intelligence is not excluded.

[0043] Trained network(s), system(s) or model(s) can be used to generate new data. By stimulating a generative network, in particular an artificial intelligence, trained rules can be used to generate new image data and / or sensor data and associated activity areas.

[0044] In particular, through the characteristics of GANs (Generative Adversarial Networks), simple arithmetic operations can influence the generated objects. For example, it is also possible to obtain: Avenue-Trees = Simple Street, or Straight Street + Curve = Curved Road.

[0045] Particularly preferably, instead of a typical convolutional network, an LSTM network, in particular a convolutional LSTM network, is used for one or more generative networks and for one or more adversarial networks. The advantages of using an LSTM network are primarily that a time-dependent sequence (of image data or sensor data and associated activity range) is obtained and that noise influences (non-physical artifacts) are avoided by the LSTM. An LSTM network should be understood in particular as a network that takes into account past states. LSTM is a specialization of this type of network. Therefore, it is also possible to jointly model temporal correlations, which in particular require image sequences and activity ranges or action channels. An LSTM network that can be used within the scope of the present invention is described, for example, in the 1997 article "Long Short Term Memory" (LSTM) Neural Computation 9 (8): 1735-1780 by SeppHochreiter and Jürgen Schmidhuber.

[0046] The method is extended and allows the convolutional LSTM network to quickly collect recurring previous states in order to train the artificial intelligence provided by it, including in particular the GAN system, i.e. to obtain image data, in particular image sequences and / or sensor data, and the associated activity range. This allows the dynamic behavior of the test scene to be modeled.

[0047] The processing method according to the present invention provides a variety of advantages. On the one hand, it can significantly reduce the scope of human interaction or human intervention when testing complex algorithms. Because a very large amount of test data can be easily provided, the test coverage (English: test coverage) of complex algorithms can be significantly improved.

[0048] Since the testing can be particularly extensive and intensive, particularly reliably functioning algorithms can be obtained, which is a significant advantage or a must in the field of image-assisted control, such as may be carried out in the context of advanced driver assistance or autonomous driving. As a result, the quality and reliability of such systems can be significantly increased, and this with very manageable effort. Another advantage is that according to the invention, the test armor and test vectors are generated in only one preprocessing step.

[0049] In a preferred embodiment, at least one test data set is generated for an algorithm that receives an image sequence as input and outputs an activity range associated with the image sequence as output, in particular for an algorithm for a system for autonomous vehicle control or a preferred advanced driver assistance system, wherein, in particular in step 2.1, a learning data set is provided, which provides real image data recorded by means of at least one image detection device and an activity range associated with the sequence of images, the activity range in particular representing permissible control parameters of a system for automated image-assisted control of the device, and, in step 1.3, a trained artificial intelligence, in particular at least one generative network of the artificial intelligence, generates at least one test data set, which includes a new image sequence and an activity range associated with the new image sequence.

[0050] It should be noted that the processing method according to the invention is particularly suitable for the field of advanced driver assistance and autonomous driving, since in this field it is necessary to output control commands based on image sequences, such as videos, but the control commands are not limited to these two examples at all. Rather, test data sets can also be generated for algorithms from other fields, for example for algorithms used in the context of image-assisted automated robot control or automated sorting processes (for example on conveyor belts). Algorithms for non-image-assisted control, but based on sensor data, can also be tested in the manner according to the invention.

[0051] Preferably, raw, real image data, in particular image sequences and / or sensor data and associated activity ranges are used as learning data sets in order to adjust the artificial intelligence, in particular the GAN system.

[0052] Accordingly, it is preferred to first detect at least one image data set, in particular an image sequence, and / or at least one sensor data set, which can be achieved with the aid of well-known image detection devices (such as one or more cameras or sensor devices). For example, during the travel of the vehicle or the operation of the robot, a camera arranged in the vehicle or at the robot can be used to record. The obtained image sequence or the obtained video is then in particular a test vector. The activity range assigned to the real image data or sensor data is determined, and the image data or sensor data and the activity range are associated with each other, and a test data set is obtained. Instead of first creating a real test data set in this way, an existing test data set can also be used as long as a suitable test data set is available.

[0053] In a next step, the artificial intelligence, in particular its generating network, is preferably trained with real test data and can subsequently be stimulated, for example, with one or more random signals in order to generate new image data, in particular image sequences and / or sensor data and associated activity ranges.

[0054] Then, (or in addition to the learning data set) newly obtained synthetic test data can be fed to the algorithm to be tested in order to check the algorithm.

[0055] In particular, for the case where at least one test data set is generated for an algorithm of an advanced driver assistance system or for a system for autonomous vehicle control, it is proposed in another preferred design that the learning data set provided to the artificial intelligence for training includes an image sequence, which includes images of driving situations, and / or the image sequence includes a driving trajectory for each image and, in a preferred design, includes a target channel assigned to the corresponding driving trajectory as an assigned activity range, which target channel defines a target range around the driving trajectory.

[0056] In addition, in an improved solution, at least one test data set is generated in step 1.3, which test data set includes an image sequence, which includes images of driving situations, and / or the image sequence includes a driving trajectory for each image and in particular a target channel assigned to the corresponding driving trajectory as an assigned activity range, wherein the target channel defines a target range around the driving trajectory.

[0057] Another subject of the invention is a method for testing an algorithm, comprising the following steps:

[0058] 9.1 Provide an algorithm to be tested, which algorithm receives image data as input and outputs an activity range associated with the image data as output and / or which algorithm receives sensor data as input and outputs an activity range associated with the sensor data as output,

[0059] 9.2 carrying out the method according to the invention in order to obtain at least one test data set having image data and activity ranges assigned to the image data and / or having sensor data and activity ranges assigned to the sensor data,

[0060] 9.3 Provide at least one test dataset,

[0061] 9.4 feeding image data and / or sensor data of at least one test data set to the algorithm to be tested,

[0062] 9.5 Implement the algorithm and output the activity range as output,

[0063] 9.6 Compare the range of activities output by the algorithm as output with the range of activities of at least one test data set,

[0064] 9.7 Infer the quality of the tested algorithm from the results of the comparison.

[0065] In a preferred embodiment, in step 9 . 1 , an algorithm of an advanced driver assistance system or an algorithm for a system for autonomous vehicle control is provided as the algorithm to be tested.

[0066] One embodiment of the method for testing an algorithm according to the present invention is characterized in that, after executing steps 9.1 to 9.7, steps 1.2 and 1.3 of the method for generating a test data set according to the present invention are re-executed to generate at least one additional test data set, which is different from the previously generated test data set, and at least one additional test data set is provided and steps 9.4 to 9.7 are repeated, in particular to test other aspects of the algorithm.

[0067] This procedure has proven to be particularly advantageous. It is generally possible to repeat the generation step once or several times, in particular if new and additional aspects of the algorithm to be tested are repeatedly checked in a known manner.

[0068] Another subject of the invention is a method for operating a system for automatically controlling a device, in particular a vehicle or a robot, wherein the system comprises at least one image capture device, in particular a camera, and / or at least one sensor device as well as an evaluation and control unit and components for controlling the device, wherein an algorithm is stored or can be stored in the evaluation and control unit, which algorithm receives image data as input and outputs an activity range associated with the image data as output and / or the algorithm receives sensor data as input and outputs an activity range associated with the sensor data as output, the activity range in particular representing permissible control parameters, wherein the algorithm is tested while carrying out the method according to the invention for testing the algorithm, and if the test shows that the algorithm is not functionally sufficient, the algorithm is adapted in a predefined manner and the device is controlled using the algorithm.

[0069] The system can be, for example, a system for autonomous vehicle control or for advanced driver assistance used in a vehicle, such as a passenger car.

[0070] If a device (e.g. a vehicle or a robot) is controlled based on an algorithm which is checked and, if necessary, adapted using at least one test data set generated in the manner according to the invention, in particular while carrying out the method according to the invention for testing the algorithm, control can be carried out particularly reliably and the safety of the automated control can be significantly increased.

[0071] In a preferred embodiment, the method for the operation of a system for automatically, in particular image- and / or sensor-dependently controlling a device is characterized in that the system comprises at least one image acquisition device and by means of the at least one image acquisition device of the system a sequence of images is acquired, an activity range is associated with the images of the sequence, and in particular in step 2.1, the image sequence with the associated activity range is provided as a learning data set for training an artificial intelligence. Alternatively or additionally, it can be provided that the system comprises at least one sensor device and by means of the at least one sensor device of the system sensor data are acquired, an activity range is associated with the sensor data, and in particular in step 2.1, the sensor data with the associated activity range is provided as a learning data set for training an artificial intelligence. Step 2.1 is in particular the first step of the method according to the invention for generating a test data set, which is performed in step 9.2 of the method according to the invention for testing an algorithm in order to obtain a test data set.

[0072] Another subject of the invention is the use of a trained artificial intelligence for generating at least one test data set for an algorithm which receives image data as input and outputs an activity range associated with the image data as output, which activity range in particular represents an admissible control parameter, and / or which receives sensor data as input and outputs an activity range associated with the sensor data as output, which activity range in particular represents an admissible control parameter, wherein the artificial intelligence in particular comprises a generative adversarial network or is provided by a generative adversarial network. It should be noted that all features described above for the artificial intelligence and / or the at least one test data set can also be implemented individually or in combination within the scope of the application according to the invention.

[0073] Another subject of the present invention is a device which is constructed and designed to carry out the method according to the present invention for generating a test data set for an algorithm or for carrying out the method according to the present invention for testing an algorithm or for carrying out the method according to the present invention for operating a system for automatically, in particular image-relatedly, controlling a device.

[0074] The device according to the invention preferably comprises at least one processor and in particular a data memory. The device can, for example, comprise a computer (in particular a PC) or be designed as such a computer.

[0075] The device according to the invention can, for example, also form part of a control system for a device, in particular a mobile tool or a robot.

[0076] Likewise, the subject matter of the present invention is a computer program product comprising program code components for executing the method according to the invention for generating a test data set for an algorithm or for executing the method according to the invention for testing an algorithm or for executing the method according to the invention for operating a system for automated, image-dependent control of a device.

[0077] The invention further provides for the production of a data carrier on which at least one test data set is stored, which test data set is stored during the execution of the method according to the invention for producing at least one test data set.

[0078] Finally, the subject of the present invention is a computer-readable medium comprising instructions which, when executed on at least one computer, cause at least one computer to perform the steps of the method according to the present invention for generating a test data set for an algorithm, or for performing the steps of the method according to the present invention for testing an algorithm, or for performing the steps of the method according to the present invention for operating a system for automatically, image-dependently controlling a device.

[0079] The computer readable medium can exist, for example, in the form of a data carrier (e.g., CD-ROM, DVD, flash memory, or USB memory). It should be noted that the computer readable medium should not be understood only as a physical medium, but can also exist, for example, in the form of a data stream and or a signal representing a data stream. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Other features and advantages of the method according to the invention will become clearer from the following exemplary description with reference to the accompanying drawings.

[0081] Figure 1 A purely schematic illustration shows a process for running an exemplary embodiment of the method according to the invention for testing an algorithm,

[0082] Figure 2 A block diagram is shown for purely schematic illustration of the adjustment of the artificial intelligence,

[0083] Figure 3 A purely schematic representation of a learning data set with image sequences and associated activity areas in the form of associated driving trajectories and corridors is shown.

[0084] Figure 4 A block diagram is shown for purely schematic illustration of the generation of a test data set and a test algorithm.

[0085] Figure 5 A purely schematic representation of three test data sets is shown, each with an image sequence and an associated range of motion in the form of an associated driving trajectory and corridor. DETAILED DESCRIPTION

[0086] Figure 1 A purely schematic illustration of the process of an exemplary embodiment of the method according to the invention for testing an algorithm 1 is shown. In the drawing, the steps carried out within the scope of the exemplary embodiment are shown in the form of block diagrams or plus diagrams.

[0087] In the context of the exemplary embodiment shown, in a first step S1 an algorithm 1 to be tested is provided, which in the present case is a control algorithm for controlling an autonomous vehicle. Figure 4 An algorithm 1 for controlling an autonomous vehicle, schematically illustrated therein by box elements, is designed such that it obtains as output a sequence of images, specifically a video of the vehicle's surroundings, in a manner known per se, and outputs as output an activity range associated with the image sequence, which represents permissible control parameters for the vehicle depending on the corresponding images of the sequence.

[0088] After Algorithm 1 is provided in step S1 , an embodiment of a method for generating a test data set according to the present invention is performed.

[0089] Specifically, in step S2, a learning data set 2 is provided, which includes real image data in the form of a real image sequence 3 with a plurality of images 4, which are time-linked and currently show a plurality of photographs of a driving situation. The real image sequence 3 is obtained by means of a plurality of images 4 during the driving process. Figure 2 The image detection device (specifically, a camera arranged in a vehicle 2 which is also not shown) is recorded.

[0090] In addition to the image sequence 3 , the learning data set 2 also includes a movement range 5 assigned to the image sequence, which in the present exemplary embodiment is provided by a driving trajectory 6 corresponding to the respective image 4 of the image sequence 3 and an assigned target channel for the driving trajectory.

[0091] exist Figure 2 The learning data set 2 is schematically shown with frame elements for the image 4 and the activity range 5, and Figure 3 The same schematic diagram includes a learning data set 2, wherein nine images 4 are shown by way of example, as well as three associated driving trajectories 6 and target channels 7, which form an associated activity range 5. It should be noted that the target channel 7 defines the target range for the driving trajectory 6. The nine images 4 represent nine consecutive points in time, which are Figure 3 This is illustrated using the illustration T=1...9 on the corresponding image 4.

[0092] In step S3, artificial intelligence 8 is provided, which in the illustrated embodiment is provided by a long short-term memory generative adversarial network (LSTM-GAN). LSTM-GAN in the illustrated embodiment is present in the form of a long short-term memory generative adversarial network 9 and a long short-term memory discriminative network 10, which is Figure 2 Here again, the two networks are sketched purely by way of example by means of box elements. The architecture of the two networks is described in the article “Generative Adversarial Nets” (GANs) by Ian Goodfellow et al., published on January 10, 2014, ArXiv: 1406.2661 [statML] 661v1. In contrast, the networks are currently not designed as typical “convolutional” networks as per the article, but as “convolutional” LSTM networks, i.e. long short-term memory “convolutional” networks, as described in the article “Long Short Term Memory” (LSTM) Neural Computation 9 (8): 1735-1780, 1997 by Sepp Hochreiter and Jürgen Schmidhuber.

[0093] In step S4, the artificial intelligence 8 is adjusted using the provided learning data set 2. Figure 2 In the example described here, which is purely schematically indicated by arrows, the image sequence 3 and the activity range 5 are spliced, which is sketched by the block element symbol with the reference number 11, and transmitted to the LSTM discriminant network 10. The LSTM generative network is stimulated via the stimulus input, and its output, i.e. the product, is also transmitted to the LSTM discriminant system, which makes a correct / incorrect decision, which is indicated by the block element with the reference number 13. The stimulus input can be provided by a random signal. It is also possible that the LSTM generative network also starts at zero.

[0094] The correct / incorrect decision 13 is in particular a decision whether the product output by the generative network is consistent with or different from the data of the learning data set transmitted to the discriminative network 10. The method of the network is particularly improved until the product generated by the generative network is no longer different from the data according to the learning data set 2 as far as possible. In the context of GAN training, reference is also made to the article "Generative Adversarial Nets" (GANs) published by Ian Goodfellow et al. on January 10, 2014, ArXiv: 1406.2661 [statML] 661v1 in the introduction.

[0095] After the adjustment, in step S5, in the exemplary embodiment shown, the generator network of the adjusted artificial intelligence 8 is activated again by means of a random signal 12. Figure 4 The random signal 12 can be provided by at least one image or at least one image file which disappears or shows only noise, wherein this is to be understood as a purely exemplary embodiment.

[0096] Nine scenes are generated as responses to the examples by means of a random signal 12. For this, different dimensions are possible depending on the structure of the training. This can be seen as noise like in a TV set.

[0097] As an alternative or in addition to the random signal 12 , preferably two or more incentive images or a sequence of such incentive images or at least one incentive video, in particular with realistic content (for example showing a driving situation), can be used as incentive input.

[0098] The stimulated LSTM generative network generates at least one test data set 14, which includes a new image sequence 15 and an activity range 16 assigned to the new image sequence 15 (step S6). In the described implementation, similar to the learning data set 2, at least one test data set 14 obtained using the adjusted LSTM generative network includes an image sequence 15 (but the image sequence has a plurality of synthetically generated images 17), and the activity range 16 present for each image 17 of the sequence 15 includes an assigned driving trajectory 18 and an assigned target channel 19, which are also similar to the learning data set 2. Figure 5 In FIG. 1 , this is again explained in detail purely schematically and by way of example for a total of three generated test data sets 14 .

[0099] First, for the case where one or more stimulus videos and / or one or more stimulus images with real content (e.g., driving conditions) are fed to artificial intelligence as (multiple) input signals for stimulus, a synthetic image can be generated or obtained by a quasi-linear property for visual design or by considering or applying a quasi-linear property for visual design. In the context of quasi-linear properties for visual design, reference is also made to the article “Unsupervised Representation Learning With Deep Convolutional Generative Adversarial Networks” by Alec Redfort et al., January 7, 2016, with particular reference to Figure 7 on page 10.

[0100] In step S7, at least one test data set 14 obtained in the above-described manner is provided, and in step S8, the image sequence 15 is fed as input to the algorithm 1 to be tested, as in Figure 4 This is schematically indicated by corresponding arrows leading from the image sequence 15 to the algorithm 1 schematically shown as a box element.

[0101] In step S9 , Algorithm 1 is executed and the activity range 20 is output as output.

[0102] In step S10 , the activity range 20 output by the algorithm 1 as output is compared with the activity range 16 of the test data set 14 generated by means of the generative network. This comparison is schematically indicated in the figure by a block element provided with the reference numeral 21 .

[0103] In step S11 , the quality or functionality of the tested algorithm 1 is inferred from the comparison result.

[0104] In the case where sufficient quality or functionality of the algorithm 1 to be tested can be inferred based on the result of the comparison, the activity range 20 can be output as the correct activity range 22, which is Figure 4 The above is indicated purely schematically by frame elements (step S12 ).

[0105] The vehicle can then be controlled particularly reliably based on the correct range of motion 22 (step S13 ).

[0106] If a plurality of test data sets 14 are generated, the image sequences 15 of all test data sets 14 can be fed to the algorithm 1 to be tested, and the activity range 20 output by the algorithm as a reaction to the image sequence can be compared with the activity range 16 of the respective test data set 14, i.e. the driving trajectory 18 and the associated target corridor 19, assigned to the respective image sequence 15, and conclusions can be drawn about the quality or function of the algorithm 1. If a plurality of test data sets 14 are generated or used, the algorithm 1 can be checked particularly comprehensively.

[0107] Although the present invention has been described and illustrated in detail by preferred embodiments, the present invention is not limited to the disclosed examples and those skilled in the art can derive other variants therefrom without departing from the scope of protection of the present invention.

Claims

1. A computer-implemented method for generating at least one test data set for an algorithm (1), the algorithm receiving image data as input and emitting a range of motion associated with the image data as output, and / or the algorithm receiving sensor data as input and emitting a range of motion associated with the sensor data as output, wherein the algorithm is used to control a moving tool or a robot, and wherein the range of activity represents the permissible control parameters, and wherein, when the control parameter is executed, the travel tool or robot is triggered to move within the activity range, The method comprises the following steps: 1.1 providing a trained artificial intelligence (8), said artificial intelligence comprising at least one generative adversarial network, 1.2 stimulating the artificial intelligence (8) with at least one random signal and / or with quasi-linear properties for visual design, and 1.3 the stimulated artificial intelligence (8) generates at least one test data set, the test data set comprising image data and an activity range associated with the image data, and / or the test data set comprising sensor data and an activity range associated with the sensor data, In order to receive the trained artificial intelligence provided in step 1.1, the following steps are performed: 2.1 providing a learning data set (2), wherein the learning data set comprises real image data recorded by means of at least one image detection device and an activity range associated with the real image data, and / or the learning data set comprises real sensor data recorded by means of at least one sensor device and an activity range associated with the real sensor data, 2.2 Provide artificial intelligence, and 2.3 Adjusting the artificial intelligence (8) using the provided learning data set (2).

2. The method according to claim 1, characterized in that In step 2.1, the learning data set (2) is provided, the learning data set comprising a real image sequence recorded by means of at least one image detection device and an activity range associated with the images (4) of the sequence, and in step 1.3, at least one test data set is generated by the stimulated artificial intelligence (8), the test data set comprising a new image sequence and an activity range associated with the new image sequence, and / or in step 2.1, a learning data set (2) with a real image sequence is provided, and the activity range of the learning data set (2) comprises at least one control parameter or a value representing the control parameter for each image (4) of the image sequence, and comprises a relevant target range for at least one control parameter or a value representing the control parameter.

3. The method according to claim 1 or 2, characterized in that: At least one test data set is generated for an algorithm (1) for a system for autonomous vehicle control or for an advanced driver assistance system, and the provided learning data set (2) comprises an image sequence, which comprises images of driving situations and / or for each image (4) comprises a driving trajectory (6) as an associated range of motion.

4. The method according to claim 1 or 2, characterized in that: At least one test data set is generated for an algorithm (1) which receives an image sequence as input and emits activity ranges associated with the image sequence as output.

5. The method according to claim 1 or 2, characterized in that: In step 1.3, at least one test data set is generated, which includes an image sequence, which includes images of driving situations and / or which includes a driving trajectory as the relevant range of motion for each image.

6. The method according to claim 1 or 2, characterized in that: In step 1.1, artificial intelligence (8) is provided, which includes a long short-term memory network or is provided by a long short-term memory network.

7. The method according to claim 1 or 2, characterized in that: The artificial intelligence (8) includes at least one discriminant network (10).

8. A method for testing an algorithm (1), the method comprising the following steps: 9.1 providing an algorithm (1) to be tested, the algorithm receiving image data as input and emitting as output an activity range associated with the image data, and / or the algorithm receiving sensor data as input and emitting as output an activity range associated with the sensor data, 9.2 Executing the method according to any one of claims 1 to 7 so as to receive at least one test data set, the test data set having image data and activity ranges associated with the image data and / or the test data set having sensor data and activity ranges associated with the sensor data, 9.3 Provide at least one test data set as described above, 9.4 supplying the image data and / or the sensor data of at least one of the test data sets as input to the algorithm (1) to be tested, 9.5 Execute the algorithm (1) and emit the activity range as output, 9.6 comparing the activity range emitted as output by said algorithm with the activity range of at least one of said test data sets, 9.7 The quality of the tested algorithm (1) is inferred from the result of the comparison.

9. The method according to claim 8 is characterized in that after executing steps 9.1 to 9.7, steps 1.2 and 1.3 of the method according to any one of claims 1 to 7 for generating at least one test data set are re-executed to generate at least one additional test data set that is different from the previously generated test data set, and at least one of the additional test data sets is provided and steps 9.4 to 9.7 are repeated.

10. A method for operating a system for automatically controlling a device, wherein: The system comprises at least one image capture device and / or at least one sensor device, as well as an evaluation and control unit and components for controlling the devices, wherein an algorithm (1) is stored in the evaluation and control unit, which receives image data as input and emits an activity range associated with the image data as output, or the algorithm receives sensor data as input and emits an activity range associated with the sensor data as output, wherein the algorithm (1) is tested when executing the method according to claim 8 or 9, and if the test shows that the algorithm (1) is insufficient in function, the algorithm (1) is adapted in a predetermined manner and the device is controlled using the algorithm (1).

11. The method according to claim 10, characterized in that The system comprises at least one image detection device and detects a sequence of images (4) by means of the at least one image detection device of the system, associates activity ranges with the images (4) of the sequence, and provides the image sequence with the associated activity ranges as a learning data set (2) for training artificial intelligence (8), and / or the system comprises at least one sensor device and detects sensor data by means of the at least one sensor device of the system, associates activity ranges with the sensor data, and provides the sensor data with the associated activity ranges as a learning data set for training artificial intelligence (8).

12. A computer program product comprising instructions which, when executed on at least one computer, cause the at least one computer to perform the method according to any one of claims 1 to 7.

13. A computer-readable medium comprising instructions which, when executed on at least one computer, cause the at least one computer to perform the steps of the method according to any one of claims 1 to 7.

Images