Procedure for testing and providing a function

Abstract

A computer-implemented method for testing and providing a computer-generated function, wherein the method comprises the following process steps: - Providing a known sensor signal - Providing the computer-generated function - Modifying the known sensor signal with respect to a non-functional requirement - Applying the computer-generated function to the known sensor signal and generating a result - Applying the computer-generated function to the modified sensor signal and generating a result - Check whether the generated results are within the respective target range - Discarding the computer-generated function if one of the generated results is not within the expected range. - Output of the computer-generated function when all generated results are within the respective target range; Computer program product which includes the computer-generated function; computer system which is trained to perform the computer-generated function; and train which has a computer system which is trained to perform the computer-generated function.

Description

[0001] Regardless of the grammatical gender of a particular term, persons with male, female or other gender identities are included. Technical area

[0002] The invention relates to a computer-implemented method for testing and providing a computer-generated function, a computer program product which includes the computer-generated function, a computer system which is designed to execute the computer-generated function, and a train which has a computer system which is designed to execute the computer-generated function. State of the art

[0003] Artificial intelligence (AI) systems, such as machine learning models, are well known in the art. AI systems are software programs whose behavior or results are learned from data using computer generation rather than being explicitly programmed. The learning process is called "training" and requires a large amount of data and significant computing resources. The trained AI system solves a specific task for which it was trained, e.g., predicting system properties or generating a program product or function. Unlike conventional software, the learned behavior of AI systems depends to a large extent on the data, including the parameters used during training, i.e., the training data.

[0004] The performance of an AI system is high when the training data is a representative sample of the data to which the AI ​​system will later be applied in practice. However, in real-world scenarios, it is often observed that the performance of the AI ​​system shifts between training or test data and real-world application data. It is common for a trained AI system to perform well after training and for its performance to decline over time, especially when the real-world data differs from the original training data. The same applies to a program product or function generated by the AI ​​system. The program product or function achieves good performance when applied to the training or test data. However, performance may decrease or fluctuate when applied to real-world data. Description of the invention

[0005] The object of the invention is to provide an improved computer-implemented method for testing and outputting a computer-generated function. In particular, the object of the invention is to provide a computer-implemented method for testing and outputting a computer-generated function, which minimizes performance fluctuations of the computer-generated function between training data and real application data.

[0006] The problem is solved by a computer-implemented method according to claim 1.

[0007] Advantageous and expedient variants of the invention are specified in the dependent claims.

[0008] According to the invention, a computer-implemented method for testing and outputting a computer-generated function is provided, the method comprising the following method steps: - Providing a known sensor signal - Providing the computer-generated function - Changing the known sensor signal with regard to a non-functional requirement - Apply the computer-generated function to the known sensor signal and generate a result - Applying the computer-generated function to the modified sensor signal and generating a result - Check whether the generated results are within a respective target range - Discard the computer-generated function if one of the generated results is not within the target range - Output of the computer-generated function if all generated results are within the respective target range.

[0009] The computer-implemented method according to the invention can provide an improved computer-generated function. In particular, improved monitoring and control solutions can be implemented that build on and incorporate the computer-generated function achieved by the method according to the invention.

[0010] A computer-generated function is a function created by an AI system. For example, the AI ​​system generates the computer-generated function by evaluating training or test data provided to the AI ​​system.

[0011] Such training or test data includes, for example, known sensor signals. A known sensor signal is present, for example, as an image, where the image contains a known image, such as a sign, an animal, a person, or a vehicle.

[0012] Changing the known sensor signal means that the sensor signal is changed with regard to its non-functional properties. In the example of the image, a non-functional property is understood to be the brightness of the image, so that as a result of changing the known sensor signal in the example of the image, the brightness of the image is changed. However, it is also possible that a non-functional property is a color of the image and this color is changed. For example, the clothing of a person who can be seen in the image is changed. A non-functional property or requirement is therefore understood to be a property of the sensor signal that can be changed without the basic information of the sensor signal being changed for a human observer of the sensor signal.

[0013] In the example of the image, a change in brightness would, for example, still allow the human observer to retain the basic information of the image's content; the observer would, despite the change, identify a sign, an animal, a person, or a vehicle in the image.

[0014] As previously described, functions designed to determine the fundamental properties of a sensor signal from the sensor signal generally react sensitively to changes in the properties of the sensor signal. A change in the brightness of an image, for example, can affect the recognition accuracy of the function. The present computer-implemented method for testing and outputting a computer-generated function presents a process that determines and outputs a computer-generated function that is as insensitive as possible with regard to non-functional changes in the sensor signal. This increases the accuracy of the function, which comparatively increases the functional reliability of a system in which the function is used.For example, it is conceivable that by changing the sensor data, the original properties of the sensor signal are changed in such a way that a human observer can no longer derive them from the changed sensor signal, but a function can still recognize the original properties of the changed image.

[0015] In this case, it is proposed that the computer-implemented method determines the computer-generated function's accuracy with respect to the training and test data—the known sensor signal—as well as the accuracy of the training and test data modified with respect to non-functional properties by checking whether the results lie within a target range. For example, the accuracy is evaluated using statistical methods, e.g., the F-score or F-measure. The use of a so-called "saliency map metric" may also be provided.

[0016] In the example of the image, the target area would be the recognition of the basic information present in the image such as a sign, animal, person or vehicle.

[0017] For example, the computer-implemented method rejects the computer-generated function if one of the generated results is outside the target range. In the image example, this means, for example, that the computer-implemented method rejects the computer-generated function if, when the function is applied to the image with altered brightness, the basic information present in the image, such as a sign, animal, person, or vehicle, is not recognized, but rather one or more pieces of information present in the image are not recognized by the computer-generated function.

[0018] For example, the computer-implemented method outputs the computer-generated function if all generated results are within the respective target range. In the example image, the function is output if the computer-generated function also successfully recognizes the basic information of the image with altered brightness.

[0019] For example, the computer-implemented method can be provided using the GitLab software. This allows for variable adaptation of the computer-implemented method depending on the computer-generated function to be tested. For example, the use of the GitLab software enables the automatic installation of independently existing libraries, such as test libraries. It is also conceivable that this could allow the modification of the sensor signal to be configured and / or that it could allow the configuration of which modified sensor signals should be used.

[0020] For example, a defined test procedure can be created using a suitable environment, such as GitLab. Such a defined test procedure is a Machine Learning Operation (ML Ops), which combines the principles of DevOps (Development Operations) with specific considerations for machine learning workflows. ML Ops can be used to optimize and manage production environments of machine learning models.

[0021] For example, the defined test sequence can be relatively easily embedded into the development process of a control module or a monitoring module that uses the computer-generated function, ensuring the highest possible test quality for the computer-generated function. It is conceivable that the test sequence can be integrated into a computer-implemented development environment, for example, as a software module. It is also conceivable that the test sequence includes the AI ​​system that generates the computer-generated function.

[0022] For example, the computer-implemented method is designed to output the computer-generated function when all generated results are within the respective target range. The computer-generated function is output, for example, as a Docker image. It is also conceivable that the computer-implemented method is designed to store the output computer-generated function in a model registry, model database, or model directory, or to store the information that the computer-generated function has been output. It is also conceivable that the computer-implemented method is designed to store metadata of the test performed and / or the results obtained in a memory, e.g., in a metadata memory. This makes it possible to compare and evaluate the output computer-generated functions.For example, if a malfunction of the computer-generated function later emerges, the metadata can be used to identify additional computer-generated functions that may, for example, exhibit the same malfunction in use. It is also conceivable that this could improve the quality of the test and training data, particularly with regard to changes in the known sensor signal.

[0023] It is also proposed that the computer-implemented method comprises the following further method steps: - Changing the known sensor signal with regard to another non-functional requirement - Applying the computer-generated function to the modified sensor signal with respect to the further non-functional requirement and generating a result.

[0024] This can provide an improved computer-generated function, in particular by requiring the computer-generated function to meet various test scenarios.

[0025] It is also proposed that the computer-implemented method be designed such that a further computer-generated function is provided if the computer-generated function has been discarded and a previously mentioned execution of the method is subsequently executed again with the further computer-generated function. It is also conceivable that the computer-implemented method be designed such that a further computer-generated function is provided if the computer-generated function has been output and a previously mentioned execution of the method is subsequently executed again with the further computer-generated function. This makes it possible to automate the testing of the computer-generated function using the computer-implemented method.In particular, this makes it possible to create a test loop that can be used automatically by the AI ​​system that provides the computer-generated functions.

[0026] Furthermore, it is proposed that the computer-implemented method be designed such that the application of the computer-generated function to the sensor signal and to the modified sensor signal(s) is carried out in parallel. This makes it possible to accelerate the testing of the computer-generated function. It also makes it possible to simultaneously test multiple functional requirements.

[0027] It is further proposed that the sensor signal be present as an image. It is also conceivable that the sensor signal be present as a sensor signal from a train, a gas turbine, an electrical wiring system, or a vehicle, for example, a self-driving vehicle in a factory. For example, the sensor signal is a camera measurement, a distance measurement, a temperature measurement, a pressure measurement, and / or a voltage or power measurement.

[0028] For example, the computer-implemented method is designed to output the results, e.g., in the form of metadata, along with the output of the computer-generated function. This makes it possible, for example, to draw conclusions about the cause of the malfunction in the event of a malfunction of the computer-generated function during subsequent use, e.g., during further development. This inference can also improve future testing of the computer-generated function using the computer-implemented method, since it is possible to draw conclusions about the test parameters, in particular about changes in the known sensor signal.

[0029] Furthermore, a computer program product is proposed which includes the said computer-generated function which was output according to a method according to one of the above-mentioned variants.

[0030] A further possible embodiment of the invention is a computer system which is designed to execute the computer-generated function which was output according to a method according to one of the above-mentioned variants.

[0031] Likewise proposed is a train, a gas turbine, a control system of an electrical line system, a vehicle, or a self-driving vehicle with a computer system, a computer program product according to one of the aforementioned variants, wherein the train, the gas turbine, the control system, the vehicle, or the self-driving vehicle has a sensor that outputs a sensor signal, wherein the computer system is configured to apply the computer-generated function to the sensor signal. For example, the computer system is configured as a control module or as a monitoring module. By using the computer-generated function output by the computer-implemented method, improved control and / or improved monitoring of the train, the gas turbine, the control system, the vehicle, or the self-driving vehicle is realized. Character description

[0032] Several embodiments are explained in more detail with reference to the following drawings, giving further details and advantages:

[0033] They show: Fig. 1 a flowchart of an exemplary computer-implemented method; Fig. 2a, b an exemplary representation of a known sensor signal in the form of an image according to Fig. 3a, as well as a non-functional change of the image according to Fig. 3b, Fig. 3 a schematic representation of a flow chart of an exemplary test procedure, Fig. 4 a schematic representation of a flow chart of another variant of a test procedure and Fig. 5 a schematic representation of an object having a computer-generated function which was tested using a method according to the invention.

[0034] A method (20) according to the invention for testing and outputting a computer-generated function comprises the Fig. 1 process steps S1 to S8: S1: Providing the computer-generated function S2: Providing a known sensor signal S3: Changing the known sensor signal with regard to a non-functional requirement S4: Apply the computer-generated function to the known sensor signal and generate a result S5: Applying the computer-generated function to the modified sensor signal and generating a result S6: Check whether the generated results are within a respective target range S7: Discard the computer-generated function if one of the generated results is not within the target range S8: Output of the computer-generated function if all generated results are within the respective target range.

[0035] It is also conceivable that the known sensor signal is modified with respect to several different non-functional requirements. For example, several different modified sensor signals then exist. In a variant of the proposed computer-implemented method (20), the computer-generated function is also applied to these further modified sensor signals, and it is checked whether the results are within a respective target range. It is also conceivable that method steps S1 (providing the computer-generated function) and S2 (providing a known sensor signal) occur in parallel, or that method step S2 occurs, for example, chronologically, before method step S1.

[0036] Exemplarily shown in Fig. 1 through the further exemplary steps S9: Modifying the known sensor signal with regard to another non-functional requirement S10: Applying the computer-generated function to the further modified sensor signal and generating a result

[0037] It is also conceivable that the computer-generated method (20) is programmed as a loop and thus two or more different computer-generated functions can be tested one after the other (S11).

[0038] For example, the procedure is as in Fig. 1 is designed to execute steps S3 and S9, or S4, S5, and S10, in parallel. This advantageously accelerates the method.

[0039] It is conceivable that a known sensor signal is formed as an image. Fig. 2a shows an example of such a known sensor signal. In Fig. Figure 2a shows an image 1 depicting a silhouette of an elephant. A human observer can tell from the silhouette that an elephant is depicted. In Fig. 2b, the silhouette of the elephant is only partially shown. In Fig. 2b, an image 2 is thus modified with regard to a non-functional requirement. A human observer can still identify an elephant in the image, even if only partially represented. A computer-generated function, which is designed, for example, to recognize objects in an image, can, for example, be represented by the complete silhouette according to Fig. 2a identify an elephant in the picture, but fails, for example, in the partial representation according to Fig. 2b. In the present proposed method, for example, it should be checked whether the computer-generated function in Figure 2 according to Fig. 2b can identify an elephant or not. As a result, the proposed computer-implemented method is intended to help identify improved computer-generated functions in order to improve the control of a machine or the safety or safety system of a machine.

[0040] Fig. Figure 3 shows a variant of a test sequence that includes the proposed computer-implemented method. A test sequence 3 can be implemented, for example, using the GitLab software. Test sequence 3 comprises the following steps: reading in sensor data 4, developing a computer-generated function 5 based on the read-in sensor data 4. Alternatively, the computer-generated function can already exist and be read into the test sequence. Applying the computer-generated function to the read-in sensor data 6 and checking whether the generated results are within the target range 7. For example, test sequence 3 comprises the further process steps 8 and 9 in parallel to process steps 6 and 7. In process step 8, for example, test sequence 3 checks which type of computer-generated function is to be tested in the proposed computer-implemented method and activates or installs the necessary libraries.In method step 9, the read-in, known sensor data 4 are then modified with regard to a non-functional requirement, and the computer-generated function is applied to the modified sensor data and checked to see whether the results generated are within a target range.

[0041] Furthermore, test sequence 3 includes, for example, the further method step 10, in which it is checked whether all results are within the respective target ranges. If not, the computer-generated function is discarded. If yes, the computer-generated function is output as an image in a further method step 11 and / or entered into a model list 12 and / or the metadata used and / or generated in the test sequence is entered into a metadatabase 13.

[0042] Another variant of a test procedure 14 is in Fig. 4. The test procedure 14 according to Fig. 4 differs from test procedure 3 according to Fig. 3 in that, parallel to method steps 4 to 13, the test sequence 14 shows that the read-in sensor data 4 are not only changed with regard to a non-functional requirement according to method steps 8 and 9, but for example in parallel with this with regard to a further non-functional requirement in method steps 15 and 16, wherein in method step 15 it is checked which type of computer-generated function is to be tested in the proposed computer-implemented method and further necessary libraries are activated or installed and then in method step 16 the read-in, known sensor data 4 are changed with regard to a further non-functional requirement, as well as the application of the computer-generated function to the changed sensor data and the checking of whether the results generated thereby are within a target range.

[0043] In Fig. Figure 4 shows a train 17 according to the invention, which has a sensor 18. The sensor 18 is designed, for example, as a camera. The train 17 has a computer system 19, wherein the computer system 19 has a computer program product according to the invention with a computer-generated function according to the invention, wherein the computer system 19 is designed to apply the computer-generated function to sensor signals from the sensor 18.

[0044] Although the invention has been illustrated and described in detail by the above-mentioned embodiment, the invention is not limited to the disclosed examples and other variations can be derived therefrom by those skilled in the art without departing from the scope of the invention.

Claims

[1] A computer-implemented method for testing and outputting a computer-generated function, the method comprising the following method steps: - Providing a known sensor signal (S2) - Providing the computer-generated function (S1) - Changing the known sensor signal with regard to a non-functional requirement (S3) - Applying the computer-generated function to the known sensor signal and generating a result (S4) - Applying the computer-generated function to the modified sensor signal and generating a result (S5) - Check whether the generated results are within a respective target range (S6) - Discard the computer-generated function if one of the generated results is not within the target range (S7) - Output of the computer-generated function if all generated results are within the respective target range (S8). [2] Computer-implemented method according to the preceding claim 1, characterized by that the procedure comprises the following further procedural steps: - Changing the known sensor signal with regard to another non-functional requirement (S9) - Applying the computer-generated function to the modified sensor signal with regard to the further non-functional requirement and generating a result (S10). [3] Computer-implemented method according to one of the preceding claims, characterized by that a further computer-generated function is provided if the computer-generated function has been discarded and then the method according to claim 1 is carried out again with the further computer-generated function (S11). [4] Computer-implemented method according to one of the preceding claims, characterized by that the computer-implemented method is designed such that the application of the computer-generated function to the sensor signal and to the modified sensor signal(s) is carried out in parallel. [5] Computer-implemented method according to one of the preceding claims, characterized by that the sensor signal is present as an image (1). [6] Computer-implemented method according to one of the preceding claims, characterized by that together with the output of the computer-generated function the results (13) are also output. [7] A computer program product including the computer-generated function output according to a method according to any one of the preceding claims. [8] Computer system (19) adapted to execute the computer-generated function output according to a method according to any one of the preceding claims. [9] Train (17) comprising a computer system (19) according to claim 8 and a computer program product according to claim 7, wherein the train (17) comprises a sensor (18) which outputs a sensor signal, wherein the computer system (19) is adapted to apply the computer-generated function to the sensor signal.

Images