Method for checking and providing a function
Patent Information
- Authority / Receiving Office
- EP · EP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2024-06-24
- Publication Date
- 2026-04-01
AI Technical Summary
Machine learning systems often experience performance degradation when transitioning from training data to real-world application data due to sensitivity to non-functional changes in sensor signals, leading to fluctuations in performance.
A computer-implemented method that modifies sensor signals by changing non-functional properties such as brightness or color, applies the computer-generated function to both original and modified signals, and checks if the results fall within a target range, rejecting the function if they do not, thereby ensuring consistency and reliability.
This method enhances the accuracy and reliability of the computer-generated function by minimizing performance fluctuations and ensuring it remains effective across different data conditions, allowing for improved monitoring and control solutions.
Smart Images
Figure EP2024067606_16012025_PF_FP_ABST
Abstract
Description
[0001] Description
[0002] Procedure for testing and providing a function
[0003] Regardless of the grammatical gender of a particular term, persons with male, female or other gender identity are included.
[0004] Technical area
[0005] The invention relates to a computer-implemented method for testing and providing a computer-generated function, a computer program product which includes the computer-generated function, a computer system which is designed to carry out the computer-generated function, and a train which has a computer system which is designed to carry out the computer-generated function.
[0006] State of the art
[0007] Artificial intelligence (AI) systems, such as machine learning models, are well known in the art. AI systems are software programs whose behavior or results are learned from data using computer generation, rather than being explicitly programmed. The learning process is called "training" and requires a large amount of data and considerable computing resources. The trained AI system solves a specific task for which it was trained, e.g. predicting system properties or generating a program product or function. In contrast to conventional software, the learned behavior of AI systems depends to a large extent on the data, including the parameters used during training, i.e. the training data.
[0008] The performance of an AI system is high when the training data is a representative sample of the data to which the AI system will later be applied in practice. However, in real-world scenarios, it is often observed that the performance of the AI system shifts between training or test data and real-world application data. It is common for a trained AI system to perform well after training and for its performance to decline over time, particularly when the data in reality differs from the original training data. The same applies to a program product or function that the AI system has generated. The program product or function performs well when applied to training or test data. However, performance may decline or fluctuate when applied to real data.
[0009] Description of the invention
[0010] The object of the invention is to provide an improved computer-implemented method for testing and outputting a computer-generated function. In particular, the object of the invention is to provide a computer-implemented method for testing and outputting a computer-generated function, which minimizes performance fluctuations of the computer-generated function between training data and real application data.
[0011] The object is achieved by a computer-implemented method according to claim 1.
[0012] Advantageous and expedient variants of the invention are specified in the dependent claims.
[0013] According to the invention, a computer-implemented method for testing and outputting a computer-generated function is provided, the method comprising the following method steps:
[0014] - Providing a known sensor signal
[0015] - Providing the computer-generated function - Modifying the known sensor signal with regard to a non-functional requirement
[0016] - Apply the computer-generated function to the known sensor signal and generate a result
[0017] - Applying the computer-generated function to the modified sensor signal and generating a result
[0018] - Check whether the generated results are within the respective target range
[0019] - Discard the computer-generated function if one of the generated results is not within the target range
[0020] - Output of the computer-generated function if all generated results are within the respective target range.
[0021] The computer-implemented method according to the invention can provide an improved computer-generated function. In particular, improved monitoring and control solutions can be implemented that build on and incorporate the computer-generated function achieved by the method according to the invention.
[0022] A computer-generated function is a function generated by an AI system. For example, the AI system generates the computer-generated function by evaluating training or test data provided to the AI system.
[0023] Such training or test data includes, for example, known sensor signals. A known sensor signal is present, for example, as an image, where what is visible in the image is known, e.g., a sign, an animal, a person, or a vehicle.
[0024] Changing the known sensor signal means that the sensor signal is changed with regard to its non-functional properties. In the example of the image, a non-functional property is understood to be the brightness of the image, so that as a result of changing the known sensor signal in the example of the image, for example, the brightness of the image is changed. However, it is also possible that a non-functional property is a color of the image and that this color is changed. For example, the clothing of a person who can be seen in the image is changed. A non-functional property or requirement is therefore understood to be a property of the sensor signal that can be changed without the basic information of the sensor signal being changed for a human observer of the sensor signal.
[0025] In the example of the image, a change in brightness would, for example, still allow the human observer to retain the basic information of the image's content; the observer would, despite the change, identify a sign, an animal, a person or a vehicle in the image.
[0026] As described above, functions which are intended to determine the basic properties of a sensor signal from the sensor signal generally react sensitively to changes in the properties of the sensor signal. A change in the brightness of an image, for example, can have an effect on the recognition accuracy of the function. The present computer-implemented method for testing and outputting a computer-generated function presents a sequence which determines and outputs a computer-generated function which is as insensitive as possible with regard to non-functional changes in the sensor signal. This increases the accuracy of the function, which in turn comparatively increases the functional reliability of a system in which the function is used.For example, it is conceivable that by changing the sensor data the original properties of the sensor signal are changed in such a way that a human observer can no longer derive them from the changed sensor signal, but a function can still recognize the original properties of the changed image. In this case it is proposed that the computer-implemented method determines the computer-generated function with regard to its accuracy in relation to the training and test data - the known sensor signal - and also determines the accuracy of the training and test data changed with regard to non-functional properties by checking whether the results lie within a target range. For example, the accuracy is assessed using statistical methods, e.g. by using the F-score or F-measure.The use of a so-called “saliency map metric” may also be provided for.
[0027] In the example of the image, the target area would be the recognition of the basic information present in the image such as a sign, animal, person or vehicle.
[0028] For example, the computer-implemented method rejects the computer-generated function if one of the generated results is not within the target range. In the example of the image, this means, for example, that the computer-implemented method rejects the computer-generated function if, when the function is applied to the image with altered brightness, the basic information present in the image, such as a sign, animal, person, or vehicle, is not recognized, but instead, for example, one or more pieces of information present in the image are not recognized by the computer-generated function.
[0029] For example, the computer-implemented method outputs the computer-generated function if all generated results are within the respective target range. In the example image, the function is output if the computer-generated function also successfully recognizes the basic information of the image with altered brightness.
[0030] For example, the computer-implemented method can be provided using the GitLab software. This allows for variable adaptation of the computer-implemented method depending on the computer-generated function to be tested. For example, the use of the GitLab software enables the automatic installation of independently existing libraries, such as test libraries. It is also conceivable that this could allow the modification of the sensor signal to be configured and / or that it could allow the configuration of which modified sensor signals should be used.
[0031] For example, a defined test procedure can be created using a suitable environment, such as GitLab. Such a defined test procedure is, for example, a machine learning operation (ML Ops), which combines the principles of DevOps (Development Operations) with specific considerations for machine learning workflows. ML Ops can be used to optimize and manage production environments of machine learning models.
[0032] For example, the defined test sequence can be relatively easily embedded into a development process of a control module or a monitoring module that uses the computer-generated function, thus ensuring the highest possible test quality for the computer-generated function. It is conceivable that the test sequence can be integrated into a computer-implemented development environment, for example, as a software module. It is also conceivable that the test sequence includes the AI system that generates the computer-generated function.
[0033] For example, the computer-implemented method is designed to output the computer-generated function when all of the results generated are within the respective target range. The computer-generated function is output, for example, as a Docker image. It is also conceivable for the computer-implemented method to be designed to store the output computer-generated function in a model registry, model database or model directory or to store the information that the computer-generated function has been output. It is also conceivable for the computer-implemented method to store metadata of the test performed and / or the results achieved in a memory, e.g. in a metadata memory. This makes it possible to compare and evaluate the output computer-generated functions.For example, if a malfunction of the computer-generated function later emerges, additional computer-generated functions can be identified based on the metadata, which may, for example, exhibit the same malfunction in use. It is also conceivable that this could improve the quality of the test and training data, particularly with regard to a change in the known sensor signal.
[0034] It is also proposed that the computer-implemented method comprises the following further method steps:
[0035] - Changing the known sensor signal with regard to another non-functional requirement
[0036] - Applying the computer-generated function to the modified sensor signal with regard to the further non-functional requirement and generating a result.
[0037] This can provide an improved computer-generated function, in particular by requiring the computer-generated function to meet various test scenarios.
[0038] It is also proposed that the computer-implemented method be designed such that a further computer-generated function is provided if the computer-generated function has been discarded and a previously mentioned execution of the method is subsequently executed again with the further computer-generated function. It is also conceivable that the computer-implemented method be designed such that a further computer-generated function is provided if the computer-generated function has been output and a previously mentioned execution of the method is subsequently executed again with the further computer-generated function. This makes it possible to automate the testing of the computer-generated function using the computer-implemented method.In particular, this makes it possible to implement a test loop which can be used automatically by the Kl system, which provides the computer-generated functions.
[0039] Furthermore, it is proposed that the computer-implemented method be designed such that the application of the computer-generated function to the sensor signal and to the modified sensor signal(s) is carried out in parallel. This makes it possible to accelerate the testing of the computer-generated function. It also makes it possible to simultaneously test multiple functional requirements.
[0040] It is further proposed that the sensor signal be present as an image. It is also conceivable that the sensor signal be present as a sensor signal from a train, a gas turbine, an electrical wiring system, or a vehicle, for example, a self-driving vehicle in a factory. For example, the sensor signal is a camera measurement, a distance measurement, a temperature measurement, a pressure measurement, and / or a voltage or power measurement.
[0041] For example, the computer-implemented method is designed to output the results, e.g. in the form of metadata, along with the output of the computer-generated function. This makes it possible, for example, to draw conclusions about the cause of the malfunction if the computer-generated function malfunctions during further use, e.g. in further development. This conclusion can also improve future testing of the computer-generated function with the computer-implemented method, since conclusions can be drawn about the test parameters, in particular about changes in the known sensor signal.
[0042] Furthermore, a computer program product is proposed which includes the said computer-generated function which was output according to a method according to one of the previously mentioned variants.
[0043] A further possible embodiment of the invention is a computer system which is designed to execute the computer-generated function which was output according to a method according to one of the previously mentioned variants.
[0044] Likewise proposed is a train, a gas turbine, a control system of an electrical line system, a vehicle or a self-driving vehicle with a computer system, a computer program product according to one of the aforementioned variants, wherein the train, the gas turbine, the control system, the vehicle or the self-driving vehicle has a sensor which outputs a sensor signal, wherein the computer system is designed to apply the computer-generated function to the sensor signal. For example, the computer system is designed as a control module or as a monitoring module. By using the computer-generated function output by the computer-implemented method, improved control and / or improved monitoring of the train, the gas turbine, the control system, the vehicle or the self-driving vehicle is realized. Description of the Figures
[0045] Several examples of implementation are explained in more detail in the following drawings, giving further details and advantages:
[0046] It shows :
[0047] Figure 1 is a flow chart of an exemplary computer-implemented method;
[0048] Figures 2a, b show an exemplary representation of a known sensor signal in the form of an image according to Figure 3a, as well as a non-functional change of the image according to Figure 3b,
[0049] Figure 3 is a schematic representation of a flow chart of an exemplary test procedure,
[0050] Figure 4 is a schematic representation of a flow chart of another variant of a test procedure and
[0051] Figure 5 is a schematic representation of an article having a computer-generated function which was tested using a method according to the invention.
[0052] A method (20) according to the invention for testing and outputting a computer-generated function comprises the method steps S 1 to S 8 shown in Figure 1:
[0053] Sl : Providing the computer-generated function
[0054] S2 : Providing a known sensor signal
[0055] S3 : Modifying the known sensor signal with respect to a non-functional requirement S4 : Applying the computer-generated function to the known sensor signal and generating a result
[0056] S5 : Applying the computer-generated function to the modified sensor signal and generating a result
[0057] S 6 : Check whether the generated results are within a respective target range
[0058] S7 : Discard the computer-generated function if one of the generated results is not within the target range
[0059] S8 : Output of the computer-generated function if all generated results are in the respective target range.
[0060] It is also conceivable that the known sensor signal is modified with regard to several different non-functional requirements. For example, several different modified sensor signals then exist. In a variant of the proposed computer-implemented method (20), the computer-generated function is also applied to these further modified sensor signals and a check is carried out to determine whether the results are within a respective target range. It is also conceivable that method steps S1 (providing the computer-generated function) and S2 (providing a known sensor signal) take place in parallel or that method step S2 takes place, for example in time, before method step S1.
[0061] Shown as an example in Figure 1 by the further exemplary steps
[0062] S9 : Modifying the known sensor signal with regard to another non-functional requirement
[0063] S10: Applying the computer-generated function to the further modified sensor signal and generating a result. It is also conceivable that the computer-generated method (20) is programmed as a loop and thus two or more different computer-generated functions can be tested one after the other (S il).
[0064] For example, the method as shown in Figure 1 is designed to carry out steps S3 and S9, or S4, S5 and S10, in parallel. This advantageously accelerates the method.
[0065] It is conceivable that a known sensor signal is in the form of an image. Figure 2a shows an example of such a known sensor signal. Figure 2a shows an image 1 which depicts the silhouette of an elephant. A human observer can tell from the silhouette that an elephant is being depicted. In Figure 2b, the silhouette of the elephant is only partially depicted. In Figure 2b, an image 2 is therefore modified with regard to a non-functional requirement. A human observer can still identify an elephant in the image, even if it is only partially depicted. A computer-generated function which is designed, for example, to recognize objects in an image can, for example, identify an elephant in the image using the complete silhouette according to Figure 2a, but fails, for example, with the partial depiction according to Figure 2b.In the present proposed method, for example, it is intended to test whether the computer-generated function in image 2 according to Figure 2b can identify an elephant or not. As a result, the proposed computer-implemented method is intended to help identify improved computer-generated functions in order to improve the control of a machine or the safety or safety system of a machine.
[0066] Figure 3 shows a variant of a test sequence which comprises the proposed computer-implemented method. A test sequence 3 can be implemented, for example, using the GitLab software. The test sequence 3 has the following steps: reading in sensor data 4, developing a computer-generated function 5 on the basis of the read in sensor data 4. Alternatively, the computer-generated function can already exist and be read into the test sequence, applying the computer-generated function to the read in sensor data 6 and checking whether the generated results are within the target range 7. For example, the test sequence 3 has the further process steps 8 and 9 in parallel to process steps 6 and 7. In process step 8 in test sequence 3, for example, it is checked which type of computer-generated function is to be tested in the proposed computer-implemented method and the necessary libraries are activated or installed.In process step 9, the read-in, known sensor data 4 are then changed with regard to a non-functional requirement, and the computer-generated function is applied to the changed sensor data and checked to see whether the results generated are within a target range.
[0067] Furthermore, the test sequence 3 includes, for example, the further method step 10, in which it is checked whether all results are within the respective target ranges. If not, the computer-generated function is discarded. If yes, the computer-generated function is output as an image in a further method step 11 and / or entered into a model list 12 and / or the metadata used and / or generated in the test sequence is entered into a metadatabase 13.
[0068] A further variant of a test procedure 14 is shown in Figure 4. The test procedure 14 according to Figure 4 differs from the test procedure 3 according to Figure 3 in that, in parallel to the method steps 4 to 13 in the test procedure 14, it is shown that the read-in sensor data 4 are not only modified with regard to a non-functional requirement according to method steps 8 and 9, but for example in parallel with this with regard to a further non-functional requirement in method steps 15 and 16, wherein in method step 15 it is checked which type of computer-generated function is to be tested in the proposed computer-implemented method and further necessary libraries are activated or installed and in method step 16 then a modification of the read-in, known sensor data 4 with regard to a further non-functional requirement,and applying the computer-generated function to the modified sensor data and checking whether the results generated are within a target range.
[0069] Figure 4 shows a train 17 according to the invention, which has a sensor 18. The sensor 18 is designed, for example, as a camera. The train 17 has a computer system 19, wherein the computer system 19 has a computer program product according to the invention with a computer-generated function according to the invention, wherein the computer system 19 is designed to apply the computer-generated function to sensor signals from the sensor 18.
[0070] Although the invention has been illustrated and described in detail by the above-mentioned embodiment, the invention is not limited to the disclosed examples and other variations can be derived therefrom by a person skilled in the art without departing from the scope of the invention.
Claims
Patent claims 1. A computer-implemented method for testing and outputting a computer-generated function, the method comprising the following method steps: - Providing a known sensor signal (S2) - Providing the computer-generated function (Sl) - Changing the known sensor signal with regard to a non-functional requirement (S3) - Applying the computer-generated function to the known sensor signal and generating a result (S4) - Applying the computer-generated function to the modified sensor signal and generating a result (S5) - Check whether the generated results are within a respective target range (S6) - Discard the computer-generated function if one of the generated results is not within the target range (S7) - Output of the computer-generated function if all generated results are in the respective target range (S8 ).
2. Computer-implemented method according to the preceding claim 1, characterized in that the method comprises the following further method steps: - Changing the known sensor signal with regard to another non-functional requirement (S9) - Applying the computer-generated function to the modified sensor signal with respect to the further non-functional requirement and generating a result (S10).
3. Computer-implemented method according to one of the preceding claims, characterized in that a further computer-generated function is provided if the computer-generated function has been discarded and then the method according to claim 1 with the further computer-generated function is executed again (Sil).
4. Computer-implemented method according to one of the preceding claims, characterized in that the computer-implemented method is designed such that the application of the computer-generated function to the sensor signal and to the modified sensor signal(s) is carried out in parallel.
5. Computer-implemented method according to one of the preceding claims, characterized in that the sensor signal is present as an image (1).
6. Computer-implemented method according to one of the preceding claims, characterized in that the results (13) are also output together with the output of the computer-generated function.
7. A computer program product including the computer-generated function output according to a method according to any one of the preceding claims.
8. Computer system (19) configured to execute the computer-generated function output according to a method according to any one of the preceding claims.
9. Train (17) comprising a computer system (19) according to claim 8 and a computer program product according to claim 7, wherein the train (17) comprises a sensor (18) which outputs a sensor signal, wherein the computer system (19) is adapted to apply the computer-generated function to the sensor signal.