Methods, devices, products and computer programs for operating a technical system

Abstract

The invention relates to a method (30) for operating, for example, a robot. The method (30) comprises the steps of detecting a sequence of records (23) of a space around the robot, in which a potential object is present. Determining, by means of a first machine learning system (21), each first variable (24) characterizing the potential object from each record (23). Determining, by means of a second machine learning system (22), at least one second variable (26) characterizing the potential object from the plurality of first variables characterizing the potential object. The robot is controlled in accordance with the second variable (26) characterizing the potential object. The invention also relates to a computer program and a device for executing the method and to a machine-readable memory element on which the computer program is stored.

Description

Technical Field

[0001] The invention relates to a method, an apparatus and a product for operating a technical system by means of two machine learning systems. Furthermore, the invention relates to a computer program and a machine-readable memory element. Background Art

[0002] DE 10 2011 087 791 A1 discloses a method in a maneuvering assistance system in a vehicle and a corresponding maneuvering assistance system. The method comprises the following steps: identifying a situation according to which a vehicle driver wants to perform a manual action with respect to an object, and providing support when orienting the vehicle with respect to the object.

[0003] DE 199 34 670 B4 discloses an object detection system which is composed of a combination of three object detectors. Summary of the invention

[0004] Advantages of the present invention

[0005] During operation of a deep neural network, for example for image classification, lighting conditions or the perspective of a traffic scene may change drastically. This may lead to the deep neural network incorrectly classifying objects during such changes in environmental properties. In addition, insufficiently large amounts of training data or incomplete training data, which do not contain all possible different perspective displays or exposures of the objects, may have been used to train the deep neural network. As a result, isolated objects may not be correctly classified or detected. In contrast, the method of independent claim 1 has the advantage of using a further deep neural network, which re-performs the classification according to the determined deep neural network classification. The further deep neural network uses temporally contained information for this purpose, which may be distributed along the temporally previously determined classification. This temporally contained information is introduced into the reclassification, whereby the accuracy of the classification can be improved.

[0006] Disclosure of the Invention

[0007] In a first aspect, a method for operating a technical system, in particular a robot, an at least partially autonomous vehicle, a tool or a factory machine, is proposed according to independent claim 1 .

[0008] Advantageously, by using a plurality of determined first variables characterizing a potential object, additional information is provided to the second machine learning system, which additional information allows a more accurate and more reliable characterization, for example a classification of the potential object. In addition, incorrect classifications of the first machine learning system can thus be identified and, if necessary, can be disregarded for the operation of the technical system.

[0009] In the following, a potential object is understood to be an object that has been detected, but it is not known whether the object is a real object located in the space around the technical system or a falsely detected object, such as a reflection or a ghost object (English: ghost target). In addition, it is not known for a potential object to which object class or object category the potential object can be classified.

[0010] In addition, it is proposed that the first machine learning system includes a convolutional neural network (English: Convolutional Neural Network) and the second machine learning system includes at least one recurrent neural network. The advantage here is that the object is characterized by the recurrent neural network using time information that can be present along the temporally continuously detected recording sequence. Because the recurrent neural network reuses the temporally previously determined intermediate variables, the information that is temporally present in the temporally previously determined first variables characterizing the potential object is used. Because the recurrent neural network reuses its intermediate results, a higher object characterization accuracy can be achieved.

[0011] Alternatively, the second machine learning system may comprise a further deep neural network, in particular a full mesh or convolutional neural network. The advantage of this alternative is that here, with the aid of a further deep neural network, the temporal integration of information from the sequence is used to characterize the potential object. The temporal integration of information along the previously determined first variables characterizing the potential object can be provided to the second machine learning system by arranging these variables in a chronological order.

[0012] Furthermore, it is proposed to determine a plurality of second variables characterizing the potential object and to determine a confidence level as a function of the plurality of second variables characterizing the potential object.

[0013] In the following, a confidence level is understood to be a measure that indicates how reliable a variable characterizing a potential object is, for example, for a subsequent classification of the potential object. This has the advantage that, based on the confidence level, it can be decided whether a further first variable characterizing the potential object must also be determined in order to perform a reliable classification of the potential object. This makes it possible, for example, to avoid unnecessary and computationally expensive further processing of the records.

[0014] Furthermore, it is proposed that a first variable characterizing a potential object is determined from a correspondingly recorded segment. The segment contains the potential object and the segment can be scaled to a predeterminable size. Scaling can be understood to mean that the segment is interpolated into a larger segment, but also reduced into a smaller segment.

[0015] The advantage here is that by selectively using each recorded segment, less input data is provided to the first machine learning system and thus a reduction in computationally expensive image processing is achieved. Scaling of the segments is advantageous because the segments are adapted to a uniform size so that the input to the first machine learning system does not have to be adapted separately to potential objects of different sizes.

[0016] It may be provided that the segments are generated based on bounding boxes that include potential objects, since these bounding boxes include the objects as accurately as possible and can be determined with the aid of further deep neural networks.

[0017] Furthermore, it can be provided that the segment is generated as a function of at least one first variable characterizing the potential object. Furthermore, it can be provided that the bounding box is determined as a function of the variable characterizing the potential object. This has the advantage that the segment can be specifically adapted to the size of the object.

[0018] In a second aspect, a method for teaching a machine learning system is proposed. The machine learning system is taught in such a way that a first and a second variable characterizing a potential object respectively characterize the potential object, in particular classify the potential object. Alternatively, the machine learning system can be taught in such a way that a control variable can be determined based on the second variable characterizing the potential object. In a further development of the second aspect, the machine learning system is taught independently of each other. Additionally or alternatively, after the machine learning system has been taught, the machine learning system can be taught dependently of each other.

[0019] Independent teaching can be understood as teaching the machine learning systems separately from each other so that the parameters of the respective machine learning systems are adapted. In this case, the parameters of the other machine learning system remain unchanged. Dependent teaching can be understood as the parameters of the machine learning systems being adapted dependently on each other or dependent on a cost function that is dependent on the parameters of the machine learning systems.

[0020] In another aspect, a computer program is proposed, which is designed to perform one of the above methods, i.e., comprises instructions which, when the computer program is run on a computer, cause the computer to perform one of the methods and all of the steps thereof, and a machine-readable memory module on which the computer program is stored. Furthermore, a device, in particular a computer, is proposed, which is designed to perform one of the methods.

[0021] Furthermore, a product is proposed, which is produced by means of carrying out the method according to the second aspect of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] An exemplary embodiment is shown in the drawings and explained in detail in the following description.

[0023] Figure 1 A schematic diagram of an at least partially autonomous vehicle is shown;

[0024] Figure 2 A schematic diagram of a first and a second machine learning system is shown;

[0025] Figure 3 A schematic diagram of an embodiment of a method for operating a technical system is shown. DETAILED DESCRIPTION

[0026] Figure 1 A schematic diagram of an at least partially autonomous robot is shown, which in this embodiment is provided by at least one partially autonomous vehicle 10. In another embodiment, the at least partially autonomous robot can be a maintenance robot, an assembly robot or a stationary production robot. The at least partially autonomous vehicle 10 can include a detection unit 11. The detection unit 11 can be, for example, a camera for detecting the environment of the vehicle 10. The detection unit 11 can be connected to a classification system 12. The classification system 12 determines output variables based on the input variables provided, for example with the aid of the detection unit 11 and with the aid of at least two machine learning systems, which are Figure 2 The output variable may characterize a potential object detected in the environment of the vehicle 10. For example, the classification system 12 may classify the potential object as a pedestrian or a cyclist. In addition, the classification system 12 may also classify the potential object as "no object" in order to reject an erroneously detected object.

[0027] The output variables may be forwarded to the actuator control unit 13. The actuator control unit 13 may manipulate the actuators according to the output variables of the machine learning system 12, for example to avoid a collision with an object. The actuator may be, for example, an engine of the vehicle 10. Alternatively, the actuator control unit 13 may determine control variables according to the output variables of the classification system 12, which are then used to control the actuators.

[0028] In another embodiment, the actuator control unit 13 includes a release system. The release system decides whether an object (e.g., a detected robot or a detected person) is approaching the area based on the output variable of the machine learning system 12. Preferably, the actuator can be controlled based on the decision of the release system. In addition, the vehicle 10 includes a computing unit 14 and a machine-readable memory element 15.

[0029] A computer program may be stored on the memory element 15, which computer program includes instructions which, when executed on the computing unit 14, cause the computing unit 14 to perform the method according to the invention. It is also conceivable that a download product or an artificially generated signal, which may respectively include a computer program, after being received at a receiver of the vehicle 10, causes the computing unit 14 to perform the method according to the invention.

[0030] In another embodiment, the classification system 12 is used in a measurement system not shown in the figure. Figure 1 The robot of is different in that the measuring system does not comprise an actuator control unit 13. The measuring system may store or display (eg by means of a visual or acoustic display) the output variables of the classification system 12 instead of forwarding them to the actuator control unit 13.

[0031] Furthermore, the classification system 12 may be used for tools or factory machines. Potential objects may be workpieces, the materials of which are classified with the aid of the classification system 12. Here, the actuator may be, for example, a motor operating a grinding head.

[0032] It is also conceivable that in a further development of the measuring system, the detection unit 11 detects an image of the human or animal body or a part thereof. This can be done, for example, by means of optical signals, or by means of ultrasound signals, or by means of MRT / CT methods. In this further development, the measuring system can include a machine learning system that is taught to output a classification depending on the input variables, for example, which clinical image may be present based on the input variables.

[0033] exist Figure 2 , two machine learning systems 21, 22 of a classification system 12 and their collaboration are shown by way of example. The first machine learning system 21 receives a sequence of records 23. The record 23 can be, for example, a camera image of the environment of the vehicle 10, in which, for example, a potential object has been detected. The first machine learning system 21 then determines an output variable 24 from each record 23. The output variable 24 characterizes the potential object, in particular classifies the object into a plurality of predefined categories. If the sequence exemplarily contains 50 records 23, the first machine learning system 21 continuously determines 50 output variables 24. Preferably, the first machine learning system 21 comprises a convolutional deep neural network (English: Convolutional Neural Network) and is taught so that the determined output variable 24 characterizes the potential object.

[0034] The second machine learning system 22 then determines at least one second output variable 26 based on the input variables 25 of the second machine learning system 22. The second output variable 26 also characterizes the potential object. The input variables 25 of the second machine learning system 22 include a plurality of output variables 24 of the first machine learning system 21. For example, the input variables 25 from 50 consecutively determined output variables 24 may include 30 output variables 24. The output variables 25, 26 may each be a feature vector, the number of entries of which corresponds to the number of different possible categories to which the potential object may be assigned.

[0035] exist Figure 2 In an embodiment, the first machine learning system 21 includes a convolutional neural network and the second machine learning system 22 includes a full-mesh deep neural network. In this embodiment, the full-mesh neural network determines at least one second output variable 26 based on all output variables 24 provided by the first machine learning system 21. For example, if the sequence includes 50 records 23 and the first machine learning system 21 determines 50 output variables 21, the second machine learning system 22 obtains these 50 output variables 24 as input variables 25 and determines at least one output variable 26 based on these input variables 25.

[0036] exist Figure 2 In an alternative embodiment, the second machine learning system 22 includes a recurrent neural network (English: Recurrent Neural Network). In this embodiment, the recurrent neural network determines at least one second output variable 26 from each determined output variable 24 of the first machine learning system 21. If the sequence includes 50 records 23 and the first machine learning system 21 determines 50 output variables 24, the second machine learning system 22 continuously determines a second output variable 26 from each determined output variable 24. It is therefore conceivable that once the first machine learning system 21 determines an output variable 24, the output variable 24 is then used by the second machine learning system 22 to determine the second output variable 26. Optionally, the 50 determined second output variables 26 can be stored so that confidence levels can be determined based on these output variables.

[0037] The recurrent neural network is preferably a Recursive Weighted Average (RWA) model, alternatively a Long Short-Term Memory (LSTM) or a Gated Recurrent Unit (GRU) model.

[0038] Figure 3 A schematic diagram of a method 30 for operating a technical system, such as a vehicle 10, is shown. It should be noted that the method 30 can also be used to operate a measuring system. For this purpose, it must be taken into account that the measuring system processes the output variables of the classification system 12 differently, and this must be taken into account accordingly in the method 30 described below as an example.

[0039] The method 30 starts with step 31. In step 31, the classification system 12 is taught. The teaching of the classification system 12 can be performed in such a way that the parameters of the machine learning systems 21, 22 of the classification system 12 are determined based on the labeled training data and by means of a cost function and a gradient descent method, so that the classification system 12 classifies the potential object based on the detected record. In an alternative embodiment of step 31, the first machine learning system 21 can be taught independently of the second machine learning system 22. In this embodiment, the parameters of the machine learning systems are determined while being taught separately from each other. For example, the parameters of the first machine learning system 21 are first determined and then the parameters of the second machine learning system are determined, each based on a different cost function. Alternatively or additionally, the two machine learning systems 21, 22 can also be taught together, i.e. the parameters of the two machine learning systems 21, 22 are determined dependently on each other, so that the machine learning systems 21, 22 are coordinated with each other, for example by means of a total cost function. The total cost function can be defined by means of an additive logical connection of the corresponding cost functions of the machine learning systems 21, 22.

[0040] After completion of step 31, step 32 follows. In step 32, the sequence of records 23 is provided to the classification system 12. The sequence of records 21 may be detected by means of the detection unit 11, for example.

[0041] In a subsequent step 33, the classification system 12 determines the classification of the potential object, alternatively determines an output variable characterizing the potential object. To this end, the first machine learning system 21 first determines an output variable 24 characterizing the potential object based on each record 23. Next, at least one second output variable 26 is determined based on the multiple output variables 24 of the first machine learning system 21 with the help of the second machine learning system 22. If multiple second output variables 26 are determined, a confidence level can be determined based on them. In the event that the confidence level exceeds a predeterminable threshold, the potential object can be assigned to a category based on the determined second output variable 26. Alternatively, the potential object can also be assigned to a category if the potential object is assigned to the same category multiple times in a row based on the second output variable 26.

[0042] Step 33 is followed by step 34. In this case, the actuator of the technical system is controlled as a function of the second output variable 26 of the second machine learning system 22. Alternatively, in step 34 a control variable can be determined as a function of the second output variable 26, which control variable is then used to control, for example, an actuator of the vehicle 10. In this case, the actuator can be controlled so that the vehicle 10 does not collide with an object in the environment.

[0043] This ends the method 30. In an additional or alternative embodiment, step 32 or 33 of the method 30 can be executed again, and optionally these steps can be repeated several times in succession until a predefinable termination criterion is met.

Claims

1. A method (30) for operating a technical system, the method comprising the following steps: Providing a record of multiple temporally consecutive detections of the space surrounding a technical system (23) in which potential objects are present; Determining, by means of a first machine learning system (21), a plurality of first variables (24) characterizing the potential object, respectively, from one of the records (23); Determining at least one second variable (26) characterizing the potential object based on at least two first variables characterizing the potential object by means of a second machine learning system (22); Determining confidences of at least two first variables characterizing the potential object based on at least one second variable characterizing the potential object, and determining based on the confidences whether it is necessary to determine an additional first variable characterizing the potential object with the aid of the first machine learning system; A control variable is determined based on a second variable (26) characterizing the potential object in order to actuate an actuator of the technical system.

2. The method according to claim 1, wherein the first machine learning system (21) comprises at least one convolutional neural network, and the second machine learning system comprises at least one recurrent neural network, in, The second machine learning system (22) continuously determines at least one second variable characterizing the potential object based on at least one first variable (24) characterizing the potential object among a plurality of first variables (24) characterizing the potential object.

3. The method according to claim 2, wherein: The first machine learning system (21) includes at least the convolutional neural network, and the second machine learning system (22) includes at least one other deep neural network, The second machine learning system (22) determines a second characterizing variable (26) at least once based on a plurality of first variables (24) characterizing the potential object.

4. The method according to claim 3, wherein: Assigning a predeterminable time to each of the temporally consecutively detected records (23) and assigning to each first variable (24) characterizing the potential object the same time as the record from which the first variable (24) characterizing the potential object was determined, Therein, first variables (24) characterizing the potential objects are arranged according to their respectively assigned moments.

5. The method according to any one of claims 1 to 4, wherein: determining a first variable (24) characterizing the potential object in each case based on the fragments of the corresponding record (23), wherein said segment contains said potential object, The segments are each scaled to form a uniform segment having at least one predeterminable size, and the movement of the potential object is tracked and the segments are determined as a function of the movement of the object.

6. The method according to any one of claims 1 to 4, wherein: The machine learning system (21, 22) is taught so that the first and second variables characterizing the potential object respectively classify the potential object.

7. The method according to claim 6, wherein: During teaching, the machine learning systems (21, 22) are taught independently of each other, Wherein after teaching the machine learning systems (21, 22), the machine learning systems (21, 22) are taught dependently on each other.

8. The method according to claim 3, wherein: The further deep neural network is a fully meshed or convolutional neural network.

9. The method according to claim 1, wherein: The technical system is a robot, an at least partially autonomous vehicle, a tool or a factory machine.

10. A computer program product comprising instructions designed to, when executed on a computer, cause the computer to perform the method of any one of claims 1 to 9.

11. A machine-readable memory element having stored thereon a computer program product according to claim 10.

12. A device designed to carry out the method according to one of claims 1 to 9.

13. A product obtainable by a process according to claim 6 or 7.

Images