Method for computing quality metrics for evaluating object detection algorithms

Abstract

The present invention describes a method for calculating a quality measure of a computer-implemented object detection algorithm, which method can be used in particular to enable the object detection algorithm for use in a partially automated, highly automated or fully automated robot, wherein the method comprises the following steps: · causing the determined object detection to correspond to a label, wherein the object detection and / or the label corresponds to a bounding box; · determining (102) a deviation, in particular a distance, of the label to the object detection corresponding thereto; · calculating (103) a quality measure of the object detection algorithm based on the determined deviation, wherein the quality measure represents the probability that the deviation of the object detection from the label corresponding thereto is above or below a predefined threshold value with the said probability.

Description

Technical Field

[0001] The present invention relates to a method for calculating a quality metric for evaluating a computer-implemented object detection algorithm. A method for carrying out a method of a computer program for carrying out the method, a device configured to carry out the method, a computer program for carrying out the method, and a machine-readable storage medium on which the computer program is stored. Background Art

[0002] Computer-implemented object detection algorithms are often used as part of environmental recognition for partially, highly, or fully automated robots, particularly for autonomously operated vehicles. The algorithms used for this purpose are not perfect and can lead to more or less severe false detections. For example, an object detection algorithm in an autonomously operated vehicle may detect an object at a different location than it actually is, thereby generating an incorrect model of the environment. Therefore, to fully operationalize such systems, it is absolutely necessary to evaluate the quality of the object detection algorithm and classify it as sufficiently good.

[0003] Typically, average metrics, such as the intersection over union (IoU), are used to evaluate object detection algorithms. However, from a safety perspective, average metrics are critical because they evaluate the safest and most dangerous behaviors on average. Consequently, these average metrics are no longer sufficient for the commissioning of safety-critical products. Summary of the Invention

[0004] The present invention describes a computer-implemented method for calculating a quality measure of a computer-implemented object detection algorithm, which method can be used in particular for commissioning the object detection algorithm for a partially automated, highly automated, or fully automated robot, wherein the method comprises the following steps:

[0005] Assigning the determined object detections to the annotations, wherein the object detections and / or the annotations correspond to bounding boxes;

[0006] Determine the deviation, in particular the distance, from the annotation to the corresponding object detection;

[0007] Calculating a quality measure of the object detection algorithm based on the determined deviations, wherein the quality measure represents the probability that the deviation of an object detection from its corresponding annotation is above or below a predetermined defined threshold value with said probability.

[0008] In an optional step, the object detection algorithm may be enabled for use, in particular for use in an at least partially automatically operating robot, when the object detection algorithm is above or below a predefined quality metric threshold.

[0009] A robot may be understood to mean, for example, an industrial robot, an automated factory machine, or an automatically operated vehicle. In particular, it may be understood to mean a partially automated, highly automated, or fully automated vehicle that can perform driving maneuvers, in particular adaptation of longitudinal and / or lateral movements, at least temporarily without human intervention.

[0010] The method particularly uses data sets with annotations, wherein annotations are particularly understood to be bounding boxes. A bounding box can particularly be understood to be a rectangle that encloses the object to be detected. For example, in the case of video-based person detection, a bounding box can mark the area of ​​the image in which the person is located. Alternatively, if the object is to be detected in three-dimensional space, the bounding box can also be square. This can be useful, for example, if, as in the example mentioned above, the position coordinates of the person in the real world are to be detected directly. For each data point in the data set, there may be multiple objects to be detected and therefore multiple annotations. For example, multiple people can be seen in one image, all of which should be detected.

[0011] Various data can be used. In particular, image data recorded by one or more cameras can be used. However, data from other sensors (such as radar sensors, lidar sensors, ultrasonic sensors, or microphones) can also be used. When using acoustic signals, in particular, visualized noise spectra can be used as a basis for object recognition.

[0012] The sources of the annotations can be different. For example, if the data sets are obtained from external sources such as the Internet, they are usually already provided with annotations, which can then be read out accordingly. Alternatively, the annotations can be created manually and associated with the data sets. Another alternative is to create the annotations automatically and / or semi-automatically. In the case of semi-automatic annotation, the images are labeled by an annotation algorithm, wherein, in a second step, only the correctness of the labels is verified by a human.

[0013] The task of an object detection algorithm is to detect objects as accurately as possible. To this end, the object detection algorithm calculates object detections, where an object detection is particularly understood as a bounding box. Generally speaking, both annotations and object detections can be represented by a bounding box. The difference is that the bounding box of an annotation defines the object to be detected, while the bounding box of an object detection represents the bounding box determined by the object detection algorithm.

[0014] To determine how accurate a given object detection algorithm is at detecting objects, it can first be applied to a selected data set to calculate object detections for that data set. The resulting object detections can then be assigned to the annotations of the data set. This can be done by assigning the annotation to the object detection that has the highest overlap with the annotation. Alternatively, the distance between the object detection and the annotation can be used as the assignment criterion by assigning the annotation with the smallest distance to the object detection.

[0015] After the assignment is performed, three possible scenarios arise. An object detection may not be assigned to any annotation, for example, because it does not overlap with any of the annotations. This is called a false positive. A second possibility is that an annotation does not correspond to any object detection. This is called a false negative. Finally, an assignment is performed and a pair of object detection and annotation exists. This is called a match.

[0016] For all object detections and annotations that meet the third condition, a quality measure can now be determined, which characterizes the accuracy with which the object detection identified the annotation and thus the object to be detected.

[0017] In the following, a quality metric may be understood as the probability that the distance between an object detection and its corresponding annotation is below a predetermined distance threshold with this probability. Here, a distance may be understood as the distance between a point on a side of an object detection and a point on a side of the associated annotation. In particular, it may be the minimum or maximum distance between an object detection and a side of the associated annotation.

[0018] The advantage of the present invention is that it is possible to determine the probability with which an object detection algorithm may cause a safety risk during object detection. A safety risk can be understood as the probability that an object detection no longer completely encompasses the correspondingly associated annotation of the object detection. This situation is particularly critical for robots, drones, and other autonomously operating conventional vehicles that use object detection algorithms as part of their environment modeling and motion planning. Conversely, if the determined probability is below a predefined value, the present invention can be used to classify the object detection algorithm as safe during object detection.

[0019] In a further embodiment of the present invention, the deviation is the distance between a detected point on the object and a corresponding marked point.

[0020] An advantage of this embodiment is that the object detection algorithm is not limited to object detections whose edges are parallel to the edges of the annotation. For example, the object detection algorithm can output object detections that are rotated relative to their corresponding annotation. In this case, the deviation can be understood as the distance between a corner of the annotation and a side of the object detection.

[0021] In another embodiment of the method, the deviation is an offset between a point detected on the object and a corresponding marked point, wherein the offset is a signed scalar whose value represents the distance and whose sign represents the direction in which the marked point is offset from the point detected on the object.

[0022] The advantage of this extension is that, for example, the minimum offset can be used to determine how far a label is located inside the corresponding object detection, or how far the label protrudes from the object detection. If the object detection completely surrounds the label, the minimum offset is greater than zero. If the label is partially outside the object detection, the minimum offset is less than zero. This can be used to determine how likely it is that a portion of the label or the entire label is outside the object detection.

[0023] In a further embodiment of the method, the deviation corresponds to the smallest offset from the set of offsets.

[0024] The advantage of this embodiment is that a given object detection can be characterized by its deviation from the annotation that is—from a safety point of view—most dangerous. For safety reasons, the most dangerous deviations from all object detections can be considered and statistically analyzed.

[0025] In a further embodiment of the method, the set of offsets consists of offsets of the marked edges to the corresponding edges of the corresponding object detection, wherein the offsets are orthogonal to the corresponding edges. Corresponding edges are understood to be edges of the object detection and annotation that represent the same boundary. In the case of two-dimensional object detection and annotation, these are the left, right, top and bottom edges, respectively. For example, the left edge of the object detection corresponds to the left edge corresponding to it. In order to determine the minimum offset between corresponding edges, for each pair of corresponding edges, an offset is determined parallel to the corresponding marked edge, which is offset from the corresponding edge of the object detection by this offset. The minimum offset is then the offset of the minimum length.

[0026] The advantage of this extension is that it allows for determining how far a label can protrude from an object detection. This allows for an estimate of the most safety-critical deviation for each pair of label and corresponding object detection. Conversely, it allows for determining how much wiggle room the object detection algorithm has before it makes a potentially safety-critical error.

[0027] In another embodiment of the method, the deviation is understood to be the area corresponding to the marked portion that does not overlap with the object detection. This embodiment has the advantage that the area can better illustrate the extent to which multiple deviations from the marking (e.g., height and width) may be safety-critical. In the case of three-dimensional objects, the deviation can be represented by a volume, wherein the deviation is represented by the volume that does not overlap with the object detection volume.

[0028] In a further embodiment of the method cited above, the probability is calculated based on a model, which is ascertained based on the determined deviations.

[0029] The model may for example involve a representation of a probability distribution The model can also be determined based on a specific deviation. For example, its parameters can be determined based on known methods, in particular maximum likelihood estimation or Bayesian parameter estimation. Alternatively, the parameters can be adjusted based on expert knowledge so that the model exhibits the desired behavior. The advantage is that the model selected in this way can be integrated into the determination of the probability with appropriate assumptions.

[0030] Alternatively, the model can extract knowledge only from certain deviations and output probabilities accordingly. For this purpose, known machine learning methods, in particular neural networks, can be used.

[0031] The advantage is that this type of model can incorporate additional and / or fewer assumptions into the probability determination and extract information based solely on the data (i.e., the determined deviations). This can be meaningful, for example, when no meaningful assumptions about the deviation distribution are known.

[0032] In a further embodiment, the model described above is a parameterizable model, in particular a parameterizable probability distribution, the parameters of which can be ascertained from the determined deviations.

[0033] The advantage of this embodiment is that the assumptions about the selected family of probability distributions are clearly listed and the actual distribution of the determined deviations can be easily determined using known methods, for example, by means of maximum likelihood estimation. Alternatively, a Bayesian method can be used to incorporate additional assumptions about the parameters into the determination.

[0034] In another embodiment, the parameterizable model described above is an embodiment of a general extreme value distribution, wherein the parameters define a specific distribution.

[0035] The advantage of this embodiment of the method is that a general extreme value distribution models rare events very well. In general, it can be assumed that the deviations described above follow an extreme value distribution. To prove this in a specific case, statistical tests, in particular the Kolmogorov-Smirnoff test, can be used.

[0036] Furthermore, protection is claimed for a computer-implemented method that can be used to adapt a computer-implemented object detection algorithm for determining object detection. The method comprises the following steps:

[0037] Obtaining labels for objects to be detected using an object detection algorithm;

[0038] Obtaining object detection with the aid of an object detection algorithm;

[0039] Calculating a quality metric for the object detection algorithm according to one of the methods described above for calculating a quality metric for an object detection algorithm;

[0040] Based on the calculated quality measure, the detection algorithm is adapted in such a way that a renewed execution of the object detection algorithm results in a scaling of the object detection ascertained with the aid of the object detection algorithm.

[0041] The ascertainment of the annotations and object detections, as well as the calculation of the quality measure, can be performed in a manner analogous to the embodiments of the method described above. For example, a data set can be used from which the annotations are extracted. Subsequently, in the method, an object detection algorithm is used to predict the objects to be detected by object detection. The object detections are then associated with the annotations, and the predictions are evaluated using one of the methods described above for determining the quality measure.

[0042] The calculated quality metric can be used in the adaptation step of the object detection algorithm to make the underlying object detection algorithm safer. As already explained, a predicted object detection by the object detection algorithm can be considered safety-critical if the annotation corresponding to the object detection is completely or partially outside the object detection. To modify the object detection algorithm to make it safer, the predicted object detection can, for example, be scaled (i.e., its shape and size are changed) so that the corresponding annotation is completely enclosed.

[0043] The advantage of this method is that the object detection algorithm can be measurably adapted by improving the quality metric so that the object detection algorithm achieves better or safer object detection. Therefore, the method can be used as a functional block for enabling safety-critical products based on object detection algorithms (e.g., automated driving functions and / or driver assistance functions).

[0044] In a further embodiment, the steps of ascertaining object detection, calculating a quality measure, and adapting an object detection algorithm are repeated with the respectively adapted object detection algorithm until the quality measure falls below or is equal to a predefined quality value and / or a predefined number of repetitions is reached.

[0045] The advantage of this embodiment is that object detection algorithms based on iterative methods can be very easily adapted in order to increase the safety of the predicted object detection.

[0046] In a further embodiment of the method, the object detection algorithm is based on a parameterizable model, in particular a neural network.

[0047] The advantage of this embodiment is that the currently highest-performing object detection algorithms are based on neural networks. This embodiment allows the safety of the neural network to be assessed using one of the quality metrics described above.

[0048] In another embodiment, scaling is performed based on the ascertained properties of the object detections, in particular, their size, size ratio, and / or position in the image. For example, it may be provided that smaller object detections must be scaled differently than larger object detections because deviations from the corresponding annotations are more safety-critical for larger annotations than for smaller ones, and / or vice versa. Alternatively and / or additionally, the position of the object detections may also be used to determine scaling. For example, in the case of an autonomous vehicle, it may be determined that objects at the upper edge of the video image are further away from the vehicle itself and therefore less safety-critical.

[0049] In a further embodiment of the method, the scaling is performed independently of the ascertained object detection, in particular based on a predefined factor.

[0050] The advantage of this embodiment is that the factor can be optimized without any assumptions, solely based on a quality measure, and represents a computationally-saving measure by which existing object detection algorithms can be made measurably safer in a simple and rapid manner.

[0051] In a further embodiment of the method, the object detection algorithm is based on a parameterizable model, in particular a neural network, wherein the change of the parameters based on the parameterizable model is adapted, the method comprising the following steps:

[0052] Determining a scaled annotation based on the determined annotation;

[0053] Obtain object detection with the help of detection algorithms;

[0054] Aligning object detection with the scaled annotation based on the obtained annotation;

[0055] Determine the error between the object detection and the corresponding scaled annotation;

[0056] Reduce errors by matching parameters.

[0057] The core idea of ​​this embodiment of the method is to train the neural network so that it outputs scaled object detections that no longer require subsequent scaling to encompass the corresponding annotations. To do this, scaled annotations are first required. Scaled annotations are defined as annotations generated by scaling the initially extracted annotations. These scaled annotations can then be used to train the neural network, thereby instructing the neural network to inherently scale.

[0058] Experience has shown that neural networks are currently the most powerful object detection algorithms. Therefore, the advantage of this embodiment is that, in addition to high performance, a high degree of predictive reliability in object detection can be achieved.

[0059] In a further embodiment of the method, the individual steps of the preceding embodiment are repeated with respectively adapted parameters until a predefined error threshold is fallen below and / or a predefined number of repetitions is reached.

[0060] The advantage of this embodiment is that the neural network can be adapted iteratively. It is known that this iterative method for training a neural network achieves the best prediction performance.

[0061] Furthermore, protection is claimed for a computer program comprising instructions which, when the computer program is executed by a computer, cause the computer to carry out one of the methods specified above.

[0062] Furthermore, protection is claimed for a machine-readable storage medium on which the computer program is stored.

[0063] Furthermore, protection is claimed for a device which is configured to carry out one of the methods described above. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 A schematic diagram of a method for determining a quality measure of an object detection algorithm is shown.

[0065] Figure 2 The relationship between labeling, object detection, and the scale of object detection is shown as an example.

[0066] Figure 3 The annotations and the offsets of the corresponding edges of the corresponding object detections are shown as examples.

[0067] Figure 4A general extreme value distribution with a threshold value is schematically shown.

[0068] Figure 5 An illustrative process for improving the quality metric of an object detection algorithm is shown. DETAILED DESCRIPTION

[0069] In a first embodiment, a quality measure of an object detection algorithm is determined using a computer-implemented method. The object detection algorithm is designed so that it can recognize predefined objects by marking them with bounding boxes in the image data recorded by the camera. This is done, for example, in Figure 2 a schematically shows that in this Figure 2 A vehicle is depicted in a with a label 201 and a bounding box 202a determined with the aid of an object detection algorithm.

[0070] In order to determine a measure of the quality of the algorithm or the accuracy of object recognition, in this embodiment, a set of images is used in which objects are annotated and the object detection algorithm determines bounding boxes for the annotated objects. Figure 1 A method for determining a quality measure for an object recognition algorithm is schematically shown in FIG.

[0071] In step 101 of the method, the object detection determined by means of the object detection algorithm is assigned to the annotation (201) contained in the image data. In this case, the annotation can generally protrude beyond the associated object detection (202a), as exemplified in Figure 2 Another possibility is that the annotation is completely surrounded by the object detection (202b), which is shown in Figure 2 b and Figure 3 For the following steps, the special case of marking the object detection may optionally correspond to the Figure 2 In this embodiment, the correspondence between annotations and object detections is performed by the so-called intersection-over-union (i.e., the ratio of the overlap of two bounding boxes to the combined area of ​​the two bounding boxes).

[0072] (In an alternative embodiment, the distance between the centers of the two bounding boxes may also be used at this point to enforce correspondence.)

[0073] In step 102, the minimum deviation is determined for each pair of annotations and corresponding object detections. In this case, the minimum deviation is determined from the set of deviations between the object detection and the associated annotations. Figure 3. In this embodiment, the deviation is the corresponding offset of the object detection from the corresponding edge of the annotation corresponding to it. This means that the offset is calculated for the corresponding edges on the left (301), above (302), right (303) and below (304). Here, the offset is always parallel to the corresponding edge of the annotation (201). In addition, the sign of the offset indicates the direction in which the object detection is offset from the annotation (201). For the case where the annotation (201) protrudes from the object detection on one side, the corresponding offset is negative (301). In other cases, the offset is positive (302, 303, 304). Then, the smallest offset is calculated from the four offsets (301, 302, 303, 304).

[0074] In step 103, a quality measure is calculated. For this purpose, a model (401) is determined from the deviations determined in step 102, which model represents the distribution of the deviations. In this embodiment, a general extreme value distribution is used for this purpose.

[0075] In order to obtain the parameters of the general extreme value distribution, the maximum likelihood estimation method is used.

[0076] To calculate the quality metric, the cumulative distribution function of the general extreme value distribution is analyzed at the value 0 (402). Figure 4 In the figure, the deviation is plotted on the X-axis and the probability density of the extreme value distribution is plotted on the Y-axis. The result of the analysis corresponds to the probability that the label stands out from the object detection corresponding to it.

[0077] In the second embodiment, the same steps as in the first embodiment are performed, however, Bayesian parameter estimation is performed in step 103 instead of maximum likelihood estimation.

[0078] exist Figure 5 In a third exemplary embodiment schematically shown in FIG, the object detection algorithm is modified in such a way that it becomes safer.

[0079] For this purpose, a manual annotation is generated for a data set based on sensor data of a camera in step 501. Alternatively, the annotation can also be generated semi-automatically or fully automatically.

[0080] In step 502, object detections are determined for the sensor data by an object detection algorithm, and these object detections are then assigned to the annotations in step 503. The assignment occurs here as in the first exemplary embodiment.

[0081] In step 504, a quality measure of the object detection algorithm is determined. This occurs as in the first embodiment.

[0082] In step 505, the object detection algorithm is adapted so that the probability of a label protruding from its corresponding object detection becomes smaller. To this end, all object detections are scaled by a fixed factor so that they surround their corresponding labels.

[0083] In the fourth embodiment, the same steps as in the third embodiment are performed, however, using lidar-based sensor data instead of camera-based sensor data. The remaining steps are performed identically.

[0084] In the fifth embodiment, the same steps as in the third embodiment are performed, with step 505 modified as follows: the object detection is scaled by a fixed factor, and a quality metric is calculated for the scaled object detection. If the quality metric does not meet a predefined threshold, the scaled object detection is scaled by a factor such that the object detection becomes larger. This adaptation of the variable using the scaling factor is continued until the quality metric falls below a predefined probability.

[0085] In a sixth embodiment, the object detection algorithm is based on a neural network. The same steps as in the third embodiment are performed, with step 505 modified as follows: the neural network is trained using the sensor data and the annotations of the second data set so that it outputs inherently larger object detections. To this end, the annotations of the second data set are scaled so that they become larger. During subsequent training with the scaled annotations, the neural network then learns to predict larger object detections. After training, the modified neural network is applied again to the first data set and the quality metric is re-determined. If the quality metric lies above a predefined probability value, the neural network is trained based on the second data set with even larger scaled annotations. The matching of the neural network and the evaluation of the quality metric are repeated in this manner until the quality metric falls below the predefined probability value.

Claims

1. A method for computing a quality metric for a computer-implemented object detection algorithm for image data, In the method, an object detection is ascertained for a selected data set using the object detection method (202a, 202b, 305), In the method, the object detection is assigned to the annotation (201) of the selected data set according to a predefined assignment criterion. in, forming an object detection and annotation pair, wherein the object detection (202a, 202b, 305) and / or the annotation (201) corresponds to a bounding box, In this case, the highest overlap or the smallest distance between the annotation and the object detection in terms of area or volume is used as the assignment criterion. It is characterized by the following steps: For all pairs of object detections and annotations, determining (102) the deviation (301, 302, 303, 304) of the annotation to the object detection corresponding to the annotation; calculating (103, 504) a quality metric of the object detection algorithm based on the determined deviations (301, 302, 303, 304), wherein the quality metric represents the probability that the deviation of an object detection from a label (201) corresponding to the object detection is above or below a predefined threshold (402) with the said probability, The calculation of the probability is based on a model (401) which is determined based on the determined deviation.

2. The method according to claim 1, wherein The deviation represents an offset (203a, 203b, 301, 302, 303, 304) between a point of the object detection (202a, 202b, 305) and a point of the annotation (201) corresponding to the object detection, wherein the offset is a scalar with a positive or negative sign, the value of the scalar represents a distance and the positive or negative sign of the scalar represents the direction in which the point of the annotation (201) is offset from the point of the object detection (202a, 202b, 305).

3. The method according to claim 2, wherein: The deviation is the smallest offset from the set of ascertained offsets ( 301 , 302 , 303 , 304 ).

4. The method according to claim 3, wherein: The set consists of offsets (301, 302, 303, 304) of the edges of the annotation (201) to corresponding edges of the corresponding object detection, wherein the offsets are orthogonal to the corresponding edges.

5. The method according to claim 1, wherein The deviation represents an area that corresponds to the portion of the annotation that does not overlap with the object detection.

6. The method according to any one of claims 1 to 5, wherein The model (401) is a parameterizable probability distribution.

7. The method according to claim 1, wherein The method is used to enable the object detection algorithm for use with a partially automated, highly automated, or fully automated robot.

8. A method for adapting a computer-implemented object detection algorithm for determining object detection (202a, 202b, 305), the method comprising the following steps: a) determining (501) a label (201) of an object to be detected by means of the object detection algorithm based on the selected data set; b) determining (502) an object detection based on the selected data set by means of the object detection algorithm; c) calculating a quality metric of the object detection algorithm according to the method according to any one of claims 1 to 7; d) Based on the calculated quality measure, the object detection algorithm is adapted ( 505 ) such that a renewed execution of the object detection algorithm results in a scaling of the object detection ascertained by means of the object detection algorithm.

9. The method according to claim 8, wherein Steps b to d are repeated with the respectively adapted object detection algorithm until the quality measure is below or above a predefined quality value (402) and / or a predefined number of repetitions has been reached.

10. The method according to any one of claims 8 or 9, wherein The scaling is performed based on the ascertained properties of the object detection (202a, 202b, 305).

11. The method according to claim 10, wherein: The scaling is performed based on the ascertained size, size ratio and / or position of the object detection ( 202 a , 202 b , 305 ) in the image.

12. The method according to any one of claims 8 or 9, wherein The scaling is performed independently of the ascertained object detection.

13. The method according to claim 12, wherein: The scaling is performed based on a predefined factor.

14. The method according to claim 9, wherein The object detection algorithm is based on a parameterizable model, wherein the matching is based on a change of a parameter of the parameterizable model, the method comprising the following steps: e. obtaining a scaled annotation based on the obtained annotation; f. Obtaining object detection by means of the object detection algorithm (202a, 202b, 305); g. Aligning the object detection (202a, 202b, 305) with the scaled annotation based on the obtained annotation (201); h. Obtaining an error between the object detection and the scaled annotation corresponding to the object detection; i. Reduce the error by matching the parameters.

15. The method according to claim 14, wherein The parameterizable model is a neural network.

16. The method according to claim 14, wherein Steps f to i are repeated with the respectively adapted parameters until a predefined error threshold is fallen below and / or a predefined number of repetitions is reached. 17 . A computer program product comprising instructions which, when executed by a computer, cause the computer to carry out the method according to claim 1 . 18 . A machine-readable storage medium having a computer program stored thereon, the computer program comprising instructions which, when executed by a computer, cause the computer to carry out the method according to claim 1 .

Images