Method for controlling robotic device, control device, computer program and medium

Abstract

According to various embodiments, there is provided a method, a control device, a computer program and a computer-readable medium for controlling a robotic device, the method having: training a control strategy for training the robotic device on the basis of a large number of training control conditions; and for each of the plurality of control time points: ascertaining information about the surroundings of the robot device for the control time point, a first consistency measure for the information ascertained for the control time point is ascertained by comparing a control situation, which describes the information ascertained for the control time point, with a training control situation, and at least one second consistency measure for the information ascertained for the control time point is ascertained, ascertaining a combined consistency measure by combining the first consistency measure with the at least one second consistency measure; and ascertaining one or more control actions for the robotic device according to the combined consistency metrics.

Description

Technical Field

[0001] The present disclosure relates to a method for controlling a robotic device. Background Art

[0002] Control strategies for robotic devices, such as autonomous vehicles - when artificial intelligence is used for this purpose - are usually trained on the basis of training data, i.e., they are trained for training control situations (or training scenarios). The control strategy trained in this way can then be used under the assumption that it also provides good results for control situations that were not seen during training, i.e., it can generalize sufficiently. In the case of suitable training data, this assumption is usually proven to be correct, but it may nevertheless occur that the control strategy is faced with a control situation for which it provides poor results, because the input data obtained by the control strategy describe a control situation to which it cannot generalize, or because the input data are of low quality, i.e., they do not describe the corresponding control situation well, for example. Some approach is therefore desirable in order to still achieve suitable (e.g., safe) control in this case.

[0003] The publication “Motion Planning for Connected Automated Vehicles at Occluded Junctions with External Perception” by Johannes Christian Müller (Institute for Measurement, Control and Microtechnology, Ulm University, Vol. 41, 2022, hereinafter referred to as reference [1]) describes a method based on subjective logic for evaluating the reliability of external perception in automated driving. Summary of the invention

[0004] According to various embodiments, a method for controlling a robotic device is provided, the method comprising: training a control strategy for the robotic device based on a large number of training control situations, and for each of a plurality of control time points:

[0005] determining information about the surroundings of the robotic device for the control time point;

[0006] determining a first consistency measure for the information ascertained for the control time point by comparing a control situation with a training control situation that describes the information ascertained for the control time point;

[0007] By taking the information obtained for this control time point

[0008] a prediction of a state of the surroundings based on information about the surroundings of the robotic device for one or more previous control time points,

[0009] and prior knowledge (Vorwissen),

[0010] each other, and / or

[0011] The information obtained for one or more previous control points in time and the information obtained for one or more previous control points in time

[0012] Performing a comparison to determine at least one second consistency measure of the information determined for the control time point;

[0013] Obtaining a combined consistency metric by combining the first consistency metric with at least one second consistency metric; and

[0014] One or more control actions for the robotic device are obtained based on the combined consistency measure.

[0015] The method described above enables safe control, since it takes into account the situation that the current information about the surroundings of the robotic device to be controlled is inconsistent in different aspects (e.g. with each other, with prior knowledge or with training conditions). If a low consistency is detected (i.e., for example, the combined consistency measure is below a predefined threshold value), control can be carried out according to a safe mode (e.g., driving cautiously), and thus safe control can be achieved even when there is information that is not suitable for the corresponding control.

[0016] Various embodiments are described below.

[0017] Embodiment 1 is a method for controlling a robot as described above.

[0018] Embodiment 2 is a method according to Embodiment 1, wherein the first consistency metric and at least one second consistency metric are subjective logic opinions (Subjective-Logic-Opinion), and the combination of the first consistency metric and at least one second consistency metric is a fusion in the sense of at least one subjective logic fusion operator (Subjective-Logic-Fusionsoperator).

[0019] With the help of subjective logic, epistemic uncertainty can be explicitly taken into account in the consistency assessment. It is therefore suitable for modeling and analyzing situations (in the present application, control situations) with sources of uncertainty and unreliability (in the present application, for example, sensors). In the following, common English terms are used in conjunction with subjective logic (i.e. “Subjective-Logic”), in particular for operators, as they are also used, for example, in reference [1].

[0020] Embodiment 3 is a method according to embodiment 2, wherein the first consistency metric and at least one second consistency metric are combined by applying uncertainty-weighted-average-belief-fusions operators.

[0021] This enables that uncertainty is taken into account when evaluating consistency and avoids that an (uncertain) evaluation indicating good consistency has a negative impact on control, e.g. not leading to cautious driving behavior even though cautious driving behavior may be appropriate because the consistency is actually low.

[0022] Embodiment 4 is a method according to embodiment 1, wherein the first consistency metric and the second consistency metric are Dempster-Shafer-Belief-Massen.

[0023] Embodiment 5 is a method according to embodiment 1, wherein the first consistency metric and the second consistency metric are determined by means of Bayesian inference.

[0024] Dempster Shafer and Bayesian inference are alternatives to using subjective logic that can also provide a measure of consistency in order to achieve appropriate (eg, safe) control.

[0025] Embodiment 6 is a method according to any one of embodiments 1 to 5, wherein combining a first consistency metric with at least one second consistency metric comprises: time filtering the first consistency metric and / or the second consistency metric, and combining the first consistency metric or the filtered first consistency metric with the second consistency metric or the filtered second consistency metric.

[0026] Temporal filtering enables filtering out jumps in the consistency evaluation caused by erroneous (eg particularly pessimistic) consistency evaluations. Before the (actual) combination occurs, all consistency measures or only one or (in the case of multiple second consistency measures) a portion of the consistency measures may be filtered.

[0027] Embodiment 7 is a method according to embodiment 4, wherein the first consistency metric and at least one second consistency metric are subjective logical opinions, and the first consistency metric and / or at least one second consistency metric are time-filtered by applying cumulative-belief-fusion-operators.

[0028] In the case of a jump, the uncertainty of the consistency evaluation usually increases sharply and is therefore weighted less strongly by the fusion operator, taking into account that immediately after a jump, the persuasiveness of the corresponding consistency measure is usually low.

[0029] Example 8 is a method according to Example 7, wherein, when time filtering is performed on a first consistency metric and / or at least one second consistency metric, for a corresponding consistency metric, if the degree of conflict (Degree-of-Conflict) with a previous filtering result is higher than a predetermined threshold, the current value of the consistency metric is used as the result of time filtering.

[0030] Embodiment 9 is a method according to any one of embodiments 1 to 8, wherein a first consistency measure and / or at least one second consistency measure is obtained based on multiple measurement values, wherein the importance of the measurement values ​​is reduced according to the relevance of the measurement values ​​by reducing the confidence level.

[0031] For example, it may be possible to take into account that multiple measurements come from the same data source (eg, from the same sensor) and avoid errors in the data source from distorting the consistency measure.

[0032] Embodiment 10 is a method according to any one of embodiments 1 to 9, wherein the information has an environmental model.

[0033] In particular, the consistency of information derived from the sensor data (eg object detection or trajectory prediction) can be evaluated.

[0034] Embodiment 11 is a method according to any one of embodiments 1 to 10, wherein obtaining one or more control actions based on the combined consistency measure comprises: obtaining one or more control actions based on a quality function, the quality function evaluates the control actions, and the combined consistency measure enters the quality function.

[0035] In other words, the possible control actions are evaluated, for example, according to a quality function (into which the combined consistency measure enters), and one or more control actions (with the best evaluation) are selected accordingly. In this way, the combined consistency measure can be effectively taken into account without necessarily setting hard limits (such as threshold comparisons for consistency).

[0036] Embodiment 12 is a control device configured to execute the method according to any one of embodiments 1 to 11.

[0037] Embodiment 13 is a computer program having instructions. When the instructions are executed by a processor, the instructions cause the processor to execute the method according to any one of embodiments 1 to 11.

[0038] Embodiment 14 is a computer-readable medium storing instructions. When the instructions are executed by a processor, the instructions cause the processor to execute the method according to any one of embodiments 1 to 11. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In the drawings, like reference characters generally refer to the same parts throughout the different views. The drawings are not necessarily to scale, with emphasis instead generally being placed on illustrating the principles of the invention. In the following description, various aspects are described with reference to the following drawings.

[0040] Figure 1 A vehicle is shown.

[0041] Figure 2 It shows that VAE (Varational Autoencoders) is used to obtain the consistency measure.

[0042] Figure 3 The fusion of consistency metrics according to one embodiment is shown.

[0043] Figure 4 A flow chart is shown which illustrates a method for controlling a robotic device according to one specific embodiment. DETAILED DESCRIPTION

[0044] The following detailed description relates to the accompanying drawings, which are shown to illustrate the specific details and aspects of the present disclosure in which the present invention may be implemented. Other aspects may be used and structural, logical and electrical modifications may be performed without departing from the scope of protection of the present invention. Different aspects of the present disclosure are not necessarily mutually exclusive, as some aspects of the present disclosure may be combined with one or more other aspects of the present disclosure to form new aspects.

[0045] Different examples are described in more detail below.

[0046] Figure 1 A vehicle 101 is shown.

[0047] exist Figure 1In the example of FIG. 1 , a vehicle 101 , such as a passenger car or a truck, is equipped with a vehicle control device 102 (such as an electronic control unit (ECU)).

[0048] The vehicle control device 102 has data processing components such as a processor (eg, a central processing unit CPU) 103 and a memory 104 for storing control software 107 according to which the vehicle control device 102 operates and data processed by the processor 103. The processor 103 implements the control software 107.

[0049] For example, the stored control software (computer program) has instructions that, when executed by the processor, cause the processor 103 to implement driver assistance functions or even autonomously control the vehicle.

[0050] Control software 107 is transferred from computer system 105 to vehicle 101, for example, via network 106 (or also by means of a storage medium such as a memory card). This can also take place during operation (or at least when vehicle 101 is at the user's location), since control software 107 is updated with new versions, for example, over time.

[0051] The control software 107 can be trained, for example, by means of machine learning (ML), i.e., the control software 107 implements one or more ML models 108 (or “machine learning models”), which are trained based on training data (in this example from the computer system 105). Thus, the computer system 105 implements an ML training algorithm for training the one or more ML models 108.

[0052] The control software 107 determines control actions of the vehicle (e.g., steering actions, braking actions, etc.) based on input data 109, which are provided to the control software and contain information about the surroundings or the control software derives information about the surroundings based on the input data (e.g., by detecting other road users, such as other vehicles). For example, the input data are sensor data, such as from a camera of the vehicle, or also information obtained by communication with other vehicles or devices at the edge of the road.

[0053] In road traffic, it often happens that the vehicle is in an unusual situation that was not anticipated when designing the control software and in particular when training the ML model or models 108. Human drivers typically have an intuition based on their experience, whereby they drive particularly cautiously in such situations. In contrast, automated vehicles (i.e., control software) typically lack such mechanisms.

[0054] According to various embodiments, a mechanism is provided with the help of which an automated vehicle (or also another agent, in particular another type of robotic device) can react in an anticipatory manner to unusual situations. For example, the mechanism allows an automated vehicle to react to unknown or critical situations in a particularly cautious driving style. As a result, some unknown, critical situations (for example in the sense of ISO 21 448 Safety of the Intended Functionality) can be mitigated, which can achieve a safety that goes beyond safety in the sense of ISO 21 212 Functional Safety.

[0055] To this end, according to various embodiments, the information provided to the vehicle (or other agent), i.e. the input data 109 for the control software 107, but also the data derived therefrom (information about the surroundings), such as trajectory estimates for other road users or detected objects, is checked for consistency in at least one aspect (e.g. current with previous or training data). In order to check the consistency, a probabilistic consistency measure is established for each aspect. Taking into account a plurality of aspects, the individual aspects are fused into an overall result. The results of the analysis module are forwarded to the planning module and there influence the driving behavior so that the more inconsistent the information presented to the automated vehicle, the more cautious (and therefore more proactive) the automated vehicle drives.

[0056] Different methods can be used to model the probabilistic consistency measure, such as subjective logic, Dempster-Shafer theory or Bayesian inference. Different embodiments are described below based on subjective logic. However, the embodiments can be implemented similarly with other methods.

[0057] Example 1 of a consistency measure: A consistency measure derived from a comparison of known critical and non-critical situations using a variational autoencoder.

[0058] In such an embodiment, the consistency of the current information is evaluated based on practical experience (e.g. Figure 1 In the example case of input data 109) as one aspect.

[0059] To this end, a neural network (e.g., as one of the one or more ML models 108), such as a variational autoencoder (VAE) network, is first trained to achieve a compact representation of the input data in a feature space (e.g., the input data has multiple data points or data elements that are each mapped to a feature vector).

[0060] Figure 2 The use of VAE 201 to obtain a consistency measure is shown.

[0061] A training data set 202 is used to train the VAE 201, which training data set contains traffic scenes classified as critical and non-critical. The data points of the training data set are clustered according to feature points in the feature space 203, to which the data points are mapped (i.e. clustered according to the metric in the feature space 203). The feature point - the input data point (i.e. the data point of the training data set in training) is mapped to this feature point - is, for example, the center of the distribution provided by the VAE 201 for the input data point. Subsequently, a Gaussian fit is performed on the data in each cluster, so that the resulting probability density distribution (i.e. the fitted Gaussian distribution) represents the probability of belonging to the corresponding cluster. Accordingly, by construction (per Konstruktion), the probability of belonging of a data point to a cluster is derived directly from the Mahalanobis distance (Mahalanobis distance) of the feature point of the data point and the corresponding cluster.

[0062] Online (ie, during operation, in the present application example, while controlling the vehicle), input data 109 (eg, sensory information) are transformed into feature space 203 by means of an encoder of VAE 201 and a consistency measure such as the following is derived therefrom:

[0063] The consistency measure for a situation s described, for example, by a data point of the input data 109 (i.e., a measure for consistency with the training data 202) is represented in this embodiment by a binary subjective logic opinion ω(b, d, u, a). Subjective logic is a mathematical theory that extends the classical probability theory. Here, the binary cognitive subjective logic operation formula ω(b, d, u, a) is calculated for the situation s described by the data point of the input data 109 as follows:

[0064]

[0065] Here, b represents the belief mass (Belief-Mass or Glaubensmasse), which is formed by the maximum membership probability of the situation s (i.e., the corresponding data point) to the cluster Cuncritical of the situation marked as non-critical; d represents the disbelief mass (Disbelief-Mass or Zweifelsmasse), which is formed by the maximum membership probability of the situation s (i.e., the corresponding data point) to the cluster Ccritical with the situation marked as critical; and u represents the epistemic uncertainty. In order to obtain these different types of clusters, the data points are marked (i.e., labeled) as critical (i.e., representing a critical situation) or non-critical (i.e., representing a non-critical situation) in the training data set 202.

[0066] The closer a data point (or its assigned feature point) is to a cluster (eg, in terms of the Mahalanobis distance to the Gaussian distribution fitted at the cluster), the greater the probability of membership, and thus the corresponding confidence quality.

[0067] The prior distribution is characterized by the parameter aprior, which provides the possibility of introducing expert knowledge or prior knowledge. In the absence of specific prior knowledge, the parameter aprior is set to 0.5, which represents the absence of prior knowledge in subjective logic theory. In the following, aprior is also referred to as a.

[0068] Figure 2 The method thus makes it possible to determine a subjective logical opinion about the criticality of the current situation s for use in a subjective logical criticality estimation.

[0069] By normalizing accordingly, the sum of the individual probabilities b and d does not add up to 1, but leaves a residual probability u that the new data point does not belong to any known cluster. This probability constitutes the statistical uncertainty u of the subjective logical opinion.

[0070] Instead of the VAE network 201, a Gaussian mixture model, a latent Dirichlet allocation, a Dirichlet Boltzman machine, a deep belief network, an adversarial generative neural network, or a variant of an autoencoder network may also be used. In addition, the decoder 204 of the VAE 201 may also be used to determine an error metric, which can then also be converted into a cognitive subjective logical opinion (which can be used as a consistency metric for the corresponding input data point), instead of using the feature space 203 (latent space), i.e., obtaining a consistency metric by operating in the feature space 203 as described above.

[0071] Example 2 of consistency measure: Reliability of predicted trajectory through consistency check

[0072] In this embodiment, the reliability of the trajectory prediction of other road users is evaluated (if necessary additionally) by checking the consistency of temporarily stored, previous predictions (which the control software 107 has determined, for example, based on previous input data 109) with current and / or previous input data 109 (i.e., for example, current or past measurements of the actual behavior of other road users). For the trajectory prediction, the control software includes, for example, a prediction module (for example, as one of one or more ML models), for example a multiple trajectory prediction neural network.

[0073] The consistency measure in this case (now for the trajectory prediction of the other traffic participant and the input data 109 , such as the current or previous measurement of the actual trajectory of the other traffic participant) can again express a binary cognitive subjective logical opinion.

[0074] For this purpose, for example, an earlier trajectory prediction (predicted trajectory hypothesis) is represented as a probability distribution, and the probability of the current measured trajectory of the traffic participant (or, depending on the application, another object represented by one or more measurement states, such as a road sign) belonging to the closest trajectory hypothesis of the earlier prediction is selected as the confidence quality b. The probability that the measured trajectory does not match any trajectory hypothesis of the earlier prediction is selected as the distrust quality d, wherein the epistemic uncertainty u is obtained as a residual sum with 1 as before.

[0075] Alternatively, a frequentist opinion can be generated by representing each trajectory hypothesis as a trajectory point (landmark) over time and checking whether the distance of the measured state (i.e. the state from the measured trajectory) to the closest trajectory point is below or exceeds a predetermined threshold. These binary results form samples for which a frequentist opinion is constructed via relative frequencies. For the special case of a single trajectory prediction, see, for example, reference [1].

[0076] The trajectory prediction for one or more traffic participants can generally be regarded as a surroundings model (i.e. a model for the surroundings) of the robotic device (in this case, the vehicle). Such a surroundings model can be determined for each time point in a series of time points (e.g., control time points, for which one or more control actions are respectively selected).

[0077] Example 3 of consistency measure: Consistency between perception and semantic graph

[0078] In this embodiment, the consistency between input data 109 (e.g., sensory information) and the map is checked (if necessary additionally). For this purpose, for example, the sensory information from different vehicle sensors is converted into a bird's-eye view representation and compared with a digital map stored in the vehicle. With the help of preprocessing modules such as lane recognition, a consistency measure can again be determined as described above.

[0079] If input data 109 provide information that the vehicle is moving at a location at which there is a building according to the map, the consistency of input data 109 with the map may be evaluated as low, for example.

[0080] A map can be considered as an example of reference information about the surroundings, which is constant over multiple control points in time (because the map remains unchanged until it is updated). In this sense, this reference information therefore represents (possibly static) a priori knowledge for the current control point in time. Priori knowledge can be understood in this context as information that is already known before a certain task is performed (in this case, the control performed at a control point in time or at least at the current control point in time). In the present application, this can be information that is known before the start of a journey or a determined route and helps the vehicle to better understand the surroundings of the vehicle.

[0081] Example 4 of consistency metric: consistency of objects detected by different sensor modalities

[0082] In this embodiment, the consistency of the sensor data elements (e.g., a plurality of camera images) contained in the input data 109 is evaluated (if necessary additionally) by, for example, evaluating the consistency of the object detections from different camera images (or sensor data elements of different sensors) which were recorded at the same time or also evaluating the consistency of the object detections over time (in each case from one camera image in a series of camera images). For this purpose, the object detections of each of two sensor data elements (from different sensors or different times, one of which corresponds to the current input data 109) are checked for consistency in a common visible range (i.e., for this purpose, a measure is determined: how well the object detections of the two sensor data elements match). This can be done, for example, in the case of camera images, by enclosing features (e.g., detected objects) with tolerances in each case and counting how many features in one camera image have a corresponding object (“matches”) in another camera image (partial matching). In this case, it is also possible to count mismatches (i.e., features that do not have a corresponding object) and then determine a subjective logical opinion by dividing the matches and mismatches by the total number of features plus two. The greater the probability of the projection, the better. In addition, the newly arrived object hypothesis of the (current) time step can also be checked for consistency with the time-filtered objects of the previous time step. This can be done as follows: the object hypothesis of the previous time step is projected onto the current time step, and how many objects in the previous time step can be associated (belief) with the object hypothesis in the current time step. Then count how many objects are neither likely to be associated nor to have come from the "spawn zone", i.e., the area where it is expected that new objects usually "appear" (disbelief). The uncertainty is derived from the total number of associations. The greater the probability of the projection, the better. The consistency measure can then be determined as described for the above example.

[0083] The above exemplary embodiments of consistency measures describe different possibilities of how consistency measures of current input data 109 (i.e. the information determined for each (“current”) control time point) can be determined in different aspects (wherein this may be determined on the basis of data derived therefrom, such as object detection).

[0084] According to different implementations, multiple consistency metrics are obtained (for example, through control software) (for example, obtained according to multiple embodiments in the above-mentioned embodiments, that is, obtained in multiple aspects), and these consistency metrics are fused into a "combined" (or "fused" or "overall") consistency metric.

[0085] To further improve the results and increase their persuasiveness, this can include time-varying filtering (i.e. taking into account not only the consistency measure of the information about the surrounding environment for the current control time point, but also the consistency measure (possibly combined consistency measure) for one or more earlier (i.e. previous) control time points (in a series of control time points).

[0086] Figure 3 The fusion of consistency metrics according to one embodiment is shown.

[0087] exist Figure 3 In the example of rel A consistency measure in the form of a measure of reliability of perception (see Examples 3 and 4 of consistency measure) and a consistency measure in the form of a subjective logical opinion on the severity of the situation based on practical experience ω SC (See Example 1 of the consistency metric) Fusion is performed (based on the time filtering performed, each marked with a control time index k).

[0088] First, with the help of the corresponding application of the cumulative trust fusion operators 301, 302, the consistency measure ω rel (k) and ω SC (k) is the corresponding value ω in the previous time step rel (k-1) or ω SC (k-1) are fused and the consistency measure is combined with its corresponding value ω in the previous time step with the help of the corresponding application of the conflict degree operators 303, 304. rel (k-1) or ω SC (k-1) is checked for consistency.

[0089] In the consistency measure ω rel =(b rel , d rel ,u rel , a rel ) and ω SC =(b SC , d SC ,u SC , a SC ) in this example (index k is omitted here for simplicity), by

[0090]

[0091]

[0092] Cumulative trust fusion operators 301 and 302 are given.

[0093] In this example, by

[0094]

[0095] Give the degree of conflict.

[0096] If the degree of conflict is lower than a predetermined threshold, the previous and current subjective logical opinions are respectively classified as consistent, and the values ​​generated by the corresponding cumulative trust fusion operators 301 and 302 (i.e., the fusion of the previous and current subjective logical opinions) are transmitted through the corresponding selection functions 305 and 306. Otherwise, the corresponding selection functions 305 and 306 transmit the current value of the subjective logical opinion.

[0097] Here, the corresponding consistency measure ω in the previous time step rel (k-1) or ω SC (k-1) itself can be used for the fusion results of the cumulative trust fusion operators 301, 302 of the previous time step, so that the consistency measure is summarized over time (except for the "reset" when the current consistency measure is inconsistent with the consistency measure in the previous time step, because the corresponding selection function 305, 306 then passes the current consistency measure as described).

[0098] The outputs of the selection functions 305 and 306 are combined into the fused (overall) subjective logical opinion ω with the help of the uncertainty weighted average trust fusion operator total (k).

[0099] If ω A =(b A , d A ,u A , a A ) and ω B =(b B , d B ,u B , a B ) represents the output of the selection function 305, 306, then by

[0100]

[0101] An uncertainty weighted average trust fusion operator 307 is given.

[0102] The general versions of equations (2), (3) and (4) are given in reference [1].

[0103] If more than two consistency measures are combined, then Figure 3 The fusion mechanism may be, for example, arkadiert.

[0104] In reference Figure 3 The temporal filtering included in the described fusion mechanism enables aggregating information over long time windows (if the corresponding consistency measure ω in the previous time step is used as described above). rel (k-1) or ω SC (k-1) itself can in turn be used for the fusion results of the accumulated trust fusion operators 301, 302 of the previous time step) and thereby (potentially) achieve high noise reduction.

[0105] On the other hand, due to the switching behavior (depending on the consistency of successive consistency measures by selecting functions 305, 306), the temporal filtering can react quickly to sudden changes in the consistency measure. In the case of a jump, the uncertainty of the result increases sharply initially and is therefore not so strongly weighted by the fusion operator (i.e., the uncertainty weighted average trust fusion operator according to formula (4)). This effect is desirable because immediately after a jump, the persuasiveness of the corresponding consistency measure is usually not yet high.

[0106] If the correlation of the individual data points in the input data 109 is not negligibly small but can be estimated, the consistency measure can be corrected by using a trust discount operator. For example, if a single trajectory is checked based on multiple measurements (i.e. data points in the input data 109), the individual measurements (data points) are not statistically independent because they relate to, for example, the same data source. Nevertheless, an added value is derived here from the multiple measurements. The correlation can be estimated here by expert knowledge. In addition, this can prevent the epistemic uncertainty u from decreasing too quickly and prevent the fusion (i.e. the cumulative trust fusion operator according to formula (2)) from taking less and less into account the newly arrived data.

[0107] The subjective logical opinion after integration total (k) is converted into a probability (e.g., by control software). For implementations based on subjective logic, the probability p projected by the subjective logic opinion can be either directly ant = b + au to calculate the probability, or by integrating the Dirichlet distribution corresponding to the subjective opinion over a pre-given confidence interval. The Softmax function can be used for confidence scores. Below, the probability p generated in this way ant (k) is regarded as the fused consistency measure for the kth control time point, where p is also used below. ant (t), where t is the time corresponding to the corresponding control time point.

[0108] For example, the fused consistency measure is the result of the analysis module and is further processed by the planning module (for example, these two modules are part of the control software 107). Therefore, input data 109, such as sensor information (from sensors or other vehicles) and possible prior knowledge (such as maps) enter the analysis module, which provides the fused consistency measure (and, if necessary, individual consistency measures). The planning module generates a trajectory (i.e., a planned driving route) from this, and the control device controls the vehicle so that the vehicle follows the generated trajectory.

[0109] In parallel with the analysis module, one or more further analysis modules may be provided, which preprocess the perception information for the planning module. The information provided to the analysis module (e.g. the perception information) may be the result of the preprocessing, i.e. it is not necessary (only) to provide the input data 109 itself to the analysis module, but the result of the preprocessing may also (if necessary additionally) be provided to the analysis module. For example, lane recognition may support a consistency check of the map and the vehicle perception by the analysis module. In addition to the output of the analysis module, the planning module may have further inputs.

[0110] In the planning module, the fused consistency measure p is used in this application example of vehicle control ant (t) in order to influence driving behavior. This can be done in many ways. In addition, the planning module can be divided differently. For example, the planning module can be subdivided into strategic planning, tactical planning, and operational planning. The consistency measure p after fusion can be used to ant (t) affecting one or more of the components of the planning module.

[0111] For example, the planning module can compare the combined consistency measure with a threshold value (which is determined depending on the scale of the consistency measure or the scale of the corresponding indicator, and the consistency measure is determined based on the corresponding indicator) and switch to a safe control mode depending on whether the consistency described by it is lower than the minimum consistency described by the threshold value.

[0112] In one embodiment, the planning module takes the fused consistency measure as one of its inputs and uses an optimization-based method to thereby determine a trajectory that is output by the planning module (and according to which the control device 102 controls the vehicle). For example, the fused consistency measure is entered into the optimization function as follows:

[0113] J ges =J gew +w ant ∫v(t)·(1-p ant (t))dt (5)

[0114] Here, J ges is a modified quality function, in addition to the traditional quality function J gew In addition to the term, the quality function contains an additive term (i.e., a one-dimensional penalty term in the cost function), which is calculated by optimizing the weight w ant The weighted pair consists of the planned speed v(t) and p ant (t) is the integral of the product obtained by calculating the inverse probability (Gegenwahrscheinlichkeit). Therefore, by making J ges minimizes, preferring trajectories that, in the case of low consistency (close to zero), result in the selection of a slower trajectory for the automated vehicle. On the contrary, for very high consistency measures (close to one), the additive term becomes negligibly small, thus according to the traditional quality function Jx ew The planning cost is advantageous.

[0115] In another embodiment, the planning module also uses an optimization-based approach to determine the trajectory, but a multidimensional probability distribution f(x, t) different from the above embodiment enters the planning module, which describes the probability that a certain state x is not allowed at a certain time point t. Here, the probability distribution does not need to take into account all aspects in which the state may be invalid. For example, the probability distribution can map the current and predicted states of other traffic participants and be provided by another module. The integral of the probability distribution along the planned trajectory enters the quality function of the planning module so that the resulting trajectory has the lowest possible probability of containing invalid states. In addition, the integral of the probability distribution over the trajectory is limited to the maximum allowed residual probability range.

[0116] The fused consistency measure is then fed into the probability distribution. For example, the original version of the probability distribution f(x, t) provided by another module is modified as follows:

[0117]

[0118] Where n is a normalization constant. Because With the consistency probability p ant (t) (i.e., the probability that the trajectory contains an unallowed state) decreases, so in this case, by Based on the optimization, we can produce trajectories that represent more cautious behavior overall (compared to when using f(x, t)). ant In the limiting case of (t) = 0, f therefore ignores the overall distribution information in f, as this has been evaluated as completely unconvincing.

[0119] In another embodiment, a consistency measure is introduced as a boundary condition into the planning performed by the planning module, wherein the planning module determines a required consistency measure (e.g. as a confidence measure) or a plurality of required consistency measures in different aspects for the planned trajectory. The planning module only allows trajectories for which a required (e.g. combined) consistency measure or required consistency measures are sufficient to continuously maintain the required reliability represented by the consistency measure or the plurality of consistency measures. For example, for driving behaviors that depend on sensory information of other traffic participants or infrastructure units (i.e. require external sensory information), the planning module can require a minimum consistency measure with respect to the external sensory information.

[0120] As another example, the planning module may require a higher overall consistency for more complex driving maneuvers. If this cannot be achieved, the planning module falls back to less complex, but potentially less popular, maneuvers. Thus, for example, the ego vehicle is prevented from overtaking a vehicle traveling ahead in situations where visibility is poor and therefore the consistency between the radar and the camera is low.

[0121] In summary, a method is provided according to different embodiments, such as in Figure 4 As shown in .

[0122] Figure 4 A flow chart 400 is shown which illustrates a method for controlling a robotic device according to one embodiment.

[0123] At 401 , a control strategy for a robotic device is trained based on a large number of training control situations.

[0124] It should be noted that the control strategy does not necessarily need to directly output a control signal, but may also output a control action at a higher level (such as a trajectory, a road sign, etc.), and then obtain a suitable control signal for the control action.

[0125] In 402, for each control time point in a plurality of control time points:

[0126] In 403 , information about the surroundings of the robot device is obtained for the control time point;

[0127] In 404 , by comparing a control situation with a training control situation, which describes the information determined for the control time point, a first consistency measure of the information determined for the control time point is determined (see, for example, embodiment 1 of the consistency measure);

[0128] ● In 405, by combining the information obtained for the control time point

[0129] ■ a prediction of the state of the surroundings based on information about the surroundings of the robotic device for one or more previous control time points (e.g. trajectory prediction, see e.g. embodiment 2 of the consistency measure),

[0130] ■ with each other (i.e. internal consistency, e.g. measurements or information derived therefrom (e.g. object detection) match each other, e.g. see example 4 of the consistency measure) with prior knowledge (e.g. maps, e.g. see example 3 of the consistency measure), and / or

[0131] ■ with information determined for one or more previous control points in time with information determined for one or more previous control points in time (e.g. object detection with previous object detection, see, for example, Example 4 of the consistency measure)

[0132] Performing a comparison to determine at least one second consistency measure of the information determined for the control time point;

[0133] At 405, by comparing the first consistency metric with at least one second consistency metric

[0134] Combining (fusion) to obtain the consistency measure after the combination; and

[0135] In 406, one or more control actions for the robot device are determined based on the combined consistency measure (and if necessary, the robot device is controlled based on the determined control actions).

[0136] the robot device).

[0137] The consistency measure and the combined consistency measure can be obtained through the analysis module, and the planning module can obtain the control action according to the output of the analysis module.

[0138] For example, a comparison of data or information is understood to mean the determination of a distance in the sense of a corresponding distance measure (or indicator), using which a consistency measure between two corresponding data or information is determined, as described in the above exemplary embodiments of a consistency measure.

[0139] In other words, according to different embodiments, in different aspects, one or more consistency measures are combined into a combined consistency measure, such as the consistency of the current input data with the trajectory prediction, or the consistency of the current input data with the map (i.e., the prior knowledge), and the consistency of the current control state (as reflected in the input data) with the training state. Control is then performed according to the combined consistency measure. If the combined consistency (as the combined consistency measure described by it) is below a predetermined threshold, control can be performed, for example, according to a safety mode, for example, the autonomous vehicle can be driven cautiously.

[0140] For example, one or more consistency measures in the consistency measure may be obtained as a binary cognitive subjective logical opinion. For example, a multimodal trajectory prediction may also be obtained, and its consistency is evaluated based on the measurement (measurement value).

[0141] For example, forecasts can be taken into account in the control. In this context, the determination of a consistency measure for the information determined can also be considered as an evaluation of the forecast, i.e. a low consistency between the forecast and the information determined can indicate not only a low quality of the information determined, but also a poor forecast. In both cases, careful control can be justified.

[0142] Figure 4 The method can be performed by one or more computers having one or more data processing units. The term "data processing unit" can be understood as any type of entity capable of processing data or signals. For example, data or signals can be processed according to at least one (i.e., one or more) special functions performed by a data processing unit. The data processing unit may include an analog circuit, a digital circuit, a logic circuit, a microprocessor, a microcontroller, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a field programmable gate array (FPGA) integrated circuit or any combination thereof or consist of these. Any other means for implementing the various functions described more accurately herein may also be understood as a data processing unit or a logic circuit component. One or more of the method steps described in detail herein may be implemented (e.g., realized) by one or more special functions by a data processing unit, and the special functions are performed by a data processing unit.

[0143] Therefore, the method according to various embodiments is in particular computer-implemented.

[0144] The term "robotic device" may be understood to refer to any technical system having mechanical parts whose movements are controlled, such as computer-controlled machines, vehicles, household appliances, power tools, manufacturing machines, personal assistants or access control systems.

[0145] Different embodiments may receive and use sensor signals from different sensors (e.g., video, radar, lidar, ultrasound, motion, thermal imaging, etc.) as information about the surrounding environment or as a basis for information about the surrounding environment, for example, in order to obtain sensor data on the state of the corresponding system (robotic device and one or more objects in the surrounding environment) and configuration and scene aspects. The sensor data may be processed to obtain information about the surrounding environment. This may include classifying the sensor data or performing semantic segmentation on the sensor data, for example, in order to detect the presence of objects (in the surrounding environment where the sensor data is obtained). The embodiments may be used to train machine learning systems and control robots (e.g., autonomously by a robotic manipulator) to achieve different manipulation tasks in different scenarios. In particular, the embodiments can be applied to control and monitor the execution of manipulation tasks, for example, on an assembly line.

Claims

1. A method for controlling a robotic device (101), the method comprising: training a control strategy for training a robotic device (101) based on a large number of training control situations, and For each control time point in the plurality of control time points: Obtaining information about the surrounding environment of the robot device (101) at the control time point; determining a first consistency measure for the information ascertained for the control time point by comparing a control situation that describes the information ascertained for the control time point with a training control situation; By taking the information obtained for the control time point and a prediction of a state of the surrounding environment based on information about the surrounding environment of the robotic device (101) for one or more previous control time points, With prior knowledge, each other, and / or and information sought for one or more previous control points in time and information sought for one or more previous control points in time, Performing a comparison to determine at least one second consistency measure of the information determined for the control time point; Obtaining a combined consistency metric by combining the first consistency metric with the at least one second consistency metric; as well as Based on the combined consistency measure, one or more control actions for the robotic device (101) are obtained.

2. The method according to claim 1, wherein: The first consistency measure and the at least one second consistency measure are subjective logical opinions, and the combination of the first consistency measure and the at least one second consistency measure is fused in the sense of at least one subjective logical fusion operator (301, 302, 307).

3. The method according to claim 2, wherein: The first consistency metric and the at least one second consistency metric are combined by applying an uncertainty weighted average trust fusion operator (307).

4. The method according to claim 1, wherein: The first consistency metric and the second consistency metric are Dempster Shafer confidence qualities.

5. The method according to claim 1, wherein: The first consistency measure and the second consistency measure are determined by means of Bayesian inference.

6. The method according to any one of claims 1 to 5, wherein: Combining the first consistency metric with the at least one second consistency metric comprises: temporally filtering the first consistency metric and / or the second consistency metric, and combining the first consistency metric or the filtered first consistency metric with the second consistency metric or the filtered second consistency metric.

7. The method according to claim 4, wherein: The first consistency metric and the at least one second consistency metric are subjective logical opinions, and the first consistency metric and / or the at least one second consistency metric are temporally filtered by applying a cumulative trust fusion operator (301, 302).

8. The method according to claim 7, wherein: When performing time filtering on the first consistency metric and / or the at least one second consistency metric, for the corresponding consistency metric, if the result of the conflict degree (303, 304) with the previous filtering result is higher than a predetermined threshold, the current value of the consistency metric is used as the result of time filtering.

9. The method according to any one of the preceding claims 1 to 8, wherein: The first consistency measure and / or the at least one second consistency measure are determined based on a plurality of measured values, wherein the importance of the measured values ​​is reduced based on their relevance by means of a reduced confidence level.

10. The method according to any one of the preceding claims 1 to 9, wherein: The information has a surrounding environment model.

11. The method according to any one of the preceding claims 1 to 10, wherein: Obtaining the one or more control actions according to the combined consistency measure comprises: obtaining the one or more control actions according to a quality function, the quality function evaluating the control actions, and the combined consistency measure entering the quality function.

12. A control device (102) configured to carry out the method according to any one of claims 1 to 11. 13 . A computer program comprising instructions which, when executed by a processor, cause the processor to execute the method according to claim 1 .

14. A computer-readable medium storing instructions, which, when executed by a processor, cause the processor to execute the method according to any one of claims 1 to 11.

Images