Method for controlling a robot device

Abstract

According to various embodiments, a method for controlling a robot device is provided, comprising training a control strategy for the robot device (101) based on a plurality of training control situations and, for each of several control time points, determining information about an environment of the robot device (101) for the control time point, determining a first consistency measure for the information determined for the control time point by comparing a control situation specifying the information determined for the control time point with the training control situations, and determining at least one second consistency measure for the information determined for the control time point.Determining a combined consistency measure by combining the first consistency measure with the at least one second consistency measure and determining one or more control actions for the robot device (101) depending on the combined consistency measure.

Description

[0001] The present disclosure relates to methods for controlling a robotic device.

[0002] Control strategies for robotic devices, such as autonomous vehicles, are typically trained using artificial intelligence based on training data; that is, they are trained for training control situations (or training scenarios). A control strategy trained in this way can then be used under the assumption that it will also perform well for control situations not encountered during training, i.e., that it can generalize sufficiently. With suitable training data, this assumption is usually justified. Nevertheless, it can happen that a control strategy is confronted with a control situation for which it performs poorly because the input data it receives describes a control situation to which it cannot generalize, or because the input data is of low quality.The respective control situations are poorly described. Therefore, approaches are desirable to achieve suitable (e.g., safe) control in such cases.

[0003] The publication “Motion Planning for Connected Automated Vehicles at Occluded Junctions with External Perception” by Johannes Christian Müller, Schriftenreihe des Instituts für Mess-, Regel- und Mikrotechnik Band 41, Universität Ulm, 2022, hereinafter referred to as Reference [1], describes a Subjective-Logic-based method to evaluate the reliability of external perception in automated driving.

[0004] According to various embodiments, a method for controlling a robot device is provided, comprising training a control strategy for the robot device based on a multitude of training control situations and, for each of several control time points. • Gathering information about the robot device's environment for the control timing; • Determining an initial consistency measure for the information obtained for the control point by comparing a control situation that specifies the information obtained for the control point with the training control situations; • Determine at least one second consistency measure for the information obtained for the control time by comparing the information obtained for the control time. ◯ with a prediction for the state of the environment from information about the environment of the robot device for one or more previous control times; ◯ with prior knowledge; ◯ among each other; and / or ◯ with the information determined for one or more previous control times; • Determining a combined consistency measure by combining the first consistency measure with at least one second consistency measure; and • Determining one or more control actions for the robot device depending on the combined consistency measure.

[0005] The method described above enables safe control because it takes into account the case where current information about the environment of the robot device to be controlled is not consistent with regard to various aspects (e.g., with each other, with prior knowledge, or with training situations).

[0006] If low consistency is detected (e.g., a combined consistency measure below a predefined threshold), a safe mode can be used for control (e.g., driving carefully), thus achieving safe control in cases where the information is unsuitable for the respective control system.

[0007] The following are various examples of implementation.

[0008] Exemplary embodiment 1 is a method for controlling a robot, as described above.

[0009] Embodiment 2 is a method according to embodiment 1, wherein the first consistency measure and the at least one second consistency measure are subjective logic opinions, and combining the first consistency measure with the at least one second consistency measure is a fusion in the sense of at least one subjective logic fusion operator.

[0010] Subjective logic allows for the explicit consideration of epistemic uncertainty in consistency assessments. It is therefore suitable for modeling and analyzing situations (in this application, control situations) involving uncertainty and unreliable sources (in this application, for example, sensors). In the following discussion of subjective logic, the standard English terms, particularly for the operators, will be used, as also found, for example, in reference [1].

[0011] Embodiment 3 is a method according to embodiment 2, wherein the first consistency measure and the at least one second consistency measure are combined by applying the Uncertainty-Weighted-Average-Belief fusion operator.

[0012] This allows for the consideration of uncertainty in the consistency assessment and avoids a situation where an (uncertain) assessment indicating good consistency negatively influences the control, e.g., does not lead to cautious driving behavior even when this would be appropriate, because the consistency is actually low.

[0013] Embodiment 4 is a method according to embodiment 1, wherein the first consistency measure and the second consistency measure are Dempster-Shafer belief masses.

[0014] Embodiment 5 is a method according to embodiment 1, wherein the first consistency measure and the second consistency measure are determined by means of Bayesian inference.

[0015] Dempster-Shafer and Bayesian inference represent alternatives to the use of Subjective Logic, which can also provide consistency measures to achieve suitable (e.g. safe) control.

[0016] Exemplary embodiment 6 is a method according to one of the exemplary embodiments 1 to 5, wherein the combination of the first consistency measure with the at least one second consistency measure comprises the temporal filtering of the first consistency measure and / or the second consistency measure and combining the first consistency measure or filtered first consistency measure with the second consistency measure or filtered second consistency measure.

[0017] Temporal filtering makes it possible to filter out jumps in the consistency score caused by an incorrect (e.g., particularly pessimistic) consistency assessment. All, one, or (in the case of multiple secondary consistency measures) a subset of the consistency measures can be filtered before the (actual) combination takes place.

[0018] Embodiment 7 is a method according to embodiment 4, wherein the first consistency measure and the at least one second consistency measure are Subjective-Logic-Opinions and the first consistency measure and / or the at least one second consistency measure are time-filtered by applying the Cumulative-Belief-Fusion-Operator.

[0019] In the case of a jump, the uncertainty of the consistency assessment typically increases significantly and is therefore weighted less by this fusion operator, thus taking into account that the significance of the corresponding consistency measure is typically low immediately after a jump.

[0020] Exemplary embodiment 8 is a method according to exemplary embodiment 7, wherein, when filtering the first consistency measure and / or the at least second consistency measure over time, a current value of the consistency measure is taken as the result of the filtering over time if the result of a degree-of-conflict with a previous filtering result is above a predetermined threshold.

[0021] Embodiment 9 is a method according to one of embodiments 1 to 8, wherein the first consistency measure and / or the at least second consistency measure is determined on the basis of several measured values, wherein the significance of the measured values ​​is reduced depending on their correlation by means of a confidence reduction.

[0022] For example, it can be taken into account that multiple measurements come from the same data source (e.g., from the same sensor) and that errors in the data source can be avoided in distorting the consistency measure.

[0023] Embodiment 10 is a method according to one of embodiments 1 to 9, wherein the information includes an environment model.

[0024] In particular, the consistency of information derived from sensor data, such as object detections or trajectory predictions, can be evaluated.

[0025] Embodiment 11 is a method according to one of embodiments 1 to 10, wherein the determination of the one or more control actions depending on the combined consistency measure comprises the determination of the one or more control actions on the basis of a quality function that evaluates control actions, in which the combined consistency measure is included.

[0026] In other words, for example, possible control actions are evaluated based on a quality function that incorporates the combined consistency measure, and one or more control actions (with the best rating) are selected accordingly. This allows the combined consistency measure to be effectively considered without necessarily requiring hard limits (such as a threshold comparison for consistency).

[0027] Embodiment 12 is a control device configured to perform a method according to one of embodiments 1 to 11.

[0028] Embodiment 13 is a computer program with instructions which, when executed by a processor, cause the processor to perform a method according to one of embodiments 1 to 11.

[0029] Embodiment 14 is a computer-readable medium that stores instructions which, when executed by a processor, cause the processor to perform a method according to one of embodiments 1 to 11.

[0030] In the drawings, similar reference numerals generally refer to the same parts in all the different views. The drawings are not necessarily to scale, with the emphasis generally placed on illustrating the principles of the invention. Various aspects are described in the following description with reference to the drawings. Fig. Figure 1 shows a vehicle. Fig. Figure 2 illustrates the use of a VAE (Variational Autoencoder) to determine a consistency measure. Fig. Figure 3 illustrates a fusion of consistency measures according to one embodiment. Fig. Figure 4 shows a flowchart illustrating a method for controlling a robot device according to one embodiment.

[0031] The following detailed description refers to the accompanying drawings, which illustrate specific details and aspects of this disclosure in which the invention can be implemented. Other aspects may be used, and structural, logical, and electrical modifications may be made without deviating from the scope of the invention. The various aspects of this disclosure are not necessarily mutually exclusive, as some aspects of this disclosure may be combined with one or more other aspects of this disclosure to form new aspects.

[0032] Several examples are described in more detail below.

[0033] Fig. Figure 1 shows vehicle 101.

[0034] In the example of Fig. 1 is a vehicle 101, for example a car or truck, equipped with a vehicle control device (e.g. an Electronic Control Unit (ECU)) 102.

[0035] The vehicle control unit 102 has data processing components, e.g., a processor (e.g., a CPU (central processing unit)) 103 and a memory 104 for storing control software 107, according to which the vehicle control unit 102 operates, and data that is processed by the processor 103. The processor 103 executes the control software 107.

[0036] For example, the stored control software (computer program) contains instructions which, when executed by the processor, cause the processor to perform 103 driver assistance functions or even control the vehicle autonomously.

[0037] The control software 107 is transferred, for example, from a computer system 105 to the vehicle 101, e.g., via a network 106 (or also with the help of a storage medium such as a memory card). This can also happen during operation (or at least when the vehicle 101 is with the user), since the control software 107 is updated to new versions over time.

[0038] The control software 107 can, for example, be trained using machine learning (ML). This means that the control software 107 implements one or more ML models 108 (or "machine learning models") that are trained on the basis of training data, in this example from the computer system 105. The computer system 105 therefore implements an ML training algorithm for training the one or more ML models 108.

[0039] The control software 107 determines control actions for the vehicle (such as steering, braking, etc.) from input data 109 available to it, which contains information about the environment or from which it derives information about the environment (such as by detecting other road users, e.g., other vehicles). This includes, for example, sensor data such as from a camera on the vehicle or information obtained through communication with other vehicles or devices at the roadside.

[0040] In road traffic, it frequently happens that a vehicle finds itself in an unfamiliar situation that was not anticipated during the design phase of the control software, and especially during the training phase of the one or more ML 108 models. Human drivers typically have a gut feeling based on their experience, which leads them to drive particularly cautiously in such situations. Automated vehicles (i.e., control software), on the other hand, typically lack such a mechanism.

[0041] According to various embodiments, a mechanism is provided that enables automated vehicles (or other agents, particularly other types of robotic devices) to react anticipatorily to unusual situations. For example, the mechanism allows an automated vehicle to respond to unknown or critical situations with particularly cautious driving. This can mitigate some unknown, critical situations (e.g., in the sense of ISO 21448 Safety of the Intended Functionality), thus enabling a level of safety that goes beyond the safety defined in ISO 21212 Functional Safety.

[0042] For this purpose, according to various embodiments, the information available to the vehicle (or other agent), i.e., the input data 109 for the control software 107 but also derived data (information about the environment), such as trajectory estimates for other road users or detected objects, is checked for consistency (e.g., current with previous or training data) with respect to at least one aspect. To check the consistency, probabilistic consistency measures are established for the individual aspects. In the case of several aspects considered, the individual aspects are merged into an overall result. The result of the analysis module is passed on to a planning module and influences the driving behavior there, so that the automated vehicle drives more cautiously (and thus more predictively) the less consistent the information available to it is.

[0043] Various approaches can be used to model probabilistic consistency measures, such as subjective logic, Dempster-Shafer theory, or Bayesian inference. Several implementation examples based on subjective logic are explained below. However, these examples can be implemented analogously using the other approaches.

[0044] Example 1 for consistency measure: Consistency measure from comparison with known critical and non-critical situations using a variational autoencoder

[0045] In this embodiment, one aspect is the consistency of the available information (e.g., the input data 109 in the case of the example of Fig. 1) evaluated with regard to experiential knowledge.

[0046] For this purpose, a neural network (e.g., as one or more ML models 108) is first trained, for example, a Variational Auto Encoder (VAE) network, to achieve a compact representation of the input data in a feature space (e.g., the input data has several data points or data elements, each of which is mapped to a feature vector).

[0047] Fig. Figure 2 illustrates the use of a VAE 201 to determine a consistency measure.

[0048] For training the VAE 201, a training dataset 202 is used, which contains traffic scenarios classified as critical and non-critical. The data points of the training dataset are clustered based on the feature points in feature space 203 to which they are mapped (i.e., the clustering is performed according to a metric in feature space 203). For example, the feature point to which an input data point (i.e., a data point of the training dataset during training) is mapped is the midpoint of the distribution that the VAE 201 provides for the input data point. Subsequently, a Gaussian fit of the data in the respective clusters is performed, so that the resulting probability density distributions (i.e., fitted Gaussian distributions) represent the probability of membership in the respective cluster.Accordingly, the probability of a data point belonging to a cluster is derived directly from the Mahalanobis distance of the feature point of the data point to the corresponding cluster.

[0049] Online (i.e., during operation, in the present application example during vehicle control), the input data 109 (e.g., perception information) are transformed into the feature space 203 using the encoder of the VAE 201, and a consistency measure is derived from it, for example, as follows:

[0050] The consistency measure for a situation s, described, for example, by a data point of the input data 109 (i.e., the measure of consistency with the training data 202), is expressed in this embodiment by a binomial subjective logic opinion ω(b,d,u,a). Subjective logic is a mathematical theory that extends classical probability theory. The binomial epistemic subjective logic opinion ω(b,d,u,a) is calculated for the situation s, described by a data point of the input data 109, as follows: b=maxCuncriticalp(s∈Cuncritical),d=maxCuncriticalp(s∈Cuncritical),u=1−b−d,a=[aprior,1−aprior]

[0051] Here, b denotes the belief measure, which is derived from the maximum probability of the situation s (i.e., the corresponding data point) belonging to a cluster C. unkritischis formed from situations labelled as uncritical, d the disbelief measure (or “doubt measure”), which is derived from the maximum probability of the situation s (i.e., the corresponding data point) belonging to a cluster C kritisch The epistemic uncertainty is determined by identifying situations labelled as critical. To identify these different types of clusters, 202 data points in the training dataset are marked (i.e., "labeled") as critical (i.e., representing a critical situation) or non-critical (i.e., representing a non-critical situation).

[0052] The closer a data point (or its associated feature point) is to a cluster (e.g., in the sense of the Mahalanobis distance to the Gaussian distribution fitted to the cluster), the greater the probability of membership and thus the respective belief measure.

[0053] The prior distribution is determined by the parameter a priorThis parameter is characterized by the ability to contribute expert knowledge or prior knowledge. Without specific prior knowledge, the parameter a prior set to 0.5, which in subjective logic theory represents the absence of prior knowledge. In the following, a prior also simply denoted by 'a'.

[0054] The approach of Fig. 2 thus enables the determination of a Subjective-Logic opinion about the criticality of a current situation s for a criticality estimation using Subjective-Logic.

[0055] Due to appropriate normalization, the sums of the individual probabilities b and d do not add up to one, but leave a residual probability u that a new data point does not belong to any known cluster. This probability constitutes the statistical uncertainty u of the subjective logic opinion.

[0056] As an alternative to a VAE network 201, a Gaussian Mixture Model, a Latent Dirichlet Allocation, a Dirichlet Bolzman Machine, a Deep Belief Network, an Adversarial Generative Neural Network, or variations of Auto Encoder networks can also be used. Furthermore, instead of using the feature space 203 (Latent Space), i.e., determining the consistency measure through operations in the feature space 203 as described above, the decoder 204 of the VAE 201 can also be used to determine an error measure, which can then also be transformed into an Epistemic Subjective-Logic Opinion (which can be used as a consistency measure for the respective input data point).

[0057] Example 2 for consistency measure: Reliability of trajectory prediction through consistency testing

[0058] In this embodiment, the reliability of the trajectory prediction for other road users is (optionally additionally) evaluated by checking the consistency of cached, past predictions (which the control software 107 has determined, for example, from previous input data 109) with the current and / or previous input data 109 (i.e., current or past measurements of the actual behavior of the other road users). For trajectory prediction, the control software includes, for example, a prediction module (e.g., as one or more machine learning models), such as a multiple trajectory prediction neural network.

[0059] In this case (now for trajectory predictions for other road users with input data 109 such as current or previous measurements of the actual trajectories of the other road users), the consistency measure can again be formulated as a binomial epistemic subjective logic opinion.

[0060] For example, previous trajectory predictions (predicted trajectory hypotheses) are represented as probability distributions, and the probability that a currently measured trajectory of a road user (or, depending on the application, another object expressed by one or more measured states such as waypoints) belongs to the trajectory hypothesis of a previous prediction that is closest to it is chosen as the belief measure b. The probability that the measured trajectory does not fit any trajectory hypothesis of the previous prediction is chosen as the disbelief measure d, with the epistemic uncertainty u being, as before, a remainder equal to one.

[0061] Alternatively, a frequentist subjective logic opinion can be generated by representing each trajectory hypothesis as trajectory points (waypoints) over time and checking whether the distance of a measured state (i.e., a state from a measured trajectory) to the nearest trajectory point falls below or exceeds a predefined threshold. These binary results form a sample for which the frequentist subjective logic opinion is constructed via the relative frequency. See, for example, reference [1] for the special case of a single-trajectory prediction.

[0062] A trajectory prediction for one or more road users can generally be viewed as an environmental model (i.e., a model of the surroundings) of a robotic device (in this case, a vehicle). This environmental model can be determined for each point in a sequence of points in time (e.g., control points, for each of which one or more control actions are selected).

[0063] Example 3 for consistency measure: Consistency of perception with semantic map

[0064] In this embodiment, the consistency between the input data 109 (e.g., perception information) and a map is checked (optionally additionally). For this purpose, perception information from various vehicle sensors is, for example, converted into a bird's-eye view representation and compared to a digital map stored in the vehicle. Using preprocessing modules such as lane detection, a consistency measure can then be determined as described above.

[0065] The consistency of input data 109 with a map can be rated as low, for example, if the input data 109 provides the information that I am moving the vehicle over a place where, according to the map, there is a building.

[0066] A map can be considered an example of reference information about the environment that remains constant across multiple control points (since the map stays the same until it is updated). This reference information thus represents (potentially static) prior knowledge for the current control point. Prior knowledge can be understood as information that is already known before a specific task is performed (in this case, the control at the control points, or at least the current control point). In this particular application, this could be information known before the start of a journey or a specific route, which helps the vehicle to better understand its surroundings.

[0067] Example 4 for consistency measure: Consistency of detected objects from different sensor modalities

[0068] In this embodiment, the consistency of sensor data elements (e.g., multiple camera images) contained in the input data 109 is evaluated (optionally additionally) by assessing, for example, the consistency of object detections from different camera images (or sensor data elements from different sensors) captured simultaneously, or the object detections over time (each from one camera image of a sequence of camera images). For this purpose, the object detections of any two sensor data elements (from different sensors or from different times, one of which corresponds to the current input data 109) within the common field of view are checked for consistency (i.e., a measure is determined of how well the object detections for the two sensor data elements match). In the example of camera images, this can be done, for instance, by evaluating features (e.g.,Each detected object is surrounded by a tolerance, and the number of features in one camera image that have corresponding matches in another camera image is counted (spatial matching). Mismatches (features without a corresponding feature) can also be counted, and a subjective logic opinion can then be determined by dividing the number of matches and mismatches by the total number of features plus two. The higher the projected probability, the better. Additionally, newly added object hypotheses from a (current) time step can be checked for consistency with the time-filtered objects from the previous time step. This can be done by projecting object hypotheses from the previous time step onto the current time step and counting how many objects from the previous time step can be associated with object hypotheses from the current time step (belief).Then, the number of objects that could neither be associated nor originate from a "spawn zone"—that is, an area where new objects are typically expected to "emerge" (disbelief)—is counted. The uncertainty is obtained from the total number of associations. The higher the projected probability, the better. The consistency measure can then be determined as described for the examples above.

[0069] The preceding examples of the consistency measure describe various ways in which a consistency measure for current input data 109 (i.e., for information determined for a given (“current”) control time) can be determined with respect to various aspects (possibly based on derived data such as object detection).

[0070] According to various embodiments, several consistency measures (e.g., determined according to several of the embodiments described above, i.e., consistency measures with regard to several aspects) are determined (e.g., by the control software) and these are fused into a "combined" (or "fused" or "overall") consistency measure.

[0071] To further improve the result and increase its explanatory power, this can involve filtering over time (i.e., not only consistency measures for information about the environment for a current control point are included, but also consistency measures (possibly the combined consistency measure) for one or more earlier (i.e., previous) control points (in a sequence of control points).

[0072] Fig. Figure 3 illustrates a fusion of consistency measures according to one embodiment.

[0073] In the example of Fig. 3. A fusion mechanism is used in which a consistency measure in the form of a subjective logic opinion ω is used. rel regarding the reliability of perception (see examples for consistency measure 3 and 4) and a consistency measure in the form of a subjective logic opinion regarding the criticality of the situation from experiential knowledge ω SC (see embodiment 1 for the consistency measure) are merged (due to the temporal filtering that is carried out, each indexed with the control time index k).

[0074] First, the consistency measures ω are determined. rel (k) and ω SC (k) using a respective application of the cumulative belief fusion operator 301, 302 with their respective value from the previous time step ω rel (k-1) or ω SC(k-1) is fused and, using a respective application of the degree-of-conflict operator 303, 304, is made consistent with their respective value from the previous time step ω. rel (k-1) or ω SC (k-1) checked.

[0075] The cumulative belief fusion operator 301, 302 is the consistency measure ω in the present example. rel =(b rel , d rel , u rel , a rel ) and ω SC =(b SC , d SC , u SC , a SC ) (Index k is omitted here for simplicity) given by ωrel⊕ωSC=(brel+uSC+bSCurelurel+uSC−ureluSC,drel+uSC+dSCurelurel+uSC−ureluSC,ureluSCurel+uSC−ureluSC,areluSC+aSCurel−(arel+aSC)ureluSCurel+uSC−ureluSC)

[0076] The degree of conflict in this example is given by DC(ωrel,ωSC)=12(|brel+urelarel−(bSC+uSCaSC)|+|drel+urel(1−arel)−(dSC+uSC(1−aSC))|)(1−urel)(1−uSC)

[0077] If the degree of conflict falls below a predefined threshold, the preceding and current subjective logic opinions are considered consistent, and the value generated by the respective cumulative belief fusion operator 301, 302 (i.e., the fusion of the preceding and current subjective logic opinions) is passed on by a respective selection function 305, 306. Otherwise, the respective selection function 305, 306 passes on the current value of the subjective logic opinion.

[0078] Here, the respective consistency measure can be taken from the previous time step ω. rel (k-1) or ω SC(k-1 ) itself uses the fusion result of the cumulative belief fusion operator 301, 302 for the previous time step, so that the consistency measures are aggregated over time (except for "resetbacks" if the current consistency measure is not consistent with the consistency measure from the previous time step, since then, as described, the respective selection function 305, 306 passes on the current consistency measure).

[0079] The outputs of the selection functions 305, 306 are merged into a fused (total) subjective logic opinion ω using the uncertainty-weighted average belief fusion operator. total (k) summarized.

[0080] Are the outputs of the selection functions 305, 306 with ω A = (b A , d A , u A , a A ) or ω B = (b B , d B , u B , a B) denoted, the Uncertainty-Weighted-Average-Belief fusion operator 307 is given by ωA⊕^ωB=(bA(1−uA)uB+bB(1−uB)uA(1−uB)uA+(1−uA)uB,dA(1−uA)uB+dB(1−uB)uA(1−uB)uA+ (1−uA)uB,((1−uA)+(1−uB))uAuB(1−uB)uA+(1−uA)uB,aA(1−uA)+aB(1−uB)(1−uA)+(1−uB))

[0081] General versions of formulas (2), (3) and (4) are given in reference [1].

[0082] If more than two consistency measures are to be combined, the fusion mechanism of Fig. 3, for example, can be arcaded.

[0083] The one in which with reference to Fig. The temporal filtering contained in the fusion mechanism described in point 3 makes it possible to aggregate information over long time windows (if, as described above, the respective consistency measure from the preceding time step ω rel (k-1) or ω SC(k-1 ) itself uses the fusion result of the cumulative belief fusion operator 301, 302 for the preceding time step) and thereby (potentially) achieves high noise suppression.

[0084] On the other hand, the temporal filtering, through its switching behavior (via selection functions 305, 306, depending on the consistency of the successive consistency measures), can react quickly to abrupt changes in the consistency measures. In the case of a jump, the uncertainty of the result initially increases sharply and is therefore weighted less heavily by the fusion operator (i.e., the uncertainty-weighted average-belief fusion operator according to equation (4)). This effect is desirable because, immediately after a jump, the significance of the corresponding consistency measure is usually not yet high.

[0085] If the correlation between individual data points in the input data 109 is not negligibly small but estimable, the consistency measures can be corrected using the trust discounting operator. For example, if a single trajectory is checked against multiple measurements (i.e., data points in the input data 109), the individual measurements (data points) are not statistically independent, as they may refer to the same data source. Nevertheless, multiple measurements provide added value. The correlation could be estimated using expert knowledge. This also prevents the epistemic uncertainty u from decreasing too rapidly and the fusion (i.e., the cumulative belief fusion operator according to equation (2)) from taking newly arriving data into account less and less.

[0086] The fused subjective logic opinion ω total(k) is converted into a probability (e.g., by the control software). For implementations based on subjective logic, the probability can either be derived directly from the projected probability p. ant = b + au from the subjective logic opinion, or by integrating the Dirichlet distribution corresponding to the subjective logic opinion over a given confidence interval. A softmax function can be used for confidence scores. The probability p thus generated will be discussed below. ant (k) is considered the fused consistency measure for the k-th control point, where p is also referred to below. ant (t) is used, where t is the time corresponding to a given control point in time.

[0087] The fused consistency measure, for example, is the result of an analysis module and is further processed by a planning module (both modules are, for example, part of the control software 107). The analysis module receives input data 109, such as perception information (from sensors or other vehicles) and, if applicable, prior knowledge (like a map), and delivers the fused consistency measure (and, if applicable, the individual consistency measures). The planning module then generates a trajectory (i.e., a planned route) from this data, and the control unit steers the vehicle to follow the generated trajectory.

[0088] In parallel to the analysis module, one or more additional analysis modules can be provided to process the perception information for the planning module. The information (e.g., perception information) fed to the analysis module can be the result of preprocessing; that is, the analysis module does not (only) need to receive the input data itself, but can also (optionally additionally) receive the result of preprocessing. For example, lane detection can support the consistency check of the map and vehicle perception by the analysis module. The planning module can have additional inputs besides the output from the analysis module.

[0089] In the planning module, the fused consistency measure p is used in the present application example of a vehicle control system. ant(t) is used to influence driving behavior. This can be done in various ways. The planning module can also be structured in different ways. For example, the planning module can be subdivided into strategic, tactical, and operational planning. One or more of these components of the planning module can be represented by the fused consistency measure p. ant (t) are influenced.

[0090] The planning module can, for example, compare the combined consistency measure with a threshold (which is defined depending on the scaling of the consistency measure or the respective metric by which it is determined) and switch to a safe control mode depending on whether the consistency indicated by it is below a minimum consistency specified by the threshold.

[0091] In one embodiment, the planning module takes the fused consistency measure as one of its inputs and uses an optimization-based approach to determine a trajectory, which is then output (and according to which the control unit 102 controls the vehicle). For example, the fused consistency measure is fed into the optimization function as follows: Jges=Jgew+want∫v(t)⋅(1−pant(t))dt

[0092] J is involved ges a modified quality function, which, in addition to the terms of a conventional quality function J gew contains an additive term (i.e., a one-dimensional penalty term in the cost functional) consisting of the optimization weight w ant weighted integral over the product of the planned velocity v(t) and the complementary probability to p ant (t) exists. By minimizing J gesConsequently, trajectories are preferred which, at low consistency (close to zero), lead to the selection of a slower trajectory for the automated vehicle. For a very high consistency measure (close to one), however, the additive term becomes vanishingly small, thus reducing the planning costs according to the conventional efficiency function J. gew predominate.

[0093] In another embodiment, the planning module also uses an optimization-based approach to determine a trajectory. However, unlike the embodiment described above, this module incorporates a multidimensional probability distribution f(x,t) that describes the probability that a specific state x is invalid at a specific time t. This probability distribution does not need to consider all aspects under which a state can be invalid. For example, the probability distribution can represent the current and predicted states of other road users and be provided by another module. The integral of the probability distribution along the planned trajectory is incorporated into the planning module's performance function, ensuring that the resulting trajectory has the lowest possible probability of containing invalid states.Furthermore, the integral of the probability distribution over the trajectory is limited to a maximum permissible residual probability.

[0094] This probability distribution incorporates the merged consistency measure. For example, an original version of the probability distribution f(x, t) provided by the other module is modified as follows: f˜(x,t,pant)=n⋅[(1−pant(t))+pant(t)⋅f(x,t)], where n is a normalization constant. Since f̃(x,t,p) ant ) with decreasing probability of consistency p ant (t), i.e., the probability that a trajectory contains inadmissible states increases, is in this case optimized based on the probability distribution f̃(x,t,p). ant ) generates a trajectory that represents a more cautious overall behavior (than when f(x, t) is used). In the limiting case p ant(t) = 0 would therefore mean that f would ignore all distribution information from f, since this was assessed as completely meaningless.

[0095] In another embodiment, the consistency measure is incorporated as a boundary condition into the planning performed by the planning module. The planning module determines a required consistency measure (e.g., a confidence measure) or measures with respect to various aspects for the planned trajectory. By allowing only trajectories for which one or more required consistency measures are sufficient, the planning module can maintain the required reliability represented by the consistency measure(s). For example, the planning module can require a minimum consistency measure with respect to external perception information for driving behavior that relies on perception information from other road users or infrastructure units.

[0096] As another example, the planning module for more complex driving behaviors might require higher overall consistency. If this is not achieved, the planning module reverts to a less complex, but possibly less preferred, behavior. For example, this would prevent the ego vehicle from overtaking a vehicle ahead in poor visibility conditions, where there is low consistency between the radar and camera data.

[0097] In summary, a method is provided according to various embodiments, as described in Fig. 4 shown.

[0098] Fig. Figure 4 shows a flowchart 400, which represents a method for controlling a robot device according to one embodiment.

[0099] In 401, a control strategy for the robot device is trained based on a variety of training control situations.

[0100] It should be noted that the control strategy does not necessarily have to directly output control signals, but can also output control actions at a higher level (e.g. trajectories, waypoints, etc.), for which suitable control signals are then determined.

[0101] In 402, for each of several control points • in 403 information about an environment of the robot device for the control time determined; • in 404 a first consistency measure for the information determined for the control time is determined by comparing a control situation that specifies the information determined for the control time with the training control situations (see e.g. embodiment 1 for the consistency measure); • in 405 at least a second consistency measure for the information determined for the control time by comparing the information determined for the control time • with a prediction for the state of the environment from information about the environment of the robot device for one or more previous control times (e.g. a trajectory prediction, see e.g. embodiment 2 for the consistency measure). • with prior knowledge (such as a map, see e.g. embodiment 3 for the consistency measure) among themselves (i.e., internal consistency, such as that measurements or derived information like object detections match each other, see e.g. embodiment 4 for the consistency measure) and / or • determined with the information obtained for one or more previous control times (such as an object detection, see e.g. example 4 for the consistency measure) • in 405 a combined consistency measure is determined by combining (fusion) the first consistency measure with at least one second consistency measure; and • in 406 one or more control actions for the robot device depending on the combined consistency measure (and, if necessary, control of the robot device according to the determined control actions) are determined.

[0102] The consistency measures and the combined consistency measure can be determined by an analysis module, and depending on its output, a planning module can determine the control actions.

[0103] Comparing data or information includes, for example, determining a distance in the sense of a respective distance measure (or metric) with which the consistency measure between the two respective data or information is determined, as described in the above examples of the consistency measure.

[0104] In other words, according to various embodiments, one or more consistency measures regarding different aspects, such as consistency of current input data with trajectory predictions, or consistency of current input data with a map (i.e., prior knowledge) with consistency of a current control situation (as reflected in input data) with training situations, are combined into a combined consistency measure. Control is then performed based on this measure. If the combined consistency (as indicated by the combined consistency measure) is below a predefined threshold, control can, for example, be implemented in a safe mode; for instance, an autonomous vehicle can drive cautiously.

[0105] One or more of the consistency measures can be determined, for example, as binomial epistemic subjective logic opinions. Alternatively, multimodal trajectory predictions can be determined, the consistency of which is evaluated based on measurements (measured values).

[0106] Predictions can be considered, for example, in control mechanisms. In this respect, determining a consistency measure for the information obtained can also be seen as an evaluation of the prediction; that is, low consistency between prediction and information obtained can indicate both poor quality of the information obtained and poor prediction. In both cases, cautious control may be advisable.

[0107] The procedure of Fig.4. This can be performed by one or more computers with one or more data processing units. The term "data processing unit" can be understood as any type of entity that enables the processing of data or signals. The data or signals can, for example, be handled according to at least one (i.e., one or more than one) special function performed by the data processing unit. A data processing unit can include or be composed of an analog circuit, a digital circuit, a logic circuit, a microprocessor, a microcontroller, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), an integrated circuit, a programmable gate array (FPGA), or any combination thereof.Any other method for implementing the respective functions described in more detail herein may also be understood as a data processing unit or logic circuit arrangement. One or more of the process steps described in detail herein may be executed (e.g., implemented) by a data processing unit through one or more special functions performed by the data processing unit.

[0108] According to various explanations, the process is therefore implemented primarily using computers.

[0109] The term "robot device" can be understood as referring to any technical system (with a mechanical part whose movement is controlled), such as a computer-controlled machine, a vehicle, a household appliance, a power tool, a manufacturing machine, a personal assistant, or an access control system.

[0110] Various embodiments can receive and use sensor signals from different sensors, such as video, radar, LiDAR, ultrasound, motion, thermal imaging, etc., as information about an environment or as a basis for obtaining information about the environment. This can be done, for example, to obtain sensor data regarding the state of the respective system (robot device and object or objects in the environment) and configurations and scenarios. The sensor data can be processed to determine information about an environment. This can include classifying the sensor data or performing semantic segmentation on the sensor data, for example, to detect the presence of objects (in the environment where the sensor data was obtained). Embodiments can be used to train a machine learning system and control a robot, for example.They can be used autonomously by robot manipulators to accomplish various manipulation tasks under different scenarios. In particular, embodiments are applicable to controlling and monitoring the execution of manipulation tasks, e.g., in assembly lines. QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited non-patent literature

[0000] Motion Planning for Connected Automated Vehicles at Occluded Junctions with External Perception” by Johannes Christian Müller, Publications of the Institute for Measurement, Control and Microtechnology Volume 41, University of Ulm, 2022

[0003]

Claims

[1] A method for controlling a robot device (101), comprising: Training a control strategy for the robot device (101) based on a plurality of training control situations; and For each of several control points Determining information about an environment of the robot device (101) for the control time; Determining a first consistency measure for the information determined for the control time by comparing a control situation indicating the information determined for the control time with the training control situations; Determining at least one second consistency measure for the information determined for the control time by comparing the information determined for the control time with a prediction for the state of the environment from information about the environment of the robot device (101) for one or more previous control times; with prior knowledge; among themselves; and / or with the information determined for one or more previous control points in time with the information determined for one or more previous control points in time; Determining a combined consistency measure by combining the first consistency measure with the at least one second consistency measure; and Determining one or more control actions for the robot device (101) depending on the combined consistency measure. [2] The method of claim 1, wherein the first consistency measure and the at least one second consistency measure are subjective logic opinions and combining the first consistency measure with the at least one second consistency measure is a fusion in the sense of at least one subjective logic fusion operator (301, 302, 307). [3] The method of claim 2, wherein the first consistency measure and the at least one second consistency measure are combined by applying the uncertainty weighted average belief fusion operator (307). [4] The method of claim 1, wherein the first consistency measure and the second consistency measure are Dempster-Shafer belief masses. [5] The method of claim 1, wherein the first consistency measure and the second consistency measure are determined by Bayesian inference. [6] Method according to one of claims 1 to 5, wherein combining the first consistency measure with the at least one second consistency measure comprises temporally filtering the first consistency measure and / or the second consistency measure and combining the first consistency measure or filtered first consistency measure with the second consistency measure or filtered second consistency measure. [7] The method according to claim 4, wherein the first consistency measure and the at least one second consistency measure are subjective logic opinions and the first consistency measure and / or the at least one second consistency measure are temporally filtered by applying the cumulative belief fusion operator (301, 302). [8] Method according to claim 7, wherein during the temporal filtering of the first consistency measure and / or the at least second consistency measure for the respective consistency measure, a current value of the consistency measure is taken as a result of the temporal filtering if the result of a degree of conflict (303, 304) with a previous result of the filtering is above a predetermined threshold value. [9] Method according to one of claims 1 to 8, wherein the first consistency measure and / or the at least second consistency measure is determined on the basis of a plurality of measured values, wherein the significance of the measured values ​​is reduced by means of a confidence reduction depending on their correlation. [10] Method according to one of claims 1 to 9, wherein the information comprises an environment model. [11] Method according to one of claims 1 to 10, wherein determining the one or more control actions depending on the combined consistency measure comprises determining the one or more control actions based on a quality function that evaluates control actions, into which the combined consistency measure is included. [12] Control device (102) which is arranged to carry out a method according to one of claims 1 to 11. [13] A computer program comprising instructions which, when executed by a processor, cause the processor to perform a method according to any one of claims 1 to 11. [14] A computer-readable medium storing instructions which, when executed by a processor, cause the processor to perform a method according to any one of claims 1 to 11.

Images