Method, computer program and apparatus for determining the actions of an automation device based on the undentification of the environmental

By utilizing the confidence information of environmental states and the statistical characteristics of intermediate states, combined with unscented transformation technology, the problem of action decision-making for automated equipment under uncertain environmental states is solved, thereby improving the accuracy and safety of action selection.

CN118984996APending Publication Date: 2024-11-19SONY GROUP CORP
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202380028751.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-03-31
Filing Date
2023-03-29
Publication Date
2024-11-19

AI Technical Summary

Technical Problem

Existing technologies struggle to effectively handle the action decisions of automated equipment under uncertain environmental conditions, especially when environmental condition measurements have limited confidence levels, leading to inaccurate action selection or potential safety hazards.

Method used

By using confidence information of environmental states and statistical properties of intermediate states, combined with unscented transformation techniques, the actions of automated equipment are determined. This includes using sigma point feedback to the policy library and performing nonlinear transformations to handle the uncertainty of state estimation.

Benefits of technology

It improves the accuracy and safety of automated equipment in selecting actions in the real world, especially when environmental conditions are uncertain, enabling it to better select the best or safest actions and reduce the risk of incorrect decisions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118984996A_ABST
    Figure CN118984996A_ABST
Patent Text Reader

Abstract

Examples relate to a method, computer program and apparatus for determining an action of an automation device based on an environmental state of the automation device. A method for determining an action of an automation device based on an environmental state of the automation device. The method comprises: obtaining information about an environmental measurement, the measurement having a limited confidence; estimating information about an environmental state based on the information about the environmental measurement, the information about the environmental state including information about a confidence of the environmental state; determining a representation of the environmental state based on the information about the environmental state and based on the information about the confidence of the environmental state, the representation of the environmental state comprising two or more intermediate states representing the environmental state and the confidence of the environmental state; and determining information about the action of the automation device based on the representation.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] Examples relate to methods, computer programs and apparatus for determining actions of automation devices based on environmental states of the automation devices, and more particularly, but not exclusively, to the concept of statistical information for implementing determined actions of automation devices. Background Art

[0002] Artificial intelligence and machine learning concepts have been developed and applied in many different technical fields, for example, in the field of autonomous vehicles or industrial robots. For example, for the purposes described later, training data is used to train a neural network, wherein the later performance of the trained neural network also depends on the quality of the training and the quality of the training data. The trained neural network provides its output based on input from the environment (for example, based on sensor data from the environment). Some concepts utilize continuous learning or training, for example, reinforcement learning can use rewards or other quality measures to judge the decisions made by the neural network. Based on the rewards or quality measures, the neural network keeps learning about its decisions, and thus a continuous learning or training concept can be established. Summary of the invention

[0003] Examples are based on the discovery that statistical properties of environmental data can be used to improve the use of machine learning methods. For example, confidence information about the environment can be used to generate or determine an improved output of a neural network. Another discovery is that the state of the environment can be represented using two or more intermediate states that also represent statistical properties of the environment. These intermediate states can then be used to determine multiple actions or outputs of the neural network, which also have statistical properties. Due to the nonlinear operation of the neural network, the statistical properties of the actions or outputs are different from the statistical properties of the environment, for example, decisions are made by the neural network. Examples can be based on using two or more intermediate states as representations of the state of the environment to provide improved actions or outputs.

[0004] One example relates to a method for determining an action of an automation device based on an environmental state of the automation device. The method includes: obtaining information about an environmental measurement result, the measurement result having a limited confidence. The method also includes: estimating information about the environmental state based on the information about the environmental measurement result. The information about the environmental state includes information about the confidence of the environmental state. In addition, the method includes: determining a representation of the environmental state based on the information about the environmental state and based on the information about the confidence of the environmental state. The representation of the environmental state includes two or more intermediate states representing the environmental state and the confidence of the environmental state. For example, the confidence of the state represents the strength of evidence supporting each possible state. The method also includes: determining information about the action of the automation device based on the representation. The example may allow for determining an improved action by considering a representation of the environment, which is based not only on the environmental state but also on the confidence of the environmental state.

[0005] Further examples may also include: determining confidence information of the action. The confidence of the action may enable further judgments or measurements in the system, such as specific manipulations in low confidence situations and improvements in training data.

[0006] For example, each of the two or more intermediate states includes one or more sigma points representing statistical characteristics of information about the state of the environment. The two or more sigma points may form a statistical basis for determining an action also based on the statistical characteristics of the environment.

[0007] The method may also include: using one or more strategies to obtain two or more intermediate actions based on two or more intermediate states. The strategy can implement a decision process and provide a nonlinear transformation from the intermediate state to the intermediate action. In an example, the strategy can be implemented as a machine learning or neural network that can make decisions on actions.

[0008] For example, the method can use the same strategy to obtain an intermediate action for each intermediate state. In an example, multiple copies of the same strategy can be used in the strategy library. In this way, the same decision process can be used in parallel for two or more intermediate states that produce two or more intermediate actions. In addition, the resulting distribution of intermediate actions can represent a transformation of the distribution of intermediate states through the strategy. The distribution of intermediate actions can serve as a basis for determining confidence information for the determined action.

[0009] For example, the strategy may be influenced by reinforcement learning or evolutionary search. The one or more strategies may involve nonlinear transformations to obtain two or more intermediate actions based on two or more intermediate states. Therefore, the decision process can be implemented using one or more strategies. In addition, the statistical properties of the intermediate states may be completely different from the statistical properties of the intermediate actions. The example can achieve the determination of the statistical properties of the intermediate actions, although the strategy is nonlinear. The method may also include: determining the statistical properties of the distribution of two or more intermediate actions. This enables the determination of the confidence level about the action.

[0010] In some examples, the method includes determining a statistical property of the two or more intermediate actions using an unscented transformation. The unscented transformation may enable an efficient means for determining a statistical property of an outcome of a nonlinear strategy.

[0011] Determining the information about the action may also be based on statistical characteristics of the distribution of two or more intermediate actions, and the statistical characteristics of the distribution of the two or more intermediate actions include confidence information of the intermediate actions. For example, determining the information about the action may be based on confidence information of the intermediate actions.

[0012] In at least some examples, the method may also include determining the information about the safety action as the information about the action if the confidence information of the intermediate action indicates that the confidence level of the intermediate action is below a predefined confidence threshold. In this way, if the confidence level of the action is too low, this situation can be detected and safety measures can be applied.

[0013] In a further example, further training of the policy may be applied, wherein the training of the policy is influenced by intermediate states identified for intermediate actions that exhibit a predefined statistical characteristic. For example, the predefined statistical characteristic is a confidence threshold.

[0014] The automated device may be an autonomous vehicle or an industrial robot, and the action may be a controlled action for the autonomous vehicle or the industrial robot. The action may include or correspond to one or more elements of the group of: manipulation, movement, acceleration, deceleration, steering command, stop command, and emergency command. Thus, in an example, an emergency command or safety action may be a command to stop the automated device.

[0015] A further example is a computer program with a program code for performing any of the methods described herein, when the computer program runs on a computer, a processor or a programmable hardware component.

[0016] Another example is an apparatus for controlling an automated device, the apparatus comprising a control unit or a module for performing one of the methods described herein. Yet another example is an automated device comprising an example of the apparatus, such as an automated or self-driving vehicle or an industrial robot. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Some examples of apparatus and / or methods will be described below, by way of example only, and with reference to the accompanying drawings, in which:

[0018] Figure 1 A block diagram illustrating a flow chart of an example of a method for determining an action of an automation device based on an environmental state of the automation device;

[0019] Figure 2 A block diagram showing an example of an apparatus for controlling an automation device and an example of an automation device;

[0020] Figure 3 Shows how the learned policy in the example uses the state of the environment to determine the best action;

[0021] Figure 4 An exemplary implementation of using a state estimator to determine a point estimate of the current state as input to a learned policy is shown;

[0022] Figure 5describes an example of using a set of sigma points from an estimated distribution of states, which are fed back into a policy library;

[0023] Figure 6 shows how, in an example, a policy is first trained in simulation with perfect state information and then deployed in the real world using a probabilistic state estimator;

[0024] Figure 7 A block diagram illustrating a flow diagram of an exemplary implementation in which, after computing a distribution of actions using an unscented transformation, an action that accounts for state estimate uncertainty is selected using a resulting estimate of uncertainty;

[0025] Figure 8 A block diagram illustrating a flow chart of an exemplary implementation in which, after calculating the distribution of actions using an unscented transformation, the resulting estimate of uncertainty is used to trigger an emergency stop;

[0026] Fig. 9 Still another block diagram illustrating a flow diagram of an exemplary implementation in which, after computing an action distribution using an unscented transform, a resulting estimate of uncertainty is used to decide whether to commit to an action or wait for more state evidence; and

[0027] Fig.10 A block diagram illustrating a flowchart of an example implementation in which, after computing a distribution of actions using an unscented transformation, actions outside the resulting distribution are used for further training of a bias policy. DETAILED DESCRIPTION

[0028] Some examples are now described in more detail with reference to the accompanying drawings. However, other possible examples are not limited to the features of these embodiments described in detail. Other examples may include modifications of features and equivalents and substitutions of features. In addition, the terms used to describe certain examples in this article should not limit other possible examples.

[0029] In the description of the accompanying drawings, the same or similar reference numerals represent the same or similar elements and / or features, which may be the same or implemented in a modified form while providing the same or similar functions. For clarity, the thickness of the lines, layers and / or regions in the drawings may also be exaggerated.

[0030] When two elements A and B are combined using "or", this should be understood to disclose all possible combinations, i.e., only A, only B and A and B, unless otherwise explicitly defined in individual cases. As an alternative expression of the same combination, "at least one of A and B" or "A and / or B" can be used. The same applies to combinations of more than two elements.

[0031] If singular forms such as "a", "an", and "the" are used, and the use of only a single element is not explicitly or implicitly defined as mandatory, additional examples may also use several elements to implement the same functionality. If a function is described below as being implemented using multiple elements, additional examples may use a single element or a single processing entity to implement the same functionality. It should also be understood that the terms "include", "including", "comprise" and / or "comprising" when used describe the presence of specified features, wholes, steps, operations, processes, elements, parts and / or groups thereof, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, processes, elements, parts and / or groups thereof.

[0032] Figure 1 A block diagram of a flow diagram of an example of a method 10 for determining an action of an automation device based on an environmental state of the automation device is shown. The method 10 includes obtaining 12 information about environmental measurements. The measurements have a limited confidence level. For example, obtaining 12 may include sensing certain physical parameters or quantities in the environment that are subject to noise and distortion. These parameters may include temperature, pressure, humidity, radiation, mechanical stress, etc. Obtaining 12 may also include receiving information about the environment (e.g., in the form of messages or digital information), such as traffic or environmental information communicated between vehicles. Any such information about the environment may have a limited reliability or confidence level.

[0033] The method 10 further comprises estimating 14 information about the state of the environment based on the information about the environmental measurements. The information about the state of the environment comprises information about the confidence of the state of the environment. The confidence about the estimated state may depend on the confidence of the measurements and the interrelationship of the quantities determining the state of the environment. Although the perfect state of the environment may be known in simulation or training, there is always uncertainty involved in the real world. Furthermore, depending on the application area, the environment may change over time. The confidence of the measurements or the information about the environment may deteriorate over time. Thus, the sensing and processing time may also affect the confidence.

[0034] like Figure 1As further shown in , the method 10 includes: determining 16 a representation of the state of the environment based on information about the state of the environment and based on information about the confidence of the state of the environment. The representation of the state of the environment includes two or more intermediate states representing the state of the environment and the confidence of the state of the environment. For example, the intermediate states may be determined or selected such that an average of the intermediate states matches the state of the environment and a change in the intermediate states matches or is based on the confidence of the information related to the confidence of the state of the environment. The method 10 also includes: determining 18 information about an action of the automation device based on the representation. As will be described in further detail later, the action of the automation device may be determined based on a distribution or confidence of the intermediate actions.

[0035] Figure 2 An example of an apparatus 20 for controlling an automation device and a block diagram of an example of an automation device 200 are shown. The apparatus 20 for controlling an automation device includes a control module 24 for performing one of the methods described herein. Optionally, the apparatus 20 includes one or more interfaces 22 coupled to the control module 24 and configured to communicate with the environment (e.g., other devices, sensors, etc.). Optional components are shown in dashed lines.

[0036] like Figure 2 As further shown, the device 20 may include one or more interfaces 22 coupled to a corresponding control module 24. One or more interfaces 22 may correspond to one or more inputs and / or outputs for receiving and / or transmitting information, which may be digital (bit) values ​​according to a specified code within a module, between modules, or between modules of different entities. For example, the interface 22 may include an interface circuit configured to receive and / or transmit information. In an example, the interface 22 may correspond to any device for obtaining, receiving, transmitting, or providing analog or digital signals or information, for example, any connector, contact, pin, register, input port, output port, conductor, channel, etc. that allows providing or obtaining signals or information. The interface 22 may be configured to communicate (transmit, receive, or transmit and receive) in a wireless or wired manner, and may be configured to communicate with other internal or external components, that is, transmit and / or receive signals, information. One or more interfaces 22 may include additional components capable of communicating in a mobile communication system, such components may include transceiver (transmitter and / or receiver) components, such as one or more low noise amplifiers (LNA), one or more power amplifiers (PA), one or more duplexers, one or more diplexers, one or more filters or filter circuits, one or more converters, one or more mixers, accordingly, adapted radio frequency components, etc.

[0037] The control module 24 may be implemented using one or more processing units, one or more processing devices, any means for processing, such as a processor, a computer, or a programmable hardware component operable with correspondingly adapted software. In other words, the functions of the control / processing module 24 described may also be implemented in software, which is then executed on one or more programmable hardware components. Such hardware components may include general purpose processors, digital signal processors (DSPs), microcontrollers, etc. Another example is an automated device 200 including an example of the apparatus 20, such as an automated / autonomous driving vehicle or an industrial robot.

[0038] Examples can be improved by combining probabilistic state estimation methods with learned nonlinear control strategies. For example, by applying an unscented transformation to the probability distribution of the input to the learned control strategy, an approximate probability distribution of actions can be obtained, from which a better choice of actions can be derived. For example, reinforcement learning (RL) enables the use of learned control strategies in robotic systems. In classical RL, an agent that receives states from an environment learns which actions to take by interacting with its environment driven by a reward signal.

[0039] Figure 3 Shown in an example, how the learned policy uses the state of the environment to determine the best action. Figure 3 An environment 30 at the bottom and a policy 40 at the top are shown. The policy 40 receives information about the state of the environment and decides on actions that affect the environment. Thus, the learned policy 40 uses the state of the environment to determine the best action. However, one of the challenges in physical implementations of RL is how to incorporate the inherent uncertainty of state signals from the environment. While some implementations use an end-to-end approach, directly using measurements from sensors as input to the control policy, many implementations employ probabilistic state estimation algorithms, such as recursive filtering (e.g., using an extended Kalman filter, an unscented Kalman filter, etc.) to estimate a Gaussian probability distribution of the current state.

[0040] The probability distribution of this state is then used to derive a point estimate, typically the mean of a Gaussian distribution, which is used as input to the learned control policy. Figure 4 An example implementation is shown that uses a state estimator 50 to determine a point estimate of the current state as input to the learned policy 40. This results in a single choice of action that a) does not reflect the uncertainty of the state estimate and b) does not approximate the optimal action for a non-linear policy well.

[0041] Thus, the example uses a representation of the state of the environment that utilizes two or more intermediate states that reflect the uncertainty of the state estimate. For example, an unscented transform (UT) may be used to approximate a distribution of actions from which a preferred action may be derived. Then, method 10 includes: determining confidence information for the action.

[0042] Figure 5 An example of using a set of sigma points from an estimated distribution of states that are fed back to a policy library is depicted. The state estimator 60 obtains sensor measurements from the environment 30 and determines two or more sigma points. For example, each of the two or more intermediate states includes one or more sigma points representing statistical characteristics of information about the state of the environment. These sigma points are then provided to the policy library 70. Method 10 includes: using one or more strategies 70 to obtain two or more intermediate actions based on two or more intermediate states (sigma points). For example, the same strategy is used to obtain an intermediate action for each intermediate state. One or more strategies 70 involve nonlinear transformations to obtain two or more intermediate actions based on two or more intermediate states.

[0043] Given a probability estimate for the current state, e.g. from a recursive filter in the state estimator 70, a set of sigma points is selected, for each of which the learned policy 70 is evaluated. The resulting point estimates are then combined into an approximate probability distribution over (intermediate) actions. Figure 5 In the example of FIG. 1 , a set of sigma points from an estimated distribution of states is fed back to a policy library 70 (e.g., multiple parallel implementations of the same policy). The outputs of the policy 70 are then combined by the UT 80 into an approximate distribution of (intermediate) actions. The action selector 90 uses this distribution to determine the action. For example, the mean or median action of this distribution may be selected, and its variance may be used as confidence information.

[0044] like Figure 5 As shown, method 10 may further include: determining statistical characteristics of the distribution of two or more intermediate actions. Figure 5 In the example of , the method 10 determines the statistical characteristics of two or more intermediate actions using an unscented transformation 80. Thus, this example implements an unscented policy transformation for a learned control policy with uncertain inputs.

[0045] Determining 18 the information about the action may also be based on statistical properties of the distribution of the two or more intermediate actions.The statistical properties of the distribution of the two or more intermediate actions may include confidence information of the intermediate actions.

[0046] Using this setup, the policy can be trained using perfect state information in simulation and then deployed in the real system using probabilistic state estimates, as in Figure 6 As further shown in . Figure 6 It is shown how, in an example, a policy is first trained in simulation with perfect state information and then deployed in the real world using a probabilistic state estimator. Figure 6The training environment is shown on the left hand side. Based on the executed policy training 70a, the simulated environment 30a is used to generate perfect state information. The result of the policy training 70a is an action, which in turn affects the simulated environment 30a. From the policy training 70a, policy parameters 70b can be extracted and passed to the policy library in the real world deployment.

[0047] The policy 70 may first be trained in simulation with perfect state information, and then deployed in the real world using the probabilistic state estimator 60. Examples may be implemented using a hardware accelerator or hardware implementation of the control policy 70 and / or the state estimator 60.

[0048] The example may provide benefits for deploying learned RL policies in the real world, where there is uncertainty in state estimates and the example may provide improved action selection. The example may allow the best action to be better approximated by the learned control policy based on the uncertainty in the state estimate. The classical approach feeds a point estimate of the state back to a single copy of the RL policy and executes the resulting action. However, given the uncertainty in the state estimate, the point-estimated best action may be completely different from the best action. Therefore, the example considers the uncertainty in the state estimate and how this affects the predictions of the policy. By using an untraceable policy transformation, a set of sigma points can be transmitted to each copy of the policy, and the resulting action distribution can be observed. Then, an action can be selected from this distribution, and it is more confident that the action takes into account the uncertainty present in the state estimate. Importantly, due to the nonlinearity of the policy, the action can be completely different from the action generated by the point estimate of the state.

[0049] Figure 7 A block diagram illustrating a flow diagram of an exemplary implementation in which, after computing a distribution of actions using an unscented transformation, an action that accounts for state estimate uncertainty is selected using the resulting estimate of uncertainty. Figure 7 The sensor measurements are collected 702 at the top, which corresponds to obtaining 12 information about the environment measurements described above. From this, an estimate of the state of the environment is updated in 704, which corresponds to estimating 14 information about the state of the environment described above. Figure 7 As further noted in , the standard approach at this point would be to use a point estimate of the state of the environment as input to a single policy. Figure 7 In the example shown, a set of sigma points are calculated 708, which corresponds to the above-mentioned determination 16 of a representation of the state of the environment, including two or more intermediate states representing the state of the environment and the confidence of the state of the environment. The policy can now be evaluated 710 for each sigma point, and the untraceable transformation can be calculated 712. From the untraceable transformation, an average action or sample action can be performed 714, which corresponds to the above-mentioned determination 18 of information about the action based on the intermediate actions. The determination 18 of information about the action is based on the confidence information of the intermediate actions.

[0050] Examples can improve safety in the real world. A wrong choice of action by a learned policy can have serious consequences, since physical systems controlled by such a policy can cause serious damage (e.g., for industrial robots in a factory) or even pose a danger to their environment (e.g., for drones or self-driving cars). Therefore, in many cases, when the uncertainty relative to the action is too large, it is preferable to implement a safe action (e.g., an emergency stop), see Figure 7 By using unscented policy shifts, this uncertainty in actions can be quantified with respect to the policy and can be used as a threshold for execution. Note that learned policies are often nonlinear and large uncertainty in actions does not necessarily correspond to large uncertainty in states.

[0051] Figure 8 A block diagram showing a flow diagram of an exemplary implementation in which, after computing the distribution of actions using an unscented transformation, the resulting estimate of uncertainty is used to trigger an emergency stop. Figure 8 In the flowchart of Figure 7 Once the distribution and uncertainty of the intermediate actions are known, a decision can be made based on the uncertainty. For example, if the uncertainty is large, an emergency stop 802 can be caused, while if the uncertainty is small, the best action 804 can be selected. Generally, in the example, the method 10 can also include: if the confidence information of the intermediate action indicates that the confidence level of the intermediate action is below a predefined confidence threshold, determining the information about the safe action as the information about the action.

[0052] Examples may allow decisions about when to act. In a similar manner to improving safety, untraceable policy changes may be used to decide when to perform an action. This is particularly important in scenarios where there is high uncertainty in the state estimate and high confidence in the outcome of the action is desired. By waiting until the uncertainty in the action distribution is below a certain level, there is a way to measure whether enough evidence has been accumulated for the state estimate. This is advantageous when there is a high cost to changing actions online or when the state is estimated from multiple sources that all contribute to the uncertainty level on different time scales. Fig. 9 Yet another block diagram of a flow diagram of an exemplary implementation is shown, wherein after computing an action distribution using an unscented transform, the resulting estimate of uncertainty is used to decide whether to commit to an action or wait for more state evidence. Fig. 9 The first steps 702-712 in the flowchart shown are similar to those described above for Figure 7 As further shown, if the uncertainty is small, then action 904 may be performed, and if the uncertainty is large, then step 902 waits for more evidence, ie, to reduce the uncertainty.

[0053] Examples enable automatic curriculum learning. Effective training courses can be constructed by biasing training towards regions of state space that induce significantly different action predictions to the surrounding neighborhood. By using untraceable policy transformation, sigma points that produce actions that are far beyond the general distribution of actions or beyond the predicted action manifold can be identified. The state estimates responsible for these actions can then be used to bias further training of the original policy network. This can also be viewed as a form of adversarial training via untraceable policy transformation. Fig.10 A block diagram illustrating a flowchart of an example implementation in which, after computing a distribution of actions using an unscented transformation, actions outside the resulting distribution are used for further training of a bias policy. Fig.10 The first steps 702-712 in the flowchart shown are similar to those described above for Figure 7 Same as described. Fig.10 Further shown is the different modification of the training data results depending on whether the action is outside the distribution or within the distribution. If the action is within the distribution, the probability of using the associated sigma point for further training can be reduced 1002. If the action is outside the distribution, the probability of using the associated sigma point can be increased 1004.

[0054] Therefore, the method 10 may further include: training the strategy, wherein the training of the strategy is influenced by the intermediate states for the intermediate action recognition exhibiting predefined statistical characteristics. For example, the predefined statistical characteristics are confidence or uncertainty thresholds.

[0055] In an example, the automated device may be an autonomous vehicle or an industrial robot. The action may be a controlled action for the autonomous vehicle or the industrial robot. For example, the action may include one or more elements of the group of manipulation, motion, acceleration, deceleration, steering command, stop command (emergency stop command), and emergency command.

[0056] Some examples are summarized below:

[0057] (1) A method for determining an action of an automation device based on an environmental state of the automation device. The method comprises: obtaining information about environmental measurements, the measurements having a limited confidence level; estimating information about the environmental state based on the information about the environmental measurements, the information about the environmental state including information about the confidence level of the environmental state; determining a representation of the environmental state based on the information about the environmental state and based on the information about the confidence level of the environmental state, the representation of the environmental state including two or more intermediate states representing the environmental state and the confidence level of the environmental state; and determining information about the action of the automation device based on the representation.

[0058] (2) The method according to (1), comprising: determining confidence information of the action.

[0059] (3) The method according to one of (1) or (2), wherein each of the two or more intermediate states includes one or more sigma points representing statistical characteristics of information about the environment state.

[0060] (4) The method according to any one of (1) to (3), further comprising using one or more strategies to obtain two or more intermediate actions based on two or more intermediate states.

[0061] (5) The method according to (4) also includes using the same strategy to obtain an intermediate action for each of the intermediate states.

[0062] (6) A method according to one of (4) or (5), wherein the one or more strategies involve nonlinear transformations to obtain two or more intermediate actions based on two or more intermediate states.

[0063] (7) The method according to one of (4) to (6), further comprising determining statistical characteristics of the distribution of the two or more intermediate actions.

[0064] (8) The method according to (7) also includes determining statistical characteristics of the two or more intermediate actions using an unscented transformation.

[0065] (9) A method according to one of (7) or (8), wherein determining information about the action is also based on statistical characteristics of the distribution of two or more intermediate actions, and wherein the statistical characteristics of the distribution of the two or more intermediate actions include confidence information of the intermediate actions.

[0066] (10) The method of (9), wherein determining information about the action is based on confidence information of the intermediate action.

[0067] (11) The method according to (9) or (10) also includes: if the confidence information of the intermediate action indicates that the confidence level of the intermediate action is lower than a predefined confidence threshold, determining the information about the safe action as the information about the action.

[0068] (12) The method according to one of (9) to (11) also includes a training strategy, wherein the training of the strategy is influenced by intermediate states for intermediate action recognition that exhibit predefined statistical characteristics.

[0069] (13) The method according to (12), wherein the predetermined statistical feature is a confidence threshold.

[0070] (14) A method according to one of (1) to (13), wherein the automated device is an autonomous vehicle or an industrial robot, and wherein the action is a controlled action for the autonomous vehicle or the industrial robot.

[0071] (15) The method according to (14), wherein the action includes one or more elements from the group consisting of manipulation, movement, acceleration, deceleration, steering command, stop command, and emergency command.

[0072] (16) A computer program having a program code, which is used to execute the method according to any one of (1) to (15) when the computer program is executed on a computer, a processor or a programmable hardware component.

[0073] (17) An apparatus for controlling an automation device, comprising a control unit or a module for executing one of the methods according to one of (1) to (15).

[0074] Aspects and features described with respect to a particular example in the previous examples may also be combined with one or more of the other examples to replace the same or similar features of that other example or to otherwise introduce the features into that other example.

[0075] Examples may also be or relate to (computer) programs, which include program codes that execute one or more of the above methods when the program is executed on a computer, a processor, or other programmable hardware components. Therefore, the steps, operations, or processes of the different methods in the above methods may also be performed by a programmed computer, a processor, or other programmable hardware components. Examples may also cover program storage devices, such as digital data storage media, which are machine-readable, processor-readable, or computer-readable and encode and / or contain machine-executable, processor-executable, or computer-executable programs and instructions. Program storage devices may include or be digital storage devices, magnetic storage media (e.g., disks and tapes), hard drives, or optically readable digital data storage media. Other examples may also include computers, processors, control units, (field) programmable logic arrays ((F)PLA), (field) programmable gate arrays ((F)PGA), graphics processor units (GPUs), application-specific integrated circuits (ASICs), integrated circuits (ICs), or system-on-chip (SoC) systems programmed to perform the steps of the methods described above.

[0076] It should be further understood that the contents of several steps, processes, operations or functions disclosed in the specification or claims should not be interpreted as implying that these operations necessarily depend on the order described, unless explicitly stated in individual cases or necessary for technical reasons. Therefore, the foregoing description does not limit the execution of several steps or functions to a specific order. In addition, in other examples, a single step, function, process or operation may include and / or be decomposed into several sub-steps, sub-functions, sub-processes or sub-operations.

[0077] If some aspects have been described with respect to a device or system, these aspects should also be understood as descriptions of corresponding methods. For example, a block, device or functional aspect of a device or system may correspond to a feature of a corresponding method, such as a method step. Thus, aspects described with respect to a method should also be understood as descriptions of corresponding blocks or corresponding blocks, corresponding elements, characteristics or functional features of a corresponding system.

[0078] The following claims are hereby incorporated into the detailed description, wherein each claim may stand alone as a separate example. It should also be noted that, although in the claims, a dependent claim refers to a specific combination with one or more other claims, other examples may also include a combination of a dependent claim with the subject matter of any other dependent claim or independent claim. Such combinations are hereby explicitly proposed, unless in individual cases a specific combination is not intended. Furthermore, features of a claim shall also be included in any other independent claim, even if the claim is not directly qualified as dependent on the other independent claim.

Claims

1. A method for determining an action of an automation device based on an environmental state of the automation device, the method comprising: obtaining information about environmental measurements, the measurements having a limited degree of confidence; estimating information about an environment state based on the information about the environment measurement result, the information about the environment state including information about a confidence level of the environment state; determining, based on the information about the state of the environment and based on the information about the confidence level of the state of the environment, a representation of the state of the environment, the representation of the state of the environment comprising two or more intermediate states representing the state of the environment and the confidence level of the state of the environment; as well as Based on the representation, information about the action of the automation device is determined.

2. The method of claim 1 further comprising determining confidence information for the action.

3. The method according to claim 1, wherein: Each of the two or more intermediate states includes one or more sigma points representing a statistical characteristic of information about the environmental state. 4 . The method of claim 1 , further comprising using one or more strategies to obtain two or more intermediate actions based on the two or more intermediate states. 5 . The method of claim 4 , further comprising using the same strategy to obtain an intermediate action for each of the intermediate states.

6. The method according to claim 5, wherein: The one or more strategies involve non-linear transformations to obtain the two or more intermediate actions based on the two or more intermediate states.

7. The method of claim 6, further comprising determining a statistical characteristic of a distribution of the two or more intermediate actions.

8. The method of claim 7, further comprising using an unscented transform for determining the statistical characteristics of the two or more intermediate actions.

9. The method according to claim 8, wherein: Determining the information about the action is also based on statistical characteristics of a distribution of the two or more intermediate actions, and wherein the statistical characteristics of the distribution of the two or more intermediate actions include confidence information about the intermediate actions.

10. The method according to claim 9, wherein: Determining information about the action is based on confidence information of the intermediate action.

11. The method according to claim 10, further comprising: If the confidence information of the intermediate action indicates that the confidence level of the intermediate action is lower than a predefined confidence threshold, the information about the safe action is determined as the information about the action.

12. The method of claim 11, further comprising training the strategy, wherein: The training of the policy is effected by identifying intermediate states for intermediate actions that exhibit predefined statistical characteristics.

13. The method according to claim 12, wherein: The predefined statistical feature is a confidence threshold.

14. The method according to claim 13, wherein: The automated device is an autonomous vehicle or an industrial robot, and wherein the action is a controlled action of the autonomous vehicle or the industrial robot.

15. The method according to claim 14, wherein: The action includes one or more elements from the group of maneuvering, movement, acceleration, deceleration, steering command, stop command and emergency command. 16 . A computer program having a program code for executing the method according to claim 1 when the computer program is executed on a computer, a processor or a programmable hardware component. 17 . An apparatus for controlling an automation device, comprising a control module for executing the method according to claim 1 .