Method for operating robotic device

Abstract

According to various embodiments, a method for operating a robotic device is described, comprising: transmitting sensor data to a monitoring device external to the robotic device, to monitor, by the external monitoring device, an actual value of a physical quantity on the basis of a target value of the physical quantity occurring in or on the robotic device during operation of the robotic device; activating a protected operating mode of the robotic device if it is determined during the monitoring that a deviation between the actual value and the target value exceeds a predetermined threshold value; checking whether one or more functions for determining a target value and an actual value and for monitoring the actual value have been performed, the functions being at least partially to be performed in the external monitoring device; and activating the protected operating mode if it is detected that the one or more functions are not performed.

Description

Technical Field

[0001] The present disclosure relates to methods for operating a robotic device. Background Art

[0002] The functionality implemented in software to run robotic devices has reached considerable size for some applications such as vehicle control.

[0003] For example, the publication “Standardisiertes E-Gas für Benzin und Diesel Motorsteuerungen" (Standardized electronic throttle monitoring concept for gasoline and diesel engine control, EGAS working group, July 8, 2015, version 6.0, hereinafter referred to as Reference 1) describes various functions for engine control and its monitoring, which are divided into three levels ("Level 1" to "Level 3").

[0004] It would be desirable to have methods that would enable such functionality to be implemented efficiently and reliably in robotic devices, especially in (eg, mobile) robotic devices with limited processing resources. Summary of the invention

[0005] According to various embodiments, a method for operating a robotic device is provided, comprising: sending sensor data to a monitoring device external to the robotic device so that the external monitoring device monitors the actual value of a physical quantity based on a target value of the physical quantity that appears in or at the robotic device when the robotic device is operating; activating a protected operating mode of the robotic device if, during monitoring, it is determined that the deviation between the actual value and the target value exceeds a preset threshold value; checking whether one or more functions for determining target values ​​and actual values ​​and monitoring actual values, which functions are at least partially executed in the external monitoring device, have been executed; and activating the protected operating mode if it is detected that one or more functions have not been executed.

[0006] The method described above ensures security when transferring the monitoring function to an external monitoring device, ie, an external system such as an edge computing network or a cloud.

[0007] Various embodiments will be described below.

[0008] Embodiment 1 is a method for operating a robotic device as described above.

[0009] Embodiment 2 is a method according to embodiment 1, further comprising: checking whether a communication connection exists between the monitoring device and the robot device, via which the monitoring device can trigger activation of safety measures; and activating a safe operation mode if the communication connection is interrupted.

[0010] This further increases the safety of the transfer of the monitoring function, since it is avoided that the safety operating mode is not activated (even if it is required) due to a loss of the connection between the robotic device and the external monitoring device.

[0011] Embodiment 3 is a method according to embodiment 1 or 2, wherein one or more functions are executed in sequence, and it is checked whether one or more functions have been executed by checking whether a request message delivered to the sequence and processed by one of the functions when the function is executed has been processed by the sequence as an expected response.

[0012] This enables efficient testing of processing chains including externally executed functions, ie including the flow control of the transferred functions.

[0013] Embodiment 4 is a method according to any one of Embodiments 1 to 3, comprising: before monitoring, checking whether an external processing device is able to activate the protected operating mode of the robot device; and outputting an error signal if the external processing device cannot activate the protected operating mode of the robot device.

[0014] This ensures that the protected operating mode can be activated when monitoring indicates that the protected operating mode should be activated.

[0015] Embodiment 5 is a method according to embodiment 4, comprising: if the external processing device cannot activate the protected operating mode of the robot device, monitoring is performed in the robot device.

[0016] This makes it possible to carry out a monitored operation even in the case where the external processing device cannot activate the protected operating mode.

[0017] Embodiment 6 is a robot monitoring system configured to perform the method according to any one of Embodiments 1 to 5.

[0018] The robot monitoring system includes components of the robot apparatus and may also include external monitoring devices.

[0019] Embodiment 7 is a computer program having instructions, which, when executed by a processor, cause the processor to perform the method according to any one of embodiments 1 to 5.

[0020] Embodiment 8 is a computer-readable medium storing instructions, which, when executed by a processor, cause the processor to perform the method according to any one of embodiments 1 to 5. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In the accompanying drawings, like reference numerals generally refer to the same parts in different views. The accompanying drawings are not necessarily to scale, with emphasis instead generally being placed on illustrating the principles of the invention. In the following description, various aspects will be described with reference to the following drawings.

[0022] Figure 1 A vehicle is shown.

[0023] Figure 2 An architecture is shown that transfers monitoring functionality from the vehicle to an external system (cloud or edge) via a wireless connection.

[0024] Figure 3 An embodiment is shown in which a diagnostic function is performed on the input data in an external system.

[0025] Figure 4 An embodiment is shown in which, in addition to diagnostic functions, application functions are also moved to an external system and monitored there.

[0026] Figure 5 A flow chart illustrating a method for operating a robotic device is shown. DETAILED DESCRIPTION

[0027] The following detailed description refers to the attached drawings, which illustrate the present disclosure for illustrating specific details and aspects in which the present invention may be performed. Other aspects may be used, and structural, logical and electrical changes may be made without departing from the scope of protection of the present invention. Various aspects of the present disclosure are not necessarily mutually exclusive, as some aspects of the present disclosure may be combined with one or more other aspects of the present disclosure to form new aspects.

[0028] Various examples are described in more detail below.

[0029] Figure 1 A vehicle 101 is shown.

[0030] exist Figure 1 In the example of FIG. 1 , a vehicle 101 (eg a passenger car (PKW) or a truck (LKW)) is equipped with a vehicle control device 102 (eg composed of one or more electronic control units (ECU)).

[0031] The vehicle control device 102 has a data processing component, such as a processor (e.g., CPU (central processing unit)) 103 and a working memory 104 for storing control software 107 and storing data processed by the processor 103, and the vehicle control device 102 operates according to the control software 107. The vehicle control device 102 may have a plurality of data processing devices (e.g., ECUs) connected to each other via an internal communication network (e.g., CAN bus). These data processing devices may also execute the control software 107 in a distributed manner.

[0032] For example, the stored control software (computer program) has instructions that, when executed by the processor (or when multiple processors execute the instructions in a distributed manner), cause the processor 103 (or multiple processors) to perform driver assistance functions (or collect driving data) or even autonomously control the vehicle.

[0033] The vehicle E / E architecture of the future is expected to be highly networked with external systems (e.g. cloud, edge, other vehicles, smart devices) because, for example, based on 5G technology, communication between systems can be carried out with extremely low latency. This offers the possibility of utilizing resources outside the vehicle during driving operation, for example in order to transfer functions or to incorporate external sensor devices into the construction of the own environmental model. Depending on the degree of transfer, it is conceivable that the monitoring of vehicle functions (e.g. engine control) can also be delegated to the outside (e.g. cloud, edge) or supported by external systems.

[0034] For example, the vehicle 101 may be networked (here via a communications network 106) with an external system 105 (e.g., one or more servers of a cloud or edge computing platform) (actually continuously, or at least for most of the time during which it is in operation), and may provide low-latency communications (e.g., with the aid of 5G) between the vehicle 101 (typically a robotic device, and in particular a mobile robotic device) and the external system 105.

[0035] According to various embodiments, the transfer comprises in particular a monitoring function which compares the actual value with a target value or a threshold value (which represents, for example, a maximum deviation from the target value), i.e. Figure 1 In the example of , the process is executed in an external system 105 (eg, a cloud) rather than within the vehicle 101 .

[0036] This includes safety functions for cloud-based (or edge-based) (functional) monitoring of vehicle systems, such as engine control. In the following, an architecture is described as an example, with which functional monitoring (i.e., level 2 functions in the sense of reference 1) can be moved from a vehicle 101 (typically a robotic device) to an external system 105 (e.g., the cloud).

[0037] This saves resources for the level 2 part of the controller (software (SW) and hardware (HW), such as reserved memory areas) in the vehicle 101. In this case, the resources of the external system 105 can also be used multiple times, for example alternately for the running vehicle or even for multiple vehicles in parallel.

[0038] By transferring, the impact of potential errors within the vehicle (such as permanent damage to storage areas) and transient errors (such as temporary changes to storage areas (such as caused by alpha particles)) can also be reduced, which may lead to reduced availability of vehicle internal resources because fewer vehicle internal HW resources are used (such as for monitoring reserved storage areas).

[0039] Thus, due to the physical isolation via the wireless interface (i.e., network 106), the necessary measures within the vehicle for ensuring "freedom from interference" (FFI) between ASIL X safety software (level 2) and QM (quality management) application software are further reduced. In addition, interventions by level 1 SW in level 3 SW (according to reference 1) in the cloud can be identified.

[0040] Adaptations to the monitoring functionality that become necessary due to changes in the application software (SW update or configuration change) can be made purely in the external system 105 .

[0041] All HW-related parts that are monitored (e.g. I / O pins, buses, uC) can remain unchanged in the field (i.e. in the vehicle in use) and can be fully protected by the described architecture.

[0042] By offloading the monitoring functions, those monitoring functions themselves can be designed to be more comprehensive and detailed, and thus more computationally intensive, because resources in the external system 105 (eg, the cloud) are far less constrained than within the vehicle 101 .

[0043] When determining an appropriate (safe) error reaction, contextual information (e.g. the vehicle environment) available (only) in the external system 105 can be taken into account (e.g. if there is no danger, no immediate shutdown in the event of an error). This can also be used in hybrid variants: the external system 105 intervenes with stricter monitoring limits only when the vehicle is in a rare E1 or E2 situation (classification of situation frequency according to ISO 26262), which requires stricter limits.

[0044] The monitoring functionality may be provided by the external system 105 as a service (eg, with a corresponding pricing model such as pay-per-use).

[0045] According to various embodiments, the architecture includes an abstracted interface between the vehicle 101 and the external system 105, thereby supporting the use of replaceable ECU-HW modules (e.g. SoCs with higher computing power) in each vehicle, since functional monitoring can be performed in the external system 105 independently of the specific vehicle internal HW.

[0046] Figure 2An architecture for transferring monitoring functionality 201 from the vehicle to an external system (cloud or edge) via a wireless connection 202 is shown.

[0047] The functions of the first, second and third levels in the sense of reference 1 are denoted by L1, L2 and L3. The functions below the dashed line representing the connection 202 are provided in the vehicle, while the functions above the dashed line are provided in the external system.

[0048] In the vehicle, a monitored input 203 is generated and provided, for example via a bus interface (such as CAN), a digital sensor interface (SPI), an analog sensor interface (with an analog-to-digital converter ADC) or via variable values ​​in RAM.

[0049] These inputs are then first checked by HW input diagnostics 204 (eg, in the case of redundant sensors, whether the sensor data is consistent). HW input diagnostics 204 has functions such as bus diagnostics 205, interface diagnostics 206, ADC diagnostics 207, and RAM diagnostics 208.

[0050] HW input diagnostics device 204 provides the checked results to connection control unit (CCU) 218 ​​, which transmits them as data 209 to monitoring function 201 , if necessary together with metadata, such as requirements of systems in the vehicle.

[0051] Furthermore, the HW input diagnostics device 204 provides the checked input data to the application software 210. The application software generates one or more application results, such as target values ​​(eg torque set by means of the accelerator pedal). These application results are provided by the connection control unit 218 to the external system.

[0052] The data 209 and the application result 211 are provided to the external system by the connection control unit 218 as a service, for example.

[0053] If the monitoring function 201 determines an error (e.g. the torque set by means of the accelerator pedal deviates greatly from the torque generated as determined from the measurement data contained in the data 206), the monitoring function sends an error signal to the shutdown manager 212. The shutdown manager will inform the watchdog (WD) 214 in the vehicle via the watchdog communication function 213 provided by the connection control unit 205, which in the event of an error performs a shutdown or limitation 215 of the vehicle's functions (i.e. activates a protected operating mode), for example by means of one or more hardware signals to relays, power stages, bus transceivers, switches (such as IGBTs), etc. For example, the speed of the vehicle can be reduced or, in the case of an autonomous vehicle, the vehicle can be driven to the side of the road and parked.

[0054] The watchdog communication function 213 and the watchdog 214 thus provide a safe, externally activatable shutdown path or transition path for transitioning to a safe state (eg switching off the power output stage, deactivating bus communication) from an external system.

[0055] The external system, in particular the components implementing the monitoring function 201 and the shutdown manager 212 may be regarded as a monitoring device external to the robotic device.

[0056] Provision can be made to check the shutdown path for potential errors (ie to check whether shutdown or limiting 215 can be performed via watchdog 214 ). Such a check is performed, for example, at the beginning of an operating cycle and can also be performed from an external system, for example by shutdown manager 212 .

[0057] Furthermore, it can be provided that the monitoring function 201 also includes the behavior of the digital twin (DT stands for Digital Twin) 222 in the monitoring, or that a shutdown or restriction request is sent from the digital twin itself to the shutdown manager 212, and the shutdown manager notifies the watchdog 214 accordingly.

[0058] The communication module, i.e. the connection control unit 218, communicates with the outside and forwards messages from the external system to the inside. In the connection control unit 218, for example, a SW module is integrated, which provides a watchdog communication function 213, i.e., communicates with a watchdog 214, wherein when the communication module fails (and thus the intervention of the vehicle from the external system fails), the watchdog 214 is automatically triggered, so that the watchdog 214 performs a shutdown or limitation 215.

[0059] Alternatively, the watchdog 214 can also be part of the CCU 218 , or the CCU 218 can intervene directly in the shutdown path (eg output stage, communication bus).

[0060] The vehicle is also provided with a hardware diagnosis / FFI device 216 having a function 217 for diagnosing internal hardware components (such as a processor (CPU, TPU, GPU, etc.)) and ensuring FFI.

[0061] Figure 3 An implementation form is shown, in which a diagnostic function is performed on the input data in the external system 301. For example, the external system can perform the following diagnostic functions:

[0062] ● Bus (CAN, SPI): checked via end-to-end protection measures, such as checksum (correctness) and / or frame counter (timing).

[0063] ●Sensor data: plausibility check of single or multiple sensors based on redundancy and / or physical models, e.g. two independent measurements (speed, current, voltage, torque, ...).

[0064] ●RAM: For example, a comparison of dual storage.

[0065] Thus, if the corresponding prerequisites are met (eg end-to-end protection measures in bus messages, availability of safety metadata, plausibility checking possibilities via the physical model), the diagnosis of monitored input variables can also be partially or completely moved to the external system 105 .

[0066] Figure 4 An embodiment is shown in which, in addition to diagnostic functions 401 , application functions 402 (level 1 ) are also moved to an external system and monitored there (level 2 ).

[0067] Regardless of which functions are transferred, system-internal functions and interfaces are provided in the external system according to different implementation forms to perform process control. These functions and interfaces ensure that system-internal functions for (e.g. HW) diagnostics and ensuring FFI safety (such as functions 205, 206, 207, 208, 218) can be executed. For this purpose, the architecture has a PFC and metadata monitoring device 219.

[0068] exist Figure 2 In the figure, flow control (PFC stands for Program Flow Check) is represented by a dotted message flow.

[0069] The watchdog sends a request message 220 ("Query", e.g., an 8-bit value) 501 to an internal component of the system and expects a response ("Response", e.g., 32 bits) within a time window. Here, the watchdog first sends the request message 220 to the hardware diagnostic / FFI device 216, which forwards the request message to the HW input diagnostic device 204. The HW input diagnostic device 204 further sends the request message 220 to the monitoring function 201, which sends it to the PFC and metadata monitoring device 219. As shown in Figure 2 As shown in FIG. 1 , the functions of the hardware diagnosis / FFI device 216, the HW input diagnosis device 204 and the monitoring function 201 all have a PFC function, which processes the request message 220 in a corresponding preset manner, so that when the request message 220 has traversed all functions, that is, all these functions have been executed, the request message 221 processed in this way has a specific expected value. Only when all the functions linked by PFC in this way are called in the correct order, the processed request message 221 is correct (that is, contains the expected value).

[0070] The PFC and metadata monitoring device 219 compares the expected values ​​with the values ​​contained in the processed request message 221 received by it and sends a response to the watchdog accordingly (via the shutdown manager 212). The response indicates to the watchdog that the functions have been executed when the values ​​are consistent, or that at least one function has not been executed in the case of deviations. In the latter case, the watchdog can react appropriately, for example shutting down or limiting 215 the function of the vehicle again, or at least increasing an error counter and shutting down or limiting 215 the function of the vehicle when the counter exceeds a threshold value. The watchdog can also react in this way when the response is not timely.

[0071] exist Figure 2 In the example of , the response is (at least partially) calculated in the external system. Thus, the availability of the external system is also checked by PFC. However, the PFC function of the PFC and metadata monitoring device 219, i.e. the comparison of the processed request message 221 with the expected value, can also be undertaken by the watchdog. Nevertheless, the request message 220 can be directed via the external system in order to monitor the availability of the external system by means of PFC.

[0072] Process control can be similarly targeted Figure 3 and Figure 4 The architecture is implemented as shown therein by the PFC functions and the dotted paths through the different units.

[0073] For the functions described above, appropriate networking is provided in the entire range of use of the robotic device, i.e. a stable, ideally even redundant connection from the vehicle 101 (typically the robotic device) to the cloud (typically the external system), for example via 5G, 5G+, 6G, or the functions described above are used only when such networking exists. For example, before starting a journey, a predictive assessment of the achievable QoS (quality of service) can be made along the planned route.

[0074] The networking is provided here (or the networking is checked to meet this requirement before transferring monitoring) so that communication between the cloud and the vehicle has a sufficiently low latency to ensure that the overall fault tolerance time (including debouncing) is not exceeded (e.g. engine control, HV battery, hybrid powertrain approximately 500ms; electric drive approximately 200ms).

[0075] An initial synchronization of vehicle and cloud functions can be set, for example when the vehicle is started. In addition, regular resynchronization can be performed during operation, for example to keep the two functional parts synchronized even after switching between base stations for communication (for example in 5G networks). Alternatively, the cloud (i.e., typically an external system) and the vehicle (typically a robotic device) can also use the same absolute time reference.

[0076] In summary, a method is provided according to different implementation forms, such as in Figure 5 as shown in .

[0077] Figure 5 A flow chart 500 is shown illustrating a method for operating a robotic device.

[0078] In 501, sensor data is sent to a monitoring device external to the robotic device to monitor the actual value of the physical quantity occurring in or at the robotic device during operation of the robotic device via the external monitoring device, based on the target value of the physical quantity. This means that the sensor data enables the monitoring device to determine the actual value and / or the target value, so that the monitoring device can monitor the actual value.

[0079] In 502 , a protected operating mode of the robotic device is activated if during monitoring it is determined that a deviation between the actual value and the target value exceeds a predefined threshold value.

[0080] In 503 , it is checked whether one or more functions for determining a target value and an actual value and for monitoring the actual value have been executed, said functions being at least partially executed in the external monitoring device.

[0081] At 504 , if it is detected that one or more functions are not being performed, a protected operating mode is activated.

[0082] The protected operating mode may include various safety measures as described above. These measures need not be consistent in all cases of the protected operating mode, i.e., the safety measures taken when the deviation exceeds a preset threshold may be different from the measures taken when it is detected that one or more functions are not executed.

[0083] The robotic device is, for example, mobile. In this case, "external" may mean that the monitoring device does not necessarily move with the robotic device (i.e. the monitoring device is fixed or at least can be moved independently of the robotic device). External may also be understood to mean that the components of the robotic device and the monitoring device are contained in separate housings (wherein they do not need to be contained in only one housing each). The monitoring device is connected to the robotic device via a communication connection, for example via a communication network (e.g. a wireless communication network, such as a mobile radio network).

[0084] Figure 5The method can be performed by one or more data processing units (inside and outside the robot device). The term "data processing unit" can be understood as any type of entity that enables processing of data or signals. Data or signals can be processed, for example, according to at least one (i.e., one or more) specific functions performed by the data processing unit. The data processing unit may include an analog circuit, a digital circuit, a logic circuit, a microprocessor, a microcontroller, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a programmable gate array integrated circuit (FPGA), or any combination thereof, or be composed of an analog circuit, a digital circuit, a logic circuit, a microprocessor, a microcontroller, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a programmable gate array integrated circuit (FPGA), or any combination thereof. Any other means for implementing the corresponding functions described in detail herein can also be understood as a data processing unit or a logic circuit device. One or more method steps described in detail herein can be performed (e.g., implemented) by a data processing unit by one or more specific functions performed by the data processing unit.

[0085] The term "robotic device" may be understood as any technical system (having mechanical parts, the movement of which is controlled), such as a computer-controlled machine, a vehicle, a household appliance, a power tool, a manufacturing machine, a personal assistant or an access control system.

[0086] Different implementations may receive and use sensor signals from different sensors (eg, video, radar, lidar, ultrasound, motion, thermal imaging, etc.), for example, to determine target values ​​and / or actual values.

[0087] The interface between the robotic device and the external monitoring device can be designed to abstract which hardware components from the set of possible hardware components are used in the robotic device, i.e., to make the exact hardware of the robotic device transparent to the external monitoring device. In particular, the hardware of the robotic device can be replaced without affecting the monitoring.

[0088] Although specific implementations are shown and described herein, those skilled in the art recognize that the specific implementations shown and described may be replaced with a variety of alternative and / or equivalent implementations without departing from the scope of protection of the present invention. This application is intended to cover any adaptation or change of the specific implementations discussed herein. Therefore, the present invention is limited only by the claims and their equivalents.

Claims

1. A method for operating a robotic device, comprising: sending sensor data to a monitoring device external to the robotic device to monitor, by the external monitoring device, an actual value of the physical quantity based on a target value of the physical quantity occurring in or at the robotic device when the robotic device is operating; activating a protected operating mode of the robotic device if during monitoring it is determined that a deviation between the actual value and the target value exceeds a predefined threshold value; checking whether one or more functions for determining the target value and the actual value and for monitoring the actual value have been performed, the functions being at least partially to be performed in the external monitoring device; as well as If it is detected that the one or more functions are not being performed, the protected operating mode is activated.

2. The method according to claim 1, further comprising: checking whether a communication connection exists between the monitoring device and the robotic device, via which the monitoring device can trigger activation of a safety measure; and If the communication connection is interrupted, the safe operating mode is activated.

3. A method according to claim 1 or 2, wherein the one or more functions are to be executed in sequence, and checking whether one or more of the functions have been executed is performed by checking whether the request message delivered to the sequence and processed by each of the functions when the function is executed has been processed by the sequence as an expected response.

4. The method according to any one of claims 1 to 3, comprising: Prior to monitoring, checking whether the external processing device is capable of activating a protected operating mode of the robotic device; and If the external processing device is unable to activate the protected operating mode of the robotic device, a fault signal is output.

5. The method according to claim 4, comprising: If the external processing device is unable to activate the protected operating mode of the robotic device, monitoring is performed in the robotic device.

6. A robotic monitoring system configured to perform the method according to any one of claims 1 to 5.

7. A computer program having instructions which, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 5.

8. A computer-readable medium storing instructions, which, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 5.

Images