Service robot system, robot and method for operating a service robot

Abstract

The invention relates to a service robot system comprising a robot and a learning method for the system. The robot has a drive system for moving the robot to a target position and at least one effector. The processing unit of the system comprises a task determination unit for determining a task to be performed by the robot and is configured to control the drive system and the effector according to the task based on action definitions. The processing unit automatically retrieves action definition candidates from a database, evaluates the retrieved action definition candidates for a success score indicating a likelihood that an action according to the action definition candidate contributes to successfully fulfilling the task, executes an action according to the action definition candidate having the highest probability equal to or higher than a predefined threshold value and sends a request for assistance via a communication interface if the success score is less than a preset threshold value. The system learns from instructions and additional information received in response to such a request.

Description

Technical Field

[0001] This invention generally relates to service robot systems and the operation of such service robots. Background Technology

[0002] Service robots, which provide services to their owners, are becoming increasingly popular. However, in most cases, the advantages of such service robots are quite limited. One reason is the lack of operational flexibility. For example, it is easy to instruct a service robot on how to mow a lawn within clearly defined outer boundaries. This type of automatic lawnmower randomly travels across the lawn to be mowed until it reaches the wire mesh indicating the boundaries of its work area or encounters an obstacle. However, changes in environmental conditions often overload the capabilities of this type of automatic lawnmower. When the lawnmower encounters a situation that cannot be resolved by changing its direction of travel, it will stop operating to maintain a safe state. This stop will continue as long as the system operator (often the garden owner) clears the situation by re-establishing the environmental conditions set for the automatic lawnmower. Although this system is very useful as long as the automatic lawnmower does not encounter unexpected situations, its inherent lack of flexibility limits its efficiency in assisting its owner.

[0003] On the other hand, systems in which users remotely operate robots are known. This remote operation of the robot is based on images captured by a camera mounted on the robot, which are transmitted to a remote control station where a display reproduces the captured images. Thus, the reproduced image presents information about the robot's current environment to its user / operator. Based on this, the operator can remotely control the robot. EP 2 363774 A1 describes a system that specifically addresses the problem of latency when remotely controlling a robot. Although the solution proposed in EP 2363 774A1 improves the results of remote operation of service robots, it still requires the user or operator to be able to continuously operate the service robot. This may be a solution when only human intervention is needed to avoid humans entering harmful or dangerous situations. However, this does not alleviate the burden on humans performing actions themselves, meaning that humans may not be able to perform other tasks simultaneously.

[0004] Therefore, there is a need to provide an improved service robot system that provides the ability to handle new situations during its regular operation and even learn for the future. Summary of the Invention

[0005] This task is accomplished by the service robot system, service robot, and method for operating the service robot as defined in the independent claims. Advantages and further features are defined in the dependent claims.

[0006] The service robot system according to the present invention includes a service robot and a processing unit, the processing unit including a task determination unit for determining tasks to be performed by the robot. Therefore, the robot's operation is highly flexible at the location where the task to be performed is determined. The robot prepares to perform different tasks defined in the task determination unit. In the simplest embodiment, the task determination unit communicates with an interface to receive instructions (e.g., verbal instructions) from a user of the robot, extracting information about the task to be performed from the instructions. Alternatively, the task determination unit determines the task based on information sensed by the robot or received from other information sources (e.g., a clock or calendar for time information or information about events (calendar entries)).

[0007] The robot is capable of movement using its drive system. The drive system moves the robot to a target location, where at least one actuator performs at least one pick-up and place-down operation. The end effector can be any type of actuator that enables the robot to manipulate its environment (e.g., pick up and hold objects). The robot, including the drive system and end effector, is controlled by a processing unit, which may be located within the robot but also at a remote location and connected to the robot, to transmit signals based on which control signals can be generated for the corresponding actuators of the drive system and end effector.

[0008] The processing unit may consist of a single processor or include multiple processors that can be distributed across the system. Specifically, parts of the processing unit may be located inside the robot, while other parts are located at a remote location and communicate with said parts of the robot. Depending on the task to be performed, the processing unit is configured to automatically retrieve action definition candidates from a database. Action definition candidates are definitions of actions that can be performed by the robot and are estimated to be applicable to the current situation of the service robot. Action definition candidates are searched in the database based on objects identified in the robot's environment. To identify objects in the robot's environment, the system includes at least one sensor for entity sensing of the robot's environment. The sensor signals are then processed to identify the objects. As an example, image processing is performed on images captured by a camera acting as a sensor.

[0009] The retrieved action definition candidates are then evaluated, and success scores are calculated. A success score is a measure of the likelihood that an action from a candidate action definition will contribute to the successful completion of the task. The determination of the success score does not depend on the confidence level of correctly identifying the current situation in which the robot should act. Individual success scores for action definition candidates can be calculated by considering actions corresponding to the same candidate action definition performed in comparable past situations. Once the action definition candidate with the highest score is selected, the system distinguishes between cases where it can be assumed that the action corresponding to the selected candidate action definition will successfully contribute to the completion of the task, and cases where it cannot be assumed that the action corresponding to the selected candidate action definition will successfully contribute to the completion of the task.

[0010] The success score of each selected action candidate is compared to a threshold. Selected actions with a success score above the threshold are considered helpful in successfully performing the task. Actions with a success score below or equal to the threshold are considered not helpful in successfully performing the task. In cases where the evaluation reveals a success score below or equal to the threshold, the controller sends a request for assistance via the communication interface. In cases where the success score is estimated to be above a preset threshold, the processing unit is configured to automatically generate a signal to cause the service robot to perform the action corresponding to the selected action candidate.

[0011] In response to a request for assistance, the system reads input received from the operator responsible for remotely controlling the robot and / or inputting information for teaching the robot. Examples of such input will be explained in more detail later. The robot then performs an action based on the input provided by the remote operator. According to a preferred embodiment, actions performed based on remote control operations are evaluated for their potential to be beneficial for future tasks. In cases where it is determined that a particular action can be successfully applied to a future task, the corresponding action definition is stored in a database. Then, if the processing unit searches for action definition candidates for a future task, the added action definition is also available. If the action definition is selected because it has the highest success score, the newly learned action can be performed by the robot. Therefore, any action definitions generated based on operator input in response to requests sent by the processing unit (since the current situation might not be handled without operator assistance) increase the service robot system's "knowledge base," and the system thus learns during its daily operation.

[0012] According to an advantageous implementation, the success score of each action definition candidate is determined based on information about the success of past executions of actions corresponding to the respective action definition candidate. Therefore, each time an action according to an action definition candidate is executed, the system stores information about the execution of that action and the circumstances associated with whether the execution of that action was successful. Based on knowledge about the success of that particular action when the specific action corresponding to the action definition candidate was executed, the processing unit can then calculate the success score of the action definition candidate in the current situation. This success score is based solely on historical knowledge of the action definition candidate. The success score does not reflect whether the actual situation has been correctly determined. Instead, the determination begins by assuming that the current situation has been correctly determined, such that the currently experienced situation can be compared with similar situations experienced in the past. This similarity comparison then enables the calculation of the success score for the current situation. The calculation may preferably consider a measure of similarity between the currently experienced situation and the stored situations.

[0013] An advantage is that the controller is configured to break down the defined task into a series of actions and their corresponding action definitions. This is advantageous when the entire task is quite complex. Therefore, the actions corresponding to the segments are relatively primitive, and thus, it is easier to predict whether the execution of a single action will be successful. As an example, the task "put the book back on the shelf" can be broken down into: 1. Move the robot to the workbench; 2. Move the arm to the gripping position; 3. Close the gripper to pick up the book; 4. Place the book on the tray; 5. Move the robot to the shelf; 6. Pick up the book from the tray; 7. Move the arm to the release position; and 8. Open the gripper to place the book on the shelf. It should be noted that breaking down the task to be performed may result in a greater number of actions than the number of actions produced by the segmentation in the example given above, but it may also result in a smaller number of actions.

[0014] It should be noted that a preferred form of "manipulation of the robotic environment" is pick-and-place operations, which will be further explained below. However, this does not limit the invention. For such pick-and-place operations, the robot has a gripper and may have one or more auxiliary tools. Suction devices or expanders may be used as tools. In a preferred embodiment, these tools and / or end effectors may be interchanged to allow for adaptation to the task.

[0015] Furthermore, preferably, the system is configured to initiate action execution only when the probability that the action can be completed uninterruptedly for the action definition is higher than a predefined threshold. It should be noted that dynamic objects will exist in the environment, which may change position (i.e., a moving dog) or change state (i.e., a person filling a cup with water), imposing new constraints on the environment. According to a preferred embodiment, the processing unit is configured to dynamically predict potential changes in the environment and adjust or select actions while still maintaining the final safe target state. Specifically, in cases where a series of actions are required to complete a task, this method ensures that the robot is always in a safe state once a single action has been completed. The action definition always defines this safe state at its end. Therefore, the robot will always be in a state in which it can receive and execute remote control commands. In cases where the processing unit is far from the robot, the signals transmitted from the processing unit to the robot always include any information necessary for the robot to complete the action corresponding to the transmitted information. Even in cases where the connection between the robot and the processing unit is interrupted, the robot has sufficient information to enter a safe state. It should be noted that in any case, the robot itself has processing capabilities, which enables the robot to use signals emitted from the processing unit to generate control signals for driving the drive system and / or actuators.

[0016] Further preferably, the robot is equipped with at least one sensor for physical sensing of its environment and includes an interface for transmitting sensor outputs to a processing unit and / or an operator interface. Transmitting sensor signals to the processing unit allows the controller to analyze the robot's current situation. Transmitting sensor outputs to the operator interface allows the operator to analyze the robot's situation. In both cases, the sensor signals are the basis for determining the robot's next action. Particularly preferably, the sensor includes a camera, or even more preferably, multiple cameras.

[0017] While the operator interface used to provide information about the current situation in which the robot needs to be controlled can be a display showing images captured by sensors mounted on the robot (or even by stationary sensors, such as surveillance cameras in a house), according to an advantageous implementation, the operator interface includes an Extended Reality (XR) suite, for example, for virtually controlling the robot to perform operations, even before remote control signals are transmitted to the robot to perform the same operations in the real world. In this embodiment, the entire action can be performed virtually by the operator before the corresponding information is transmitted to the robot and control signals are generated by the robot. This has the significant advantage that the remote control operation of the robot is not subject to lag time, which typically causes overshoot in the movements performed by the robot. In other cases, augmented reality can be used instead of virtual reality. An exemplary case could be an assistant doctor / nurse who is in the same room as the robot and, in this scenario, acts as the operator to teach the robot.

[0018] It should be noted that an "action definition" is a general definition of any kind of motion performed by a robot or a part of a robot. Therefore, an action definition can define the motion of actuators, any kind of additional tools attached to the robot, and / or its drive system. Action definitions enable the robot to adjust the execution of actions (e.g., the distance traveled in a case where the drive system is controlled based on the action definition), thereby defining the final trajectory. This adjustment is situational and performed under conditions where the robot's environmental conditions are known. To detail the action after retrieving and selecting action definition candidates from the database for execution and before finally generating control signals, the processing unit calculates how to execute the action based on information extracted from the sensed environment to calculate the final trajectory of the robot and / or its actuators. As an example, the motion of an actuator (an arm with a gripper / hand attached at its distal end) depends on the height of the object to be picked up. Based on images captured from around the service robot, the processing unit can calculate the correct position of the object and adjust the motion parameters in the action definition accordingly. It should be noted that in this context, the term "image" refers not only to images captured by cameras that produce two-dimensional (2D) images, but also to information derived from any type of sensor that allows the production of representations of the environment. Examples of such sensors are radio detection and ranking (RADAR) and light detection and ranking (LIDAR) sensors that produce three-dimensional (3D) point clouds.

[0019] According to a preferred embodiment of the service robot, in addition to the at least one actuator, the robot also includes at least one support structure, which, along with the actuator, is mounted to at least one lift. While in a simple embodiment the support structure and actuator may be mounted to the same lift, it is preferable that the heights of the support structure and the end effector are individually adjustable. In this case, two lifts are mounted on the robot's mobile platform (robot base). Providing such a support structure reduces the distance that needs to be covered by the actuator. Using the support structure, multiple items can be collected and placed on it. The support structure can be a basket, box, flatbed, pallet, or similar device, or even a forklift carrying the box or platform. This can be done by lowering the support structure (e.g., in the case of collecting multiple items from the floor), and when these items must be placed in a higher shelf, the support structure and actuator can typically be raised before the items are returned to the shelf. Therefore, the support structure allows for a significant reduction in the movement required by the service robot. Obviously, this also applies to situations where the starting position and the destination or target position are at the same height, but if each item will be transported separately, the service robot will need to move back and forth.

[0020] To enable the operator or processing unit to have a detailed and accurate understanding of the robot's current operational status, cameras or other sensors capable of imaging the robot's environment are attached to the actuators used to perform the task. Therefore, the perspective of the captured images can be optimized by moving the end effector, or in more complex embodiments, by moving the camera (sensor) relative to the actuator to which it is attached. The service robot carries at least one additional camera (sensor) mounted at different locations on the robot and / or with different zoom factors to provide an overview beyond the details of the scene.

[0021] Alternatively, the camera attached to the actuator can be moved, thereby capturing multiple images from different positions that provide both detailed and overview views. Obviously, this slows down the overall operation, so actuator movement is necessary to obtain sufficient information about the conditions that allow the robot to operate safely.

[0022] Besides requesting remote control operation without knowing how the operator will respond, it is preferable that the request include at least one action definition candidate retrieved from a database. This is particularly useful in situations where one or more action definition candidates exist, which are estimated to potentially contribute to the execution of the overall task, but which do not have a success score that clearly indicates the corresponding action can be successfully executed. In this case, action definition candidates that result in a success score below a preset threshold can be included in the request and offered to the operator as a suggestion. The operator then selects one of the suggested action definition candidates, and the robot continues to execute the action according to the operator's instructions. In this case, the operator does not need to control the entire action, as the robot can execute it automatically once the action is defined. Alternatively, if the multiple action definition candidates receive success scores above a threshold, but the success scores are not significantly different enough to identify a preference, these candidates can be included in the request, leaving the final decision to the operator.

[0023] As described above, evaluating action definition candidates involves comparing the calculated success score for each candidate with a preset threshold. For example, the task is to put a book back on the bookshelf. Initially, the system needs to identify and recognize the book on the table. Therefore, the system first calculates the probability that the table is correctly identified and the probability that the object being grasped is a book. Based on the certainty of correct identification of the object, a success score can then be calculated for the action involving the object to be successfully performed.

[0024] This success score is then compared to a preset threshold, which can be calculated based on one or more tags stored in association with the object involved in the action to be performed. In the example above, the object is a book. A “tag” is a piece of information associated with an object determined based on sensor signals. In cases where the sensor includes a camera, image processing is performed to identify the object (e.g., a cup on a table). This cup may be made of different materials (e.g., metal or porcelain), and therefore handling the cup may require different levels of care. Thus, if the cup identified from the captured image is identified as being made of porcelain, the threshold can be set higher than that for a cup made of metal. Obviously, the multiple tags in different categories can be associated with objects that the system can recognize. These tags can be added by the operator when requested and when the operator identifies an object or its characteristics in response. Inputting additional information associated with the identified object and storing that information will result in an improved database.

[0025] It should be noted that requests sent by the controller enable remote control of how the service robot performs actions, and also allow the addition of information about objects identified in the robot's environment. The latter automatically enables the robot to perform actions because enhanced situational understanding is achieved through additional information about the objects. Since the processing unit permanently re-evaluates the situation and potential actions that might be performed for the success of the task, the information about the objects involved in task execution is immediately considered for evaluation, and even for selecting the next action. Therefore, the selection of action definition candidates is dynamically implemented, taking into account changes in the robot's environment (e.g., when the robot moves to different locations).

[0026] Furthermore, preferably, the system stores encountered situations by generating combinations of context descriptors associated with past instructions used to perform tasks. Storing the history of tasks and the encountered situations defined by combinations of multiple context descriptors allows for the automatic inference of which task is most likely to be executed based on future identified situations. This information about the history can include not only the encountered situations and the instructions given in those situations, but also the frequency of encountering similar situations without given instructions. Attached Figure Description

[0027] To better understand, the system will now be described with reference to the accompanying drawings, in which...

[0028] Figure 1 A simplified diagram of a service robot is shown.

[0029] Figure 2 This is a block diagram showing the main units of a service robot system.

[0030] Figure 3 This is a flowchart illustrating the main methodological steps during task execution. Detailed Implementation

[0031] Figure 1 This is a simplified diagram of the service robot 1 used in the service robot system according to the present invention. Figure 1 The robot 1 shown is merely an example, and it goes without saying that other structures of robot 1 can also be used.

[0032] Robot 1 will be used to assist people in performing flexibly definable tasks. Therefore, a fundamental requirement is the ability to provide assistance at different locations. This is achieved by moving Robot 1 to the location requiring assistance. Robot 1 includes a robot base 2 designed to house a drive system. In the figures, only wheels 3 of the drive system are shown. Motors and energy sources such as batteries are not shown and are arranged inside the robot base 2. The specific structure of the drive system is irrelevant to this invention, and various different structures optimized for the working area of ​​Robot 1 are known in the art. The drive system enables the entire robot 1 to move from an initial position to a target position. The drive system includes all necessary components for driving and changing direction in response to received control signals.

[0033] To perform the task, robot 1 includes an end effector 4 attached to an arm 5 for positioning an end effector 4. Arm 5 consists of at least a first element 5.1 and a second element 5.2, which are connected to each other via a joint, allowing adjustment of the relative angle between the first element 5.1 and the second element 5.2 to move the end effector 4 to a desired position and orientation. In the illustrated embodiment, the end effector 4 is designed as a grasping tool mimicking the human hand. Such an end effector 4 appears to be the optimal tool for efficient assistance in the daily lives of those being assisted. Many tasks include picking up objects, moving objects, and storing objects in designated locations. On the other hand, such a grasping tool can also assist nursing staff or doctors in hospitals. For example, the grasping tool can be used to deliver food to patient rooms without the nursing staff entering the room. Therefore, the need for disinfection of the clothing, hands, etc., of nursing staff or doctors can be reduced.

[0034] It should be noted that, in addition to the movements of the first element 5.1 and the second element 5.2 indicated by the arrows in the diagram, more degrees of freedom can be achieved for the first element 5.1, the second element 5.2, and the end effector 4. Specifically, rotational movement of the first element 5.1 about its longitudinal axis and rotational movement of the second element 5.2 about its longitudinal axis are possible. Particularly preferred is that the end effector 4 can also perform rotational movement about at least two axes perpendicular to each other.

[0035] Arm 5, together with end effector 4 attached to the distal end of arm 5, is mounted to first lift 6. First lift 6 is fixed to robot base 2 and, in the illustrated embodiment, includes a fixed element 6.1, which may be a hydraulic cylinder or a pneumatic cylinder, and a movable element 6.2. In the illustrated embodiment, movable element 6.2 supports strut 8, which supports the proximal end of the first element 5.1 of arm 5. First lift 6 is adjustable in its overall length to raise strut 8 above robot base 2, and thus also raise the proximal end of arm 5.

[0036] The first lift 6 enables the end effector 4 to reach a higher position relative to the ground where the robot 1 stands. The first lift 6 can be designed to be quite rigid, and its ability to lift the proximal end of the arm 5, the first element 5.1 and the second element 5.2, and the end effector 4 can be designed to be lightweight and relatively small without sacrificing any operating range.

[0037] A preferred application of the service robot 1 is to collect items and return them to their designated locations, such as putting toys back on shelves in a children's room or plates from a table back in the kitchen. This requires considerable back-and-forth movement of the robot 1 when the location where items must be collected is far from their respective designated locations. Moving around is even more difficult when the items to be returned to their designated locations are on the ground. This is avoided with the robot 1 according to the invention by providing a second lift 7, which also includes a fixed element 7.1 and a movable element 7.2. A support structure 9 is attached to the movable element 7.2. In a simplified embodiment, the support structure 9 generally comprises a platform 10. The platform 10 serves as a temporary repository, allowing multiple items collected using the end effector 4 to be placed on the platform 10. Once all items have been collected, the robot 1 is moved to a target location from which the items are placed one by one into their designated locations.

[0038] Often, the designated location is not close to the ground, and therefore, the first elevator 6 is used to raise the arm 5 to an elevated position so that the end effector 4 can reach the target location of the corresponding item. Similarly, in cases where the platform 10 is positioned at a fixed height above the ground, it is necessary to repeat the operation of the first elevator 6 for each item. According to the robot 1 of the invention, the platform 10 can also be raised. Preferably, the first elevator 6 and the second elevator 7 are designed such that even at the maximum height of the first elevator 6, the arm 5 with the end effector 4 can reach the platform 10 without further operation of the first elevator.

[0039] Preferably, such as Figure 1 The robot 1 shown includes a first lift 6 and a second lift 7. However, the support structure 9 can also be attached to the first lift 6 so that, in any case, the platform 10 and the strut 8 are in a fixed relative position to each other, thus ensuring that the end effector 4 can reach the platform 10.

[0040] The use of hydraulic or pneumatic cylinders in lifts 6 and 7 should be understood as merely examples. Linear actuators or spindles may also be used. Furthermore, the figures show a single-stage lift, but two or more stages may be used in cases where a greater height above the ground is required.

[0041] The benefits of using service robot 1 to assist people increase with its independence, meaning that the number of situations in which robot 1 can perform automated operations increases. This independence can only be achieved when the service robot system is aware of the environment in which service robot 1 operates. One or more sensors are used to obtain knowledge about the current location and state of the environment. As an example of usable sensors, Figure 1 The embodiment shown includes a first camera 11 and a second camera 12. The first camera 11 is attached to and moves with the second element 5.2 of the arm 5. Therefore, the image captured by the first camera 11 will only show details of the environment of the robot 1 near the end effector 4. Since the robot 1 (more precisely its processing unit, as will be explained later) also needs an overview of the robot 1's environment, the second camera 12 is configured to capture images of a larger area of ​​the robot 1.

[0042] although Figure 1 The diagram shows a first camera 11 and a second camera 12, but other sensors capable of physical sensing of the environment and thus obtaining information about the environment to generate a representation of it can also be used. Examples include RADAR sensors, LIDAR sensors, ultrasonic sensors, and similar sensors. Furthermore, the entire service robot system can utilize sensors fixedly mounted in the environment in which the robot 1 operates. As an example, a surveillance camera can be used. Finally, the number of sensors is not limited to the two shown. Specifically, in cases where there is a sensor attached to the arm 5 or any other movable support that allows the sensor to be moved to different positions, a single sensor is sufficient. Information about the environment of the service robot 1 is then collected, thereby obtaining information about different positions and / or orientations from a single sensor. Of course, even when using multiple sensors to generate a representation of the service robot 1's environment, more than one capture from each sensor can be used. The amount of information obtained from the environment can be adjusted in response to the analysis of the environment performed by the processing unit, as will be explained later. For example, in situations where the reliability of information obtained from sensing the environment (the certainty of correctly identifying objects) seems insufficient, robot 1, or at least an element carrying movable sensors (such as the first camera 11), can be moved to obtain additional information by sensing the same environment of robot 1 from different perspectives. Furthermore, methods such as sensor fusion, 3D reconstruction, and overcoming occlusion can be used.

[0043] Figure 2 This is a simplified block diagram showing the overall layout of the service robot system. (Refer to...) Figure 1The service robot 1, as detailed in the description, is a major component of the entire system. Robot 1 includes sensors, which, as described above, may be cameras 11 and 12. Furthermore, robot 1 includes an interface 13 connected to the sensors to transmit signals, including information about the sensed environment, to a processing unit 14. Needless to say, in cases where some or all of the sensors are located externally to the robot, separate interfaces are required for transmitting the sensor signals to the processing unit 14.

[0044] In the illustrated embodiment, the processing unit 14 is arranged outside the robot 1. However, it is also possible to include the processing unit 14 within the robot 1. Positioning the processing unit 14 outside the robot 1 allows its processing capabilities to be utilized not only for a single service robot 1 but also for multiple robots. Communication with other robots is indicated by dashed arrows. Furthermore, the processing unit 14 does not need to be implemented by a single processor, but can also be multiple processes that collaboratively process received signals. It is also possible that such multiple processors jointly establishing the processing unit 14 are distributed, with some processors arranged within the robot 1 and others arranged at remote locations and communicating with each other via corresponding interfaces.

[0045] It is even possible that each of the multiple robots 1 utilizes at least one shared processor, but also includes individual processors for internal signal processing. Such internal processors may be specifically used to generate control signals, as instructed by driver 15, which generates control signals for actuator 16 based on information received from processing unit 14 regarding the actions to be performed. Actuator 16 will instruct all individual actuators required for driving the wheels 3, elevators 6, 7, and positioning the arm 5, including the end effector 4.

[0046] Processing unit 14 is connected to database 17, in which action definitions are stored. Action definitions define general algorithms for performing specific actions by robot 1 without requiring precise trajectory definition. Actions are, for example: grasping an object, moving to a different location, placing an object on a table and releasing it, etc. Furthermore, action definitions can be organized in a hierarchical structure, such that starting with basic action definitions, these basic action definitions can be combined to build higher-level action definitions. Action definitions can be organized in just one, two, or more layers of the hierarchical structure. Starting with the action definitions given above as examples, a higher-level action definition could be, for example, "placing a dropped object back on the table," which includes the aforementioned action. Of course, the actions given above as examples can even have higher granularity.

[0047] A key aspect of the operation of the service robot system of the present invention is the communication between the robot 1 (including its "intelligence" in the form of a processing unit 14) and the operation center, where an operator supports the operation of the robot 1 in situations where the robot 1 (including its "intelligence") cannot handle the situation independently. This is achieved by connecting the processing unit 14 to at least a display 18 (preferably multiple displays) and an input device 19 via an operation center interface 21. The display 18 is used to output information to the human operator, enabling the operator to estimate the current status of the robot 1. Assistance in the operation of the robot 1 can be implemented in different ways:

[0048] - Directly control the robot's movement

[0049] - Select the appropriate action definition from the database, or

[0050] - Add information for object identification and / or object representation to database 17.

[0051] Display 18 shows a representation of the robot 1's environment, including the robot's position identified from sensor outputs and nearby obstacles. In cases where the sensors include one or more cameras 11, 12, the captured images can also be displayed as live camera feeds. When using multiple cameras 11, 12, the operator can switch between different cameras to facilitate control of the robot 1. Alternatively or additionally, the display can be used to provide information about identified or unidentified objects and the current state of the assessment of the situation being performed by the robot 1. As will be explained later, this may allow suggestions for actions the robot might perform, but it may also lead to questions in situations where responses received from the operator cause further ambiguity.

[0052] To begin operating robot 1, the task to be performed needs to be determined. This determination is also performed using a corresponding software module in processing unit 14. Therefore, the determination unit is a software module executed by processing unit 14. Similarly, a dedicated processor can be used to execute the software module of the determination unit. To determine the task to be performed, signals from sensors are processed. One sensor may be, for example, a microphone 22, allowing analysis of verbal instructions from the user to robot 1 to derive the task. Another way to determine the task to be performed is by using the history of previously performed tasks and information about the circumstances surrounding their execution. In this case, the determination unit uses so-called context interpreters to compare the current situation encountered by robot 1 with information about past executions of specific tasks. The associations are also stored in database 17. Based on the signals from the sensors, processing unit 14 compares the current situation of robot 1 with the situations stored in database 17, where the situations are defined by combinations of context interpreters. In cases where a similarity exceeding a threshold is identified, the task associated with that situation is determined to be expected.

[0053] Once the task is determined by the task-determining unit, the processing unit 14 attempts to find a solution for automatically executing the task. The processing unit 14 retrieves one or more action definition candidates from a database, which are assumed to be helpful for successful task execution based on information about the environment derived from sensor signals. After objects in the robot 1's environment have been identified based on analysis of signals received from sensors, action definition candidates are searched in database 17 by searching for labels that describe the objects' characteristics. Operator support for task execution is only requested if automated operation of the robot 1 is impossible or at least unreasonable. Details of the analysis that leads to the decision of whether to request operator assistance will be given later.

[0054] When processing unit 14 concludes that automatic task execution is impossible or unreasonable, it sends a request for assistance to a remote operation center via communication interface 21. In response to the request received from processing unit 14, a human operator either controls robot 1 or adds information to the system's database 17. As described above, controlling robot 1 can be a direct control of its movement (e.g., using controls similar to those used in computer games as input devices, or selecting appropriate action definitions from database 17), based on a proposal made by processing unit 14. According to a more preferred embodiment, the system also includes a virtual reality kit 20, which allows the operator to see virtual objects and virtually navigate robot 1 towards them. The virtual reality kit 20 includes not only a virtual reality headset but also a virtual reality controller, which allows the remote operator to determine, for example, a grasping posture in virtual reality before sending the corresponding action definition to robot 1. Therefore, using the virtual reality kit avoids any collisions that might occur with robot 1 due to lag time in cases where robot 1 is directly controlled.

[0055] To control robot 1 using virtual reality kit 20, it is advantageous to use a 3D camera as the first camera 11 to generate a 3D representation of the object intended to be grasped by end effector 4 and its environment. The correct grasping posture of end effector 4 can be determined when a model of the physical grasper is assigned to a virtual reality controller representation in the virtual environment (or augmented reality). A remote operator can then place the 3D representation of the physical grasper next to the virtual object. A second controller is used for remote control of the operation of the drive system.

[0056] In the following text, we will use, for example Figure 3 The simplified flowchart shown illustrates in detail the typical process in the case where the initial setup of the system has been performed.

[0057] First, in step S1, at least one sensor is used to sense the environment of the service robot 1. This sensor can be mounted on the service robot 1 or at a suitable location to allow for analysis of the environment. In step S2, objects in the robot 1's environment are identified based on the sensor output. The sensors can be of the same or different types. The service robot system now needs to determine the task. Determining the task to be performed by the service robot system in step S3 can be done in several different ways. A preferred approach is to automatically move the robot 1, which includes automatically determining the task to be performed. However, in many cases, the robot 1 will not be able to identify what it will do on its own and therefore requires assistance. A typical way to instruct the robot 1 to do what it will do next is through direct communication between the user and the robot system. This communication with the robot 1 can use methods already referenced... Figure 2 The microphone and speaker 23 are used to enable the robot 1 to output information to the user. Instead of the microphone and speaker 23, an additional interface can be provided to connect the robot 1 to a user device, such as a smartphone. In this case, a smartphone or similar device can be used to input instructions that enable the robot system to determine the task to be performed. Generally, information can be supplied to the robot system in a variety of ways, including Short Messaging Service (SMS), apps, verbal commands, or human gestures. Some of these "information channels" may require additional interfaces to connect the robot system to another IT system (e.g., a camera providing signals that can be processed by the service robot system) to identify and analyze gestures, thereby determining the task.

[0058] Using microphone 22, robot 1 can receive verbal commands from a user. Processing unit 14 can then determine the task the user intends robot 1 to perform based on the received verbal commands. Similarly, the user can give commands using SMS or gestures (e.g., pointing to an object). Pointing to an object can be understood by the system as focusing the determination of the task on that specific object. In the case where the pointed object is a trash can, the determined task is identified as a potential action that can be performed on that specific object. In the case of a trash can, this might be "empty the trash can".

[0059] More preferably, robot 1 determines (using knowledge about its environment and potential additional information gathered from sources connected via the interface) the task most likely to be performed. When the probability of a particular task being performed exceeds a given threshold (which may be adjustable), robot 1 can automatically begin determining the necessary actions to be performed to execute the determined task. Potential tasks can be stored in database 17 and can be searched to find tasks that may be suitable for the current situation. If the probability of determining the correct task to be performed is not high enough, robot 1 can, for example, ask a question to the user via speaker 23 or to the operator via other interface 21. The user or operator can then instruct robot 1 by confirming the task suggested by robot 1, or, in the case where robot 1 does not offer a suggestion, by directly instructing and defining the task to be performed.

[0060] The following will explain how robot 1 can determine a task from a user via microphone 22, which is directly mounted on robot 1 or anywhere in robot 1's work area and connected to the service robot system. This allows processing unit 14 (and therefore the determination module) to receive verbal instructions and derive information about the expected task to be performed based on these received verbal instructions. For example, a user of robot 1 might instruct robot 1 to "clean the children's room." In this case, robot 1 receives this clearly defined task, and robot 1 does not need to assess its environment in order to determine the task in step S3.

[0061] It should be noted that Robot 1 not only uses microphone 22 to listen for any verbal instructions, but Robot 1 will always "listen" to whether the user has given a verbal instruction. This is especially important because instructions that directly address Robot 1 always override any task or action that Robot 1 is currently performing. This ensures that Robot 1 does not continue performing tasks that the user deems inappropriate. This interruption will allow the robot to directly enter the next safe state. An example could be cleaning a child's room, but the child's room is currently needed because the child is still playing there. To avoid conflicts of interest, previously determined tasks (whether instructed by the user or automatically determined by Robot 1, as described below) are overridden by the last instruction given by the operator or user.

[0062] Robot 1 analyzes the current situation and compares it with prototype situations stored in database 17. Prototype situations are past encounters that led to the execution of the same task. Prototype situations can also be predefined when the service robot system is designed and programmed. To analyze the situation, processing unit 14 collects data describing the current situation. Each piece of information is stored in the form of a descriptor. Multiple descriptors combine to form a prototype situation. In cases where a task is instructed to be performed by robot 1, information about the combination of descriptors in this situation is stored in association with the instructed task. When a similar situation occurs in the future, a comparison between the stored information about the situation (the combination of descriptors) and information derived from sensor signals sensing the robot 1's environment in the current situation reveals a high degree of similarity between the encountered situation and the prototype situation. It can then be concluded that the stored task associated with the prototype situation is to be performed again. Therefore, processing unit 14 calculates a measure of similarity between the stored prototype situation and the currently experienced situation based on information collected in the current situation and sensed by sensors (and / or retrieved from other sources). In cases where this metric exceeds an adjustable threshold, the processing unit 14 retrieves tasks stored that are associated with prototype situations identified as sufficiently similar to the current situation experienced by the robot 1.

[0063] Even if no instructions are received in these situations, the system can still detect the occurrence of recurring situations. Observations can be transmitted to the operator, who can then mark them as new prototype situations, and preferably, associate tasks with these new prototype situations.

[0064] The “situation” is defined using the interpreters described above. A combination of interpreters, including multiple independent interpreters, constitutes this prototype situation. These interpreters can include not only information about location, time of day, workday, or the difference between the current state and the target state of the environment, but also information obtained from other information sources via the interface. For example, an information source could be a user's calendar. Entries in the calendar can frequently trigger specific tasks that will be performed by robot 1. An example (still referring to the situation of cleaning a child's room) is an entry in the calendar similar to “playday.” When attempting to determine the next task to be performed, robot 1 (more precisely, its processing unit 14) searches for information sources that it communicates with. When robot 1 determines, based on a comparison of the current time of day with the entry in the calendar, that the playday should have ended, it is likely that the child's room needs to be cleaned. Even the social environment may be included in the decision of which task to perform. For example, cleaning the room might be done differently before the children's friends arrive and before the grandmother visits.

[0065] Since task execution is stored in association with the descriptors of the situation in which Robot 1 is instructed to perform the task, the service robot system gains experience over time. However, analysis of the current situation often leads to ambiguous results, and therefore Robot 1 cannot automatically determine the task to be performed. Although the frequency of such ambiguity decreases over time, an auxiliary determination unit is needed to identify the task to be performed. To avoid Robot 1 performing a task that is unreasonable in the current situation, it is insufficient to always select and display the task corresponding to the prototype situation with the highest similarity to the stored prototype situation. In this case, it may become necessary to ask the user or operator of Robot 1 to clearly define the task to be performed. A second threshold below the first threshold can be introduced. The first threshold should be set high enough to ensure that only one task is suitable for the current situation of Robot 1. With the second threshold set low, there is a possibility that multiple stored prototype situations may show a similarity to the current situation that is lower than the first threshold but higher than the second threshold. In this case, the service robot system can output a question to the user or operator (a request for assistance). The question can suggest a task associated with the prototype situation, for which similarity has been determined to be above a second threshold but below a first threshold. This question could be asking for confirmation of one of these tasks, or it could be an open-ended question such as, “What should I do?” (in situations where the prototype situation cannot be identified based on the information available and the associated task cannot be determined).

[0066] In the case where the request for the task determined by the auxiliary determination unit in step S3 is sent to the operator to obtain information about the task to be performed, all information collected by the processing unit 14 from all available sensors and other information sources is also forwarded to the operator, for example via the display 18 or Figure 2 A speaker (not shown) is outputting information. Thus, the operator gains knowledge about the status of robot 1, which prompts questions to be asked. The operator can then even mark specific combinations of descriptors of the status to more easily respond to requests received from robot 1 in the future.

[0067] Once the task to be executed has been determined, the process proceeds to step S4, where analysis of the task to be executed begins. Database 17 includes multiple action definitions. Each of these action definitions can be constructed from multiple actions, which themselves may include multiple further actions. This means that each action definition stored in database 17 can be combined with one or more other action definitions to create a new action. While it is possible to store new, higher-level action definitions for any possible combination of executable action definitions, this approach is impractical. More preferably, the task to be executed is divided into smaller blocks, each of which may correspond to a single action, for which the action definition is stored in database 17. Since the action definitions stored in database 17 can have a multi-level structure, such a single action definition can obviously be composed of combinations of other action definitions. Once the task has been divided into multiple actions to be executed (if necessary), processing unit 14 begins retrieving potential action definitions as candidates in step S5, and evaluates these action definition candidates from database 17 in steps S6 and S7.

[0068] Processing unit 14 has an evaluation function that enables it to determine whether a specific action is likely to be successfully applied in the current situation. Processing unit 14 calculates a success score for each action (action definition candidate) that would be applicable in the current situation. Based on this success score, an action can be selected to perform the task. The success score is a metric that allows comparison of the likelihood of success of different actions when applied to the current situation. For example, the score can be a probability or likelihood. Once a success score has been determined for an action definition candidate, the score is compared to a threshold. If the score exceeds the threshold, it is concluded that the success of the action is sufficient for the current situation if it is performed. This evaluation, which considers the objects involved in the action by adjusting the threshold, will be explained later with an example. In step S6, the evaluation of one or more action definition candidates is performed.

[0069] Task execution is carried out by performing actions one after another. Therefore, after dividing the task to be executed in step S4 and evaluating the action definition candidates in step S6, further task execution can be carried out in two different ways. If the success score of the action definition for the next action to be executed is higher than a given threshold (comparison step S7), and this action definition is the candidate with the highest success score among all evaluated action definition candidates, the action will be automatically executed in the next step S8 based on this action definition. After the execution of this action is completed, it is determined whether the initially determined task is completed. Therefore, in step S9, the result achieved by the action is compared with the target state defined by the determined task. If the task is completed, the program ends in step S10. If the task is not completed, the program returns to step S7, where step S7 evaluates the next action to be executed. It should be noted that performing actions may change the environmental conditions of robot 1. Therefore, contrary to the arrow leading from step S9 to step S7, the changed environmental conditions allow for a search for new action definition candidates, retrieval of new action definition candidates from database 17, and evaluation of the new action definition candidates before comparison of success score and threshold can be performed in step S7.

[0070] To learn from the system's operational history, the system stores the situations in which the system has been operated. For each situation, the system stores the action corresponding to the action definition candidate associated with information about whether the action was successful in the corresponding situation. This allows it to conclude that the action corresponding to the action definition candidate, when performed again in a consistent situation, will again be successful. In the case where the success score is a value between 0 and 1, the success score obtained in this simple example can be 1. If no similarity is identified between the current situation and any of the stored situations in database 17, the success score obtained will be 0. Since this is highly unlikely for every actual situation and every consistent situation that has occurred in the past, even for such a new situation, a success score should be determined. The success score is determined using a calculation of the similarity between the currently experienced situation and the situations stored in database 17. For the example described above where the success score is between 0 and 1, the higher the similarity between the stored situation and the actual situation, the closer the obtained success score will be to 1. Conversely, the closer the success score is to 0, the lower the similarity between the two situations. Of course, not only the stored cases with the highest similarity to the currently experienced situation can be considered, but multiple stored cases can also be considered. Similarity calculation can take into account visual parameters, such as classified objects, and the influence of the environment (such as the presence of people). Similarity calculation does not consider any ambiguity when determining the currently experienced situation, but rather assumes that the output of situation perception is correct.

[0071] In situations requiring multiple actions to perform a task collaboratively, each corresponding action definition candidate is evaluated as described above. Furthermore, the sequence of action definition candidates needed to perform the entire task can also be evaluated. This can be achieved by combining the success scores of each action definition candidate involved in performing the entire task.

[0072] It should be noted that, in order to perform the overall task, success scores for action definition candidates need to be determined at a later point in time, predicting the future outcome achievable by performing the preceding actions. This outcome is then used for similarity calculations to determine the success scores for subsequent action definition candidates. This evaluation is repeated continuously, such that if the predicted future outcome cannot be achieved through the performed actions (e.g., due to inaccurate positioning or changes in environmental conditions), then an evaluation can be considered.

[0073] The target state is defined by the task. Information about the target state of a task can be changed by an operator, for example, by adding details to the target state. The target state can also be indirectly defined by labels (indicating their regular positions or states) of the objects involved in the situation.

[0074] If a sequence of actions is determined by a success score above a threshold in order to perform the defined task, the service robot system will be able to automatically execute the entire defined task. However, it is possible that during the execution of the action sequence, the next action to be executed, for which a score above the given threshold has been calculated, may not arrive. This clearly indicates that the system cannot execute actions independently without risking system failure. In this case, after the comparison of the success scores of the corresponding action definition candidates in step S7 reveals that the score does not exceed the given threshold, the service robot system sends a request to the operations center in step S11. The operator will receive the request and, using the display 18, input device 19, and / or augmented / virtual reality suite 20, will analyze the situation and provide appropriate suggestions to the service robot system by inputting information and / or control signals for direct control of the robot 1 or instructing the use of action definitions stored in the database 17.

[0075] Requests sent from the robot to the operator may include an emergency indicator and / or information selected based on the emergency indicator, and this information is transmitted along with the request to assist the operator in handling multiple robots 1 simultaneously. The emergency indicator informs the operator of the actual need for assistance. The emergency indicator can be calculated considering other parameters (e.g., the label of the object involved) based on the maximum achievable success score in the actual situation of robot 1. The greater the difference between the calculated success score and the maximum achievable success score of the determined action definition candidate, the higher the value of the emergency indicator (assuming a higher emergency indicator value means more urgent assistance in this situation). However, even for the same success score calculated for the current situation, specific aspects of the situation or the task to be performed can affect the level of urgency. An example is selecting an object from a table. Initially, this task might result in the same success score, but there is a significant difference between picking up a fragile or non-fragile object. Therefore, a higher emergency indicator can be used for objects labeled "fragile".

[0076] In situations where the entire robot 1 in the system is supervised by multiple operators, emergency indicators can also be used to help assign requests to different operators.

[0077] Emergency indicators can also be used to define the information presented to the operator. The higher the urgency, the more information is presented to the operator. For example, in situations requiring immediate operator assistance, all the information needed to quickly identify the situation and determine the necessary assistance is provided. In addition to images captured by Robot 1's camera, distance or other sensor values ​​may be presented. Conversely, situations with low urgency may only require transmitting camera images, allowing the operator to monitor the current situation and interrupt operations in case of unforeseen developments. Generally, the type and amount of information regarding Robot 1's situation are adapted to the urgency level as described above.

[0078] To directly control robot 1 to perform a specific action, the operator uses controls to instruct robot 1 to execute the desired action to continue performing the defined task (step S12). In step S13, it is determined whether the operator's input can be directly converted into the robot 1's action or movement. If so, the action instructed by the operator is executed according to the remote control input received from the operator. Therefore, after the action defined by the operator has been completed, the process jumps to step S8 and will proceed automatically, as previously described for actions that can be executed automatically.

[0079] In step S12, if the operator's input is not a direct instruction for the movement and maneuvering of the service robot 1 and therefore there is no definition of the action to be directly performed, the process proceeds to step S14. As mentioned above, the operator can directly instruct the robot 1 on how to continue performing the action, but alternatively, the operator can add information to the database 17. This added information then allows the system to evaluate and improve the action definition candidates, and to "reconsider" the task segmentation and / or the selection of action definition candidates. This is indicated by dashed lines in the flowchart.

[0080] It should be noted that direct input of applicable instructions to robot 1 can be implemented in two different ways: First, robot 1 can immediately and directly execute any instructions input by the operator. Second, the virtual reality suite 20 can be used to pre-define actions by defining only the actions to be performed, and after completing such action definition, it can be sent to robot 1, which will then automatically execute the newly defined actions. In the case of using augmented reality, instructions can be generated in advance, but they can also be applied directly.

[0081] The evaluation results of the action definition candidates largely depend on the robot system's knowledge of its environment. Therefore, the robot's knowledge base is enhanced using additional information input by the operator in step S14. Depending on the operator's input, information added to the objects involved in the action to be performed may be sufficient for the subsequent processing unit 14 to handle the situation without further operator input. Thus, such input (which could be additional characteristics of the object or conditions acting on the object) can lead to an increase in the success score of the evaluated action. If so, the process proceeds to step S8, as described above. If the information added by the operator is still insufficient to increase the success score of the robot 1's automated operation, step S7 generates a new request sent to the operations center. The operator can then decide again whether direct control of the robot 1 is appropriate, or whether he prefers to improve the database 17 by adding further information.

[0082] The retrieval of action definition candidates in step S5 is based on identification certainty and also on information available for each identified object involved in the execution of the action. For example, in a situation where a cup is on a table and needs to be placed in different locations, the first action to be performed would be grasping the cup. Information included in the label of the object "cup" can define the action performed for the identified object. For cups, the information in the label may even include characteristics such as (for example) sensitivity to mechanical stress. Porcelain cups require more care than metal cups.

[0083] As explained above, the operator will input additional information in step S14, thereby improving the robot's "understanding of the real world." The operator will attempt to input additional information about all objects involved in the scene currently perceived by robot 1. This will result in two different ways improving the operation of the entire robot system over time: on the one hand, the action definitions stored in database 17 are improved, and a wider variety of action definitions will be available for future evaluation of new situations for robot 1. On the other hand, robot 1's knowledge base is improved by adding information about identified objects in robot 1's sensed environment. By adding this information, the information used by processing unit 14 in predicting the success of expected actions is improved. Therefore, as any new information is added to database 17, robot 1's ability to automatically perform tasks will be improved.

[0084] It should be noted that this invention is explained with reference to "objects." However, objects are merely used to enhance understanding. In fact, objects are given only as instances of "entities," which can consist of multiple objects, or even multiple lower-level entities, which can then be combinations of objects. From a hierarchical perspective, a house includes a kitchen, a roof, ..., and a kitchen includes a stove, a dishwasher, etc. It should be noted that in this example, these higher-level entities can be inferred from the presence of other objects, and its kitchen can be inferred as well. This method is called bootstrapping. Determining the location of the "kitchen" then makes it even possible to determine objects within the kitchen that are partially obscured.

[0085] In addition to action definitions, database 17 stores all information collected over time. Processing unit 14 considers this information when evaluating the current situation and decides whether to contact the operator or whether robot 1 can successfully perform the next action. This information can also be referred to as "world knowledge." World knowledge is improved each time information is added to any object or entity composed of objects in the stored models in database 17. The models enable the identification of objects present in robot 1's environment. Identification uses sensor outputs compared to the models stored in the database. An object is considered identified when sufficient similarity can be discerned between the stored model and the information derived from the sensors. Each object stored in database 17 may include multiple associated information fragments (referred to as tags). Each tag is a specific information fragment and can be linked to an action that enables performance on the object or entity. Whenever the robot system cannot automatically handle the current situation, additional information is added to database 17, and the operator inputs additional information in response to a request for assistance received from robot 1. All currently available information is presented to the operator, and processing unit 14 also uses all currently available information to search and evaluate action definition candidates. The operator then determines whether adding information to objects or entities in the form of additional tags will improve the robot system’s world knowledge in a way that improves overall performance, or whether direct control of robot 1’s actions is necessary.

[0086] The additional information added by the operator is not limited to adding new labels or correcting labels on existing objects that have already been identified. In cases where the system is unable to identify an object at all, the system will also contact the operations center and request assistance. The operator, provided with an image (representation) of the object under consideration, can then add a new model (new entry) to the database along with all information that the operator currently possesses and deems useful. Therefore, when an object is first identified as an object by the robot system, the system will automatically contact the operator to improve the system's world knowledge. After input by the operator, the additional knowledge is immediately available to processing unit 14. Thus, even if an unknown object is identified for the first time during task execution, the additional information can be used in the next step of task execution. To assist the operator in adding information to objects, the system can also propose available labels that have been added to similar and previously known objects. The operator can then select the most suitable label from the proposed list. However, operator-defined entries are also possible.

[0087] The information added to an object is not limited to its characteristics, but also includes executable actions or the conditions for performing those actions. For example, in the case where the system first perceives a shelf with a door, display 18 shows a corresponding representation (likely an image). The operator can then represent the object as a shelf, adding information about the shelf, including the door, and information about a series of actions that need to be performed when the object is placed on the shelf. The sequence would include first opening the door. Thus, if the object is recognized by the system in the future, the system knows that the object can be placed on the shelf, but first it needs to grasp the handle to open the door.

[0088] Although the above explanations all require the system to possess at least basic world knowledge, it is clear that the initially configured system possesses no such world knowledge other than the stored models of real-world objects already existing in database 17. During the system design phase, the most common objects are stored before the robot system is installed in its actual working environment. When configuring the system in its actual working environment, it may be helpful if not only the operator but also the user of robot 1 can input information into the system. Therefore, it is advisable to provide robot 1 with an interface to a user device (e.g., a smartphone, tablet, or similar device). During the configuration phase, robot 1 will then collect data about objects that can be sensed in the environment. A representation of these objects is presented to the user, who then inputs information.

[0089] To add information that enables Robot 1 to determine how to perform the task, it is also necessary to define, for example, the target state for each individual object. This target state is essential for interpreting the task. For instance, the task "cleaning the room" might mean that each object is placed back in its designated location. When the designated location is stored in a tag associated with each object, the robot system can thus know the designated location of each object and move the object to that location after recognizing that its current location differs from its designated location.

[0090] Starting with basic knowledge that enables Robot 1 to recognize, for example, different rooms and furniture, layouts, and typical objects in a home, Robot 1 then learns and generates improved world knowledge. This world knowledge is stored in a hierarchical structure. This means that, for example, a house (entity) consists of multiple rooms (lower-level entities). Each room includes multiple objects within that room. Objects can be stationary or movable. An example of a stationary object might be an oven or refrigerator specifically for the room type "kitchen." Other objects may be movable and therefore can be encountered in multiple rooms. Movable objects can move on their own (e.g., a pet bird) or be moved by others (e.g., Lego bricks). For these movable objects, preferred rooms can be defined by the user or operator, or can be learned over time.

[0091] Starting with this basic knowledge of the working environment of Robot 1, the first task can then be assigned to Robot 1. Improvement of world knowledge can be achieved progressively whenever the operator intervenes. As the world knowledge of each Robot 1 increases over time, it is possible that the operator will eventually be able to supervise a considerable number of robots. However, in the early stages of operating a newly configured Robot 1, the operator may focus only on a single Robot 1. It is particularly advantageous that the multiple robots can use the same "intelligence" (processing unit 14 and database 17 or a portion thereof), because adding information to database 17 improves the knowledge base of the multiple Robot 1s. Conversely, whenever one of the Robot 1s requests assistance, the added information is immediately available to all Robot 1s by referring to the same database 17.

[0092] The following will provide examples that enable understanding of how a label added to a particular object allows control over which action robot 1 performs or can perform. For illustration, the action performed by robot 1 is picking up a cup. This action, required to clean a kitchen or table, might need to be performed in many different situations:

[0093] First, consider situations where the system is already aware of the facts, either because it has successfully performed the task in the past, or because the identification of any objects involved in the situation can be performed without ambiguity, and the context is also known. In this case, robot 1 can perform the action of "picking up the cup" completely automatically. No operator contact is required.

[0094] Then, there exists a situation where at least part of the cup that should be picked up is unknown. In this case, the system can automatically determine that the confidence level of the correct action is lower than in the first case. However, if the label "unbreakable" can be found in the description of the object "cup" in this case, robot 1 can still attempt to perform the most probable action. This is a trial-and-error scenario where robot 1 can perform an action, and the result is analyzed after the action has been performed. A risk assessment is performed by processing unit 14 based on additional information included in the label of the object. The label may also include instructions to obtain additional information before making a final decision on whether the action can be performed. For example, in the case where the object to be picked up is a cup, the label may include instructions to determine whether there is still liquid in the cup. When it is determined that there is liquid in the cup, this may lead to prohibition of trial and error, even if trial and error is allowed in the case of an empty cup.

[0095] The third scenario involves picking up a cup, where only medium-risk situations are permitted. This might occur when the cup is identified as being made of porcelain, and therefore, if the action fails, the cup could break. In this scenario, the identified action can be suggested to the operator, who then responds by confirming the action can be performed or taking over control. When the picking problem is solved through an action defined based on the robot system's suggested action, the system prompts the operator to label the object accordingly. An example is: the system cannot clearly identify the board, but it suggests a picking action, which the operator confirms, and the action is ultimately successfully executed. In this scenario, the system prompts the operator to add a specific picking action that is a potential action to be performed on the board.

[0096] Finally, a completely unknown situation may arise where the system cannot automatically select an action. This occurs whenever the system is unable to identify an object that requires operational processing. In this case, processing unit 14 cannot identify and retrieve any potential action definitions available in database 17 for step S5, and therefore directly requests assistance from the operator.

[0097] The concept of picking up an object implies that any possible action can occur: there is always a range, from completely automatic actions to situations where the system cannot even suggest any action. This becomes apparent when considering the placement of objects: for many objects, multiple orientations are possible. For example, in the case of a book being placed back on a shelf, the book can stand upright as usual, but it can also be placed on its wide side surface. The decision of which possible orientation is correct depends on the specific context. In the label representing an object, the conditions that allow for the correct determination of orientation when placing the object can be defined. In the case of a book, the condition might be: if the target location is a table (or more broadly: a large surface), then the book will be placed on its wide side surface. Conversely, when the target location is a shelf and the book will be placed in the gap between other books, the orientation must be vertical.

[0098] Similar to the explanation of the picking operation given above, risk assessment can also be used for object placement. In cases where low risk is determined based on all available information, a trial-and-error approach can be adopted, and when the estimated risk exceeds a certain threshold, the operator participates by sending a corresponding request.

[0099] Sometimes, situations may arise where Robot 1 can learn from decisions made by the operator. For example, the robot is given the task of clearing a table on which many objects can be found. Therefore, the evaluation of possible actions reveals that no definite sequence is determined with sufficient probability. This is true not only for picking up different objects, but also for picking up objects, as successful picking up an object is predictable. In this scenario, the system identifies two distinct actions with comparable success scores. In such cases, the system will also request operator assistance in selecting the action to be performed first. Since both actions meet the requirements for automatic execution by the system, this may trigger a request for assistance, forcing the operator to input rules for prioritizing actions (objects) in similar situations in the future.

[0100] When answering a request (e.g., the question "Why?"), the operator can enter additional information. In the example described, possible reasons could be: least effort required, easiest to access, preferred cup, etc. This can also be indicated by the operator if they cannot define a specific reason. In future scenarios, the robot system may randomly select the action to be performed first.

Claims

1. A service robot system, comprising a robot, The robot (1) has a drive system (3) for moving the robot (1) to a target location and at least one actuator (4) for manipulating the robot's environment. The service robot system includes a processing unit (14) which includes a task determination unit for determining a task to be performed by the robot and is configured to control the drive system (3) and the at least one actuator (4) according to the task based on action definitions. The processing unit (14) is configured to automatically retrieve action definition candidates from a database (17), evaluate the retrieved action definition candidates for success scores indicating the likelihood that actions based on the action definition candidates will help to successfully perform the task, execute multiple actions based on the action definition candidates with the highest success score equal to or higher than a preset threshold until the next action to be executed with a success score lower than the preset threshold is reached, and then send a request for assistance via a communication interface (21).

2. The service robot system of claim 1, wherein the processing unit (14) is configured to determine the success score of the corresponding action definition candidate based on information about the operational success of actions according to the corresponding action definition candidate in the past.

3. The system according to claim 1 or claim 2, wherein the processing unit (14) is configured to divide the task into a series of action definitions.

4. The service robot system according to claim 1, wherein the robot (1) is configured to perform actions with a success score equal to or higher than the preset threshold only for actions that may be fully performed.

5. The service robot system of claim 1, wherein the processing unit (14) is configured to dynamically predict potential changes in the environment and adjust actions or select actions.

6. The service robot system according to claim 1 further includes an operator interface (18, 19, 20), the operator interface (18, 19, 20) for remotely controlling the robot (1) to perform actions and / or input information to enhance the database (17) in response to the request.

7. The service robot system according to claim 6, wherein the definition of the remotely controlled action is stored as a new action definition in the database (17).

8. The service robot system according to claim 1, wherein the robot (1) is equipped with at least one sensor (11, 12) for physical sensing of the robot environment and is configured to transmit the output of the sensor to the processing unit (14) and / or the operator interface (18, 19, 20).

9. The service robot system according to claim 6, wherein the operator interface (18, 19, 20) includes an extended reality suite (20), the extended reality suite (20) including a headset and controls, the controls being used to virtually control the robot (1) to perform operations before transmitting remote control signals to the robot (1) to perform actions accordingly.

10. A method for improving the automated execution of tasks by a service robot, comprising the following steps: - The environment of the service robot (1) is sensed using at least one sensor (11, 12) (S1). - Identify objects in the service robot's environment (S2) and associate the identification certainty with the identified objects. - Determine the task to be performed by the service robot (1) (S3). - Retrieve action definition candidates (S5) from the database (17). - Evaluate the retrieved action definition candidates (S6) to determine a success score that is successfully executed and contributes to the successful fulfillment of the task. - Execute multiple actions (S8) with the highest success score equal to or higher than a preset threshold until the next action to be executed with a success score lower than the preset threshold is reached, and then send a request for external assistance via the communication interface.

11. The method of claim 10, wherein the success score of the action definition candidate is calculated based on information regarding the operational success of actions performed according to the corresponding actions of the action definition candidate in the past.

12. The method according to claim 10 or claim 11, wherein the task is divided (S4) into a plurality of actions to be performed sequentially and / or in parallel, and for each of the plurality of actions, an action definition candidate is retrieved (S5).

13. The method of claim 10, wherein the request includes at least one action definition candidate.

14. The method of claim 10, wherein the preset threshold of the success score is set separately for each action to be performed based on one or more tags of one or more objects involved in the action to be performed.

15. The method of claim 10, wherein the service robot (1) is remotely controlled and / or information input by an operator interface is stored in the database (17) in response to the request.

16. The method of claim 15, wherein a new action definition is created based on the remote operation of the service robot (1) and the new action definition is added to the database (17).

17. The method of claim 10, wherein in response to the request, information about an entity involved in the action to be performed is read and the information is added to a tag or added as an additional tag for the entity.

18. The method of claim 10, wherein the task to be performed is determined (S3) based on instructions received via an operator interface, or based on the current situation encountered by the service robot (1).

19. The method of claim 18, wherein the determination of the task to be performed is based on the association between a previously received instruction and context information of the previously received instruction extracted from the circumstances under which the instruction was received.

Images