Service robot system, robot, and method for operating a service robot

Abstract

To provide a service robot system including a robot, and a learning method for the system. The robot (1) has a drive system (3) for moving the robot (1) to a target position and at least one effector (4) for manipulating the robot's environment. The system further includes a processing unit (14) including a task determination unit for determining a task to be performed by the robot and configured to control the drive system (3) and at least one effector (4) according to the task based on an action definition. The processing unit (14) is configured to automatically retrieve candidate action definitions from a database (17), evaluate the retrieved candidate action definitions for a success score indicating the likelihood that an action according to the candidate action definition will contribute to successfully completing the task, execute the candidate action definition with the highest probability equal to or greater than a predetermined threshold, and send a request for assistance via a communication interface (21) if the success score is less than a preset threshold. The system learns from instructions and additional information received in response to such requests.

Description

[Technical Field]

[0001] The present invention relates generally to service robot systems and the operation of such service robots. [Background technology]

[0002] Service robots that provide services to their owners are becoming increasingly common. However, in most cases, the benefits of such service robots are very limited. One reason is the lack of flexibility in their operation. For example, it is easy to instruct a service robot to mow the lawn within a clearly defined outer boundary. Such an autonomous lawnmower simply moves randomly around the lawn to be mowed until it reaches a wire marking the boundary of its work area or encounters an obstacle. However, changing environmental conditions typically overtax the capabilities of such autonomous lawnmowers. When the lawnmower encounters a situation that cannot be resolved by changing its direction, the robot suspends its operation to maintain a safe state. This suspension lasts until the system operator, usually the yard owner, resolves the situation by reestablishing the environmental conditions for which the autonomous lawnmower was set up. While such systems are highly beneficial for the autonomous lawnmower as long as unexpected situations do not arise, their inherent lack of flexibility limits their effectiveness in assisting the lawnmower owner. Summary of the Invention [Problem to be solved by the invention]

[0003] On the other hand, systems in which a user remotely controls a robot are known. This remote control of a robot is based on images captured by a camera mounted on the robot and transmitted to a remote control station, where the captured images are replayed using a display. The replayed images thus provide the robot's user / operator with information about the robot's current environment. Based on this, the operator can remotely control the robot. EP2363774A1 describes such a system, focusing particularly on the problems caused by time lags when remotely controlling a robot. While the solution proposed in EP2363774A1 may improve the results of remote control of service robots, it still requires the user or operator to be able to control the service robot at all times. This can be a solution when only humans are required to avoid harmful or dangerous situations. However, this still means that the human must perform the action himself, preventing him from simultaneously performing other tasks. [Means for solving the problem]

[0004] Therefore, there is a need to provide improved service robot systems that provide the ability to handle new situations during their normal operation, and even learn for the future.

[0005] This task is achieved by a service robot system, a service robot, and a method for operating a service robot as defined in the independent claims. Advantageous aspects and further features are defined in the dependent claims.

[0006] The service robot system according to the present invention comprises a service robot and a processing unit including a task determination unit for determining a task to be performed by the robot. Therefore, once in a position to determine the task to be performed, the robot's behavior is highly flexible. The robot is ready to perform various tasks defined by the task determination unit. In the simplest embodiment, the task determination unit is in communication with an interface for receiving commands from a user of the robot, e.g., spoken commands, and information on the task to be performed is extracted from this command. Alternatively, the task determination unit determines the task based on information sensed by the robot or information received from other sources, such as a clock or calendar, about time information or event information (calendar entries).

[0007] The robot can move using its drive system. The drive system can move the robot to a target location where at least one effector can perform at least a pick-and-place operation. The end effector can be any type of effector that allows the robot to manipulate its environment, such as picking up and holding an item. The robot includes the drive system and end effector and is controlled by a processing unit, which may be included in the robot or may be located remotely and connected to the robot to transmit signals based on which control signals for the respective actuators of the drive system and end effector can be generated.

[0008] The processing unit may consist of a single processor or may include multiple processors that may be distributed throughout the system. Specifically, parts of the processing unit may be located inside the robot, while other parts may be located remotely and in communication with parts within the robot. Depending on the task to be performed, the processing unit is configured to automatically retrieve candidate action definitions from a database. The candidate action definitions are definitions of actions that can be performed by the robot and are presumed to be applicable in the service robot's current situation. The candidate action definitions are searched in the database based on objects identified in the robot's environment. To identify objects in the robot's environment, the system includes at least one sensor for physically sensing the robot's environment. The sensor signal is then processed to identify the object. For example, image processing of images captured by a camera, such as a sensor, is performed.

[0009] The retrieved action definition candidates are then evaluated, and a success score is calculated. The success score is a measure of how likely it is that the action of the action definition candidate will contribute to successfully completing the task. The determination of the success score does not rely on the confidence in correctly identifying the current situation in which the robot will act. The success score for each action candidate can be calculated by taking into account actions corresponding to each action candidate that have been previously performed in similar situations. Once the action definition candidate with the highest score is selected, the system distinguishes between situations in which the action corresponding to the selected action definition candidate can be assumed to successfully contribute to completing the task and situations in which the action corresponding to the selected action definition candidate cannot be assumed to successfully contribute to completing the task.

[0010] The success score of each selected candidate action definition is compared with a threshold. Selected actions with a success score above the threshold are assumed to contribute to successfully performing the task. Actions with a success score below the threshold are assumed not to contribute to successfully performing the task. If the evaluation reveals that the success score is below the threshold, the controller will send a request for assistance via the communication interface. If the success score is estimated to be above the preset threshold, the processing unit is configured to automatically generate a signal to cause the service robot to perform an action corresponding to the selected candidate action definition.

[0011] In response to a request for assistance, the system will read input received from an operator responsible for remotely controlling the robot and / or inputting information to teach the robot. Examples of such input will be described in more detail later. The robot will then execute an action based on the input provided by the remote operator. According to a preferred embodiment, actions executed based on remotely controlled operation are evaluated for the likelihood that the action will be advantageous for future tasks. If it is determined that a particular action may be successfully applied to a future task, the corresponding action definition is stored in a database. The added action definition will then be available when the processing unit searches for candidate action definitions for future tasks in the future. If this action definition is then selected because it has the highest success score, this newly learned action may then be executed by the robot. Thus, any action definition generated from operator input in response to a request sent by the processing unit because it cannot handle a current situation without assistance from the operator will improve the "knowledge base" of the service robot system, and the system will inevitably learn during its daily operation.

[0012] According to an advantageous implementation, the success score of each candidate action definition is determined based on information about the success of past action implementations corresponding to the respective candidate action definition. Thus, each time an action according to a candidate action definition is implemented, the system stores the situation in which the action was implemented and associated with the situation, regardless of whether the implementation of the action was successful. Based on knowledge of the success of the specific action corresponding to the candidate action definition in the situation in which this particular action was implemented, the processing unit can then calculate a success score for the candidate action definition in the current situation. This success score is based only on historical knowledge about the candidate action definition. The success score does not reflect whether the actual situation was correctly determined. Rather, the determination begins with the assumption that the current situation was correctly determined, so that the currently experienced situation can be compared with similar situations experienced in the past. A similarity comparison then allows for the success score of the current situation to be calculated. The calculation preferably takes into account a measure of similarity between the currently experienced situation and the stored situation.

[0013] An advantageous aspect is that the controller is configured to segment the determined task into a sequence of actions and their respective action definitions. Segmenting the entire task into a sequence of actions is advantageous when the entire task is somewhat complex. The actions corresponding to the segments are therefore rather simple, and as a result, it is easier to predict whether the execution of a single action will be successful. As an example, the task "return the book to the shelf" can be segmented into the following: 1. move the robot to the table, 2. move the arm to the gripping position, 3. close the gripper and pick up the book, 4. place the book on the tray, 5. move the robot to the shelf, 6. pick up the book from the tray, 7. move the arm to the release position, and 8. open the gripper and place the book on the shelf. It should be noted that segmenting the task to be performed may result in a large number of actions, but may also result in a smaller number of actions compared to the number of actions resulting from the segmentation in the given example above.

[0014] It should be noted that one preferred "manipulating the robot's environment" is a pick-and-place operation, and this operation will be used below for further explanation. However, this does not limit the present invention. For such pick-and-place operations, the robot has a gripper and potentially one or more supporting tools. A suction device or a diffuser may be used as a tool. In a preferred embodiment, these tools and / or end effectors can be interchanged to allow for adaptation to the task.

[0015] Furthermore, the system is preferably configured to begin executing an action only if the probability of completing the action without interruption for the action definition exceeds a predetermined threshold. Note that there will be dynamic objects in the environment, which may change position (e.g., a moving dog) or state (e.g., a person filling a cup with water), imposing new constraints on the environment. According to a preferred embodiment, the processing unit is configured to dynamically predict potential changes in the environment and adapt actions or select actions while still maintaining a final safe goal state. Specifically, if a sequence of actions is required to complete a task, this approach ensures that the robot is always in a safe state once a single action is completed. The action definition always defines such a safe state at its end. As a result, the robot will always be ready to receive and execute remotely controlled commands. If the processing unit is far away from the robot, the signal transmitted from the processing unit to the robot always contains any information necessary for the robot to complete the action corresponding to the transmitted information. If the connection between the robot and the processing unit is interrupted, the robot has enough information to enter a safe state. In any case, it should be noted that the robot itself has processing capabilities that allow it to use signals transmitted from the processing unit to generate control signals for driving the drive system and / or effectors.

[0016] It is further preferred that the robot is equipped with at least one sensor for physically detecting the robot's environment and further comprises an interface for transmitting the sensor output to a processing unit and / or an operator interface. Transmitting the sensor signal to the processing unit allows a controller to analyze the current situation in which the robot is located. Furthermore, transmitting the sensor output to the operator interface allows an operator to analyze the situation of the robot. In both cases, the sensor signal is the basis for determining the robot's next action. It is particularly preferred that the sensor includes a camera, or even more preferably, multiple cameras.

[0017] The operator's interface for providing information about the current situation in which the robot must be controlled can be a display showing images captured by sensors attached to the robot (or even images captured by stationary sensors, such as a home security camera), but according to an advantageous implementation, the operator interface comprises an XR set for virtually controlling the robot to perform actions, even before remote control signals are transmitted to the robot to perform the same actions in the real world. In such an embodiment, the entire action can be virtually performed by the operator before the respective information is transmitted to the robot and the control signals are generated by the robot. This has the great advantage that the remotely controlled actions of the robot are not subject to time lags, which would necessarily result in overkill of the actions performed by the robot. In other situations, augmented reality can be used instead of virtual reality. An exemplary situation could be assisting a doctor / nurse, who is in the same room as the robot and instructs it as an operator in this case.

[0018] Note that an "action definition" is a general definition of any type of behavior performed by a robot or part of a robot. Thus, an action definition can define the movement of an effector, any type of additional tool of the robot, and / or the robot's drive system. The action definition allows the robot to adjust the execution of the action, such as the distance traveled if the drive system is controlled based on the action definition, thereby defining the final trajectory. The adjustment is context-sensitive and is performed with knowledge of the robot's environmental situation. After a candidate action definition is retrieved from the database and selected for execution, the processing unit calculates how the action should be performed based on information extracted from the sensed environment to calculate the final trajectory of the robot and / or its effector, before finally generating the control signal. For example, the movement of an effector (an arm with a gripper / hand attached to the distal end of the arm) depends on the height of the object to be picked up. Based on images captured from the service robot's surroundings, the processing unit can calculate the correct position of the object and adjust the movement parameters in the action definition accordingly. It should be noted that the term "image" in this context does not only refer to images such as those captured by a camera that produces a 2D image, but also covers information derived from any type of sensor that allows for the production of a representation of the environment. Examples of such sensors are RADAR and LIDAR, which produce 3D point clouds.

[0019] According to a preferred embodiment of the service robot, the robot includes at least one support structure in addition to at least one effector, with the at least one effector and the support structure mounted on at least one lifter. In a simple embodiment, the support structure and the effector may be mounted on the same lifter, but it is preferable to be able to adjust the elevation of the support structure and the end effector independently. In this case, two lifters are mounted on the robot's mobile platform (robot base). Providing such a support structure reduces the distance that must be covered by the effector. The support structure allows multiple items to be collected and placed on the support structure, which may be a basket, box, tablet, or pallet, or even a forklift carrying a storage box or platform. This can be done, for example, when multiple items are collected from the floor, with the support structure lowered. When these items need to be placed on a higher shelf, both the support structure and the effector can be raised before returning the items to the shelf. Thus, the support structure significantly reduces the movement required by the service robot. Obviously, this is also true if the starting position and the destination or target position are at the same height, but if every item is transported individually, the service robot will have to move back and forth.

[0020] To allow an operator or processing unit to understand in detail and with precision the situation in which the robot is currently operating, cameras or other sensors that allow imaging of the robot's environment are attached to the effector used to perform the task. Thus, by moving the end effector or, in more sophisticated embodiments, the camera (sensor) relative to the effector to which it is attached, the view of the captured image can be optimized. Service robots are equipped with at least one additional camera (sensor) mounted at a different position on the robot and / or with a different zoom ratio to provide an overview in addition to the details of the scene.

[0021] Alternatively, it would be possible to move a camera attached to the effector, thereby capturing multiple images from different positions that provide detailed views and overviews. Obviously, this would slow down the overall operation, as the effector would need to move in a way that captures enough information about the situation to allow secure operation of the robot.

[0022] In addition to requesting remote-controlled operation pending the operator's response, the request also preferably includes at least one action definition candidate retrieved from the database. This is particularly useful when there are one or more action definition candidates that are estimated to potentially contribute to the execution of the entire task but do not have a success score that clearly indicates that the corresponding action can be successfully executed. In such cases, action definition candidates that result in a success score below a preset threshold can be included in the request and offered as suggestions to the operator. The operator then selects one of the proposed action definition candidates, and the robot will proceed with executing the action as commanded by the operator. In such cases, the operator does not need to control the entire action, since the robot can autonomously execute the action once defined. Alternatively, if multiple action definition candidates receive success scores above the threshold but the success scores are not significantly different so that one favorite can be identified, these candidates can be included in the request so that the final decision is the operator's responsibility.

[0023] As mentioned above, the evaluation of the candidate action definitions involves comparing the success score calculated for each candidate action definition with a preset threshold. As an example, a book lying on a table must be returned to its shelf. First, the system must recognize and identify the book lying on the table. Thus, the system will first calculate the probability that the table will be correctly identified, and also the probability that the object to be grasped is a book. From the certainty of the correct identification of the object involved, a success score can then be calculated that the action involving the target can be successfully performed.

[0024] This success score is then compared to a preset threshold, which may be calculated from one or more tags stored in association with the object involved in the action to be performed. In the above example, the object is a book. A "tag" is a plurality of pieces of information associated with the object determined from the sensor signal. If the sensor includes a camera, image processing is performed to identify the object, such as a cup on a table. This cup may be made of different materials, such as metal or ceramic, and handling the cup may necessarily require different levels of care. Thus, if a cup recognized from a captured image is identified as being made of ceramic, the threshold may be set higher than for a cup made of metal. Obviously, multiple tags of different categories may be associated with an object that can be identified by the system. These tags may be added by the operator when the operator is called upon by a request and, in response, when the operator identifies an object or its characteristics. Entering and storing additional information associated with the identified object will improve the database.

[0025] It should be noted that the requests sent by the controller remotely control how actions are performed by the service robot, but can also result in adding information about recognized objects in the robot's environment. The latter can automatically result in the robot being able to perform actions due to the enhanced situational understanding achieved by the additional information about objects. Since the processing unit permanently reevaluates the situation and potential actions that can be performed for the success of the task, the added information about objects involved in the execution of the task will be immediately taken into account for the evaluation and even for the selection of the next action. Thus, the selection of candidate action definitions is performed dynamically, taking into account the robot's changing environment, for example, when the robot moves to a different position.

[0026] The system further preferably stores encountered situations by generating a combination of context descriptors along with instructions for performing past tasks. Storing the history of tasks and encountered situations defined by a combination of context descriptors allows for automatic conclusions to be drawn from future recognized situations as to which tasks are most likely to be performed. Such information about the history may include not only the situation encountered and the instructions given in this situation, but also how frequently a similar situation is encountered without instructions being given.

[0027] For improved understanding, the system will now be described with reference to the accompanying drawings. [Brief explanation of the drawings]

[0028] [Figure 1] A simplified diagram of a service robot. [Figure 2] FIG. 1 is a block diagram showing the main units of a service robot system. [Figure 3] 1 is a flowchart showing the main method steps during the execution of a task. DETAILED DESCRIPTION OF THE INVENTION

[0029] 1 is a simplified diagram of a service robot 1 used in a service robot system according to the present invention. It is obvious that the robot 1 as shown in FIG. 1 is only an example, and other configurations of the robot 1 can be used as well.

[0030] The robot 1 is used to assist people in completing tasks that can be flexibly defined. Therefore, one basic requirement is that assistance can be provided in various locations. This is achieved by moving the robot 1 to the location where assistance is needed. The robot 1 comprises a robot base 2 designed to house a drive system. In the drawing, only the wheels 3 of the drive system are shown. The motors and an energy source, such as a battery, are not shown but are located inside the robot base 2. The specific structure of the drive system is not relevant to the present invention; various different structures optimized for the working area of ​​the robot 1 are known in the art. The drive system enables the entire robot 1 to move from an initial position to a target position. The drive system comprises all the components necessary for driving and changing direction in response to received control signals.

[0031] To perform a task, the robot 1 includes an end effector 4 attached to an arm 5 for positioning the end effector 4. The arm 5 consists of at least a first element 5.1 and a second element 5.2, which are connected to each other by a joint that allows the relative angle between the first element 5.1 and the second element 5.2 to be adjusted for moving the end effector 4 to a desired position and for achieving a desired orientation. In the illustrated embodiment, the end effector 4 is designed as a gripping tool that mimics a human hand. For applications in the daily lives of assisted individuals, such an end effector 4 appears to be the best tool for efficient assistance. Many tasks involve picking up, manipulating, and placing items in designated locations. On the other hand, such a gripping tool can also assist hospital nursing staff or doctors. For example, a gripping tool may be used to deliver meals to a patient's room without the nursing staff having to enter the room. As a result, the need for disinfection of the nursing staff's or doctors' clothes, hands, etc. can be reduced.

[0032] It should be noted that in addition to the movements of the first element 5.1 and the second element 5.2 indicated by the arrows in the drawings, further degrees of freedom of the first element 5.1, the second element 5.2 and also of the end effector 4 may be realized. In particular, rotational movement of the first element 5.1 about its longitudinal axis and rotational movement of the second element 5.2 about its longitudinal axis may also be possible. It is particularly preferred that the end effector 4 may also perform rotational movements about at least two axes that are perpendicular to each other.

[0033] The arm 5 is mounted on a first lifter 6 with an end effector 4 attached to the distal end of the arm 5. The first lifter 6 is fixed to the robot base 2 and, in the illustrated embodiment, comprises a fixed element 6.1 and a movable element 6.2, which may be hydraulic or pneumatic cylinders. In the illustrated embodiment, the movable element 6.2 supports a stay 8, which in turn supports the proximal end of the first element 5.1 of the arm 5. The first lifter 6 can adjust its overall length, thereby lifting the stay 8 above the robot base 2 and, consequently, the proximal end of the arm 5.

[0034] The first lifter 6 allows the end effector 4 to reach a higher position relative to the ground on which the robot 1 is standing. The first lifter 6 can be designed to be somewhat rigid, and using its ability to lift the proximal end of the arm 5, the first and second elements 5.1, 5.2, and also the end effector 4, can be designed to be lightweight and relatively small, without reducing the range of motion in any way.

[0035] One preferred application of the service robot 1 is to collect items and return them to their designated locations, such as returning toys to shelves in a child's room or plates from a table to the kitchen. If the location where the items must be collected and their respective designated locations are far apart, this would require the robot 1 to move around considerably. Moving around is even more difficult when the items to be returned to their designated locations are on the ground. With the robot 1 according to the present invention, this is avoided by providing a second lifter 7 further comprising a fixed element 7.1 and a movable element 7.2. A support structure 9 is attached to the movable element 7.2. In a simple embodiment, the support structure 9 includes an integrated platform 10. The platform 10 serves as a temporary storage location on which multiple items collected using the end effector 4 can be placed. Once all items have been collected, the robot 1 moves to the target location, from which each item is placed, one after the other, in its designated location.

[0036] Often, the dedicated location is not close to the ground, and therefore the first lifter 6 is used to raise the arm 5 to a high position so that the end effector 4 can reach the target position of each item. Again, if the platform 10 were placed at a fixed height above the ground, it would be necessary to repeat the operation of the first lifter 6 for each item. According to the robot 1 of the present invention, the platform 10 can also be raised. The first lifter 6 and the second lifter 7 are preferably designed such that it is guaranteed that the arm 5 with the end effector 4 can reach the platform 10 without further operation of the first lifter, even for the maximum height of the first lifter 6.

[0037] 1 preferably comprises a first lifter 6 and a second lifter 7. However, it is also possible to similarly attach the support structure 9 to the first lifter 6, in any case the platform 10 and the stays 8 being in a fixed relative position to one another that ensures that the end effector 4 can reach the platform 10.

[0038] The use of hydraulic or pneumatic cylinders as lifters 6, 7 should be understood as an example only. Linear actuators or spindles may be used as well. Furthermore, although the figures show a single stage lifter, two or more stages may be used if greater heights above ground level must be achieved.

[0039] The benefit of using a service robot 1 to assist people is the robot's increased independence, increasing the number of situations in which the robot can operate autonomously. Such independence can only be achieved if the service robot system has knowledge of the environment in which the service robot 1 operates. Knowledge of the current location and the current status of the environment is obtained using one or more sensors. As an example of sensors that can be used, the embodiment shown in FIG. 1 includes a first camera 11 and a second camera 12. The first camera 11 is attached to the second element 5.2 of the arm 5 and moves together with the second element 5.2. Thus, the image captured by the first camera 11 shows only details of the robot's environment near the end effector 4. Because the robot 1 (or rather, its processing unit, as will be explained later) also needs to view the entire environment of the robot 1, the second camera 12 is configured to capture images of a wider area of ​​the robot 1.

[0040] While a first camera 11 and a second camera 12 are shown in FIG. 1 , other sensors may be used that physically sense the environment and thus acquire information about the environment that allows a representation of the environment to be generated. Examples include RADAR sensors, LIDAR sensors, and ultrasonic sensors. Furthermore, the entire service robot system may also use sensors that are fixedly attached to the environment in which the robot 1 operates. For example, a surveillance camera may be used. Finally, the number of sensors is not limited to two as shown. Specifically, a single sensor may be sufficient if it is attached to an arm 5 or any other movable support that allows the sensor to be moved to different locations. Information about the service robot 1's environment is then collected, acquiring information from different positions and / or orientations of the single sensor. Naturally, even when multiple sensors are used to generate a representation of the service robot 1's environment, more than one capture per sensor may be used. The amount of information acquired from the environment may be adjusted depending on the analysis of the environment performed by the processing unit, as will be explained later. For example, if the reliability of the information obtained from sensing the environment (certainty of correctly identifying objects) does not seem high enough, the robot 1 or at least elements equipped with a mobile sensor such as the first camera 11 may move to obtain additional information by sensing the same environment of the robot 1 from different perspectives. Furthermore, methods such as sensor fusion, 3D reconstruction, overcoming occlusions may be used.

[0041] Figure 2 is a simplified block diagram showing the overall layout of a service robot system. The service robot 1, which has been described in detail with reference to Figure 1, is one main component of the overall system. The robot 1 is equipped with sensors, which may be cameras 11, 12 as described above. Furthermore, the robot 1 is equipped with an interface 13 connected to the sensors for transmitting signals containing information about the sensed environment to a processing unit 14. It is obvious that if some or all of the sensors are located outside the robot, a separate interface is required for transmitting the sensor signals to the processing unit 14.

[0042] In the illustrated embodiment, the processing unit 14 is located external to the robot 1. However, it would also be possible to include the processing unit 14 on the robot 1. Having the processing unit 14 external to the robot 1 allows the processing power of the processing unit 14 to be used not only for a single service robot 1, but also for multiple robots. Communication with other robots is indicated by dashed arrows. Furthermore, the processing unit 14 need not be realized by a single processor, but may also be multiple processes that cooperatively process received signals. Such multiple processors that together form the processing unit 14 could also be distributed, with some of the processors located within the robot 1 and some of the processors located remotely, communicating with each other via their respective interfaces.

[0043] Each of the multiple robots 1 uses at least one common processor, but may even be equipped with individual processors for internal signal processing as well. Such internal processors may be used, inter alia, to generate control signals, as shown by driver 15, which generates control signals for actuators 16 based on information about the action to be performed received from processing unit 14. Actuators 16 represent all individual actuators required to drive wheels 3, lifters 6, 7, and to position arm 5, including end effector 4.

[0044] The processing unit 14 is connected to a database 17 in which action definitions are stored. The action definitions define the overall algorithm for executing a particular action by the robot 1, without precisely defining a trajectory. The actions can be, for example, grabbing an item, moving to a different location, placing the item on a table, and releasing the item. Furthermore, the action definitions can be organized in a hierarchical structure, starting from simple action definitions, such that such simple action definitions can be combined to establish higher-level action definitions. The action definitions can be organized in a hierarchical structure with only one, two, or more levels. Starting from the action definition given as an example above, a higher-level action definition can be, for example, "putting a dropped item back on the table," which includes the actions mentioned above. Naturally, the actions given as an example above can have even higher granularity.

[0045] An essential aspect of the operation of the service robot system of the present invention is communication between the robot 1 (including its "intelligence" in the form of the processing unit 14) and an operations center, where an operator supports the operation of the robot 1 when the robot 1 (including its "intelligence") is unable to handle a situation on its own. This is achieved by connecting the processing unit 14 to at least a display 18 (preferably multiple displays) and input devices 19 via an operations center interface 21. The display 18 is used to output information to a human operator that allows the operator to estimate the current situation of the robot 1. Supporting the operation of the robot 1 can be implemented in different ways: - Direct control of the robot's movements, - Selecting the appropriate action definition from the database, or - Adding information identifying and / or characterizing the object to the database 17.

[0046] The display 18 displays a representation of the robot 1's environment, including the robot's location and nearby obstacles identified from the sensor output. If the sensors include one or more cameras 11, 12, the captured images can be displayed as a live camera feed as well. When multiple cameras 11, 12 are used, the operator can switch between different cameras to facilitate control of the robot 1. Alternatively or additionally, the display can be used to provide information about identified or unidentified objects, as well as the current status of the situation assessment performed by the robot 1. As will be explained later, this can result in making suggestions about actions that can be performed by the robot, but it can also result in asking questions if the responses received from the operator introduce further ambiguity.

[0047] To initiate the robot 1's operation, the task to be performed must be determined. The determination of the task to be performed is also performed in the processing unit 14 using a respective software module. The decision unit is thus a software module executed by the processing unit 14. Again, a dedicated processor may be used to execute the software modules of the decision unit. To determine the task to be performed, signals from sensors are processed. One sensor may be, for example, the microphone 22, which can analyze verbal commands from the user of the robot 1 and derive the task therefrom. Another way to determine the task to be performed is to use the history of tasks previously performed and information about the situations in which this task was previously performed. In this case, the decision unit uses so-called context descriptors to compare the situation currently encountered by the robot 1 with information about the situations in which a specific task was previously performed. The association is also stored in the database 17. Based on the signals from the sensors, the processing unit 14 compares the robot 1's current situation with the situations stored in the database 17, where the situation is defined by a combination of context descriptors. If a similarity above a threshold is identified, the task associated with the situation is determined to be intended.

[0048] Once the task is determined by the decision unit, the processing unit 14 attempts to find a solution for autonomously executing the task. Based on information about the environment derived from sensor signals, the processing unit 14 retrieves from a database one or more candidate action definitions that are assumed to contribute to the successful execution of the task. The candidate action definitions are searched for in the database 17 by searching for tags that describe the objects by defining their properties after objects in the robot 1's environment are identified from an analysis of signals received from the sensors. The operator will only be requested to assist in the execution of the task if it is not possible, or at least not feasible, for the robot 1 to operate autonomously. Details of the analysis leading to the decision on whether an operator is requested to assist will be presented later.

[0049] If the processing unit 14 concludes that it is neither possible nor feasible to perform a task autonomously, the processing unit 14 will send a request for assistance to a remote operations center via the communications interface 21. In response to the request received from the processing unit 14, a human operator will control the robot 1 or add information to the system's database 17. As described above, controlling the robot 1 may involve directly controlling the robot's behavior using controls, such as input devices similar to those used in computer games, or selecting an appropriate action definition from the database 17 may be based on suggestions made by the processing unit 14. According to a preferred embodiment, the system also includes a virtual reality set 20, which allows the operator to view virtual objects and virtually navigate the robot 1 to their immediate vicinity. The virtual reality set 20 not only includes a virtual reality headset, but also a virtual reality controller, which allows the remote operator to determine, for example, a grasping pose in virtual reality before sending a corresponding action definition to the robot 1. Thus, using the virtual reality set makes it possible to avoid any collisions of the robot 1, which would likely occur due to time lags if the robot 1 were controlled directly.

[0050] To control the robot 1 using the virtual reality set 20, it is advantageous to use a 3D camera as the first camera 11 to generate a 3D representation of the object intended to be grasped by the end effector 4, and also a 3D representation of its environment. The correct grasping pose of the end effector 4 can be determined when a model of the physical gripper is assigned to a virtual reality controller's representation in the virtual environment (or augmented reality). The remote operator can then place the 3D representation of the real gripper next to the virtual object. A second controller is used to remotely control the operation of the drive system.

[0051] In the following, a typical process will be described in detail using a simplified flowchart as shown in FIG. 3, in the situation where the initial setup of the system has already been carried out.

[0052] First, in step S1, the environment of the service robot 1 is sensed using at least one sensor, which may be attached to the service robot 1 or to a suitable location that allows the service robot 1 to analyze its environment. Based on the sensor output, objects in the robot 1's environment are identified in step S2. The sensors may be of the same or different types. The service robot system must now determine a task. In step S3, determining the task to be performed by the service robot system can be implemented in several different ways. The most preferred is an autonomously acting robot 1, which involves autonomously determining the task to be performed. However, in many cases, the robot 1 does not know what to do by itself and therefore requires assistance. A typical way to instruct the robot 1 on what to do next is direct communication between the user and the robot system. Such communication with the robot 1 uses a microphone, as already described with reference to FIG. 2, but can also be implemented using a speaker 23, which allows the robot 1 to output information to the user. Instead of the microphone 22 and speaker 23, an additional interface may be provided to connect the robot 1 to a user device, such as a smartphone. In this case, inputting commands that allow the robot system to determine the tasks to be performed may be performed using a smartphone or equivalent device. In general, information can be provided to the robot system in several different ways, including SMS, an app, verbal commands, or human gestures. Some of these "information channels" may require additional interfaces to connect the robot system with other IT systems, such as a camera that provides signals that can be processed by the service robot system to determine gestures and analyze them to determine tasks.

[0053] Using the microphone 22, the robot 1 can receive verbal commands from the user. The processing unit 14 can then determine the task that the user intends to be performed by the robot 1 based on the received verbal command. Similarly, commands can be given by the user using SMS or by making a gesture, such as pointing at an object. Pointing at an object can be understood by the system as focusing the task determination on this particular object. If the pointed object is a trash can, the determined task is identified as a potential action that can be performed on the specific object. For a trash can, this could be "empty the trash can."

[0054] More preferably, the robot 1 will determine itself (using knowledge of its environment and potentially additional information gathered from sources connected via interfaces) the task that is most likely to be performed. If the likelihood that a particular task will be performed exceeds a given threshold, which may be adjustable, the robot 1 can automatically begin determining the necessary actions to be performed to execute the determined task. Potential tasks can be stored in a database 17, which can be searched for a task that is likely to fit the current situation. If it is determined that the correct task is not likely to be performed, the robot 1 can ask a question, for example, to the user via the speaker 23 or to the operator via the other interface 21. The user or operator can then instruct the robot 1 by confirming the task proposed by the robot 1, or, if no suggestion is made by the robot 1, by directly commanding and defining the task to be performed.

[0055] From here it will be explained how the robot 1 can determine a task from a user by means of a microphone 22 mounted either directly on the robot 1 or somewhere in the working area of ​​the robot 1 and connected to a service robot system, so that the processing unit 14 (and therefore the decision module) can obtain the verbal command and derive information about the task to be performed based on such received verbal command. For example, the user 1 of the robot can command the robot 1 by saying, "Clean the children's room." In such a case, the robot 1 receives such a well-defined task, and no evaluation of the environment of the robot 1 needs to be performed by the robot 1 to determine the task in step S3.

[0056] Note that robot 1 not only listens for any verbal commands using microphone 22, but also always "listens" to whether the user issues a verbal command. This is particularly important because commands spoken directly to robot 1 always override any tasks and actions currently being performed by robot 1. This ensures that robot 1 does not proceed with the execution of a task that is deemed inappropriate by the user of robot 1. Such an interruption would cause the robot to proceed directly to the next safe state. An example might be cleaning the child's room, which is currently necessary for the child who is still playing there. To avoid conflicts of interest, previously determined tasks, whether commanded by the user or determined autonomously by robot 1 as described below, are overridden by the most recent command issued by the operator or user.

[0057] The robot 1 analyzes the currently encountered situation and compares it with a prototype situation stored in the database 17. A prototype situation is a previously encountered situation in which the same task was to be performed. A prototype situation can also be defined a priori when the service robot system is designed and programmed. To analyze the situation, the processing unit 14 collects data describing the current situation. Each piece of information is stored in the form of a descriptor. Multiple descriptors are combined into a prototype situation. When a task is commanded to be performed by the robot 1, information about the combination of descriptors in this situation is stored in association with the commanded task. When a situation similar to this prototype situation occurs in the future, a comparison of the stored information about the situation (the combination of descriptors) with information derived from sensor signals that sense the environment of the robot 1 in the currently encountered situation can reveal a high similarity between the encountered situation and the prototype situation. It can then be concluded that the task stored in association with the prototype situation will be performed again. Thus, processing unit 14 calculates a measure of similarity between the stored prototype situation and the currently experienced situation based on information collected in the currently experienced situation and perceived by sensors (and / or retrieved from other sources). If this measure exceeds a threshold, which may be adjustable, processing unit 14 retrieves a stored task associated with the prototype situation identified as being sufficiently similar to the currently experienced situation of robot 1.

[0058] The occurrence of recurring situations can also be perceived by the system even if no instructions are received in these situations. The observation can be communicated to the operator, who can then label this as a new prototype situation and preferably associate a task with this new prototype situation.

[0059] A "situation" is defined using descriptors as described above. A combination of descriptors, including multiple independent descriptors, will make up such a prototype situation. These descriptors may not only include information about the location, time, day of the week, or the difference between the current state of the environment and the goal state. Descriptors may also include information obtained from other sources via an interface. For example, the information source may be the user's calendar. Calendar entries can often trigger specific tasks to be performed by the robot 1. One example is a calendar entry such as "Playdate," still referring to the situation of cleaning the child's room. When attempting to determine the next task to be performed, the robot 1 (or rather, its processing unit 14) searches for information sources in communication with the processing unit 14. When the robot 1 recognizes that playdate should be over based on a comparison of the current time with the calendar entry, it is highly likely that the child's room needs to be cleaned. Even social contexts may be included in the determination of which task to perform. Cleaning the room can be performed differently, for example, before the child's friends arrive and before Grandma comes to visit.

[0060] The service robot system experiences tasks over time, as they are stored in association with the descriptors of the situations in which the robot 1 was instructed to perform them. However, analysis of the currently encountered situation often results in ambiguity, and the robot 1 is inevitably not in a position to autonomously determine the task to be performed. Although the frequency of such ambiguous situations will decrease over time, it is necessary to assist the decision unit in identifying the task to be performed. To avoid the robot 1 performing a task that is inappropriate for the current situation, it is not sufficient to always select the task corresponding to the prototype situation that shows the highest similarity to the stored prototype situation. In such cases, it may be necessary to ask the user or operator of the robot 1 to clearly define the task to be performed. A second threshold lower than the first threshold can be introduced. The first threshold should be set high enough to ensure that only one task fits the situation in which the robot 1 is currently located. If the second threshold is set lower, there is a good chance that multiple stored prototype situations may show a similarity to the currently encountered situation that is below the first threshold but above the second threshold. In such a situation, the service robot system can output a question (request for assistance) directed to the user or operator. The question can suggest tasks associated with prototype situations determined to have similarity above a second threshold but below the first threshold. The question can ask for confirmation of one of these tasks, or can be an open-ended question such as "What would you do?" if the prototype situation cannot be identified based on the information obtained about the situation and the associated task cannot be determined.

[0061] When a request is sent to the operator in step S3 to assist the decision unit in determining the task, in order to obtain information about the task to be performed, all information collected by the processing unit 14 from all available sensors, and also from other sources, is likewise forwarded to the operator and output, for example, via the display 18 or a speaker not shown in FIG. 2. The operator thus gains knowledge about the situation of the robot 1 that resulted in the question being asked. The operator can then even label the unique combination of situation descriptors in order to make it easier in the future to respond to requests received from the robot 1.

[0062] Once the task to be performed has been determined, the process moves to step S4, where analysis of the task to be performed begins. Database 17 contains several action definitions. Each of these action definitions may be built up by multiple actions, which may themselves contain multiple further actions. This means that each action definition stored in database 17 may be combined with one or more other action definitions, and a new action is generated by this combination. While it would be possible to store a new, higher-level action definition for every possible combination of executable action definitions, such an approach is not feasible. It is rather preferable to segment the task to be performed into smaller pieces, each of which may then correspond to a single action whose action definition is stored in database 17. Because action definitions such as those stored in database 17 may have a multi-layered structure, such a single action definition obviously consists of a combination of other action definitions. Once the task has been segmented into multiple actions to be performed (if necessary), processing unit 14 proceeds to retrieve potential action definitions as candidates in step S5 and to evaluate these candidate action definitions from database 17 in steps S6 and S7.

[0063] The processing unit 14 has an evaluation function that enables it to determine whether a specific action is likely to be successfully applied in the current situation. The processing unit 14 calculates a success score for each action (candidate action definition) that should be applicable in the current situation. Based on this success score, an action to be used to perform a task can be selected. The success score is a measure that enables a comparison between different actions regarding their likelihood of success when applied to the current situation. For example, the score may be a likelihood or probability. Once a success score is determined for a candidate action definition, the score is compared with a threshold. If the score exceeds the threshold, it is concluded that the success of the action, if executed, is sufficient for the current situation. This will be explained later with reference to an example in which such evaluation takes into account the object involved in the action by adjusting the threshold. The described evaluation of one or more candidate action definitions is performed in step S6.

[0064] The execution of a task is performed by executing one action after another. Therefore, after segmenting the task to be executed in step S4 and evaluating the action definition candidates in step S6, further execution of the task can be performed in two different ways. If the success score of the action definition of the next action to be executed exceeds a given threshold (comparison step S7) and this action definition is the candidate with the maximum success score of all evaluated action definition candidates, the action will be automatically executed in the next step S8 based on this action definition. After the execution of this action is completed, it is determined whether the initially determined task is completed. Therefore, a comparison is made in step S9 between the result achieved by the action and the goal state as defined by the determined task. If the task is completed, the procedure ends in step S10. If the task is not yet completed, the procedure returns to step S7 to evaluate the next action to be executed. Note that executing an action may change the environmental situation of the robot 1. Thus, inversely to the arrow connecting step S9 to step S7, changed environmental conditions may lead to searching for new action definition candidates, retrieving the new action definition candidates from database 17, and evaluating the new action definition candidates, after which a comparison of the success score with a threshold can be performed in step S7.

[0065] To learn from its operation history, the system memorizes the situations in which it has operated. For each situation, the system memorizes an action corresponding to a candidate action definition associated with information about whether the execution of this action in the respective situation was successful. This makes it possible to conclude that an action corresponding to such candidate action definition executed again in the same situation will be successful again. If the success score is a value between 0 and 1, the resulting success score in this simple example should be 1. If no similarity can be identified between the current situation and any of the situations stored in database 17, the resulting success score should be 0. Since it is highly unlikely that a single success score can be determined for every actual situation and for an identical situation that has already occurred in the past, even for such a new situation, the determination of the success score uses a calculation of the similarity between the currently experienced situation and the situations already stored in database 17. For the above-described example with a success score between 0 and 1, the closer the resulting success score is to 1, the higher the similarity between the memorized situation and the actual situation. Conversely, the closer the success score is to 0, the lower the similarity between the two situations. Of course, it is possible to take into account multiple memorized situations, not just the one with the greatest similarity to the currently experienced situation. The similarity calculation can take into account visual parameters, for example, classified objects, such as the presence of people, but also the effect of context. The similarity calculation does not take into account any ambiguity when determining the currently experienced situation and assumes that the output of situation perception is correct.

[0066] When multiple actions are required to perform a task cooperatively, each corresponding candidate action definition is evaluated as described above. Additionally, the set of candidate action definitions required to perform the entire task may also be evaluated. This may be achieved by combining the success scores of each candidate action definition involved in performing the entire task.

[0067] It should be noted that in determining the success score of a candidate action definition required at a later time to perform the entire task, the future situation achieved by performing the previous action is predicted. This situation is then used for a similarity calculation to determine the success score of the next candidate action definition. For example, the evaluation is repeated continuously so that it can be taken into account whether the predicted future situation will be rendered unachievable by the performed action due to changes in the environmental situation as well as inaccuracies in positioning.

[0068] The goal states are defined by the task. Information about the goal states of a task may be modified by an operator, for example, by adding a list of goal states. The goal states may also be indirectly defined by tags on objects involved in the situation that indicate their normal location or status.

[0069] If a sequence of actions with a success score above a threshold is determined to execute the determined task, the service robot system will be able to execute the entire determined task autonomously. However, during the execution of the sequence of actions, it may occur that the next action to be executed is reached without calculating a score above the given threshold. This clearly indicates that the system cannot execute the action by itself without risking system failure. In this case, after the comparison of the success scores of each candidate action definition in step S7 reveals that the score does not exceed the given threshold, the service robot system sends a request to the operation center in step S11. The operator will receive the request and will analyze the situation using the display 18, input device 19, and / or augmented / virtual reality set 20 and provide the service robot system with respective advice by inputting information and / or control signals to directly control the robot 1 or to instruct it to use the action definitions stored in the database 17.

[0070] The request sent from the robot to the operator may include an urgency index and / or information selected based on the urgency index and transmitted along with the request to assist the operator in handling multiple robots 1 simultaneously. The urgency index indicates to the operator the actual need for the operator's assistance. The urgency index may be calculated from the maximum achievable success score in the robot's actual situation, taking into account additional parameters, such as the tag of the object involved. The greater the difference between the calculated success score of the determined candidate action definition and the maximum achievable success score, the greater the urgency index value (assuming a higher urgency index value means that assistance in this situation is more urgent). However, even with the same success score calculated for the current situation, specific aspects of the situation or the task to be performed may affect the urgency. One example may be picking up an object from a table. This task may initially result in the same success score, but this makes a big difference whether the object to be picked is fragile or not. Therefore, a higher urgency index may be used for an object tagged as "fragile."

[0071] If the set of robots 1 in a system is supervised by multiple operators, the urgency indicator can also be used to distribute requests to the different operators.

[0072] The urgency indicator may also be used to define the information presented to the operator. The higher the urgency, the more information is presented to the operator. For example, if immediate assistance from an operator is required, the operator will be provided with all the information necessary to quickly identify the situation in order to determine the necessary assistance. In addition to images taken by the robot 1's camera, distance or other sensor values ​​may be presented. Conversely, a less urgent situation may only require the transmission of camera images so that the operator can monitor the current situation and intervene in case of an unexpected event. In general, the type and amount of information about the robot 1's situation will be adapted to the urgency determined as described above.

[0073] To directly control the robot 1 to perform a particular action, the operator uses the controls to manipulate the robot 1 and perform the desired action to further the execution of the determined task (step S12). In step S13, it is determined whether the operator's input can be directly translated into a movement or behavior of the robot 1. If yes, the action as commanded by the operator is performed according to the remote control input received from the operator. Thus, the process jumps to step S8 and proceeds automatically after the action defined by the operator is completed, as previously described with respect to actions that may be performed autonomously.

[0074] If the operator's input in step S12 is not a direct instruction for the movement and operation of the service robot 1, and therefore there is no definition of an action to be directly implemented, the process proceeds to step S14. As explained above, the operator can directly instruct the robot 1 on how to proceed with the execution of the action, but alternatively, the operator can add information to the database 17. Such added information may then enable the system not only to make an improved evaluation of the candidate action definitions, but also to "reconsider" the task segmentation and / or the selection of the candidate action definitions. This is indicated by the dashed lines in the flowchart.

[0075] It should be noted that inputting directly applicable instructions for the robot 1 can be implemented in two different ways: First, any instructions input by the operator are immediately and directly executed by the robot 1; Second, the virtual reality set 20 may be used to pre-set up actions by only defining the actions to be executed, and after completing such action definition, send the action definition to the robot 1, which will then autonomously execute the newly defined actions; If augmented reality is used, the instructions may be pre-generated but also directly applied.

[0076] The outcome of the evaluation of the candidate action definitions depends heavily on the robot system's knowledge of its environment. Therefore, the additional information entered by the operator in step S14 is used to strengthen the robot's knowledge base. Information added about the objects involved in the action to be performed, for example, in response to input from the operator, may be sufficient to allow the processing unit 14 to subsequently process the situation without further input from the operator. Therefore, such input, which should be about additional characteristics of the objects or conditions for acting on the objects, may result in an increase in the success score of the evaluated action. In that case, the process then proceeds to step S8, as described above. If the information added by the operator is still not sufficient to increase the success score of the robot 1's autonomous operation, step S7 results in a new request being sent to the operations center. The operator can then decide again whether direct control of the robot 1 is appropriate or whether the operator prefers to improve the database 17 by adding further information.

[0077] The retrieval of candidate action definitions in step S5 is based on the certainty of the identification and also on the information available for each identified object involved in the execution of the action. For example, if a cup is on a table and needs to be placed in a different location, the first action to be performed should be to grab the cup. The information contained in the tag of the object "cup" defines the actions that can be performed on the identified object. For a cup, the information in the tag can even include further properties, such as sensitivity to mechanical stress. A cup made of ceramic should be handled with more care than a cup made of metal.

[0078] As the above explanation makes clear, the operator will input additional information in step S14, thereby improving the robot's "understanding of the real world." The operator will attempt to input additional information about all objects involved in the scene as currently perceived by the robot 1. This results in two different ways of improving the overall behavior of the robot system over time. On the one hand, the action definitions stored in the database 17 are improved, and a wider variety of different action definitions become available for the robot 1's future evaluation of new situations. On the other hand, the robot 1's knowledge base is improved by adding information about determined objects in the robot 1's sensed environment. Adding this information improves the information used by the processing unit 14 when predicting the success of intended actions. Therefore, any new information added to the database 17 will improve the robot 1's ability to autonomously perform tasks.

[0079] Note that the present invention is described in terms of "objects." However, objects are only used for improved understanding. In fact, objects are given only as examples for "entities," which may consist of multiple objects, or even multiple lower-level entities, which may be combinations of objects. In a hierarchical sense, a house comprises a kitchen, a roof, ..., which in turn comprises a stove, a dishwasher, etc. Note that these higher-level entities can be inferred from the presence of other objects, in this example, from the kitchen of the house. This approach is known as bootstrapping. Then, determining the location "kitchen" makes it possible to determine even objects partially confined to the kitchen.

[0080] In addition to action definitions, the database 17 also stores all information collected over time. The processing unit 14 takes this information into account when evaluating the currently encountered situation and decides whether an operator needs to be contacted or whether the robot 1 can successfully execute the next action. This information can also be referred to as "world knowledge." World knowledge improves each time information is added to one of the objects or entities whose models are stored in the database 17. The models allow the robot 1 to identify objects present in its environment. Identification uses sensor output that is compared with the model stored in the database. An object can be considered identified when sufficient similarity between the stored model and the information derived from the sensors can be recognized. Each object stored in the database 17 can contain multiple associated pieces of information, called tags. Each tag is unique and can be linked to authorized actions to be performed on the object or entity. Additional information is added to the database 17 each time the robot system is unable to handle the current situation autonomously and the operator enters additional information in response to a request for assistance received from the robot 1. All currently available information, which is also used by the processing unit 14 to search and evaluate candidate action definitions, is presented to the operator, who then decides whether adding information in the form of additional tags for objects or entities is necessary to improve the robot system's world knowledge so as to improve overall performance, or to directly control the actions of the robot 1.

[0081] The additional information added by the operator is not limited to adding new tags or correcting tags of existing objects that have already been identified. If the system is unable to recognize an object at all, it may also contact the operations center for assistance. Provided with an image (representation) of the object, the operator can then add a new model (a new entry) to the database along with all the information the operator currently possesses and believes would be useful. Thus, when an object is identified for the first time by the robotic system, the system will automatically contact the operator to improve the system's world knowledge. After the additional knowledge is entered by the operator, it becomes immediately available to the processing unit 14. Thus, even when an unknown object is recognized for the first time during task execution, the additional information can be used in the next step of task execution. To assist the operator in adding information to the object, the system can also suggest tags that are already available, tags that have already been added to similar and known objects. The operator can then select from the suggested list the one they believe best fits. However, it is also possible to provide operator-defined entries.

[0082] The added information about an object is not limited to the object's characteristics but also includes possible actions or conditions for performing the actions. For example, when a shelf with a door is first perceived by the system, a representation of the object, most likely an image, is displayed by the display 18. The operator can then represent the object as a shelf and add not only the information that the shelf has a door, but also information about a set of actions that must be performed if the object is to be placed on the shelf. The set of actions includes first opening the door. Thus, if this object is identified by the system in the future, the system knows that the object can be placed on the shelf, but that the handle must first be grasped to open the door.

[0083] While all the explanations given above require that at least basic world knowledge be present in the system, it is clear that the system, when first set up, does not have any such world knowledge except for stored models of real-world objects already present in database 17. The most common objects are stored during the system design phase before the robot system is installed in its actual working environment. When setting up the system in its actual working environment, it may be useful if the user of robot 1, as well as the operator, can input information into the system. Therefore, it may be considered to provide robot 1 with an interface to a user device, such as a smartphone, tablet, etc. During the setup phase, robot 1 will then collect data about objects that can be detected in its environment. Representations of such objects are presented to the user, who then enters information.

[0084] To add information that allows the robot 1 to determine how to perform a task, it is also necessary to define, for example, a goal state for each object. Such a goal state is necessary to interpret the task. For example, the task "clean the room" may mean that every object is returned to its dedicated location. When the dedicated location for each object is stored in a tag associated with the object, the robot system will necessarily know the dedicated location of each of the objects and be able to move the object to its dedicated location after recognizing that its current location is different from the dedicated location.

[0085] Starting from basic knowledge that allows the robot 1 to identify, for example, different rooms and furniture, layouts, and typical objects in a home, the robot 1 will then learn and generate improved world knowledge. World knowledge is stored in a hierarchical structure. This means, for example, that a house (an entity) consists of multiple rooms (lower-level entities). Each room contains multiple objects that reside within it. Objects can be stationary or movable. Examples of stationary objects might be an oven or a refrigerator, which are specific to a room type called a "kitchen." Other objects can be movable and therefore can be encountered in multiple rooms. Movable objects can be moved by themselves (e.g., a pet bird) by objects that can be moved (e.g., a Lego brick). For these movable objects, preferred rooms can be defined by a user or operator or can be learned over time.

[0086] Starting with such basic knowledge of the robot's 1 working environment, the first task may then be given to the robot. The improvement of world knowledge is thus achieved incrementally whenever an operator is involved. As each robot's world knowledge increases over time, it is possible that an operator may eventually be able to supervise a significant number of robots. However, during the early phases of operation of a newly set-up robot 1, the operator is likely to be limited to focusing on a single robot 1. It is particularly advantageous for multiple robots to use the same "intelligence" (processing unit 14 and database 17, or parts thereof) since adding information to the database 17 can subsequently improve the knowledge base of multiple robots 1. Conversely, whenever one of the robots 1 requests assistance, the added information is instantly available to all robots 1 that reference the same database 17.

[0087] In the following, an example is given that makes it possible to understand how tags added to unique objects make it possible to control which actions are or can be performed by the robot 1. For illustration purposes, the action performed by the robot 1 is to pick up a cup. This action, which is necessary to clean the kitchen or the table, may need to be performed in several different situations, for example.

[0088] Firstly, it takes into account situations that are already known to the system, either because this has already been done successfully in the past, or because the identification of any object involved in the situation can be done without any ambiguity and the context is known. In such cases, the execution of the action "pick up the cup" can be performed completely autonomously by the robot 1. The operator does not need to be contacted.

[0089] Next, there is an at least partially unknown situation in which a cup is to be picked up. In this case, the system can automatically determine the correct action with less confidence than in the first case. However, in this situation, if the robot 1 can find a tag that says "not fragile" in the description of the object "cup," it may still attempt to perform the most likely action. This is a trial-and-error scenario, in which the robot 1 can perform an action and analyze the results after the action is performed. Based on additional information contained in the object's tag, a risk assessment is performed by the processing unit 14. The tag may also contain instructions for obtaining additional information before a final decision is made on whether or not an action can be performed. For example, if the object to be picked up is a cup, the tag may contain instructions for determining whether there is still liquid in the cup. If it is determined that there is liquid in the cup, this may result in preventing a trial-and-error approach, even if a trial-and-error approach would be acceptable in the case of an empty cup.

[0090] The third case is a scenario where only a moderate risk is tolerated when a cup is picked up. This may be the case when the cup is identified as being made of ceramic and therefore may break if the action is not successfully performed. In such a case, the identified action can be suggested to the operator, who then responds to this request to confirm that the action may be performed or take over control. Once the pick-up problem is resolved by an action based on the action definition suggested by the robotic system, the system prompts the operator to tag the object accordingly. An example is a plate; the plate may not be clearly identified by the system, but the pick-up action was suggested by the system and confirmed by the operator, and the action is ultimately performed successfully. In such a case, the system prompts the operator to add a unique pick-up action as one potential action to be performed on the plate.

[0091] Finally, completely unknown situations may arise where it is not possible for the system to autonomously select an action. This will occur whenever the system is unable to identify the object that must be handled by an action. In this case, the processing unit 14 will not be able to identify and retrieve any potential action definitions available in the database 17 in step S5, and will consequently be unable to directly request assistance from the operator.

[0092] What has been described with respect to the pick-up action can occur for any possible action; there is always a range from fully autonomously executing the action to situations where perhaps no action is suggested by the system at all. This becomes clear when considering placing an object, where multiple orientations can be correct for many objects. For example, a book may be placed upright, as is customary when a book is returned to a shelf, but it may also be placed on its wide side. The determination of which of the possible orientations is correct must be made depending on the specific context. In a tag characterizing an object, conditions can be defined that allow for the correct determination of the orientation when placing the object. In the case of a book, the condition may be that the target location is a table (or more generally, a large surface), and therefore the book must be placed on its wide side. Conversely, when the target location is a shelf and the book must be placed in a gap between other books, the orientation must be upright.

[0093] Similar to the explanation given above in relation to the pick-up operation, risk assessment can also be used to place an object. If a low risk is determined from all available information, a trial-and-error approach may be chosen, with the operator getting involved by sending a respective request when the estimated risk exceeds a certain threshold.

[0094] Sometimes, situations may arise where the robot 1 could learn from decisions made by the operator. For example, the robot is given the task of clearing a table where multiple objects can be found. Therefore, evaluation of possible actions reveals that a clear sequence with sufficient probability cannot be determined. A pick-up action may be predicted to be successfully performed, but so would picking up a different object. In this case, two distinct actions are identified by the system, both with equivalent success scores. In such a situation, the system will also request assistance from the operator to select the action to be performed first. Because both actions meet the requirements to be performed autonomously by the system, this may trigger a request for assistance the next time a similar situation occurs, forcing the operator to enter rules for prioritizing actions (objects).

[0095] In response to a request, for example the question "why", additional information can be entered by the operator. In the example, possible reasons could be least effort, most accessible, favorite cup... If no clear reason can be defined by the operator, this too can be indicated by the operator. In future situations, the robotic system can then select the action that happens to be performed first.

Claims

1. A service robot system including a robot, the robot (1) has a drive system (3) for moving the robot (1) to a target position and at least one effector (4) for manipulating the environment of the robot, the system including a task determination unit for determining a task to be performed by the robot, and a processing unit (14) configured to control the drive system (3) and the at least one effector (4) according to the task based on an action definition, the processing unit (14) being configured to automatically retrieve action definition candidates from a database (17), evaluate the retrieved action definition candidates for success scores indicating the likelihood that actions according to the action definition candidates will contribute to successfully completing the task, execute actions according to the action definition candidates having a maximum success score equal to or greater than a preset threshold until a next action to be executed having a success score below the preset threshold is reached, and then send a request for external assistance via a communication interface (21); an operator interface (18, 19, 20) for remotely controlling the robot (1) to perform actions and / or for inputting information for enhancing the database (17) by an operator in response to said request; The definition of the remotely controlled action is stored in the database (17) as a new action definition.

2. The system of claim 1 , wherein the processing unit (14) is configured to determine the success score of the candidate action definition based on information about the success of past actions performed by each candidate action definition.

3. The system of claim 1 or 2, wherein the processing unit (14) is configured to segment the task into a sequence of action definitions.

4. 4. The system according to claim 1, wherein the robot (1) is configured to perform actions with a success score equal to or greater than the preset threshold only for actions that have already been successfully performed in the past and that can be performed perfectly.

5. 5. The system of claim 1, wherein the processing unit (14) is configured to dynamically predict potential changes in the environment and to adapt or select the action.

6. 2. The system of claim 1, wherein the robot (1) is equipped with at least one sensor (11, 12) for physically sensing the environment of the robot and configured to transmit a sensor output to the processing unit (14) and / or the operator interface (18, 19, 20).

7. 7. The system according to any one of claims 1 to 6, wherein the operator interface (18, 19, 20) comprises an XR set (20) including a headset and controls for virtually controlling the robot (1) to perform actions before transmitting remote control signals to the robot (1) for performing the actions accordingly.

8. 1. A method for improving the autonomous execution of a task of a service robot, comprising: A step (S1) of detecting the environment of the service robot (1) with at least one sensor (11, 12); determining (S2) objects in the environment of the service robot and associating with the determined objects a reliability of information obtained from sensing the environment; determining (S3) a task to be performed by said service robot (1); A step of extracting action definition candidates from the database (17) (S5); evaluating (S6) a plurality of the retrieved candidate action definitions to determine a success score for successful execution and contribution to successful completion of the task; Executing (S8) actions according to the candidate action definitions having a maximum success score equal to or greater than a preset threshold until a next to-be-executed action having a success score below the preset threshold is reached, and then sending a request for external assistance via a communication interface. Including, the robot (1) is remotely controlled and / or information entered by an operator is stored in the database (17) in response to a request, A method in which new action definitions are created from remote operation of the robot (1) and added to the database (17).

9. The method of claim 8 , wherein the success score of a candidate action definition is calculated based on information about the success of past actions performed by each candidate action definition.

10. 10. The method according to claim 8 or 9, wherein the task is segmented (S4) into a number of actions to be performed sequentially and / or simultaneously, and for each potential action definition is retrieved (S5).

11. The method of claim 8 , wherein the request includes at least one potential action definition.

12. 12. The method of claim 8, wherein the threshold value of the success score is set individually for an action to be performed depending on one or more tags of one or more objects involved in the action to be performed, the tags being a plurality of pieces of information associated with the objects determined from sensor signals output from sensors for physically detecting the robot's environment.

13. 13. The method of claim 12, wherein information about an entity involved in an action to be performed is read in response to the request and added to the tag of the entity or added as an additional tag.

14. 14. The method according to any one of claims 8 to 13, wherein the task to be performed is determined (S3) from instructions received via a user interface or from a current situation encountered by the robot (1).

15. The method of claim 14 , wherein the determination is based on an association of previously received instructions with contextual information for the instructions extracted from the circumstances under which the instructions were received.

Images