Control device, control method, robot device, and program

Abstract

A control device is provided, which includes: an evaluation unit configured to evaluate whether environmental information acquired from a sensor represents a relevant element; and an environmental information setting unit configured to switch the environmental information to pre-acquired environmental information that does not include the relevant element when the evaluation unit evaluates that the environmental information represents the relevant element.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of Japanese Priority Patent Application JP 2018-103203 filed on May 30, 2018, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to a control device, a control method, and a program. Background Art

[0004] Heretofore, the following PTL 1 has pointed out that a detection signal output by a sensor unit of an operation unit of an observation robot (a part of the robot) is negated.

[0005] Reference List

[0006] Patent Literature

[0007] PTL 1: JP 2009-255264A Summary of the invention

[0008] Technical issues

[0009] However, in the technology described in the above PTL 1, since the detection signal corresponding to the operating unit of the robot is negated, there is a problem that the condition of the area where the operating unit is located and the surrounding environment cannot be grasped. Therefore, the condition of the surrounding environment of the robot is unclear, and it is difficult to formulate an operation plan of the robot based on the position of obstacles around the robot, etc. In the technology described in the above PTL 1, since the detection signal corresponding to the operating unit of the robot is negated, the condition of the surrounding environment of the robot cannot be grasped after the negation, and the safe operation of the robot is hindered.

[0010] Therefore, even in the case where elements related to the robot are included or indicated in the environmental information, it is necessary to enable optimal operation.

[0011] Solution to the problem

[0012] The present disclosure is defined by the appended claims.

[0013] According to an embodiment of the present disclosure, a control device is provided, comprising: an assessment unit configured to assess whether environmental information acquired from a sensor includes or represents a relevant element (e.g., a robot arm, although other examples are given below); and an environmental information setting unit configured to switch the environmental information to pre-acquired environmental information that does not include the relevant element when the assessment unit assesses that the environmental information acquired from the sensor includes the relevant element.

[0014] In addition, according to an embodiment of the present disclosure, a control method is provided, which includes: evaluating whether the environmental information acquired from the sensor includes or represents relevant elements; and, when it is evaluated that the environmental information includes or represents relevant elements, switching the environmental information to pre-acquired environmental information that does not include relevant elements.

[0015] In addition, according to an embodiment of the present disclosure, a program is provided for causing a computer to function as: a device configured to evaluate whether environmental information acquired from a sensor includes or represents a relevant element; and a device configured to switch the environmental information to pre-acquired environmental information that does not include the relevant element when the environmental information is evaluated to include or represents the relevant element.

[0016] Beneficial Effects of the Invention

[0017] As described above, according to the embodiments of the present disclosure, even in the case where a related element is included in the environmental information, an operation can be achieved.

[0018] Note that the effects described above are not limitative. With or in place of the above effects, any of the effects described in this specification or other effects that can be grasped from this specification can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] [ Figure 1 ] Figure 1 is a schematic diagram showing a state where a robot device according to an embodiment of the present disclosure moves in a direction of an arrow;

[0020] [ Figure 2 ] Figure 2 is a schematic diagram showing an example of a bounding box;

[0021] [ Figure 3 ] Figure 3 is a schematic diagram showing a process in a case where a related element is included in a bounding box;

[0022] [ Figure 4A ] Figure 4A is a schematic diagram showing a switching range in which environmental information is switched;

[0023] [ Figure 4B ] Figure 4B is a schematic diagram showing a switching range in which environmental information is switched;

[0024] [ Figure 5A ] Figure 5Ais a schematic diagram showing a usage example of environmental information acquired at the current time and acquired in advance based on a time series;

[0025] [ Figure 5B ] Figure 5B is a schematic diagram showing a usage example of environmental information acquired at the current time and acquired in advance based on a time series;

[0026] [ Figure 5C ] Figure 5C is a schematic diagram showing a usage example of environmental information acquired at the current time and acquired in advance based on a time series;

[0027] [ Figure 6 ] Figure 6 is a flowchart showing the flow of processing in the robot device;

[0028] [ Figure 7 ] Figure 7 is a schematic diagram showing an example in which a bounding box including a grasped object is obtained;

[0029] [ Figure 8 ] Figure 8 is a schematic diagram showing an example in which two arms exist to overlap between a robotic device and a wall;

[0030] [ Fig. 9A ] Fig. 9A is a schematic diagram showing an example of a process of predicting a motion trajectory of a moving obstacle;

[0031] [ Fig. 9B ] Fig. 9B is a schematic diagram showing an example of a process of predicting a motion trajectory of a moving obstacle;

[0032] [ Fig. 9C ] Fig. 9C is a schematic diagram showing an example of a process of predicting a motion trajectory of a moving obstacle;

[0033] [ Fig.9D ] Fig.9D is a schematic diagram showing an example of a process of predicting a motion trajectory of a moving obstacle;

[0034] [ Fig.10 ] Fig.10 is a schematic diagram showing an example of unloading by a forklift;

[0035] [ Fig.11 ] Fig.11 is a block diagram showing a configuration of a control device that controls a robot device;

[0036] [ Fig.12 ] Fig.12 is a schematic flow chart illustrating a method;

[0037] [ Fig.13 ] Fig.13 is a schematic diagram showing a data processing device. DETAILED DESCRIPTION

[0038] Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. Note that, in this specification and the drawings, structural elements that have substantially the same function and structure are denoted with the same reference numerals, and repeated explanation of these structural elements is omitted.

[0039] Note that the description is given in the following order.

[0040] 1. The significance of external world information in autonomous motion technology

[0041] 2. Overview of Robotic Equipment

[0042] 3. Information replacement of areas blocked by related components

[0043] 3.1. About Bounding Box

[0044] 3.2. Specific examples of bounding boxes

[0045] 3.3. Deletion of environmental information in the bounding box

[0046] 3.4. Switching of environmental information

[0047] 3.5. Timing process of information switching

[0048] 4. Processing flow in robotic devices

[0049] 5. Cases where the relevant elements include grasped objects

[0050] 6. Example of a situation where related components overlap

[0051] 7. Dynamic prediction and processing of environmental information

[0052] 8. Other examples where pre-fetched environment information is used

[0053] 8.1. Action plan during transfer work by cargo handling vehicles such as forklifts

[0054] 8.2. Action planning of an autonomous moving body equipped with a robotic arm

[0055] 1. The significance of external world information in autonomous motion technology

[0056] In order to make a moving body such as a robot move autonomously, it is necessary to always know where the moving body itself is currently and whether there are obstacles near the moving body that hinder the movement. For an autonomous moving body, there are two kinds of obstacles. One is a static obstacle that does not change over time, such as the wall of a building. The other is a mobile obstacle whose position and posture can change over time, such as a chair. Information about these obstacles is obtained through external sensors attached to the autonomous moving body. While updating the position and posture information of the autonomous moving body, the external sensor plays an important role in enabling the autonomous moving body to reach the destination without hitting obstacles.

[0057] However, if the components of the robot attached to the robot, etc., cover or block even only a part of the surrounding environment of the external sensor, the acquired information of the external world is damaged, so that the external world cannot be correctly recognized. As a result, there is a problem of erroneously identifying the components of the robot itself as obstacles, making it difficult to reach the destination. Therefore, even if the external sensor senses the components of the robot itself, it is necessary to prevent them from being erroneously identified as obstacles. In addition, in the case where the components are components that involve contact with objects in the external world, such as a robot arm, some things must be handled, including objects during transmission, so as not to be regarded as obstacles.

[0058] 2. Overview of Robotic Equipment

[0059] Now it will be based on Figure 1 The overall schematic structure of the robot device (autonomous moving body) 1000 according to the embodiment of the present disclosure is described below. Fig.11 Let's discuss further details.

[0060] Figure 1is a schematic diagram showing a state in which the robot device 1000 according to the present embodiment moves in the direction of the arrow. The robot device 1000 includes an external sensor 200 that detects the state of the external world by detecting or acquiring or capturing environmental information, and moves toward a destination while avoiding obstacles detected by the external sensor 200. The external sensor 200 is an object recognition sensor that recognizes or at least detects an object or at least the existence of an object as an object at a certain distance from the sensor 200 and / or in a certain direction relative to the sensor 200 (for example, by being used as a distance measurement sensor configured to measure the distance to the object), and detects environmental information of the external world, wherein the sensor is, for example, a camera, a ToF (time of flight) sensor, a light detection and ranging (LIDAR), a depth sensor, or an RGB (imaging) sensor. The environmental information may include, for example, position information of objects present around the robot device 1000. As an example, if the external sensor 200 is a camera, the environmental information may include image information; if the external sensor 200 is a ToF sensor, the environmental information is distance measurement information showing the depth position of the object. In the case where the external sensor 200 is a camera, information on a depth position of an object may be acquired by using a 3D camera as in a ToF sensor.

[0061] like Figure 1 As shown, the robot device 1000 includes an arm 100. The arm 100 is used to grasp an object, detect the state of an object, or perform other things.

[0062] When the robot device 1000 moves, there is a case where the external sensor 200 senses the components of the robot device 1000 itself. For example, in a case where the arm 100 is included in the detection range of the external sensor 200 and the external sensor 200 has sensed the arm 100, the arm 100 itself is represented by the environmental information and is therefore generally identified as an obstacle to the robot device 1000. Since the arm 100 is mistakenly identified as an obstacle, the area where the arm 100 exists is determined to be an area with a high risk of movement. In this case, the operation of the robot device 1000 is determined so that the robot device 1000 does not touch the arm 100.

[0063] If the arm 100 is recognized as an obstacle and the operation of the robot device 1000 is determined so that the robot device 1000 does not touch the arm 100, restrictions are imposed on the operation of the robot device 1000 by the arm 100 which is not an obstacle. Therefore, there is a case where the operation of the robot device 1000 is excessively restricted.

[0064] Therefore, in the present embodiment, in the case where the external sensor 200 has sensed an element related to the robot device 1000 itself (hereinafter, referred to as a related element), the external sensor 200 does not recognize the related element as an obstacle, thereby enabling the operation of the robot device 1000 to continue. The related element in this context includes constituent parts included in the robot device 1000 itself such as the arm 100, a grasping object grasped by the arm 100, legs, wheels, thrusters, probes, tools, weapons, etc. Thereby, it is possible to suppress the operation of the robot device 1000 from being excessively restricted.

[0065] Specifically, in obstacle sensing when an autonomous moving body such as the robot device 1000 moves, even if a related element related to the autonomous moving body is sensed, the related element is not recognized as an obstacle, and the movement continues. Thus, even in the case where the detection range of the external sensor is blocked by the related element, the movement can be performed without being restricted by the related element.

[0066] In this case, shape approximation information (e.g., a bounding box) of a three-dimensional shape obtained by approximating the position and / or shape of the relevant element in a space obtained or detected based on the environmental information is generated based on the internal information of the robot device 1000 (e.g., preprogrammed information, information derived by the robot device, configuration information, etc.); thereby, the shape, position, and posture information of the relevant element are removed from the environmental information, and the erroneous identification of the obstacle is resolved. In these examples, the assessment of whether the environmental information includes or represents the relevant element (e.g., performed by the assessment unit 530 to be described below) is configured to assess whether the environmental information includes or represents the relevant element based on the following information: shape approximation information in which the position and shape of the relevant element are approximated; and environmental information.

[0067] The shape approximation information may be, for example, information including a simple shape of the relevant element, wherein the assessment process (e.g., performed by the assessment unit as discussed below) includes detecting the presence of a shape approximation representing the simple shape including the relevant element in the environmental information. For example, it may be information obtained by enlarging the outline of the relevant element so that the shape approximation information represents the physical shape of the outline of the relevant element surrounded by the margin area.

[0068] The environmental information setting unit may be configured to select the pre-acquired environmental information as environmental information of a region of shape approximation information in which the position and shape of the relevant element are approximated, instead of the environmental information in which the relevant element is evaluated to be included.

[0069] In addition, by switching (or selecting replacement) environmental information from which the shape, position, and posture information of the relevant element is removed to pre-acquired pre-acquired environmental information that does not include the relevant element, environmental information of the area blocked by the relevant element is estimated.

[0070] Therefore, in some examples, the environment information setting unit is configured to select pre-acquired environment information as the environment information of the area blocked by the relevant element. In some examples, the environment information setting unit is configured to select pre-acquired environment information as the environment information of the area assessed to include the relevant element and the area blocked by the relevant element.

[0071] Furthermore, by planning the action of the robot apparatus 1000 in combination with shape approximation information of the three-dimensional shape and environmental information, unintentional contact between relevant elements and obstacles is suppressed.

[0072] Through the above, even in the case where the relevant elements of the robot device 1000 block the external sensor, the operation and movement of the robot device 1000 can be continued while inferring the environment of the outside world. In addition, the relevant elements include not only the constituent parts of the robot device 1000, but also the object on which the robot device 1000 acts, for example, in the case where the arm 100 of the robot device 1000 holds and transfers the object, so that it can be used in a wide range, not just for movement.

[0073] 3. Information replacement of areas blocked by related components

[0074] 3.1. About Bounding Box

[0075] In order for the robot device 1000 to grasp the position of the relevant element, the concept of a bounding box as a geometric shape covering the entire object is used. The bounding box is calculated based on internal information (size of the relevant element, joint angle, etc.) that the robot device 1000 has inside it (or at least a part of its structure and / or function). Specifically, the bounding box is generated based on the shape, position and posture information of the relevant element calculated based on the internal information.

[0076] Figure 2 is a schematic diagram showing an example of a bounding box. Figure 2 1 shows an example in which a bounding box is found for the arm 100, and the bounding box 30 of the arm 100 is shown with the area of ​​the rectangular parallelepiped shown by the dotted line. Figure 2As shown, the bounding box 30 is a shape in which the shape of the relevant element is approximated to a simple geometric shape such as a cuboid, a cube, a cylinder, a sphere, a cone, or a pyramid. On the other hand, the bounding box 30 may be the exact shape of the arm 100, or may be a shape in which the shape of the arm 100 is expanded outward by a prescribed amount. The area of ​​the bounding box 30 is set to the same area as the area of ​​the original arm 100 or an area larger than the area of ​​the original arm 100.

[0077] In addition, in the case where the relevant element includes a plurality of constituent parts as in the arm 100 of the robot apparatus 1000 , for example, a bounding box may be generated for each constituent part.

[0078] In addition to being responsive to shape information defining the shape (and size) of the bounding box or other shape approximation of the associated element 100, the assessment of whether the associated element is included in the environmental information may also be responsive to information defining the current position of the associated element relative to the sensor (which information may be provided by the operation planning unit 570 to be discussed below).

[0079] In at least some examples, the relevant element is an element related to the robotic device (e.g., a constituent element or constituent part of the robotic device, such as the arm 100) and is capable of moving with respect to and being controlled by the robotic device according to a current element position relative to the robotic device (e.g., by one or more actuators 582 to be discussed below); the sensor 200 has a current sensor position relative to the robotic device (which current sensor position may be fixed or, for example, movable by one or more actuators 582); and the assessment of whether the relevant element is included in the environmental information involves detecting information defining the current position of the relevant element relative to the sensor in response to the current element position and the current sensor position (which information may be supplied to the assessment unit 530, for example, from the operation planning unit 570 via a data connection shown as dotted line 572). Note that, as discussed above, if the current position information is unknown, the position of the relevant element may be detected by shape and / or object detection in the environmental information itself.

[0080] 3.2. Specific examples of bounding boxes

[0081] A rectangular parallelepiped having a size in which the entire relevant element can be included is used as the shape of the bounding box; thus, calculation as to whether environmental information related to the relevant element is included in the bounding box can be easily performed.

[0082] On the other hand, there is no particular restriction on the shape of the bounding box. By using the CAD model of the relevant element, a bounding box that best fits the actual shape can be generated. On the other hand, when using a CAD model, there is a possibility that the environmental information corresponding to the relevant element will deviate from the inside of the bounding box. This may be due to errors in the CAD model or errors in internal information such as joint angles (so that the actual relevant element may occupy an area in the space that is not predicted by the shape information). By setting the bounding box to a simple shape such as a cuboid and generating the bounding box in a size larger than the size of the corresponding relevant element (so that the bounding box provides a margin area), the environmental information corresponding to the relevant element can be prevented from deviating from the inside of the bounding box due to errors in internal information such as joint angles, thereby improving robustness.

[0083] As another method for generating a bounding box using a CAD model, a bounding box in which the error of the internal information can be considered while the shape is maintained along the actual shape of the relevant element can be generated by expanding the CAD model to a certain extent or generating a margin. In addition, a plurality of bounding boxes can be generated for one component. For example, a bounding box can be formed by combining a plurality of cubes to cover one relevant element. Figure 2 In the case of the bounding box of the arm 100 shown, the bounding box of the arm 100 can be generated by generating a bounding box with respect to each of the arm body 102 and the gripping unit 104 and combining the two.

[0084] 3.3. Deletion of environmental information in the bounding box

[0085] Next, the deletion of the environmental information in the bounding box is described. As described above, the size and position of the bounding box can be obtained based on the internal information of the robot device 1000. For example, if the bounding box of the arm 100 is taken as an example, when the bounding box is set for the arm 100, the size (length, width, etc.) of the bounding box is obtained in advance. In addition, the posture (angle) of the bounding box is obtained based on the detection value of the encoder set in the movable unit (joint) of the arm 100. Therefore, the size and position of the bounding box can be grasped based on the internal information obtained by the robot device 1000.

[0086] On the other hand, environmental information showing position information of objects existing around the robot device 1000 is acquired by the external sensor 200. In the case where a relevant element is included in the detection range of the external sensor 200 and the external sensor 200 is capturing the relevant element, information of the relevant element is included in the environmental information.

[0087] When the environment information is found to exist in the bounding box based on the size and position of the bounding box grasped on the robot device 1000 side, the environment information corresponds to the information of the relevant element. Therefore, after acquiring the environment information, first, it is investigated whether the environment information corresponding to the relevant element is included in the space where the bounding box exists.

[0088] Then, in the case where the environmental information corresponding to the relevant element is included in the space of the bounding box, the environmental information in which the relevant element is excluded can be obtained by removing or deleting the above-mentioned environmental information from the interior of the space of the bounding box. In other words, once it has been assessed that the relevant element is present in or included in the environmental information, the environmental information from the area corresponding to the bounding box and / or covered or blocked by the bounding box used to make the assessment is deleted.

[0089] Figure 3 is a schematic diagram showing a process in a case where a related element is included in a bounding box. Figure 3 A bounding box 30 of a rectangular parallelepiped is shown, and a case where the related element is shown by a point group 40 is shown. If an object such as a related element or an obstacle is detected by the external sensor 200, the spatial shape and position of the object can be represented as a set of coordinates of the point group 40.

[0090] As an overview of the process, a point in a so-called "point cloud" represents, for example, a detection of the presence of something at the 3D location or position of that point in 3D space. Where a particular object is present, the points corresponding to the detection of that object will tend to form a point cluster that is roughly similar in shape to the object being detected ( Figure 3 The shape of the point group (compared to the shape of the physical object being detected) may be approximate due to factors such as the resolution of the detection (point-to-point distance), noise, detection errors, obstruction of the object or portion of the object relative to the sensor 200, etc.

[0091] Various techniques can be used to detect whether and where relevant elements exist in environmental information. In an example technique, the physical position of the relevant element relative to the physical position of the sensor is obtained (for example, using motion control information of the current position of the arm 100 through which the robot device controls the motion), and the position in the environmental data is at least predicted based on this information. In another example, a bounding box that approximates the size and shape of the relevant element is compared with a point cloud detected at various possible positions and / or directions, for example, the various possible positions and / or directions are limited to a set of possibilities by a predetermined range of motion or a configurable range of motion of the relevant element relative to the sensor, and the position and direction with the maximum correlation with the bounding box (for example, the maximum number of points in the volume or area defined by the bounding box) is selected as the assessment position of the relevant element. The second technique does not require knowledge of the current position of the relevant element relative to the sensor. The two techniques can be combined so that, for example, the correlation of the bounding box can be detected at or near the predicted position.

[0092] In some examples, once such detection is made, points in the point cloud corresponding to the interior of the bounding box are deleted or removed from the environment information. In at least some examples, this will also include the deletion of objects or points detected by the bounding box with respect to the sensor (if such points are indeed detected). Note that in at least some examples, the bounding box can be larger than the actual physical object, which can include the deletion of some points that may or may not correspond to the actual physical object.

[0093] exist Figure 3 The figure on the upper side of shows a state where a point group 40 is included in a bounding box 30. The point group 40 included in the bounding box 30 corresponds to the environmental information of the relevant element. Figure 3 As shown in the figure on the lower side of , the environmental information corresponding to the relevant element is erased by removing the point group 40 included in the bounding box 30. Therefore, when an obstacle is identified based on the environmental information, the relevant element corresponding to the bounding box 30 is not identified as an obstacle, so that erroneous identification that the relevant obstacle exists near the robot device 1000 can be prevented.

[0094] As described above, by deleting the environmental information in the bounding box, the information of the related element is not included in the environmental information. Therefore, the related element is not mistakenly recognized as an obstacle, and the robot device 1000 can be controlled based on the environmental information that does not include the information of the related element.

[0095] 3.4. Switching or selecting environmental information

[0096] In the case of controlling the operation, movement, etc. of the robot device 1000 based on the environmental information, the environmental information cannot be used for the area blocked by the relevant element. In other words, the area on the other side of the bounding box from one side of the external sensor 200 is blocked by the relevant element corresponding to the bounding box, so the environmental information cannot be acquired for the area. Therefore, in the case where there is an obstacle in the blocked area, the environmental information related to the obstacle cannot be acquired.

[0097] Therefore, for the area blocked by the relevant element, pre-acquired environmental information (hereinafter, also referred to as pre-acquired environmental information) is used or selected. Specifically, the environmental information of the area blocked by the relevant element at a certain point in time is switched to the pre-acquired environmental information acquired before the time point. Thus, environmental information can be obtained for the entire area including the area blocked by the relevant element for which environmental information has been acquired by the sensor.

[0098] The pre-acquired environmental information is environmental information that has been acquired at a certain point in time, and is basically information that does not include environmental information of related elements. Therefore, by switching the environmental information of the area blocked by the related elements to the pre-acquired environmental information, environmental information that does not include information of the related elements can also be obtained for the area blocked by the related elements.

[0099] Figure 4A and Figure 4B 3 is a schematic diagram showing a switching range 35 for switching to pre-acquisition of environmental information. Figure 4A 1 is a front view showing a state where the robot device 1000 is viewed from the front. Figure 4B is shown from above (from Figure 4A Observe in the direction of arrow AA1 Figure 4A A top view of the robot device 1000 is shown. Figure 4A and Figure 4B In the illustrated example, the arm 100 of the robot device 1000 is regarded as a relevant element, and the bounding box 30 is set for the arm 100 .

[0100] like Figure 4A and Figure 4B As shown, the area on the opposite side of the bounding box 30 from the external sensor 200 is an area blocked by the relevant element and is regarded as a switching range 35 in which switching to pre-acquiring environmental information is performed. The switching range 35 is a range surrounded by straight lines 40, 42, and 44 connecting the center of the external sensor 200 and vertices 50, 52, and 54 of the bounding box 30, a straight line 46 showing the detectable range of the external sensor 200, and a straight line 48 showing the distance limit (in Figure 4A and Figure 4BThe straight line 46 showing the detectable range of the external sensor 200 shows the range in which the external sensor 200 can detect environmental information; Figure 4A In the left region of the straight line 46, no environmental information can be detected.

[0101] The switching range 35 is substantially defined by a straight line connecting the center of the external sensor 200 and the vertices of the bounding box 30. Figure 4A In the illustrated example, among the straight lines connecting the center of the external sensor 200 and the vertices of the bounding box 30, the straight line 47 connecting the center of the external sensor 200 and the vertices 56 of the bounding box 30 is located outside (to the left) of the straight line 46 showing the detectable range of the external sensor 200. Therefore, environmental information cannot be detected in the area sandwiched by the straight line 47 and the straight line 46. Therefore, the switching range 35 is defined using the straight line 46 instead of the straight line 47.

[0102] By setting the switching range 35 as described above, Figure 4A The height direction and Figure 4B The switching range 35 is set in both horizontal directions as shown so that switching is performed in the widest range and includes the entire area hidden by the relevant element.

[0103] In addition, a straight line 48 showing a distance limit may be set to provide a limit to the switching range 35 to the extent that the subsequent action plan is not hindered. The reason for narrowing the switching range 35 by the straight line 48 showing the distance limit is to try to use the environmental information acquired at the current point in time as much as possible, so that the action of the robot device 1000 can be performed according to the actual situation.

[0104] However, in the case where it is difficult to narrow the switching range due to some reason such as computational load, and in the case where it is not necessary to narrow the switching range, switching to the pre-acquired environmental information can be performed over the entire range at the point in time when the relevant element is detected in the bounding box 30.

[0105] As described above, for the switching range 35 blocked by the relevant element, the environment information at the current time cannot be used, and therefore, switching to the latest pre-acquired environment information acquired in the past is performed. Therefore, for the switching range 35, the latest and latest (most recent) pre-acquired environment information can be used to replace the environment information at the current time. On the other hand, for the inside of the bounding box, the environment information in which the relevant element is removed can be obtained by deleting the information of the relevant element. Therefore, environment information with high precision that does not include the information of the relevant element can be generated.

[0106] Thus, in such an embodiment, the environmental information setting process (which may be performed by the environmental information setting unit 550 to be discussed below) is configured to select the following information to replace at least a portion of the newly acquired or most recently acquired environmental information from the sensor: a corresponding portion of the most recent pre-acquired environmental information (i.e., the most recent environmental information acquired before acquiring the newly acquired environmental information) for which the assessment unit assesses a portion of the environmental information that does not include the relevant element or a portion of the environmental information does not represent the relevant element. For example, the corresponding portion may be selected to provide environmental information for at least a portion of the newly acquired environmental information, such as replacing environmental information in the newly acquired environmental information that is unavailable due to the presence (or assessment of the presence) of the relevant element. The pre-acquired environmental information may be acquired with respect to the same sensor position (relative to the environment), but this is not necessarily required, as long as the pre-acquired environmental information used provides replacement environmental information for the replaced environmental information in the newly acquired environmental information. Note that there may be situations where a single instance of pre-acquired environmental information does not necessarily provide all of the environmental information to be substituted for at least a portion of the newly acquired environmental information, in which case pre-acquired environmental information from more than one acquisition is used. Note also that it is not required to use the most recent pre-acquired environmental information; rather, instances of environmental information other than the most recently acquired environmental information may be used. However, using the most recent pre-fetched environment information means that the replacement information will be as up-to-date as possible.

[0107] The term "at least a portion" may refer only to the portion that is detected as representing the relevant element (note that, as referenced Figure 2 and Figure 3 As discussed, the detection process itself may (but need not) include margins around the relevant elements). Alternatively, the term may mean that additional margins around the detected relevant elements are applied in the replacement operation. Or in other examples, the term may actually apply to discarding all newly acquired context information and replacing it with pre-acquired context information.

[0108] Thus, in the example, a "portion" may be one or more of: an area in which shape approximation information is provided for the position and shape of the relevant element; an area obscured by the relevant element; an area assessed to include the relevant element and an area obscured by the relevant element; or all instances of environmental data acquired by a sensor.

[0109] Alternatively, "at least a portion" of the newly acquired context information that is replaced may be deleted as discussed below, and replacement context information is selected to replace it.

[0110] For example, the environment information deletion unit (to be discussed below) may be configured to delete at least a portion of the newly acquired environment information. In some examples, the environment information setting unit is configured to select pre-acquired environment information to replace the deleted environment information.

[0111] Therefore, environmental information excluding information of related elements can be acquired for both the inside of the bounding box 30 and the switching range 35 .

[0112] In addition, the switching to pre-acquisition of environmental information can also be performed for the area of ​​the bounding box 30. In this case, the switching to pre-acquisition of environmental information is performed for both the switching range 35 and the area of ​​the bounding box 30. Therefore, in this case, the process of deleting the environmental information in the bounding box 30 is unnecessary, and the process can be performed more simply. In addition, the switching to pre-acquisition of environmental information can be performed for the entire area for which environmental information has been acquired by the external sensor 200.

[0113] Furthermore, switching to pre-acquisition of environmental information may be performed only for the region of the bounding box 30. In this case, although switching to pre-acquisition of environmental information is limited to only a portion, more accurate environmental information at the current time can be acquired for the region of the bounding box 30.

[0114] On the other hand, in the case where an obstacle is included in the pre-acquired environmental information corresponding to the area of ​​the bounding box 30, the pre-acquired environmental information does not reflect the state at the current time; therefore, the acquisition of the latest environmental information indicating that there is no obstacle in the bounding box 30 is achieved by removing the environmental information in the bounding box 30. In this case, as described above, it is desirable to perform switching to the pre-acquired environmental information for the switching range 35 blocked by the relevant element, and delete the environmental information for the inside of the bounding box 30.

[0115] 3.5. Timing process of information switching

[0116] Figure 5A , Figure 5B and Figure 5C is a schematic diagram showing a usage example of environmental information acquired at the current time and acquired in advance based on a time series. Figure 5A , Figure 5B and Figure 5C The robot device 1000 is shown at the center of the rectangular area, and a top view in which the robot device 1000 is viewed from above is schematically shown. In addition, Figure 5A , Figure 5B and Figure 5C The relationship between the robot device 1000 and the area where the environmental information is used is shown.

[0117] Figure 5A FIG. 1 shows the situation at time t_0, and shows the situation where the external sensor 200 is not covered or blocked by the relevant elements. That is, there is no bounding box in the environmental information, and in the entire area ( Figure 5A The information of the relevant element is not included in the environmental information (the area marked with hatching in FIG. 1 ). The robot device 1000 formulates an operation plan (as an example of execution of a control function) based on or according to the environmental information detected by the external sensor 200, without performing the deletion of the environmental information in the bounding box described above or the switching to the pre-acquired environmental information for the switching range 35 blocked by the relevant element. Then, based on the operation plan, the robot device 1000 controls its own movement, the operation of the arm 100, etc. In addition, the robot device 1000 (for example, the environmental information setting unit 540) updates all the environmental information detected at this time t_0 to the pre-acquired environmental information.

[0118] Figure 5B 1 shows a situation at time t_1 after time t_0, and shows a situation where the relevant element 300 covers or blocks a part of the detection range of the external sensor 200. In this case, Figure 5B The area A1 marked with a hatch in FIG. 1 is an area blocked by the relevant element 300, and the environment information cannot be acquired for the area at the time point of time t_1. Therefore, the pre-acquired environment information acquired at time t_0 is used for the area A1 blocked by the relevant element 300. Figure 5B In, using Figure 5A The area A1 is marked with the same shadow as that of the image, thereby indicating that the pre-acquired environmental information acquired at time t_0 is used in area A1. On the other hand, the environmental information of the area not blocked by the relevant element 300 (area B1 marked with dots in the figure) can be used as the latest environmental information at time t_1. Therefore, the environmental information detected at time t_1 is used in the area not blocked by the relevant element 300.

[0119] In addition, the robot device 1000 updates the environmental information of the area B1 not blocked by the relevant element 300 to the pre-acquired environmental information at time t_1. In this case, the environmental information of the area A1 using the environmental information (pre-acquired environmental information) at time t_0 can be regarded as the latest pre-acquired environmental information at time t_1.

[0120] In the case where the environmental information of area A1 at time t_0 (pre-acquired environmental information) is used as the latest pre-acquired environmental information at time t_1, for area A1, the pre-acquired environmental information at time t_0 is updated to the pre-acquired environmental information at time t_1. In addition, for area B1 not blocked by the relevant element 300, the environmental information at time t_1 is updated to the pre-acquired environmental information.

[0121] in addition, Figure 5C 1 shows a situation at time t_2 after time t_1, and shows a situation in which the related element 300 and the related element 310 cover or block the external sensor 200. In this case, Figure 5C The area A1 marked with a hatch in FIG. 1 is an area blocked by the related element 300, and the environmental information cannot be acquired for this area at the time point of time t_2. Figure 5C The area A2 marked with dots in FIG. 1 is an area blocked by the related element 310 , and the environmental information cannot be acquired for this area at the time point of time t_2 .

[0122] Notice, Figure 5C The example of provides a schematic representation of a situation in which composite context information generated by a process includes some newly acquired context information and some replacement context information from more than one instance of pre-acquired context information. In each case, in this example, the most recent pre-acquired context information is used for each part.

[0123] Therefore, the pre-acquisition environment information acquired at time t_0 is used for the area A1 blocked by the relevant element 300, and the pre-acquisition environment information acquired at time t_1 is used for the area A2 blocked by the relevant element 310. Figure 5C In, using Figure 5A The area A1 is marked with the same shadow as that of , thereby indicating that the pre-acquisition environment information acquired at time t_0 is used in the area A1. Figure 5B The same point as the point of area B1 is marked in area A2, thereby indicating that the pre-acquired environmental information acquired at time t_1 is used in area A2. On the other hand, the environmental information of the area not blocked by the relevant element 300 or the relevant element 310 (area B2 marked with a point in the figure) can be used as the latest environmental information at time t_2; therefore, the environmental information detected at time t_2 is used in the area not blocked by the relevant element 300 or the relevant element 310.

[0124] In addition, the robot device 1000 updates the environmental information of the area not blocked by the related element 300 or the related element 310 (for example, stored in the storage unit 560 to be discussed below) to the pre-acquired environmental information. Figure 5B , the environmental information of the area A1 using the sensor information at time t_0 and the environmental information of the area A2 using the sensor information at time t_1 may be regarded as the latest pre-acquired environmental information at time t_2.

[0125] In the case where the environmental information of the area A1 using the sensor information at time t_0 and the environmental information of the area A2 using the sensor information at time t_1 are regarded as the latest pre-acquired environmental information at time t_2, for the area A1, the pre-acquired environmental information at time t_0 is updated to the pre-acquired environmental information at time t_2, and for the area A2, the pre-acquired environmental information at time t_1 is updated to the pre-acquired environmental information at time t_2. In addition, for the area B2 that is not blocked by the relevant element 300, the environmental information at time t_2 is updated to the pre-acquired environmental information.

[0126] exist Figure 5B and Figure 5C In , for the regions of the related element 300 and the related element 310, the environmental information is removed from the interior of the bounding boxes corresponding to the related element 300 and the related element 310, respectively. Figure 5B and Figure 5C In the example, the bounding box with the environment information removed can be used to pre-acquire the environment information. Figure 5B When pre-acquired environmental information at the time point t_1 shown in FIG. 1 , the information in which the environmental information is removed from the inside of the bounding box can be regarded as the pre-acquired environmental information in the area of ​​the relevant element 300. Similarly, when updating Figure 5C When pre-acquired environmental information is obtained at the time point of time t_2 shown, information in which environmental information is removed from the inside of the bounding box may be regarded as pre-acquired environmental information in the region of the relevant element 310 .

[0127] Therefore, in these examples, the environmental information setting unit (to be described below) is configured to sequentially update the pre-acquired environmental information to information acquired based on a time series in which no relevant elements are present in the detection range of the sensor, and the environmental information setting unit is configured to select the most recent pre-acquired environmental information obtained by updating as the environmental information.

[0128] 4. Processing flow in robotic devices

[0129] Figure 61 is a flowchart showing the processing flow in the robot device 1000 of this embodiment. Figure 6 The processing executed in the robot device 1000 will be described. Figure 6 The process shown is divided into process 1 and process 2. Process 1 is a process of removing information on related elements detected by the external sensor 200 in the bounding box and switching to pre-acquired pre-acquired environmental information. In addition, process 2 is a process on the movement of the robot device 1000 and the operation of related elements.

[0130] By combining the above two processes, even when the external sensor has detected the relevant element, the erroneous recognition of the relevant element as an obstacle is suppressed, and further, an action plan for avoiding contact with the obstacle based on the position and posture information of the relevant element with respect to the three-dimensional shape can be formulated. In the following, the specific processing flow is described.

[0131] The control device 500 of the robot device 1000 repeatedly executes the Figure 6 First, in step S10, a command of a task is issued by the user to the robot device 1000. Various tasks can be conceived as tasks, such as a task of moving the robot device 1000 toward a destination and a task of transferring an object held by the arm 100. If a command of a task is issued, the robot device 1000 starts operating according to the task.

[0132] In the next step S12, the environmental information detected by the external sensor 200 is acquired. In step S12, all environmental information including relevant elements is acquired. In addition, the pre-acquired environmental information that has been acquired and stored in a memory or the like and does not include relevant elements is accessed separately from the environmental information detected by the external sensor. The robot device 1000 saves the acquired environmental information and the position and posture information of the robot device 1000, and associates the acquired environmental information with the position and posture information of the robot device 1000. The position and posture information of the robot device 1000 includes information about the position and posture of the bounding box. The processing of step S12 is continuously performed for each control period, thereby updating the environmental information and pre-acquired environmental information.

[0133] Note that in a first iteration through this process, a first instance of "pre-acquired" environmental information may be acquired. Alternatively, the process may start with a first instance of "pre-acquired" environmental information that is pre-installed at the robot, or obtained from another robot in the same location, such as through a wireless data connection, or obtained from a cloud-based or server-based data source.

[0134] In the next step S14, it is evaluated whether there is environmental information corresponding to the relevant element in the bounding box; in the case where it is evaluated that there is environmental information corresponding to the relevant element in the bounding box, the process proceeds to step S16. In step S16, the information belonging to the relevant element is erased from the environmental information. That is, in the case where the external sensor 200 has detected the environmental information in the bounding box, the environmental information in the bounding box is equivalent to the information of the relevant element, so the environmental information in the bounding box is erased. Thereby, the relevant element in the bounding box is suppressed from being mistakenly identified as an obstacle. On the other hand, in the case where it is evaluated in step S14 that there is no environmental information in the bounding box, the process proceeds to step S20 to formulate an operation plan.

[0135] Note that, as described above, the process of step S18 may be performed without performing the process of step S 16. That is, the information including the information in the bounding box may be switched to the pre-acquisition environment information acquired in advance.

[0136] After step S16, the process proceeds to step S18. In step S18, environmental information of the area blocked by the relevant element, which cannot be measured because the relevant element covers or blocks the external sensor 200, is inferred. Specifically, in step S18, for the switching range 35 blocked by the relevant element, switching to the pre-acquired environmental information pre-acquired in step S12 in the previous control period is performed.

[0137] In the above manner, the processing of step S16 and step S18 is performed; thereby, environmental information excluding information of relevant elements can be obtained both about the inside of the bounding box and the switching range 35. Therefore, an operation plan of the robot device 1000 can be formulated based on the environmental information excluding information of relevant elements.

[0138] In the processing of the next step S20 and subsequent steps, based on the environmental information excluding the information of the relevant elements as obtained above, an operation plan of the robot device 1000 is formulated and the robot device 1000 is controlled. The operation plan includes a plan about the movement of the robot device 1000 and the operation of the relevant elements such as the arm 100. First, in step S20, it is evaluated whether the position of the obstacle obtained according to the environmental information excluding the information of the relevant elements is close to the position of the robot device 1000. Then, in the case where it is evaluated that the position of the obstacle obtained according to the environmental information is close to the position of the robot device 1000, the process proceeds to step S22, and the distance between the obstacle and the robot device 1000 is calculated. The origin of the robot device 1000, for example, the coordinates of the center of the robot 1000 (the position of the center of mass, etc.) can be used as the position of the robot device 1000. In addition, the position of the external sensor 200 can be used as the origin as the position of the robot device 1000. The distance between the obstacle and the robot device 1000 can be calculated by using the origin as a reference so as to also obtain environmental information.

[0139] In the evaluation of step S20, preferably, the information of each bounding box that has been acquired by the robot device 1000 is used to evaluate whether the distance between the obstacle and each bounding box is small. The reference of the bounding box side when measuring the distance may be the center of the bounding box or the surface, contour, etc. of the bounding box, and is not particularly limited. In the case where the distance between the obstacle and any bounding box is less than or equal to a predetermined threshold, the process proceeds to step S22, and the distance between the obstacle and the bounding box is calculated. Figure 3 When the point group shown represents environmental information, the distance between each point representing an obstacle and the bounding box is calculated.

[0140] In addition, in the case where some meanings are attached so that the point group representing the obstacle can be regarded as a whole, the distance between the obstacle and any bounding box can be evaluated using the center of the set given the meaning. For example, in the case where the size and shape of the obstacle are known in advance, the distance between the obstacle and the bounding box can be evaluated by using the center of the point group representing the obstacle and taking into account the size and shape of the obstacle. In addition, in order to suppress the calculation cost, the distance can be calculated after compressing the height direction of the point group into two dimensions. Note that the environmental information is not limited to the point group.

[0141] In the next step S24, an example of executing a control function based on environmental information is represented, in which a plan of the path along which the robot device 1000 moves or a plan of the operation of related elements of the robot device 1000, such as the arm 100, is made based on the distance between the obstacle and the robot device 1000 calculated in step S22. In the case where the distance between the obstacle and the robot device 1000 is calculated using the coordinates of the center of the robot device 1000, in the case where the obstacle is relatively close to the coordinates of the center, the path of the robot device 1000 is planned to be away from the obstacle. In addition, in the case where the distance between the obstacle and the coordinates of the center is relatively small, the operation of related elements, such as the arm 100, is planned to be away from the obstacle. In addition, in the case where the distance between the obstacle and the coordinates of the center is large, the movement path of the robot device 1000 and the operation of related elements are planned without special consideration of the obstacle.

[0142] Preferably, in step S24, the information of the distance between the obstacle and the bounding box is used to plan the motion path and the operation of the related elements. For example, in the case where the distance between the obstacle and the bounding box is relatively small, the path of the robot device 1000 is planned to be away from the obstacle. In addition, in the case where the distance between the obstacle and the bounding box is relatively small, the operation of the related elements such as the arm 100 is planned to be away from the obstacle. Thus, the operation plan can be formulated in consideration of the positions and states of the constituent parts of the robot device 1000 and the related elements such as the grasping object; therefore, a safer operation plan can be formulated.

[0143] In addition, in the case where the distance between the obstacle and the bounding box does not need to be considered for some reason, for example, in the case where the distance between the obstacle and the bounding box is large, the movement path of the robot device 1000 and the operation of related elements can be planned without using the information of the distance or the information of the bounding box. After step S24, the process proceeds to step S26 or step S28.

[0144] In step S26, the robot device 1000 performs travel or stop based on the plan designed in step S24 along the path along which the robot device 1000 moves. Specifically, in step S26, the movement or stop of the robot device 1000 is performed by a mechanism such as the following wheel. Stopping also includes a case where an emergency stop is performed due to some external factor or internal factor.

[0145] In addition, in step S28, operation or stop of the relevant element such as the arm 100 is performed based on the operation plan of the relevant element designed in step S24. Stopping also includes the case of emergency stop due to some external factor or internal factor.

[0146] 5. Cases where the relevant elements include grasped objects

[0147] As described above, the relevant elements widely include elements related to the robot device 1000 such as an object grasped by the arm 100, and constituent parts included in the robot device 1000 itself such as the arm 100. The arm 100 of the robot device 1000 contacts and grasps the object, and performs a transfer work, etc. Therefore, in the case where the object is known in advance, a bounding box for the object is prepared; in the case where the arm 100 contacts the object, a process similar to the above process is performed while considering the object as a part of the robot device 1000. Therefore, a process that also considers the object becomes possible.

[0148] The object itself may be detected, for example, by detecting the width of the jaws of a robot arm used to grasp the object and mapping the width to one of a set of objects having a corresponding width or to one of a set of bounding boxes having a dimension equal to the width. Alternatively, the sensor 200 may be associated with object recognition circuitry or software to detect the object to be picked up. In the absence of recognition, a pessimistic bounding box as large as the largest object that can be picked up by the robot may then be used.

[0149] Figure 7 is a schematic diagram showing an example in which a bounding box including a grasped object is obtained, and shows a state in which the robot device 1000 is viewed from the front. Figure 7 In the example shown, the arm 100 includes a link 110, a link 120, and a gripping unit 122. In addition, a bounding box 32 and a bounding box 34 are provided corresponding to the link 110 and the link 120, respectively. In addition, a bounding box 36 corresponding to the gripping unit 122 at the end of the arm 100 and a bounding box 38 corresponding to the gripping object 60 held by the gripping unit 122 are provided.

[0150] Each of the joints that join the link 110 and the link 120 and the joints that join the gripping unit 122 and the link 120 is provided with an encoder that detects the rotation angle of the joint. The sizes and shapes of the bounding boxes 32, 34, and 36 are grasped in advance on the robot device 1000 side, and the positions and postures of the bounding boxes 32, 34, and 36 are evaluated based on the detection values ​​of the encoders.

[0151] In addition, it is assumed that the grasped object 60 is grasped at a prescribed position of the grasping unit 104. Therefore, the position of the grasped object 60 relative to the grasping unit 104 is uniquely determined; therefore, the position and posture of the bounding box 38 of the grasped object 60 can also be acquired in advance on the robot device 1000 side.

[0152] Therefore, by removing the environmental information in the bounding box 38, it is possible to prevent the grasping object 60 from being recognized as an obstacle. In addition, the switching to the pre-acquired environmental information is performed for the switching range 35 that is blocked by the bounding box 38 when viewed from the external sensor 200, so that the robot device 1000 can be controlled based on the environmental information obtained by switching to the pre-acquired environmental information. Note that, for the link 110, the link 120, and the grasping unit 122, the environmental information in the bounding box 32, the bounding box 34, and the bounding box 36 are also deleted similarly.

[0153] Examples of the grasped object 60 include PET bottles, glass, a remote controller of a TV, etc. Three-dimensional data of bounding boxes of these grasped objects are acquired in advance on the robot device 1000 side, so that the grasped object 60 can be prevented from being recognized as an obstacle.

[0154] In addition, even if a bounding box is not prepared in advance, a bounding box can be automatically generated by extracting a portion having some meaning from the environmental information and associating the meaning with the portion of the environmental information as a whole. Note that Figure 7 The illustrated configurations of the arm 100, the link 110, the link 120, the gripping unit 122, etc. are merely examples, and the robot device 1000 is not limited to the configurations. For example, the robot device 1000 may include a greater number of links, and may include a shape and manner similar to that of the robot device 1000. Figure 7 The shapes and forms of the connecting rod and the gripping unit are different from those of the connecting rod and the gripping unit.

[0155] 6. Example of a situation where related components overlap

[0156] Figure 8 1 is a schematic diagram showing an example in which the two arms 130 and 140 overlap between the robot device 1000 and the wall 62, and shows a state in which the robot device 1000 is viewed from above. Figure 8 As shown, the arm 130 includes a link 132, a link 134, and a gripping unit 136. In addition, the arm 140 includes a link 142, a link 144, and a gripping unit 146. Figure 8 In the embodiment, a bounding box 72 corresponding to the connecting rod 132, a bounding box 74 corresponding to the connecting rod 134, and a bounding box 76 corresponding to the gripping unit 136 are provided. In addition, a bounding box 82 corresponding to the connecting rod 142, a bounding box 84 corresponding to the connecting rod 144, and a bounding box 86 corresponding to the gripping unit 146 are provided.

[0157] When the external sensor 200 acquires environmental information, first, the environmental information is removed from the inside of the bounding boxes 72, 74, and 76 of the arm 130 on the side close to the external sensor 200. This prevents the arm 130 from being recognized as an obstacle.

[0158] In addition, environmental information is removed from the inside of the bounding box 82, the bounding box 84, and the bounding box 86 on the side away from the external sensor 200. In this case, when viewed from the external sensor 200 side, the bounding box 82, the bounding box 84, and the bounding box 86 are blocked by the bounding box 72, the bounding box 74, and the bounding box 76; therefore, there is not much environmental information corresponding to the arm 140 in the bounding box 82, the bounding box 84, or the bounding box 86. However, within the range not blocked by the bounding box 72, the bounding box 74, or the bounding box 76, there is environmental information in the bounding box 82, the bounding box 84, and the bounding box 86; therefore, the environmental information of these bounding boxes is removed.

[0159] Thus, the environmental information corresponding to the arm 130 and the arm 140 is removed. Then, for the switching range 35 blocked by both the arm 130 and the arm 140, switching to the pre-acquired environmental information is performed. Figure 8 In the state shown, the environmental information corresponding to the arm 130 and the arm 140 is removed, and the robot device 1000 recognizes only the wall 62 included in the pre-acquired environmental information as an obstacle. Therefore, the robot device 1000 is in a state where it can see through the rear of the arm 130 and the arm 140, and can formulate an operation plan with only the wall 62 as an obstacle in mind.

[0160] As described above, even if there are related elements that cannot be detected by the external sensor 200 due to the overlap of related elements, each bounding box is generated based on the internal data of the robot device 1000, and the environmental information is deleted from each bounding box. In addition, the information acquired in advance is used as the information of the obstacle. Figure 7 As in the example above, environmental information in which the relevant elements are excluded can be obtained, and the robot device 1000 can be controlled based on the environmental information.

[0161] 7. Dynamic prediction and processing of environmental information

[0162] In the case of a mobile obstacle, when using the pre-acquired pre-acquired environmental information, there is a possibility that the mobile obstacle will have moved from the position at the last time of the pre-acquired. Therefore, the movement trajectory of the mobile obstacle is predicted based on the change in the position of the mobile obstacle in the pre-acquired environmental information, so that the movement plan and operation plan can be formulated taking into account the movement of the mobile obstacle.

[0163] 9A to 9D is a schematic diagram showing an example of a process of predicting a motion trajectory of a moving obstacle, and shows a state in which the robot device 1000 is viewed from above. 9A to 9D1 is a top view showing the positional relationship between the robot device 1000 and the moving obstacle 70 at time t_10, time t_11, time t_12, and time t_13, respectively, based on the time series. Fig. 9A and Fig. 9B As shown, the mobile obstacle 70 moves upward in the figure. That is, at time t_11 after time t_10, the mobile obstacle 70 moves upward more than the mobile obstacle 70 at time t_10.

[0164] like Fig. 9C As shown, it is assumed that the relevant element 320 covers or blocks a part of the detection range of the external sensor 200 at time t_12 after time t_11. In this case, the pre-acquired environmental information is used in the operation plan of the robot device 1000, and the path of the robot device 1000 is planned by using the information at both time t_10 and time t_11 to grasp the fact that the mobile obstacle 70 has moved and predict the subsequent movement trajectory of the mobile obstacle 70.

[0165] Specifically, in Fig. 9C At the time point of time t_12 shown, the latest pre-acquired environment information at time t_11 is basically used for the switching range blocked by the relevant element 320. On the other hand, in the case of using the pre-acquired environment information at time t_11, the movement of the mobile body 70 from time t_11 to time t_12 is not reflected in the pre-acquired environment information. Therefore, the movement of the mobile body 70 from time t_11 to time t_12 is predicted by using the environment information at both time t_10 and time t_11, and (by Fig. 9C The predicted position of the mobile body 70 at the time point of time t_12 (shown by the dotted line in FIG. 1 ) is reflected in the pre-acquired environmental information at time t_11.

[0166] Thus, if Fig. 9C As shown, the motion plan 78 of the robotic device 1000 may be designed based on the pre-acquired environmental information at time t_11 and the predicted position of the mobile body 70 at the time point of time t_12.

[0167] Fig.9D 10 shows a state in which the robot device 1000 moves at time t_13 based on the motion plan 78 for moving the moving body 70 made at time t_12. 9A to 9D An example is described in which a path that does not come into contact with the moving obstacle 70 is generated based on the prediction of the motion trajectory for the moving obstacle 70 , but a judgment to stop the robot device 1000 may be made at a point in time when the presence of the moving obstacle 70 is recognized.

[0168] As described above, when switching to pre-acquired environmental information is performed, future situations are predicted based on pre-accumulated past transitions; thus, an operation plan can be formulated based on the pre-environmental information in which the future is predicted and the status of the robot device 1000 at the current time.

[0169] 8. Other examples where pre-fetched environment information is used

[0170] 8.1. Action plan during transfer work by cargo handling vehicles such as forklifts

[0171] As described above, in the present embodiment, even when the external sensor 200 is blocked by a relevant element, the condition of the external world can be inferred by using the pre-acquired environmental information, and the condition of the external world can be used for operation planning. Hereinafter, a specific example is described in the case of application to a cargo handling vehicle such as a forklift.

[0172] In the case of transferring cargo using a cargo handling vehicle such as a forklift, there is a possibility that the following situation will occur: information of the cargo and related elements for lifting and lowering the cargo is included in the environmental information and the environmental information is not fully acquired. Therefore, it is assumed that in the case of lowering the cargo, the condition of the table or floor as the destination for lowering the cargo cannot be fully grasped, which will lead to the occurrence of unexpected accidents, etc.

[0173] Fig.10 is a schematic diagram illustrating an example of unloading by a forklift 1010 . The forklift 1010 is an example of the robot device 1000 . Fig.10 The following situation is shown: if the forklift 1010 reaches the front of the platform 82 , the forklift 1010 moves in the direction of arrow AA2 from the left side, and the forklift 1010 puts down the cargo 80 being transferred on the platform 82 .

[0174] During the transfer, the cargo 80 transferred by the forklift 1010 is lifted by the arm 400 at the position indicated by the alternate long and two short dashed lines. In the state where the cargo 80 is lifted in this way, the related elements of the forklift 1010 such as the cargo 80, the arm 400, and the motion mechanism 402 that moves the arm 400 up and down are included in the detection range of the external sensor 200. Then, the information of the surrounding environment of the platform 82 is blocked by these related elements.

[0175] Therefore, if you try to Fig.10 In the state shown, if the cargo 80 is placed on the platform 82 without using pre-acquired environmental information, the environmental information obtained from the external sensor 200 does not include information near the platform 82; therefore, it is impossible to accurately grasp the horizontal positional relationship between the forklift 1010 and the platform 82, the vertical movement amount used to put down the cargo 80, etc.

[0176] Therefore, before the forklift 1010 approaches the platform 82, the environmental information of the platform 82 is acquired and saved as the pre-acquired environmental information. In the state before the forklift 1010 approaches the platform 82, the surrounding environment of the platform 82 is included in the detection range of the external sensor. Therefore, the environmental information acquired before the forklift 1010 approaches the platform 82 is used as the pre-acquired environmental information.

[0177] Thus, when the forklift 1010 moves and approaches the platform 82, the horizontal positional relationship between the forklift 1010 and the platform 82 can be determined, so that the forklift 1010 can be stopped. Fig.10 In addition, the vertical movement amount for putting down the cargo 80 can be grasped based on the pre-acquired environmental information, so the cargo 80 can be accurately placed on the platform 82 by putting down the cargo 80 based on the grasped vertical movement amount. As described above, the environmental state near the destination where the cargo 80 is to be put down can be inferred by using the pre-acquired environmental information acquired before the forklift 1010 puts down the cargo 80, and a stable conveying work can be performed.

[0178] As described above, even in the case where the platform 82 and its surroundings are inevitably obscured during a mission, when the robot device 1000 is away from the platform 82, the conditions of the platform 82 and its surroundings are acquired as pre-acquired environmental information, so that the desired operation can be performed when the robot device 1000 approaches the platform 82.

[0179] 8.2. Action planning of an autonomous moving body equipped with a robotic arm

[0180] The robot device 1000 including the arm 100 can be used not only for transmission purposes but also for the purpose of communicating with a person or a machine through gestures such as waving a hand, etc. In the case where the robot device 1000 approaches a person or an object while waving a hand (arm 100), a situation may occur where the external sensor 200 constantly continues to detect the arm 100; however, according to the present embodiment, the robot device 1000 can safely approach the person or the object while inferring environmental information blocked by the arm 100.

[0181] The application purposes of the embodiments of the present disclosure are wide-ranging and do not depend on the movement mode of the autonomous moving body or the shape or function of the related elements. Therefore, the movement mode of the robot device 1000 can be not only a rotating mechanism such as wheels or tracks, but also a legged mechanism or a winged mechanism including rotor blades, such as a drone. In addition, a configuration in which the related elements such as the arm 100 work without the robot device 1000 moving is possible.

[0182] 9. Configuration example of control device

[0183] Fig.11 1 is a block diagram showing the configuration of a control device 500 for controlling a robot device 1000, the robot device also including (as drawn) a sensor 200 that provides environmental information to the control device 500 and one or more motion actuators 582 that are controlled by the control device 500 and configured to cause the robot device to move based on an operation plan. The control device 500 is basically provided inside the robot device 1000 (or at least as part of the function of the robot device 1000), but may also be provided outside the robot device 1000, and may be, for example, a device provided on the cloud and performing communication with the robot device 1000. Fig.11 As shown, the control device 500 includes a bounding box information storage unit 510, a position and posture information acquisition unit 520, an evaluation unit 530, an environment information setting unit 540 (for example, configured to sequentially update the pre-acquired environment information based on a time series), an environment information storage unit 560, an environment information deletion unit 550, an operation planning unit 570, and an operation control unit 580. Note that Fig.11 Each of the constituent elements shown includes a circuit (hardware) or a central arithmetic processing device such as a CPU and a program (software) for causing the device to operate.

[0184] The bounding box information storage unit 510 stores information on bounding boxes of related elements, such as information on bounding boxes of constituent parts of the robot device 1000 such as the arm 100 and bounding boxes of the grasped object 60. The position and posture information acquisition unit 520 acquires the angles of joints of the arm 100, etc., for example, from encoders provided in the joints, etc., thereby acquiring information on the position and posture of each bounding box.

[0185] The evaluation unit 530 evaluates whether the relevant element is included in the environmental information acquired from the external sensor 200. More specifically, the evaluation unit 530 acquires three-dimensional information about the shape and size of the bounding box from the bounding box information storage unit 510, and acquires information about the position and attitude of the bounding box from the position and attitude information acquisition unit 520. Then, the evaluation unit 530 evaluates whether the relevant element is included in the environmental information according to whether the environmental information is included in the bounding box. In the case where the environmental information is included in the bounding box, the evaluation unit 530 evaluates that the relevant element is included in the environmental information; in the case where the environmental information is not included in the bounding box, the evaluation unit 530 evaluates that the relevant element is not included in the environmental information.

[0186] In the case where the evaluation unit 530 has evaluated that the environmental information acquired from the external sensor 200 includes the relevant element, the environmental information setting unit 540 switches the environmental information to the pre-acquired environmental information in which the relevant element is not included. The environmental information storage unit 560 is a constituent element that stores the pre-acquired environmental information and updates and stores the pre-acquired environmental information (or at least the environmental information other than the part that is evaluated to include the environmental information) for each control period. The environmental information setting unit 540 performs switching to the pre-acquired environmental information stored in the environmental information storage unit 560. In addition, in the case where the pre-acquired environmental information includes or the pre-acquired environmental information indicates a moving obstacle, the environmental information setting unit 540 switches the environmental information to the pre-acquired environmental information in which the relevant element is not included. 9A to 9D The technology shown is used to predict the movement of mobile obstacles and reflect the predicted movement in the pre-acquired environmental information.

[0187] In the case where the environmental information is included in the bounding box, the environmental information deleting unit 550 performs a process of deleting the environmental information in the bounding box. The environmental information setting unit 540 uses the information of the bounding box in which the environmental information is deleted as the environmental information.

[0188] The operation planning unit 570 formulates an operation plan (as an example of executing a control function) of the robot device 1000 according to or based on the environmental information (e.g., formed by the environmental information newly detected by the external sensor 200 and the pre-acquired environmental information). The operation planning unit 570 calculates the distance to the obstacle based on the environmental information detected by the external sensor 200 and the pre-acquired environmental information, and formulates or generates an operation plan based on the distance to the obstacle. The operation control unit 580 controls the operation of the robot device 1000 based on the operation plan of the operation planning unit 570. The operation of the robot device 1000 also includes the operation of a motion mechanism such as wheels and legs that moves the robot device 1000 itself (e.g., by controlling one or more motion actuators 582 such as one or more motors by the operation control unit 580 to move the device robot), so that the operation control unit is configured to control the operation of one or more motion actuators 582 based on the operation plan, which may include an operation to move the entire robot and / or an operation to move related elements such as the arm 100. The operation planning unit 570 formulates an operation plan of the robot device 1000 using, for example, a planner, machine learning, spline interpolation, or a Bezier curve.

[0189] The operation planning unit may also be configured to calculate a distance between shape approximation information in which the position and shape of the relevant element are approximated and the obstacle based on the environmental information acquired from the sensor and the pre-acquired environmental information, and generate the operation plan based on the distance.

[0190] Note that one or more motion actuators may be motion actuators that control overall motion of the robot, such as translation or rotation, and / or motion actuators that control motion of a portion or component of the robot, such as arm 100, relative to the rest of the robotic apparatus.

[0191] One or more actuators 582 may, for example, drive wheels 584 and / or related elements 100 .

[0192] therefore, Fig.11 An example of a control device 500 is provided that includes the following units: an assessment unit 530 configured to assess whether environmental information newly acquired from a sensor includes relevant elements; an environmental information setting unit 540 configured to, in a case where the assessment unit has assessed that a portion of the environmental information includes relevant elements, select pre-acquired environmental information in which the relevant elements are assessed as not being included to replace at least the portion of the environmental information newly acquired from the sensor; and a control unit 570, a control unit 580 configured to perform a control function (e.g., control of one or more motion actuators 582 and / or generation of an operation plan that provides such control) based on the environmental information.

[0193] Fig.11 For example, the configuration is as follows Figure 1 When implemented in the physical form shown, an example of a robotic device 1000 is provided that includes a control device 500 and a sensor 200 configured to provide environmental information; wherein the sensor is an object recognition sensor configured to recognize an object.

[0194] Fig.12 It is a schematic flow chart showing a control method including the following operations: (at step 1200) evaluating whether the environmental information newly acquired from the sensor includes relevant elements; in the case where the evaluation step evaluates that a part of the environmental information includes relevant elements, (at step 1210) selecting (or switching to) pre-acquired environmental information in which the relevant elements are evaluated as not included to replace at least the part of the environmental information newly acquired from the environmental sensor; and (at step 1220) executing a control function based on the environmental information.

[0195] As mentioned, Fig.12 The process or method may be performed by a suitable computer or data processing device, which performs the method by executing appropriate computer software, in which case such software and / or a machine-readable non-transitory storage medium storing such software will be regarded as an embodiment of the present disclosure.

[0196] Fig.131 is a schematic diagram showing such a computer or data processing device 1300, which includes: a central processing unit (CPU) 1310; a read only memory (ROM) 1320, which is used as an example of a machine-readable non-transitory storage medium for storing such software executed by the CPU 1310; a random access memory 1330; and an input / output (I / O) interface, for example, for interacting with the sensor 200 and / or the actuator 582. The CPU 1310, ROM 1320, random access memory 1330, and input / output (I / O) interface are all interconnected by an exemplary bus structure 1350. For example, the device 1300 can implement (under software control) Fig.11 functions other than the sensor 200 and the actuator 582 and associated mechanical devices.

[0197] As described above, according to the present embodiment, a bounding box corresponding to the relevant element is generated, and the environmental information in the bounding box is removed. In addition, the environmental information of the area of ​​the relevant element or the area covered by the relevant element is switched to the pre-acquired environmental information. Thus, the relevant element is suppressed from being identified as an obstacle based on the environmental information, and the movement and operation of the robot device 1000 can be optimally controlled.

[0198] Those skilled in the art should understand that various modifications, combinations, sub-combinations and alterations may be made according to design requirements and other factors as long as these modifications, combinations, sub-combinations and alterations are within the scope of the appended claims or the equivalents thereof.

[0199] In addition, the effects described in this specification are merely illustrative or exemplary effects, and are not restrictive. That is, together with or in place of the above effects, the technology according to the present disclosure can achieve other effects that are clear to those skilled in the art based on the description of this specification.

[0200] Additionally, the present technology may also be configured as set forth below in the following numbered paragraphs:

[0201] (1) A control device comprising:

[0202] an evaluation unit configured to evaluate whether the environmental information acquired from the sensor includes a relevant element; and

[0203] The environmental information setting unit is configured to switch the environmental information to pre-acquired environmental information in which the relevant element is not included, if the evaluation unit has evaluated that the relevant element is included in the environmental information.

[0204] (2) The control device according to (1),

[0205] The evaluation unit evaluates whether the relevant element is included in the environmental information based on the shape approximation information in which the position and shape of the relevant element are approximated and the environmental information.

[0206] (3) The control device according to (2),

[0207] Wherein, in a case where the environmental information is included in the space indicated by the shape approximation information, the evaluation unit evaluates that the relevant element is included in the environmental information.

[0208] (4) The control device according to (2) or (3),

[0209] The shape approximation information is information including a simple shape of the relevant element.

[0210] (5) The control device according to any one of (2) to (4),

[0211] The shape approximation information is information obtained by enlarging the outline of the relevant element.

[0212] (6) The control device according to any one of (1) to (5),

[0213] The environmental information setting unit switches the environmental information to the latest pre-acquired environmental information in advance when the relevant element does not exist within the detection range of the sensor, and sets new environmental information in which the relevant element does not exist within the switching target range.

[0214] (7) The control device according to any one of (1) to (5),

[0215] wherein, in a state where the related element does not exist within the detection range of the sensor, the pre-acquired environmental information is sequentially updated to information acquired based on a time series; and

[0216] The environment information setting unit switches the environment information to the latest pre-acquired environment information obtained by the updating.

[0217] (8) The control device according to any one of (1) to (7),

[0218] The environmental information setting unit switches the environmental information of the area where the relevant components exist to the pre-acquired environmental information.

[0219] (9) The control device according to (8),

[0220] The environmental information setting unit switches the environmental information of the area of ​​the shape approximation information in which the position and shape of the relevant element are approximated to the pre-acquired environmental information.

[0221] (10) The control device according to any one of (1) to (9),

[0222] The environmental information setting unit switches the environmental information of the area covered by the relevant element to the pre-acquired environmental information.

[0223] (11) The control device according to any one of (1) to (9),

[0224] The environmental information setting unit switches the environmental information of the area where the relevant element exists and the area blocked by the relevant element to the pre-acquired environmental information.

[0225] (12) The control device according to any one of (1) to (11),

[0226] The environmental information setting unit switches the environmental information of the entire area to the pre-acquired environmental information.

[0227] (13) The control device according to (11), comprising:

[0228] The environment information deleting unit is configured to delete the environment information of the area where the related element exists.

[0229] (14) The control device according to (13),

[0230] The environmental information setting unit deletes the environmental information in the space of the shape approximation information in which the position and shape of the relevant element are approximated.

[0231] (15) The control device according to (13),

[0232] Here, the environment information setting unit performs switching to the pre-acquired environment information obtained by deleting past environment information.

[0233] (16) The control device according to any one of (1) to (15),

[0234] Therein, the sensor is an object recognition sensor configured to recognize an object.

[0235] (17) The control device according to (16),

[0236] Therein, the sensor is a distance measuring sensor configured to measure the distance to the object.

[0237] (18) The control device according to any one of (1) to (17), comprising:

[0238] An operation planning unit is configured to formulate an operation plan based on the environmental information acquired from the sensor and the pre-acquired environmental information.

[0239] (19) The control device according to (18),

[0240] The operation planning unit calculates the distance to the obstacle based on the environmental information acquired from the sensor and the pre-acquired environmental information, and formulates the operation plan based on the distance to the obstacle.

[0241] (20) The control device according to (18),

[0242] The operation planning unit calculates the distance between shape approximation information in which the position and shape of the relevant element are approximated and the obstacle based on the environmental information acquired from the sensor and the pre-acquired environmental information, and formulates the operation plan based on the distance.

[0243] (21) The control device according to (18), comprising:

[0244] A control unit is configured to control the operation of the robot based on the operation plan.

[0245] (22) The control device according to any one of (1) to (21),

[0246] Wherein, in a case where a moving obstacle is included in the pre-acquired environmental information, the environmental information setting unit predicts a movement of the moving obstacle and causes the predicted movement to be reflected in the pre-acquired environmental information.

[0247] (23) The control device according to any one of (1) to (22),

[0248] The control device controls a robot, and the related elements are elements related to the robot.

[0249] (24) The control device according to (23),

[0250] The related elements are components included in the robot or objects acted upon by the robot.

[0251] (25) A control method comprising:

[0252] Assess whether the environmental information obtained from sensors includes relevant components; and

[0253] In the case where it is assessed that the relevant element is included in the environmental information, the environmental information is switched to pre-acquired environmental information in which the relevant element is not included.

[0254] (26) A program for causing a computer to:

[0255] means configured to assess whether the environmental information acquired from the sensor includes a relevant element; and

[0256] means configured to switch the environmental information to pre-acquired environmental information in which the relevant element is not included if it is assessed that the relevant element is included in the environmental information.

[0257] Additional example configurations are defined by the following numbered items:

[0258] 1. A control device, comprising:

[0259] an evaluation unit configured to evaluate whether the environmental information newly acquired from the sensor indicates a related element;

[0260] an environmental information setting unit configured to: in a case where the evaluation unit has evaluated that a part of the environmental information represents the relevant element, select pre-acquired environmental information in which the relevant element is evaluated as not to be included to replace at least the part of the environmental information newly acquired from the sensor; and

[0261] The control unit is configured to execute a control function according to the environmental information.

[0262] 2. A control device according to item 1,

[0263] The evaluation unit is configured to evaluate whether the environmental information represents the relevant element based on the environmental information and shape approximation information in which the position and shape of the relevant element are approximated.

[0264] 3. A control device according to item 2,

[0265] wherein the assessment unit is responsive to information defining a current position of the associated element relative to the sensor; and

[0266] Therein, in case the environmental information is represented by a space defined by the shape approximation information at a position dependent on information defining a current position of the relevant element relative to the sensor, the assessment unit is configured to assess that the environmental information represents the relevant element.

[0267] 4. A control device according to item 2,

[0268] Therein, the assessment unit is configured to detect in the environmental information the presence of a shape approximation representing a simple shape including the relevant element.

[0269] 5. A control device according to item 2,

[0270] The shape approximation information represents the physical shape of the outline of the relevant component surrounded by the margin area.

[0271] 6. A control device according to item 1,

[0272] The environmental information setting unit is configured to select the following most recent pre-acquired environmental information to replace at least the part of the environmental information newly acquired from the sensor: for the most recent pre-acquired environmental information, the evaluation unit evaluates that the relevant element is not represented by the part of the environmental information.

[0273] 7. A control device according to item 1,

[0274] wherein the environment information setting unit is configured to sequentially update the pre-acquired environment information to information acquired based on a time series, for which the relevant element does not exist within the detection range of the sensor, and

[0275] The environmental information setting unit is configured to select the latest pre-acquisition environmental information obtained by the updating as the environmental information.

[0276] 8. A control device according to item 1,

[0277] The part of the environmental information is a region of shape approximation information in which the position and shape of the relevant element are approximated.

[0278] 9. A control device according to item 1,

[0279] The part of the environmental information is an area blocked by the related element.

[0280] 10. A control device according to item 1,

[0281] The part of the environmental information is an area in which the relevant element is evaluated to be included and an area blocked by the relevant element.

[0282] 11. A control device according to item 1,

[0283] The part of the environmental information refers to the entire area for which the environmental information has been acquired by the sensor.

[0284] 12. A control device according to item 10, comprising:

[0285] The environment information deleting unit is configured to delete at least the part of the environment information newly acquired.

[0286] 13. A control device according to item 12,

[0287] Wherein, the environment information setting unit is configured to select the pre-acquired environment information to replace the deleted environment information.

[0288] 14. A robotic device comprising:

[0289] A control device according to clause 1; and

[0290] A sensor configured to provide environmental information;

[0291] Therein, the sensor is an object recognition sensor configured to recognize an object.

[0292] 15. A robotic device according to item 14,

[0293] Therein, the sensor is a distance measuring sensor configured to measure the distance to the object.

[0294] 16. The robotic device according to item 14, comprising:

[0295] An operation planning unit is configured to generate an operation plan for the robot device as the control function according to the environmental information.

[0296] 17. A robotic device according to item 16,

[0297] The operation plan unit is configured to calculate a distance to an obstacle based on the environmental information acquired from the sensor and the pre-acquired environmental information, and to generate the operation plan based on the distance to the obstacle.

[0298] 18. A robotic device according to item 16,

[0299] wherein the assessment unit is responsive to information defining a current position of the associated element relative to the sensor; and

[0300] Wherein, the operation planning unit is configured to calculate the distance between the shape approximation information in which the position and shape of the relevant element are approximated and the obstacle based on the environmental information acquired from the sensor and the pre-acquired environmental information, and is configured to generate the operation plan based on the distance.

[0301] 19. The robotic device according to item 16, comprising:

[0302] one or more motion actuators to move the robotic device;

[0303] The control unit is configured to control operation of the one or more motion actuators to move the robotic device based on the operation plan.

[0304] 20. The robotic device according to item 14,

[0305] Wherein, in a case where the pre-acquired environmental information indicates a moving obstacle, the environmental information setting unit is configured to predict a movement of the moving obstacle and cause the predicted movement to be reflected in the pre-acquired environmental information.

[0306] 21. A robotic device according to item 14,

[0307] in:

[0308] The related element is an element related to the robotic device, capable of moving relative to the robotic device according to a current element position relative to the robotic device, and controlled by the robotic device;

[0309] The sensor has a current sensor position relative to the robotic device; and

[0310] The evaluation unit is configured to detect information defining a current position of the associated element relative to the sensor in response to the current element position and the current sensor position.

[0311] 22. The robotic device according to item 21,

[0312] Wherein, the related elements are components of the robot device.

[0313] 23. A control method, comprising:

[0314] Assess whether the newly acquired environmental information from the sensor represents a relevant component;

[0315] In the case where the evaluating step evaluates that a portion of the environmental information represents the relevant element, selecting pre-acquired environmental information in which the relevant element is evaluated not to be included to replace at least the portion of the environmental information newly acquired from the environmental sensor; and

[0316] A control function is performed according to the environmental information.

[0317] 24. Computer software which, when executed by a computer, causes the computer to perform the method of clause 23.

[0318] 25. A non-transitory machine-readable medium storing computer software according to item 24.

[0319] Reference numerals list

[0320] 500 Control Equipment

[0321] 530 Assessment Units

[0322] 540 Environmental information setting unit

[0323] 570 Operational Planning Unit

[0324] 580 Operation Control Unit

Claims

1. A control device, comprising: an evaluation unit configured to evaluate whether the environmental information newly acquired from the sensor indicates a related element; an environmental information setting unit configured to: in a case where the assessment unit has assessed that a portion of the environmental information represents the relevant element, select pre-acquired environmental information in which the relevant element is assessed as not being included to replace at least the portion of the environmental information newly acquired from the sensor; as well as a control unit configured to perform a control function according to the environmental information; The environmental information setting unit is configured to select the following most recent pre-acquired environmental information to replace at least the part of the environmental information newly acquired from the sensor: for the most recent pre-acquired environmental information, the evaluation unit evaluates that the relevant element is not represented by the part of the environmental information.

2. The control device according to claim 1, in, The evaluation unit is configured to evaluate whether the environmental information represents the relevant element based on the environmental information and shape approximation information in which the position and shape of the relevant element are approximated.

3. The control device according to claim 2, in, The assessment unit is responsive to information defining a current position of the associated element relative to the sensor; as well as Therein, in case the environmental information is represented by a space defined by the shape approximation information at a position dependent on information defining a current position of the relevant element relative to the sensor, the assessment unit is configured to assess that the environmental information represents the relevant element.

4. The control device according to claim 2, in, The assessment unit is configured to detect in the environmental information the presence of a shape approximation representing a simple shape including the relevant element.

5. The control device according to claim 2, in, The shape approximation information represents a physical shape of an outline of the associated component surrounded by a margin area.

6. The control device according to claim 1, in, The environmental information setting unit is configured to sequentially update the pre-acquired environmental information to information acquired based on a time series, for which the relevant element does not exist within the detection range of the sensor, and The environmental information setting unit is configured to select the latest pre-acquisition environmental information obtained by the updating as the environmental information.

7. The control device according to claim 1, in, The part of the environmental information is a region of shape approximation information in which the position and shape of the relevant element are approximated.

8. The control device according to claim 1, in, The portion of the environmental information is an area blocked by the related element.

9. The control device according to claim 1, in, The part of the environmental information is an area in which the relevant element is evaluated to be included and an area hidden by the relevant element.

10. The control device according to claim 1, in, The portion of the environmental information is the entire area for which environmental information has been acquired by the sensor.

11. The control device according to claim 9, comprising: The environment information deleting unit is configured to delete at least the part of the environment information newly acquired.

12. The control device according to claim 11, in, The environment information setting unit is configured to select the pre-acquired environment information to replace the deleted environment information.

13. A robotic device comprising: The control device according to claim 1; as well as A sensor configured to provide environmental information; Therein, the sensor is an object recognition sensor configured to recognize an object.

14. The robotic device according to claim 13, in, The sensor is a distance measuring sensor configured to measure a distance to the object.

15. The robotic device according to claim 13, comprising: An operation planning unit is configured to generate an operation plan for the robot device as the control function according to the environmental information.

16. The robotic device according to claim 15, in, The operation planning unit is configured to calculate a distance to an obstacle based on the environmental information acquired from the sensor and the pre-acquired environmental information, and to generate the operation plan based on the distance to the obstacle.

17. The robotic device according to claim 15, in, The assessment unit is responsive to information defining a current position of the associated element relative to the sensor; as well as Wherein, the operation planning unit is configured to calculate the distance between the shape approximation information in which the position and shape of the relevant element are approximated and the obstacle based on the environmental information acquired from the sensor and the pre-acquired environmental information, and is configured to generate the operation plan based on the distance.

18. The robotic device according to claim 15, comprising: one or more motion actuators to move the robotic device; The control unit is configured to control operation of the one or more motion actuators to move the robotic device based on the operation plan.

19. The robotic device according to claim 13, in, In a case where the pre-acquisition environmental information indicates a moving obstacle, the environmental information setting unit is configured to predict a movement of the moving obstacle and cause the predicted movement to be reflected in the pre-acquisition environmental information.

20. The robotic device according to claim 13, in: The related element is an element related to the robotic device, capable of moving relative to the robotic device according to a current element position relative to the robotic device, and controlled by the robotic device; The sensor has a current sensor position relative to the robotic device; as well as The evaluation unit is configured to detect information defining a current position of the associated element relative to the sensor in response to the current element position and the current sensor position.

21. The robotic device according to claim 20, in, The related elements are constituent parts of the robot device.

22. A control method, comprising: Assess whether the newly acquired environmental information from the sensor represents a relevant component; In the case that the evaluating step evaluates that a portion of the environmental information represents the relevant element, selecting the pre-acquired environmental information in which the relevant element is evaluated as not being included to replace at least the portion of the environmental information newly acquired from the sensor, wherein the following most recent pre-acquired environmental information is selected to replace at least the portion of the environmental information newly acquired from the sensor: for the most recent pre-acquired environmental information, it is evaluated that the relevant element is not represented by the portion of the environmental information; as well as A control function is performed according to the environmental information.

23. A computer program product which, when executed by a computer, causes the computer to perform the method of claim 22.

24. A non-transitory machine-readable medium storing the computer program product according to claim 23.

Images