Robot control apparatus, robot control method, and robot system

Abstract

To improve recognition accuracy of an object and environment, and to perform flexible and stable operation by a robot according to a situation.SOLUTION: A robot control apparatus includes: a recognition unit for recognizing a measurement object based on distance information of a measurement target measured by a distance measuring sensor provided in a robot, and tactile information of the measurement object measured by a tactile sensor provided in the robot; and a control unit for controlling operation of the robot based on the information of the measurement object recognized by the recognition unit.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present disclosure relates to a robot control device, a robot control method, and a robot system. [Background technology]

[0002] There are methods for controlling the operation of a robot by measuring the position and posture of the robot or a measurement target of the robot (such as a grasped object) using various sensors (see Patent Documents 1 to 3). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2021 / 033509 [Patent Document 2] Japanese Patent Application Publication No. 11-28692 [Patent Document 3] International Publication No. 2022 / 030242 Summary of the Invention [Problem to be solved by the invention]

[0004] In order to perform flexible and stable operations by a robot, it is desirable to improve the accuracy of recognizing objects, environments, and other measurement targets.

[0005] It is desirable to provide a robot control device, a robot control method, and a robot system that can improve the accuracy of object and environment recognition and allow the robot to perform flexible and stable operations according to the situation. [Means for solving the problem]

[0006] A robot control device according to one embodiment of the present disclosure includes a recognition unit that recognizes the object to be measured based on distance information of the object to be measured measured by a distance sensor provided on the robot and tactile information of the object to be measured measured by a tactile sensor provided on the robot, and a control unit that controls the operation of the robot based on the information of the object to be measured recognized by the recognition unit.

[0007] A robot control method according to one embodiment of the present disclosure includes recognizing the object to be measured based on distance information of the object to be measured measured by a distance sensor provided on the robot and tactile information of the object to be measured measured by a tactile sensor provided on the robot, and controlling the operation of the robot based on the information of the recognized object to be measured.

[0008] A robot system according to one embodiment of the present disclosure includes a robot equipped with a ranging sensor and a tactile sensor, and a robot control device that controls the robot. The robot control device includes a recognition unit that recognizes the object to be measured based on distance information of the object to be measured measured by the ranging sensor provided on the robot and tactile information of the object to be measured measured by the tactile sensor provided on the robot, and a control unit that controls the operation of the robot based on the information of the object to be measured recognized by the recognition unit.

[0009] In a robot control device, a robot control method, or a robot system according to one embodiment of the present disclosure, the object to be measured is recognized based on distance information of the object to be measured measured by a distance sensor provided on the robot and tactile information of the object to be measured measured by a tactile sensor provided on the robot, and the operation of the robot is controlled based on the information of the recognized object to be measured. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a configuration diagram showing an overview of a robot system according to a comparative example. [Diagram 2] FIG. 2 is a configuration diagram showing an overview of a robot system according to a comparative example. [Diagram 3]FIG. 3 is a configuration diagram illustrating an example of a hardware configuration of a robot system according to an embodiment of the present disclosure. [Figure 4] FIG. 4 is a diagram illustrating an example of a hardware configuration of a robot system according to an embodiment. [Diagram 5] Fig. 5 is a diagram illustrating an example of a hardware configuration of a robot system according to an embodiment. [Figure 6] FIG. 6 is a diagram illustrating an example of a hardware configuration of a robot system according to an embodiment. [Figure 7] FIG. 7 is a diagram illustrating an example of a hardware configuration of a robot system according to an embodiment. [Figure 8] FIG. 8 is a diagram illustrating an example of a hardware configuration of a robot system according to an embodiment. As shown in FIG. [Figure 9] FIG. 9 is a configuration diagram illustrating an example of a sensor arrangement in a robot system according to an embodiment. [Figure 10] FIG. 10 is a configuration diagram that illustrates an example of a sensor arrangement in a robot system according to an embodiment. [Figure 11] FIG. 11 is a configuration diagram illustrating an example of a sensor arrangement in a robot system according to an embodiment. [Figure 12] FIG. 12 is a configuration diagram illustrating an example of a sensor arrangement in a robot system according to an embodiment. [Figure 13] FIG. 13 is a configuration diagram illustrating an example of a sensor arrangement in a robot system according to an embodiment. [Figure 14] FIG. 14 is a configuration diagram illustrating an example of a sensor arrangement in a robot system according to an embodiment. [Figure 15] FIG. 15 is an explanatory diagram showing an overview of a method for measuring an object by a robot system according to one embodiment. [Figure 16]FIG. 16 is an explanatory diagram illustrating an example of a method for estimating an object shape by a robot system according to an embodiment. [Figure 17] FIG. 17 is an explanatory diagram illustrating an example of distance measurement error of a distance measuring sensor in a robot system according to one embodiment. [Figure 18] FIG. 18 is an explanatory diagram that roughly shows an example of sensor noise distribution of distance measurement sensor measurement points and tactile sensor measurement points in a robot system according to one embodiment. [Figure 19] FIG. 19 is an explanatory diagram that roughly shows an example of sensor noise distribution of distance measurement sensor measurement points and tactile sensor measurement points in a robot system according to one embodiment. [Figure 20] FIG. 20 is an explanatory diagram illustrating an example of a method for estimating an object shape by a robot system according to an embodiment. [Figure 21] FIG. 21 is an explanatory diagram illustrating an example of a method for estimating an object shape by a robot system according to one embodiment. [Figure 22] FIG. 22 is an explanatory diagram illustrating an example of a method for estimating an object shape by a robot system according to one embodiment. [Diagram 23] FIG. 23 is an explanatory diagram illustrating an example of a method for estimating an object shape by a robot system according to one embodiment. [Figure 24] FIG. 24 is an explanatory diagram that illustrates a first example of active sensing by a robot system according to an embodiment. [Diagram 25] FIG. 25 is an explanatory diagram that illustrates a first example of active sensing by a robot system according to one embodiment. [Figure 26] FIG. 26 is an explanatory diagram that illustrates a second example of active sensing by a robot system according to an embodiment. [Figure 27] FIG. 27 is an explanatory diagram that illustrates a third example of active sensing by a robot system according to an embodiment. [Figure 28]FIG. 28 is an explanatory diagram that illustrates a third example of active sensing by a robot system according to an embodiment. [Figure 29] FIG. 29 is an explanatory diagram that illustrates a first method for narrowing down the candidate gripping points in consideration of gripping stability by the robot system according to one embodiment. [Diagram 30] FIG. 30 is an explanatory diagram that illustrates a second method for narrowing down the candidate gripping points in consideration of gripping stability by the robot system according to one embodiment. [Diagram 31] FIG. 31 is an explanatory diagram that illustrates a fourth example of active sensing by a robot system according to an embodiment. [Diagram 32] FIG. 32 is an explanatory diagram that illustrates a fourth example of active sensing by a robot system according to an embodiment. [Diagram 33] FIG. 33 is an explanatory diagram showing an example of an operation in which a hand is moved to approach an object to be measured in a robot system according to one embodiment. [Diagram 34] FIG. 34 is an explanatory diagram showing an example of a method for determining a gripping form using an object environment recognition result including uncertainty information in a robot system according to an embodiment. [Diagram 35] FIG. 35 is an explanatory diagram showing a specific example of the procedure 2 (rule-based) shown in FIG. [Diagram 36] FIG. 36 is an explanatory diagram showing a specific example of the procedure 2 (learning-based) shown in FIG. [Figure 37] FIG. 37 is an explanatory diagram showing a specific example of the procedure 3 shown in FIG. [Figure 38] FIG. 38 is an explanatory diagram showing a modified example of a grasping form determination method using an object environment recognition result including uncertainty information in a robot system according to one embodiment. [Figure 39] FIG. 39 is a diagram illustrating a configuration of an example in which a robot system according to an embodiment is provided with sensors other than distance measuring sensors and tactile sensors. [Diagram 40]FIG. 40 is a block diagram illustrating a first configuration example of a control block (robot control device) of a robot system according to one embodiment. As shown in FIG. [Diagram 41] FIG. 41 is a block diagram illustrating a second configuration example of a control block (robot control device) of a robot system according to one embodiment. As shown in FIG. [Diagram 42] FIG. 42 is a flowchart showing a first example of a control operation of a robot system according to one embodiment. [Diagram 43] FIG. 43 is a flowchart showing a second example of the control operation of the robot system according to one embodiment. [Diagram 44] FIG. 44 is a flowchart showing a third example of the control operation of the robot system according to one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. The description will be made in the following order. 0. Comparative Example (Figs. 1 and 2) 1. One embodiment 1.1 Overview (Figs. 3 to 8) 1.2 Specific examples of sensor placement (Figs. 9 to 14) 1.3 Specific examples of object and environment recognition (Fig. 15 to Fig. 23) 1.4 Active Sensing (Figures 24 to 32) 1.5 Action generation using object and environment recognition (Fig. 33 to 38) 1.6 Use of sensors other than distance sensors and tactile sensors (Figure 39) 1.7 Example of control block (robot control device) configuration (Fig. 40 to Fig. 41) 1.8 Control operation examples (Fig. 42 to Fig. 44) 1.9 Effects 2. Other embodiments

[0012] <0. Comparative Examples> 1 and 2 are configuration diagrams showing an overview of a robot system according to a comparative example.

[0013] 1 and 2 show, as an example of a robot, a humanoid robot 1 having a head, a body 2, an arm 3, and a cart (wheels) 4. A hand (robot hand) 10 having fingers 11 capable of grasping an object 50 is attached to the arm 3. A camera (head sensor) 5 is provided in the head of the humanoid robot 1.

[0014] In such a humanoid robot 1, in order to stably grasp an object 50, it is important to select a gripping posture and holding method that matches the object 50, and to do so, it is necessary to grasp the shape and position and posture of the object 50. With a commonly used RGB-D (Depth) camera, there are problems with occlusion, the effects of the light source environment, errors, etc., which can make it difficult to adequately recognize an object. Distance sensors such as ToF (Time of Flight) sensors are an effective means, but they have errors, and close distances before contact are often outside the dynamic range, making measurement difficult.

[0015] Due to the angle of view and distance measurement, the RGB-D camera is often attached to a position away from the object 50 (such as the head of the robot). In such cases, occlusion occurs due to the robot itself or obstacles. The greater the distance between the sensor and the object, the greater the recognition error, making accurate recognition difficult. Errors occur in the shape and position and orientation of the recognized object 50, making it difficult to determine the grasping form and posture. Gripping by an arbitrarily selected method becomes unstable.

[0016] When grasping various objects 50, detailed information (shape, size, position, and posture) of the object 50 is often unknown. As a method for detecting the object 50 with an RGB-D camera, for example, there is a method for detecting the object 50 with a camera 5 such as an RGB-D camera provided on the head of a humanoid robot 1 as shown in FIG. 1. In this case, due to restrictions on the angle of view and the distance measurement range, it is necessary to observe the target object from a position away from it. In addition, detailed information cannot always be obtained due to occlusion by the robot itself or obstacles, and disturbances due to the light source environment.

[0017] Therefore, the use of a distance measuring sensor (proximity sensor) is one of the effective means. For example, as shown in Fig. 2, there is a method of providing a distance measuring sensor 21 on the hand 10 or finger 11 of the humanoid robot 1 to detect the object 50. In this case, the distance measuring sensor 21 has an error, it is difficult to arrange it at high density, and the objects 50 that can be detected are limited.

[0018] Here, in an unstructured environment where environmental information cannot be obtained in advance, such as a food factory, grocery store, restaurant, hospital, or other such environment, manipulation is required to handle objects 50 of various shapes and positions and a variety of obstacles. In addition, it is required to observe object information (shape, position, and orientation of object 50, etc.) and determine an appropriate gripping form (gripping, pinching, etc.) and the position and orientation of the robot.

[0019] Therefore, as one embodiment, a method is proposed for estimating information on the object 50, the environment, and other measurement targets as accurately as possible using a distance sensor and a tactile sensor, even when information on the object 50, the surrounding environment, obstacles, and the like cannot be obtained in advance. Also, a method is proposed for determining the gripping form, gripping position and gripping posture, the robot's obstacle avoidance posture, and route, etc., based on the estimated information on the object 50, the surrounding environment, and obstacles.

[0020] <1. One embodiment> [1.1 Overview] 3 to 8 are configuration diagrams that generally show an example of a hardware configuration of a robot system according to an embodiment of the present disclosure.

[0021] The robot system according to one embodiment includes a robot provided with a distance measuring sensor 21 and a tactile sensor 22, and a robot control device that controls the robot.

[0022] The robot control device includes a recognition unit that recognizes the measurement target based on distance information of the measurement target measured by a distance measuring sensor 21 provided on the robot and tactile information of the measurement target measured by a tactile sensor 22 provided on the robot. The robot control device also includes a control unit that controls the operation of the robot based on the information of the measurement target recognized by the recognition unit.

[0023] In a robot system according to an embodiment, the robot control device may include a plurality of control blocks, for example, as shown in Figs. 40 and 41 described later. In a robot system according to an embodiment, the processing by the recognition unit may be realized by an object / environment recognition unit 110 shown in Figs. 40 and 41 described later. Moreover, the processing by the control unit may be realized by an operation control unit 120 shown in Figs. 40 and 41 described later.

[0024] The robot applied in the robot system according to the embodiment may be various moving bodies, legged robots, drones, etc., in addition to the above-mentioned humanoid robot 1. The robot applied in the robot system according to the embodiment may be a manipulator, a hand 10, a finger 11, etc.

[0025] The robot system according to an embodiment includes at least one sensor capable of measuring the distance to a measurement target in a non-contact manner as the distance measurement sensor 21. The distance measurement sensor 21 may be at least one of a time-of-flight sensor, a millimeter wave sensor, an ultrasonic sensor, a laser sensor, a LiDAR (Light Detection And Ranging), a stereo camera, a pattern projection sensor, and an event camera, for example.

[0026] The robot system according to one embodiment includes at least one sensor capable of measuring a measurement target by contacting the target, as the tactile sensor 22. The tactile sensor 22 may be at least one of a pressure distribution sensor, a force sensor, a vision sensor, a magnetic sensor, and the like.

[0027] In a robot system according to an embodiment, object and environment recognition is performed using proximity information (distance information) from the distance measuring sensor 21 and tactile information (contact information) from the tactile sensor 22. The distance measuring sensor 21 can measure distance without contact. The distance measuring sensor 21 has errors, the objects that can be detected are limited, and contact with a measurement target such as an object 50 or the environment cannot be accurately detected. On the other hand, the tactile sensor 22 can accurately detect contact with a measurement target. The tactile sensor 22 cannot obtain information until it comes into contact with the measurement target. In a robot system according to an embodiment, the distance measuring sensor 21 and the tactile sensor 22 are appropriately used to improve the accuracy of information such as the shape, position, and orientation of a measurement target such as an object 50 or the environment.

[0028] Fig. 3 shows an example in which a distance measurement sensor 21 and a tactile sensor 22 are provided for each of a plurality of fingers 11 in a robotic hand 10. Fig. 4 shows an example in which a distance measurement sensor 21 and a tactile sensor 22 are provided on the fingertip of one finger 11 of a robotic hand 10. Fig. 5 shows an example in which a distance measurement sensor 21 and a tactile sensor 22 are provided on a cart (wheel) 4 of a humanoid robot 1. Fig. 6 shows an example in which a distance measurement sensor 21 and a tactile sensor 22 are provided on an arm 3 of a humanoid robot 1.

[0029] The distance measurement sensor 21 and the tactile sensor 22 may be provided in other types of robots as well. In the case of a legged robot (FIG. 7), the distance measurement sensor 21 and the tactile sensor 22 may be attached to the tips of the legs 41 or the like to sense objects 50 and obstacles. The distance measurement sensor 21 and the tactile sensor 22 may also be provided on the body 2 of the humanoid robot 1. The distance measurement sensor 21 and the tactile sensor 22 may also be provided on a part of a drone 60 (FIG. 8).

[0030] In the robot system according to an embodiment, the recognition unit may recognize the shape and the position and orientation of the measurement target.

[0031] In the robot system according to an embodiment, the control unit may control at least one of the motion form of the robot and the position and posture of the robot as the motion control of the robot.

[0032] In the robot system according to an embodiment, the robot may be configured to be capable of grasping an object as a measurement target. In this case, in the robot system according to an embodiment, the control unit may control at least one of the grasping form, the grasping position, and the grasping attitude of the measurement target by the robot as the operation control of the robot.

[0033] In the robot system according to an embodiment, the robot may be configured such that at least a portion of the robot is movable. In this case, in the robot system according to an embodiment, the control unit may control at least one of the moving speed and the trajectory of the at least a portion of the robot as the operation control of the robot.

[0034] [1.2 Specific examples of sensor placement] In the following, the configuration will be described using the hand 10 as an example of a robot. However, as described above, the technology of the robot system according to one embodiment can also be applied to robots other than the hand 10.

[0035] 9 and 10 are configuration diagrams that roughly show an example of a sensor arrangement in a robot system according to an embodiment.

[0036] Ideally, the distance measuring sensor 21 and the tactile sensor 22 are attached to the whole body of the robot. Also, ideally, the distance measuring sensor 21 and the tactile sensor 22 are provided in approximately the same place, and it is desirable that the distance measuring sensor 21 and the tactile sensor 22 can measure approximately the same place of the measurement object. However, in reality, the distance measuring sensor 21 and the tactile sensor 22 compete for space to provide them, and it is difficult to provide them on the whole body. Therefore, the distance measuring sensor 21 and the tactile sensor 22 may be arranged in consideration of their respective sensor functions. For example, it is preferable to provide the tactile sensor 22 in a part that frequently comes into contact with the measurement object, and the distance measuring sensor 21 in a part that rarely comes into contact with the measurement object. In the case of the hand 10, since the object 50 is often grasped with the finger surface and the palm, it is preferable to provide many tactile sensors 22 on the finger surface and the palm as shown in FIG. 10. In addition, in cases where the frequency of contact with the object 50 is low but contact with the object 50 needs to be avoided, or where alignment with the object 50 is required, it is advisable to attach distance measuring sensors 21 to the front and back of the finger 11.

[0037] FIG. 11 is a configuration diagram illustrating an example of a sensor arrangement in a robot system according to an embodiment.

[0038] In a robot in a robot system according to an embodiment, tactile sensors 22 and distance measuring sensors 21 may be provided alternately as shown in Fig. 11(A). By alternately arranging the tactile sensors 22 and distance measuring sensors 21, the effective detection resolution of the object 50 to be measured can be increased. The tactile sensors 22 may be provided in places where the object 50 is likely to come into contact, and the distance measuring sensors 21 may be provided in parts where the object 50 is likely not to come into contact. With only the tactile sensors 22, it is difficult to measure parts that are not in contact with the object 50, so by using the distance measuring sensors 21, the spatial resolution of the object detection points can be increased as shown in Fig. 11(B).

[0039] 12 to 14 are configuration diagrams that roughly show an example of a sensor arrangement in a robot system according to one embodiment.

[0040] In a robot system according to an embodiment, the robot may have a convex portion 32 and a concave portion 31, the tactile sensor 22 may be provided on the convex portion 32, and the distance measuring sensor 21 may be provided on the concave portion 31. This facilitates contact with the object 50, and enables distance measurement even when the object is in contact. In the example of FIG. 12, the surface of the fingertip that is likely to be contacted by the object 50 is made uneven. This allows the object 50 to first contact the convex portion 32 when grasping the object, so that the contact with the object 50 can be reliably detected by the tactile sensor 22. In addition, measurement by the distance measuring sensor 21 is also possible. Since the distance measuring sensor 21 is provided in the concave portion 31, the object 50 does not come into contact with the distance measuring sensor 21.

[0041] Furthermore, when there are multiple protrusions 32, the height of each of the protrusions 32 may be changed partially as shown in Fig. 13. Furthermore, as shown in Fig. 14, the protrusions 32 may be made of a flexible material that deforms upon contact with the measurement target. By using a flexible material for the protrusions 32, contact with an object 50 having a complex shape can be detected with higher accuracy.

[0042] [1.3 Specific examples of object and environment recognition] (Measurement method) FIG. 15 is an explanatory diagram showing an overview of a method for measuring an object 50 by a robot system according to one embodiment.

[0043] Here, as shown in Figure 15 (A), an example will be described in which an unknown object 50 as a measurement target with no prior information is recognized using a distance sensor 21 and a tactile sensor 22 attached to a robot's hand 10. However, as described above, the technology of the robot system according to one embodiment can also be applied to robots other than the hand 10.

[0044] In a robot system according to one embodiment, the recognition unit first estimates the approximate shape and position and orientation of the object 50 based on distance information obtained from the measurement points (distance sensor measurement points Pr) by the distance sensor 21 in a non-contact state before contact with the object 50 (FIG. 15(B)). Next, the recognition unit stores the points where contact with the object 50 is detected by the tactile sensor 22 as contact points (tactile sensor measurement points Pt) (FIG. 15(C)). Finally, the measurement points will be a mixture of distance sensor measurement points Pr detected by the distance sensor 21 and tactile sensor measurement points Pt detected by the tactile sensor 22 (FIG. 15(D)).

[0045] The information on the measurement points by the distance measuring sensor 21 and the tactile sensor 22 may be any of the distance from the finger surface, the coordinates seen from the robot's reference point (robot coordinate system), and the coordinates in the world coordinate system. In the case of the tactile sensor 22, for example, the contact position in the sensor can be estimated and the position can be calculated from information from the robot's encoder, etc.

[0046] (Object shape estimation method 1: Interpolation) FIG. 16 is an explanatory diagram illustrating an example of a method for estimating an object shape by a robot system according to an embodiment.

[0047] In a robot system according to one embodiment, the recognition unit may estimate the shape between the measurement points (distance sensor measurement points Pr) of the object measured by the distance sensor 21 and the measurement points (tactile sensor measurement points Pt) of the tactile sensor 22 by interpolation.

[0048] As shown in FIG. 16, the recognition unit interpolates between the measurement points of the distance sensor Pr and the tactile sensor Pt without distinguishing between them, and finds the object surface y i The shape of the estimated object surface y may be estimated. Examples of the interpolation method include linear interpolation, spline interpolation, and interpolation using a radial basis function (RBF). i is expressed by the following formula: x i indicates the measurement point. θ indicates the parameter (coefficient) of the curve that indicates the shape. For example, y i =axi 3 +bx i 2 +cx i When expressed in the form of +d, θ is a, b, c, d. y i ,x i ,θ is a two- or three-dimensional vector. object surface y i =f(x i ,θ)

[0049] (Object shape estimation method 2: Weighted (nonlinear) least squares method) 17 is an explanatory diagram that roughly shows an example of a distance measurement error of the distance measuring sensor 21 in the robot system according to one embodiment. In FIG. 17, the horizontal axis shows the reference distance, and the vertical axis shows the measured distance.

[0050] In a robot system according to one embodiment, the recognition unit may estimate the shape of the object to be measured after weighting the measurement points so that the tactile sensor measurement points Pt are weighted more heavily than the distance sensor measurement points Pr.

[0051] As shown in FIG. 17, the distance measurement sensor measurement points Pr generally contain an error compared to the tactile sensor measurement points Pt. The tactile sensor measurement points Pt are more accurate measurement points with higher accuracy than the distance measurement sensor measurement points Pr. Therefore, shape estimation may be performed with priority (weighting) set according to the measurement points. Measurement points with high accuracy may be given a high priority (high weighting), and measurement points with low accuracy may be given a low priority (low weighting). The weighted sum of squared residuals (SSR) is expressed by the following equation (1). It is necessary to find the parameter θ that minimizes the SSR. Once the parameter θ is found, the object surface y i An estimate of is derived.

[0052]

number

[0053] Here, the weight W i For example, the weight of the distance sensor measurement point Pr is Wi _ proc , the weight of the tactile sensor measurement point Pt is W i _ tac Then, define in advance values ​​that satisfy the following: W i _ proc <W i _ tac

[0054] Weight W of distance sensor measurement point Pr i _ proc may be determined based on the sensor characteristics of the distance measuring sensor 21. For example, as shown in FIG. 17, the weight W i _ proc For example, as shown in the following formula (2), the error function E(d) according to the measurement distance d is defined, and the weight W of the distance measurement sensor measurement point Pr is i _ proc Here, a is a coefficient and ε is a small coefficient.

[0055]

number

[0056] (Object shape estimation method 3: Probability distribution) 18 and 19 are explanatory diagrams that roughly show an example of sensor noise distribution of distance measurement sensor measurement points Pr and tactile sensor measurement points Pt in a robot system according to one embodiment.

[0057] In a robot system according to one embodiment, the recognition unit may estimate the shape of the object to be measured based on a probability distribution of measurement points calculated based on the distance sensor measurement points Pr and the tactile sensor measurement points Pt.

[0058] As shown in Fig. 18 and Fig. 19, the sensor noise distribution of the distance measurement sensor measurement point Pr and the tactile sensor measurement point Pt is considered as a probability distribution. For the sensor noise distribution of the distance measurement sensor measurement point Pr, it is assumed that there is an error such as a Gaussian distribution (f(x) = N(μ, σ)) as shown in Fig. 19. Since the error is small at the tactile sensor measurement point Pt, the width of the assumed probability distribution is narrow. By having a probability distribution, it is possible to take into account uncertainties such as measurement errors. For example, in places where there is an error and the object surface position is not accurately known, it is possible to make control adjustments such as lowering the contact speed (e.g., the motion speed (movement speed) of the hand 10) as shown in Fig. 33 described later.

[0059] (Object shape estimation method 4: Estimating the entire shape from partial observation information (no prior information on the object)) FIG. 20 is an explanatory diagram illustrating an example of a method for estimating an object shape by a robot system according to an embodiment.

[0060] In the robot system according to one embodiment, the recognition unit may estimate the overall shape of the measurement target based on partial observation information of the measurement target obtained by the distance measuring sensor 21 and the tactile sensor 22. For example, the recognition unit may estimate the overall shape of the measurement target by performing pattern matching between information of a plurality of template objects 51 prepared in advance and partial observation information of the measurement target obtained by the distance measuring sensor 21 and the tactile sensor 22.

[0061] When there is no prior information on the object 50 to be measured, the overall shape may be estimated from partial observation information of the object 50. As shown in Fig. 20, for example, the overall shape may be estimated by pattern matching between partially observed measurement points (distance sensor measurement points Pr and tactile sensor measurement points Pt) and a template object 51 in a predefined object shape library. The shape with the highest matching score is adopted as the estimated shape of the object 50. As a pattern matching method, a general template matching method or the like can be used.

[0062] (Object shape estimation method 5: Estimating the entire shape from partial observation information (with prior object information) ) FIG. 21 is an explanatory diagram illustrating an example of a method for estimating an object shape by a robot system according to one embodiment.

[0063] In a robot system according to one embodiment, the recognition unit may estimate the overall shape and position and orientation of the object to be measured by performing pattern matching between prior information about the object to be measured and partial observation information about the object to be measured obtained by the distance sensor 21 and the tactile sensor 22.

[0064] For example, as shown in Fig. 21, when the shape and name of the object 50 to be grasped are known in advance, pattern matching is performed between the partial observation information and the object information known in advance, and the information of the unobserved parts is interpolated, so that the shape and the position and orientation of the object 50 to be measured can be estimated. As an example of a pattern matching method, for example, a method using deep learning or ICT (Iterative Closest Point) can be used.

[0065] (Object Shape Estimation Method 6: Estimating Overall Shape from Partial Observation Information (Modified Example)) ) 22 and 23 are explanatory diagrams that outline an example of a method for estimating an object shape by a robot system according to one embodiment.

[0066] In a robot system according to an embodiment, the recognition unit may perform estimation by combining the above-mentioned object shape estimation method 4 and object shape estimation method 5. As shown in Fig. 22, for example, when an object is grasped by a finger 11 provided with a distance measurement sensor 21 and a tactile sensor 22, if the object 50 to be measured has unevenness, there will be parts where the distance measurement sensor 21 or the tactile sensor 22 contacts the object 50 and parts where they do not contact each other. In this case, the distribution pattern of the measurement points Pr of the distance measurement sensor and the measurement points Pt of the tactile sensor also becomes information for recognizing the object shape.

[0067] 23, the distribution pattern of the distance measuring sensor measurement points Pr and the distribution pattern of the tactile sensor measurement points Pt may be used during pattern matching to estimate the shape and position and orientation of the object 50 to be measured. In this case, information on the distribution pattern of the measurement points may be added to the pattern matching library.

[0068] [1.4 Active sensing] (Example of active sensing 1) 24 and 25 are explanatory diagrams that roughly show a first example of active sensing by a robot system according to one embodiment.

[0069] In a robot system according to an embodiment, the robot may be configured such that at least a portion of the portion where the distance measuring sensor 21 or the tactile sensor 22 is provided is movable. In a robot system according to an embodiment, a control unit may control the operation of at least a portion of the portion where the distance measuring sensor 21 or the tactile sensor 22 is provided so that the distance measuring sensor measurement point Pr or the tactile sensor measurement point Pt is located at a location where the measurement of the measurement target by the distance measuring sensor 21 or the tactile sensor 22 is insufficient.

[0070] If the part on which the ranging sensor 21 or the tactile sensor 22 is attached can be moved, the part may be moved to measure the object 50, the environment, or other measurement target so that the measurement point by the ranging sensor 21 or the tactile sensor 22 is located in a place where measurement is insufficient or where measurement uncertainty is high.

[0071] In the example of Fig. 24(A), the shape of a part of the object 50 can be observed near the base of the finger 11 by the distance measurement sensor measurement points Pr or the tactile sensor measurement points Pt, but there are few measurement points of the object 50 near the tip of the finger 11, so the shape may not be observed accurately. In this case, by moving the tip of the finger 11 closer to the object 50 as shown in Fig. 24(B), the shape of the object 50 near the tip of the finger 11 can also be observed. The "possibility of there being few measurement points" can be determined, for example, by the interval or density of the measurement points, comparison with prior information, comparison with a shape predicted from RGB data, etc.

[0072] Furthermore, when a probability distribution is used to estimate the object shape as shown in Fig. 25(A), it is preferable to operate the fingers 11 of the hand 10 so as to search for a part with large uncertainty as shown in Fig. 25(B). At that time, since the measurement by the tactile sensor 22 has less uncertainty, it is preferable to control the fingers 11 so that the tactile sensor 22 comes into contact with the object 50.

[0073] (Active sensing example 2) Fig. 26 is an explanatory diagram that illustrates a second example of active sensing by a robot system according to an embodiment. Fig. 26(B) and Fig. 26(C) show a state viewed from the X-axis direction in Fig. 26(A).

[0074] When the part to which the distance measuring sensor 21 or the tactile sensor 22 is attached can be moved, the part can be rotated or translated to obtain information on a wider range of the measurement target such as the object 50 or the environment. For example, as shown in FIG. 26(A), consider a case where the object 50 is measured by the distance measuring sensor 21 attached to the finger 11 of the hand 10. As shown in FIG. 26(B), when the finger 11 can rotate around an axis parallel to the X-axis as the rotation axis, the finger 11 to which the distance measuring sensor 21 is attached can be rotated while performing measurement, so that one distance measuring sensor 21 can continuously obtain data on different positions (distance measuring sensor measurement points Pr) of the object 50 as shown in FIG. 26(C). This makes it possible to detect the object surface over a wide range even when the number of mounted sensors is small. In addition, multiple measurement information from multiple sensors obtained from the distance measuring sensors 21 or the tactile sensors 22 attached to different positions can be integrated to reconstruct information on the measurement target such as the object 50 or the environment.

[0075] (Example 3 of active sensing) 27 and 28 are explanatory diagrams that roughly show a third example of active sensing by a robot system according to one embodiment.

[0076] In a robot system according to an embodiment, the control unit may control the motion of the robot based on distance information measured before contacting the measurement target, and then correct the motion control of the robot based on tactile information measured after contacting the measurement target. In a robot system according to an embodiment, the robot may be configured to be able to grasp an object as a measurement target. In this case, the control unit may determine a gripping point that takes gripping stability into consideration from among a plurality of gripping point candidates Pa based on information of the measurement target recognized by the recognition unit, and control the motion of the robot so that the object as a measurement target is grasped at the determined gripping point.

[0077] For example, in the case of grasping an object, the position of the grasping point and the surface direction, rigidity, and friction coefficient at that position greatly affect the grasping stability, so it is necessary to determine the grasping point taking the grasping stability into consideration. When there are many candidate grasping points Pa, the final grasping point can be narrowed down using the distance sensor 21 and the tactile sensor 22. For example, the narrowing down can be performed by the following procedure.

[0078] (Narrowing Procedure 1) First, the approximate shape is estimated by the distance measuring sensor 21 (FIG. 27(A)). Any of the estimation methods described above may be used.

[0079] (Narrowing Procedure 2) Next, the candidate gripping points Pa are narrowed down in consideration of gripping stability (FIG. 27(B)).

[0080] Fig. 29 is an explanatory diagram illustrating a first method for narrowing down the candidate gripping points Pa in consideration of the gripping stability by the robot system according to one embodiment. Fig. 30 is an explanatory diagram illustrating a second method for narrowing down the candidate gripping points Pa in consideration of the gripping stability by the robot system according to one embodiment.

[0081] As a method of estimating grip stability, there is a method of assuming that grip stability exists if the state is in a force closure state. As a first method of narrowing down the grip point candidates Pa, there is a method of narrowing down the candidates so that the intersection of the straight line connecting the contact points Pb or the straight line extended in the normal direction from the contact point Pb exists in the region 52 surrounded by the friction cone, as shown in Figure 29 (A) and Figure 29 (B). As a second method of narrowing down the grip point candidates Pa, there is a method of collating with data of grip points from past grips, and adopting the grip position of the data with a high degree of matching as the grip point candidate Pa, as shown in Figure 30.

[0082] (Narrowing procedure 3) Next, the tactile sensor 22 contacts the narrowed down candidate gripping point Pa, and measures the exact position, rigidity, and slipperiness (friction coefficient) of that point (Fig. 28(A)). The rigidity can be estimated from the reaction force at the time of contact and the magnitude of deformation. The friction coefficient is calculated as the ratio of the shear force at the start of slipping to the normal force when a force is applied in the shear direction after contact.

[0083] (Narrowing procedure 4) Next, the final gripping point is determined from the gripping point candidates Pa (FIG. 28(B)).

[0084] (Example 4 of active sensing) 31 and 32 are explanatory diagrams that outline a fourth example of active sensing by a robot system according to an embodiment.

[0085] Fig. 31(A) shows an example of a motion of kicking a ball as an object 50 with a leg 41 of a legged robot 40. Figs. 31(B) to (D) show schematic views of the vicinity of a leg 41 of the legged robot 40 as viewed from above.

[0086] Active sensing can be applied to actions other than grasping. For example, it can be applied to the action of kicking a ball as an object 50 with the leg 41 of a legged robot 40 (FIG. 31(A)), or the action of pushing an object 50 with the hand 10 of a humanoid robot 1 (FIG. 32). For example, the distance measuring sensor 21 can narrow down the contact point and the direction of the force (FIG. 31(B)), and the tactile sensor 22 can determine the contact point and the direction of the force (FIGS. 31(C) and (D)). If the direction of the contact surface is different from the expected direction due to an error in the distance measuring sensor 21 and is found at the time of contact, the information at the time of contact can be used to adjust the direction of the force.

[0087] In addition, sensing similar to the above-described active sensing examples 1 to 4 can be applied to the operation of the arm 3 and the dolly 4 (FIGS. 5 and 6) of the humanoid robot 1 provided with the distance measuring sensor 21 and the tactile sensor 22. It can also be applied to the operation of a drone 60 (FIG. 8) provided with the distance measuring sensor 21 and the tactile sensor 22.

[0088] [1.5 Motion generation using object and environment recognition] (Trajectory planning using uncertainty information) FIG. 33 is an explanatory diagram showing an example of an operation in which the hand 10 is moved to approach an object 50 that is to be measured in the robot system according to one embodiment.

[0089] When approaching a measurement target such as an object 50 or the environment, the distance to the measurement target and its shape are initially estimated based on the distance measuring sensor measurement point Pr (FIGS. 33(A) and 33(B)). However, since the measurement by the distance measuring sensor 21 contains errors, in some cases the hand may collide with the measurement target (FIG. 33(C)). Therefore, the operation speed (movement speed) and trajectory of the hand 10 may be adjusted according to the uncertainty. For example, when there is uncertainty, the movement speed may be reduced and the hand may approach on a trajectory with a margin larger than the measured distance.

[0090] FIG. 34 is an explanatory diagram showing an example of a method for determining a gripping form using an object environment recognition result including uncertainty information in a robot system according to an embodiment.

[0091] In the case of grasping an object, various approaches (approaching motions) and grasping forms are possible depending on the degree of freedom configuration of the fingers 11 and the number of fingers 11. These should be appropriately determined depending on the shape and mass friction coefficient of the object 50. Here, we propose a method of estimating an appropriate grasping form using the distance measurement sensor 21, and then fine-tuning the grasping form using the tactile sensor 22, and changing the grasping form.

[0092] First, information on a measurement target such as an object 50 or the environment is measured by the distance measuring sensor 21 (step 1). Any of the above-mentioned measurement methods may be used. The obtained information on the measurement target includes uncertainty.

[0093] Next, candidates are narrowed down from the predefined gripping forms (step 2). Narrowing down methods include, for example, rule-based and learning-based (data-driven). In the rule-based method, parameters related to the shape of the object 50 are defined and thresholds are set for each to narrow down the gripping forms. In the learning-based method, shape information of the object 50 including uncertainty is input (not only the estimated shape but also information such as variance is possible), and a model (which may be a Neural Network (NN)) whose output is the gripping form is trained. In the learning-based method, an evaluation value is output.

[0094] Next, the gripping form is finely corrected and changed using accurate object surface information detected by the tactile sensor 22 (step 3). If updating the object shape using information from the tactile sensor 22 indicates that a different gripping form is more appropriate, an action is taken to change the gripping form. Even if the same gripping form is used, if it is better to modify the position or posture for stable gripping, this is done. The above-mentioned force closure concept (Figure 29) may be used as an index of gripping stability.

[0095] (Example of step 2 (rule-based)) FIG. 35 is an explanatory diagram showing a specific example of the procedure 2 (rule-based) shown in FIG.

[0096] In the example of FIG. 35, the gripping mode is determined to be one of Enveloping Grasp, Parallel Pinch, and Medium Wrap depending on whether the finger tips collide or not and whether the contact area is wide or narrow.

[0097] (Example of Step 2 (learning-based)) FIG. 36 is an explanatory diagram showing a specific example (learning base) of the procedure 2 shown in FIG.

[0098] The advantage of using a learning-based grip form determination method is that it is possible to create new, more optimal grip forms. In the rule-based method, which grip form to adopt is uniquely determined, resulting in a discrete classification problem. In the learning-based method, an evaluation value for each grip form is calculated. Selecting the one with the largest evaluation value results in a discrete classification problem. It is possible to generate intermediate forms of different grip forms according to the evaluation value. In the example of FIG. 36, a new intermediate grip form that matches the object 50 is generated by combining grip form A and grip form C. In the example of FIG. 36, a new intermediate grip form is generated in which grip form A is used 60% of the time and grip form C is used 30% of the time.

[0099] (How to use learning-based and rule-based methods) For example, the rule base may be used in the following cases: When object recognition results can be used to classify objects into primitive shapes (cuboid, sphere, cylinder, cone, etc.) that are predefined by humans When the user has a request regarding the gripping method (e.g., an object 50 of a specific shape should be gripped by pinching it so that the contact area is small), and the user wants to select the specified gripping form reliably

[0100] Also, for example, the learning base may be used in the following cases: When it is difficult to create rules for determining the gripping form due to a complex shape When sensors cannot be installed closely due to space limitations, or when occlusion is likely to occur, it is necessary to interpolate missing information.

[0101] As another rule for distinguishing between the two, for example, an object 50 whose gripping form cannot be determined based on the rules (does not conform to a preset rule) may be processed based on learning. Also, the gripping form determined based on the rules may be matched with the gripping form determined based on learning, and if they differ, the decision may be left to a human.

[0102] (Example of step 3) FIG. 37 is an explanatory diagram showing a specific example of the procedure 3 shown in FIG.

[0103] If another gripping form is more appropriate than the initially determined gripping form (FIG. 37(A)), an action to change the gripping form may be taken (FIG. 37(B)). In a robot system according to one embodiment, the gripping form can be quickly changed by utilizing recognition information of the measurement target such as object 50 or the environment. Note that a trajectory along object 50 may be taken near tactile sensor measurement point Pt. Since the influence of errors is large near distance measurement sensor measurement point Pr, the trajectory of finger 11 may be controlled with a margin of error (FIG. 37(C)).

[0104] (Modification of the method for determining the gripping form) FIG. 38 is an explanatory diagram showing a modified example of a grasping form determination method using an object environment recognition result including uncertainty information in a robot system according to one embodiment.

[0105] In a hand 10 having multiple fingers 11, due to space limitations, it is often difficult to mount a distance measuring sensor 21 and a tactile sensor 22 on the same finger 11. For example, when the multiple fingers 11 include a first finger 11A and a second finger 11B, the type of sensor mounted may be different for each finger, such as mounting only the distance measuring sensor 21 on the first finger 11A and only the tactile sensor 22 on the second finger 11B.

[0106] In this case, first, the first finger 11A equipped with the distance measuring sensor 21 is aligned with the object 50, and measurement is performed by the distance measuring sensor 21 (step 1A). Next, the second finger 11B equipped with the tactile sensor 22 is brought into contact with the object 50, and measurement is performed by the tactile sensor 22 (step 1B). After that, the steps from step 2 in FIG. 34 onwards may be performed.

[0107] [1.6 Use of sensors other than distance sensors and tactile sensors] In the robot system according to the embodiment, in addition to the distance measuring sensor 21 and the tactile sensor 22, other types of sensors may be used.

[0108] For example, the humanoid robot 1 may utilize an RGB(-D) sensor, LiDAR, or the like attached to the head or the surrounding environment. The RGB(-D) sensor, LiDAR, or the like may perform object recognition and shape recognition to obtain advance information on the measurement target such as the object 50 or the environment. In addition, a standard shape and size of the object 50 may be assumed from the object recognition result to be used as a predicted shape of the object 50. The generated predicted shape may also be integrated with a predicted shape generated by the distance measuring sensor 21 and the tactile sensor 22 on the fingertip.

[0109] Also, at least one of a vibration sensor, a microphone, an acceleration sensor, etc., attached to the contact point with the measurement target such as the object 50 or the environment or in the vicinity of the contact point may be used. These sensors may be used in place of the tactile sensor 22 or as an auxiliary to the tactile sensor 22. These sensors can detect vibrations generated by contact with the measurement target, making it possible to detect the contact timing with high accuracy. Also, contact can be detected even if it occurs at a point where the tactile sensor 22 is not attached.

[0110] (Examples of using sensors other than the distance sensor 21 and the tactile sensor 22) FIG. 39 shows an example in which a sound sensor (microphone) 23 is provided as a sensor other than the distance measuring sensor 21 and the tactile sensor 22 in a robot system according to one embodiment.

[0111] Sound, vibration For example, as shown in Fig. 39, by sensing the sound and vibrations that occur when an object 50 to be measured comes into contact with a finger 11 of a hand 10, it is possible to accurately grasp the timing of contact with the object to be measured. Since the sound and vibrations are transmitted through the links of the fingers, the contact can be detected even if the contact is made in a place where there is no tactile sensor 22 or distance measuring sensor 21. A microphone 23, an acceleration sensor, a force sensor, or the like can be used.

[0112] ·temperature For example, a temperature sensor can detect contact with a measurement target such as object 50 or the environment. Sensitivity is improved if the temperature sensor is attached to a part that comes into direct contact with the measurement target. A temperature sensor may be placed at a point away from the contact point, but if applied to, for example, finger 11 of hand 10, the sensor will detect the transfer of heat that travels through the links of hand 10, resulting in reduced sensitivity and responsiveness.

[0113] Torque sensors, force sensors For example, when applied to a finger 11 of a hand 10, contact with a measurement target such as an object 50 or the environment can be detected by a torque sensor attached to a finger joint or a force sensor attached to a wrist. In this case, contact can be determined from a sudden torque change or a force change at the time of contact based on a threshold value related to the magnitude of the force, speed, or acceleration. Contact with a measurement target can also be estimated based on the relationship between the fingertip position and force derived from the encoder of each axis. Since it can detect minute forces, it has the effect of improving the sensitivity of contact detection. Furthermore, even if the measurement target comes into contact with an area where there is no sensor, contact with the measurement target can be detected with high accuracy.

[0114] ·Object recognition An image sensor such as an RGB camera captures an image of a measurement target such as an object 50 or the environment, and the true identity of the measurement target can be estimated in advance by object recognition. For example, this information can be used to interpolate shape information (such as storing a representative shape and performing pattern matching with the recognition result). In addition, when applied to the fingers 11 of the hand 10, for example, the gripping form can be determined in advance in accordance with the recognition result, and the gripping form can be finely corrected based on the sensor value of the distance sensor 21 or the tactile sensor 22.

[0115] [1.7 Example of control block (robot control device) configuration] (Control block configuration example 1) FIG. 40 is a block diagram illustrating a first configuration example of a control block (robot control device) of a robot system according to one embodiment. As shown in FIG.

[0116] At least some of the control blocks in the robot system according to one embodiment may be configured by a computer including, for example, one or more central processing units (CPUs), one or more read only memories (ROMs), and one or more random access memories (RAMs). In this case, the processing of each control block may be realized by one or more CPUs executing processing based on a program stored in one or more ROMs or RAMs. Furthermore, the processing of each control block may be realized by one or more CPUs executing processing based on a program supplied from the outside via, for example, a wired or wireless network.

[0117] The robot system according to one embodiment includes an object / environment recognition unit 110 and an operation control unit 120 as control blocks.

[0118] The object and environment recognition unit 110 has an object and environment measurement unit 100 , a distance information processing unit 101 , a contact information processing unit 102 , and an environment and object shape / posture estimation unit 103 .

[0119] The movement control section 120 has a movement form determination section 200 , a trajectory calculation section 201 , a motion control section 202 , and an actuator section 203 .

[0120] The object / environment measuring unit 100 includes various sensors including a distance measuring sensor 21 and a tactile sensor 22, and a sensor signal acquisition processing unit that processes sensor signals from the various sensors. The object / environment measuring unit 100 processes the sensor signals from the various sensors and outputs data such as an RGB image, a depth image, distance, point cloud, event data, force, pressure, vibration, acceleration, amount of slippage, contact position, and contact area.

[0121] The distance information processing unit 101 performs signal processing of the sensor signal from the distance measuring sensor 21. The distance information processing unit 101 can also estimate the 3D position of the measurement target such as the object 50 or the environment from the mounting position of the distance measuring sensor 21 and the position and orientation information of the robot. The reference coordinate system may be the robot coordinate system or the world coordinate system. The distance information processing unit 101 outputs data such as a depth image, distance, and measurement point position.

[0122] The contact information processing unit 102 processes the sensor signal from the tactile sensor 22. The contact information processing unit 102 determines the contact position with respect to a measurement target such as an object 50 or the environment, using the attachment position of the tactile sensor 22 and the position and orientation information of the robot. The reference coordinate system may be a robot coordinate system or a world coordinate system. The contact information processing unit 102 outputs data such as a contact flag, a slippage amount, a contact position, and a contact area.

[0123] The environment and object shape / posture estimation unit 103 combines the distance information and contact information to output data on information (shape, position and posture, etc.) of the measurement target such as the object 50 or the environment. When matching with a shape of the measurement target such as the object 50 or the environment stored in advance, the environment and object shape / posture estimation unit 103 also stores data on the shape.

[0124] The motion form determination unit 200 calculates a motion form and outputs data of the motion form. In the case of object grasping, the motion form determination unit 200 calculates a gripping form (clenched grip, fingertip grip, etc.) as the motion form. In the case of a motion of pushing a measurement target such as the object 50 or the environment, the motion form determination unit 200 calculates the pushing position, the force direction, etc. If there are predefined motion states, the motion form determination unit 200 also includes a storage unit for them.

[0125] The trajectory calculation unit 201 calculates the trajectories of the end effector and joints of the robot based on the output data from the environment / object shape / posture estimation unit 103 and the motion form determination unit 200 .

[0126] The motion control unit 202 generates control command values ​​for the actuators. The control command values ​​for the actuators include, for example, command values ​​for the position, velocity, acceleration, and force of the joint angles.

[0127] The actuator unit 203 moves the robot based on a control command value from the motion control unit 202. The actuator unit 203 includes a movable unit that moves the robot and a control processing block for the movable unit.

[0128] (Control block configuration example 2) FIG. 41 is a block diagram illustrating a second configuration example of a control block (robot control device) of a robot system according to one embodiment. As shown in FIG.

[0129] The object / environment recognition unit 110 may further include an additional measurement point identification unit 104 and an environment / object information storage unit 105 .

[0130] The movement control section 120 may have, instead of the movement form determining section 200, a movement form determining section 300, a movement form storage section 301, and a position / posture correcting section 302.

[0131] The additional measurement point identifying unit 104 identifies the points where there is a large uncertainty in the results of estimation by the environment and object shape / posture estimation unit 103, since it is desired to measure those points again. The additional measurement point identifying unit 104 outputs 2D or 3D position and posture information data of the measurement target, such as the object 50 or the environment, shown in the world coordinate system, robot coordinate system, etc. The additional measurement point identifying unit 104 calculates the amount of joint movement when it is desired to move a joint for measurement.

[0132] The motion form determination unit 300 calculates a motion form and outputs data of the motion form. In the case of grasping an object, the motion form determination unit 300 calculates a gripping form (clutch grip, fingertip grip, etc.) as the motion form. The motion form determination unit 300 also determines the position of the contact point taking into account the gripping stability.

[0133] The motion form storage unit 301 stores information on gripping forms and the like.

[0134] The position and orientation correction unit 302 outputs data such as the amount of correction of the robot's position and orientation, and information on the corrected position and orientation.

[0135] The trajectory calculation unit 201 calculates the trajectories of the robot's end effector and joints based on output data from the additional measurement point identification unit 104 , the position and orientation correction unit 302 , the environment and object shape / orientation estimation unit 103 and the motion form determination unit 200 .

[0136] [1.8 Control operation example] (First example of control operation) FIG. 42 is a flowchart showing a first example of a control operation of a robot system according to one embodiment.

[0137] FIG. 42 shows an example of the operation when the robot trajectory once determined is not corrected. The recognition unit of the robot system first measures the object 50, the environment, or other object to be measured using the distance sensor 21 (step S101). Next, the recognition unit of the robot system processes the distance information of the object measured by the distance sensor 21 (step S102). Next, the recognition unit of the robot system estimates the shape and position and orientation of the object to be measured, such as the object 50 or the environment (step S103). Next, the control unit of the robot system determines the motion form of the robot (step S104). Next, the control unit of the robot system calculates the trajectory of the robot (step S105). Next, the control unit of the robot system calculates an actuator command (step S106).

[0138] Next, the recognition unit of the robot system determines whether or not the robot has touched the measurement object based on the sensor signal from the tactile sensor 22 (step S107). If the recognition unit of the robot system determines that the robot has not touched the measurement object (step S107; N), the robot system returns to the process of step S106.

[0139] If it is determined that the robot has come into contact with the measurement target (step S107; Y), the recognition unit of the robot system then processes the contact information of the measurement target measured by the tactile sensor 22 (step S108). Next, the recognition unit of the robot system estimates the shape and position and orientation of the measurement target, such as the object 50 or the environment (step S109). Next, the control unit of the robot system determines the motion form of the robot (step S110). Next, the control unit of the robot system calculates the trajectory of the robot (step S111). Next, the control unit of the robot system calculates an actuator command (step S112). Next, the control unit of the robot system determines whether the motion is complete (step S113). If it is determined that the motion is complete (step S113; Y), the control unit of the robot system ends the processing. If it is determined that the motion is not complete (step S113; N), the control unit of the robot system returns to the processing of step S112.

[0140] (Second example of control action) FIG. 43 is a flowchart showing a second example of the control operation of the robot system according to one embodiment.

[0141] 43 shows an example of operation in which distance information and contact information are continuously updated every control cycle, and the motion form is calculated (corrected) every control cycle. The recognition unit of the robot system first measures the measurement target such as the object 50 or the environment using the distance measuring sensor 21 or the tactile sensor 22 (step S201). Next, the recognition unit of the robot system determines whether the robot has contacted the measurement target based on the sensor signal from the tactile sensor 22 (step S202).

[0142] If it is determined that the robot has come into contact with the measurement target (step S202; N), the recognition unit of the robot system then processes the contact information of the measurement target measured by the tactile sensor 22 (step S204) and proceeds to the process of step S206. If it is determined that the robot has not come into contact with the measurement target (step S202; Y), the recognition unit of the robot system then processes the distance information of the measurement target measured by the distance sensor 21 (step S203) and proceeds to the process of step S205.

[0143] In the process of step S205, the recognition unit of the robot system estimates the shape and position and orientation of the measurement target such as the object 50 or the environment, and then proceeds to the process of step S206. In the process of step S206, the control unit of the robot system determines the operation mode of the robot, and then proceeds to the process of step S207.

[0144] Next, the control unit of the robot system calculates the trajectory of the robot (step S207). Next, the control unit of the robot system calculates an actuator command (step S208). Next, the control unit of the robot system judges whether the operation is completed or not (step S209). If it is judged that the operation is completed (step S209; Y), the control unit of the robot system ends the processing. If it is judged that the operation is not completed (step S209; N), the control unit of the robot system returns to the processing of step S201.

[0145] (Third example of control action) FIG. 44 is a flowchart showing a third example of the control operation of the robot system according to one embodiment.

[0146] The control operation in Fig. 44 corresponds to the grasping form determination method shown in Fig. 38. In the grasping form determination method shown in Fig. 38, the robot is a hand 10 having a first finger 11A and a second finger 11B as the multiple fingers 11, and only a distance measuring sensor 21 is provided on the first finger 11A, and only a tactile sensor 22 is provided on the second finger 11B.

[0147] The recognition unit of the robot system first measures a measurement target such as an object 50 or the environment using the distance measurement sensor 21 or the tactile sensor 22 (step S301). The recognition unit of the robot system processes, in parallel, distance information of the measurement target measured by the distance measurement sensor 21 provided on the first finger 11A (step S302) and contact information of the measurement target measured by the tactile sensor 22 provided on the second finger 11B (step S303).

[0148] Next, the recognition unit of the robot system estimates the shape and position and orientation of the measurement target such as the object 50 or the environment (step S304). Next, the control unit of the robot system determines the motion mode of the robot (step S305). Next, the control unit of the robot system calculates the trajectory of the robot (step S306). Next, the control unit of the robot system calculates an actuator command (step S307). Next, the control unit of the robot system judges whether the motion is complete or not (step S308). If it is judged that the motion is complete (step S308; Y), the control unit of the robot system ends the processing. If it is judged that the motion is not complete (step S308; N), the control unit of the robot system returns to the processing of step S301.

[0149] [1.9 Effects] As described above, according to the robot system of one embodiment, the target is recognized based on distance information of the target measured by the distance measuring sensor 21 provided in the robot and tactile information of the target measured by the tactile sensor 22 provided in the robot, and the operation of the robot is controlled based on the information of the recognized target. This improves the recognition accuracy of the object 50 and the environment, and enables the robot to perform flexible and stable operations according to the situation.

[0150] According to the robot system according to one embodiment, even if there is uncertainty, information on the shape and position and orientation of the measurement target such as the object 50 or the environment can be estimated by using both the distance measuring sensor 21 and the tactile sensor 22. This improves the robustness of recognition in the environment.

[0151] Furthermore, according to the robot system of one embodiment, the distance measuring sensor 21 and the tactile sensor 22 are used in stages, so that the recognition accuracy can be gradually improved while taking into account uncertainty. Since there is a trade-off between the recognition accuracy and the time and processing load required for sensing, it is possible to use different sensors depending on the required accuracy. This makes it easy to switch between a case where the accuracy is poor but the operation is fast, and a case where the accuracy is improved even if it takes time.

[0152] Furthermore, the robot system according to one embodiment does not depend heavily on the sensor method due to the method of using sensors with different properties in stages, which makes it easy to introduce sensors other than the distance measuring sensor 21 and the tactile sensor 22. Furthermore, the robot system according to one embodiment can set weights according to the noise level and reliability of the sensors when recognizing the measurement target, so that sensors with different accuracies and properties can be used in combination.

[0153] In addition, the robot system according to the embodiment utilizes the degree of freedom of the robot, so that a wide range of information can be obtained with a small number of sensors. In addition, the robot system according to the embodiment can define the basic shape and movement variations in advance, so that it can quickly adapt to robots with different movement constraints.

[0154] Furthermore, according to the robot system of one embodiment, since information from both the distance measuring sensor 21 and the tactile sensor 22 is taken into consideration, the distance measuring sensor 21 and the tactile sensor 22 do not need to be attached in the same location. This allows the robot to be made smaller and space to be used more effectively.

[0155] Moreover, according to the robot system of one embodiment, by using both the distance measuring sensor 21 and the tactile sensor 22, it is possible to stably interact with the environment or object 50 even when uncertainty occurs (grasping an object, pressing a part of the environment, etc.). Furthermore, according to the robot system of one embodiment, by using both the distance measuring sensor 21 and the tactile sensor 22, it is possible to interact with the environment or object 50 according to uncertainty, which enables the robot to operate stably and safely in an environment where humans are present.

[0156] (Comparison with prior art documents) Patent Document 1 (International Publication No. 2021 / 033509) describes a technology for estimating the position and posture of a robot based on environmental information obtained from a non-contact sensor (RGB(-D), ToF, GPS, etc.) and information obtained from a contact sensor when the robot contacts the environment. In the technology described in Patent Document 1, the position and posture of the object 50 relative to the world coordinates is a known premise, and only the position and posture of the robot can be estimated. The position and posture of the object 50 cannot be obtained, and fine features such as shape are difficult to estimate. In contrast, the robot system according to one embodiment is a technology that can estimate the position and posture and shape of the object 50, rather than the position and posture of the robot. In addition, the robot system according to one embodiment can detect finer shape features of the object 50, and more accurate information on the object 50 can be obtained. In addition, by combining the generated object information with sensor information, an appropriate gripping form and gripping position can be derived.

[0157] Patent Document 2 (JP Patent Publication 2009-28692A) describes a technique for detecting the distance and contact between an object 50 and a finger using a contact bar attached to a hand, and estimating the shape of the object 50. The technique described in Patent Document 2 assumes that the contact bar will directly contact the object 50, and therefore may damage the object 50. In addition, it is difficult to measure the object surface by tracing it, and it is difficult to adjust the position and posture of the hand during grasping. In contrast, in a robot system according to one embodiment, by using both the distance measuring sensor 21 and the tactile sensor 22, it is possible to grasp the shape of the object 50 both in a non-contact and contact manner. Therefore, it is possible to search the object surface in a non-contact manner, and to grasp the shape of the object 50 by contacting the object 50 and tracing its surface. In addition, it is possible to switch the gripping form in a non-contact manner.

[0158] Patent Document 3 (International Publication No. 2022 / 030242) describes a technology for adjusting the relative position between the object and the hand and the relative position between the obstacle and the hand around the object based on the distance measured by a distance measuring sensor attached to the surface of the robot hand, thereby grasping the object without colliding with the obstacle. In the technology described in Patent Document 3, the accurate position, posture, and shape of the object 50 are often not obtained due to limitations in distance measurement errors and distance measurement range. For example, when contact is made, it is outside the distance measurement range and difficult to detect with the distance measurement sensor, and even if the object 50 is in contact, it is erroneously determined that the object 50 is not present. In contrast, in a robot system according to one embodiment, information can be obtained in both phases before and after contact by using the distance measurement sensor 21 and the tactile sensor 22. Since contact information can also be taken into account, more accurate object information can be obtained. Therefore, a more stable gripping operation can be realized.

[0159] The advantages of using not only the distance measuring sensor 21 but also the distance measuring sensor 21 and the tactile sensor 22 in combination are as follows. -Improved object detection accuracy. By using sensors that measure different physical quantities, the measurement errors can be compensated for. In general, the error of the distance measuring sensor 21 is larger than that of the tactile sensor 22. -Object recognition is possible even after contact. It is difficult for the distance measuring sensor 21 to recognize an object after contact. Therefore, it is required to store information before contact. However, it becomes difficult to respond when the object 50 shifts while in contact. - Improved time and memory efficiency of object detection. Using the results of object detection based on distance measurement, it is possible to narrow down the locations where object recognition should be performed in more detail. This improves the accuracy of object detection in two stages, improving efficiency and reducing memory usage.

[0160] The effects described in this specification are merely examples and are not limiting, and other effects may be achieved. The same applies to the effects of other embodiments described below.

[0161] <2. Other embodiments> The technology according to the present disclosure is not limited to the above-described embodiment, and various modifications are possible.

[0162] For example, the present technology can be configured as follows. According to the present technology having the following configuration, the robot recognizes the target based on distance information of the target measured by a distance sensor provided in the robot and tactile information of the target measured by a tactile sensor provided in the robot, and controls the robot's operation based on the information of the recognized target. This improves the recognition accuracy of objects and the environment, and enables the robot to perform flexible and stable operations according to the situation.

[0163] (1) a recognition unit that recognizes the measurement object based on distance information of the measurement object measured by a distance measuring sensor provided in the robot and tactile information of the measurement object measured by a tactile sensor provided in the robot; a control unit that controls the operation of the robot based on the information of the measurement target recognized by the recognition unit; Equipped Robot control device. (2) The recognition unit recognizes the shape and the position and orientation of the measurement target. A robot control device as described in (1) above. (3) The control unit controls at least one of the motion form of the robot and the position and posture of the robot as the motion control of the robot. A robot control device according to (1) or (2) above. (4) The robot is configured to be capable of grasping an object as the measurement target, The control unit controls at least one of a gripping form, a gripping position, and a gripping attitude of the measurement target by the robot as the operation control of the robot. A robot control device according to any one of (1) to (3) above. (5) The robot is configured so that at least a part of the robot is movable, The control unit controls at least one of a moving speed and a trajectory of at least a part of the robot as the motion control of the robot. A robot control device according to any one of (1) to (4) above. (6) The control unit performs motion control of the robot based on the distance information measured before contacting the measurement object, and then modifies the motion control of the robot based on the tactile information measured after contacting the measurement object. A robot control device according to any one of (1) to (5) above. (7) the robot is configured such that at least a portion of the robot in which the distance measuring sensor or the tactile sensor is provided is movable; The control unit controls the operation of at least a part of a region where the distance measuring sensor or the tactile sensor is provided so that the measurement point of the distance measuring sensor or the measurement point of the tactile sensor is located at a location where the measurement of the measurement object by the distance measuring sensor or the tactile sensor is insufficient. A robot control device according to any one of (1) to (6) above. (8) The robot is configured to be capable of grasping an object as the measurement target, The control unit determines a gripping point taking into consideration gripping stability from among a plurality of gripping point candidate points based on the information of the measurement target recognized by the recognition unit, and controls the operation of the robot so that the object as the measurement target is gripped at the determined gripping point. A robot control device according to any one of (1) to (7) above. (9) In the robot, the tactile sensor is provided in a portion that frequently comes into contact with the measurement object, and the distance measuring sensor is provided in a portion that frequently comes into contact with the measurement object. A robot control device according to any one of (1) to (8) above. (10) In the robot, the tactile sensors and the distance measuring sensors are provided alternately. A robot control device according to any one of (1) to (9) above. (11) The robot has a convex portion and a concave portion, the tactile sensor is provided on the convex portion, and the distance measuring sensor is provided on the concave portion. A robot control device according to any one of (1) to (10) above. (12) The protrusion is made of a flexible material that deforms upon contact with the measurement object. The robot control device according to (11) above. (13) The recognition unit estimates a shape between the measurement points of the measurement object measured by the distance measuring sensor and the measurement points of the measurement object measured by the tactile sensor by interpolation. A robot control device according to any one of (1) to (12) above. (14) The recognition unit estimates the shape of the measurement target in a state where the measurement points are weighted so that the weighting of the measurement points by the tactile sensor is greater than the weighting of the measurement points by the distance measuring sensor. A robot control device according to any one of (1) to (13) above. (15) The recognition unit estimates a shape of the measurement object based on a probability distribution of the measurement points calculated based on the measurement points measured by the distance measuring sensor and the measurement points measured by the tactile sensor. A robot control device according to any one of (1) to (14) above. (16) The recognition unit estimates an overall shape of the measurement object based on partial observation information of the measurement object obtained by the distance measuring sensor and the tactile sensor. A robot control device according to any one of (1) to (15) above. (17) The recognition unit performs pattern matching between information on a plurality of template objects prepared in advance and partial observation information of the measurement object obtained by the distance measuring sensor and the tactile sensor, thereby estimating the overall shape of the measurement object. A robot control device according to any one of (1) to (16) above. (18) The recognition unit estimates the overall shape and position and orientation of the measurement object by performing pattern matching between prior information of the measurement object and partial observation information of the measurement object obtained by the distance measuring sensor and the tactile sensor. A robot control device according to any one of (1) to (16) above. (19) Recognizing the object to be measured based on distance information of the object to be measured measured by a distance measuring sensor provided on the robot and tactile information of the object to be measured measured by a tactile sensor provided on the robot; Controlling the operation of the robot based on the recognized information of the measurement target. Includes A method for controlling a robot. (20) A robot provided with a distance measuring sensor and a tactile sensor; a robot control device that controls the robot; Including, The robot control device includes: a recognition unit that recognizes the measurement target based on distance information of the measurement target measured by the distance measuring sensor provided on the robot and tactile information of the measurement target measured by the tactile sensor provided on the robot; a control unit that controls the operation of the robot based on the information of the measurement target recognized by the recognition unit; Equipped Robot system. [Explanation of symbols]

[0164] 1...humanoid robot, 2...torso, 3...arm, 4...cart (wheels), 5...camera (head sensor), 10...hand (robot hand), 11...finger, 11A...finger (first finger), 11B...finger (second finger), 21...distance measurement sensor, 22...tactile sensor, 23...sound sensor (microphone), 31...concave portion, 32...convex portion, 40...legged robot, 41...leg, 50...object (measurement target), 51...object (template object), 52...area (surrounded by friction cone), 60...drone, 100...object and environment measurement unit (sensor, sensor signal acquisition processing unit), 101...distance information processing unit, 102...contact information processing unit, 103...environment / object shape / posture estimation unit, 104...additional measurement point identification unit, 105...environment / object information storage unit, 110...object / environment recognition unit (recognition unit), 120...movement control unit (control unit), 200...movement form determination unit, 201...trajectory calculation unit, 202...motion control unit, 203...actuator unit, 300...movement form determination unit, 301...movement form storage unit, 302...position and posture correction unit, Pt...tactile sensor measurement point, Pr...distance sensor measurement point, Pa...grasping point candidate point, Pb...contact point.

Claims

1. a recognition unit that recognizes the measurement object based on distance information of the measurement object measured by a distance measuring sensor provided in the robot and tactile information of the measurement object measured by a tactile sensor provided in the robot; a control unit that controls the operation of the robot based on the information of the measurement target recognized by the recognition unit; Equipped Robot control device.

2. The recognition unit recognizes the shape and the position and orientation of the measurement target. The robot control device according to claim 1 .

3. The control unit controls at least one of the motion form of the robot and the position and posture of the robot as the motion control of the robot. The robot control device according to claim 1 .

4. The robot is configured to be capable of grasping an object as the measurement target, The control unit controls at least one of a gripping form, a gripping position, and a gripping attitude of the measurement target by the robot as the operation control of the robot. The robot control device according to claim 1 .

5. The robot is configured so that at least a part of the robot is movable, The control unit controls at least one of a moving speed and a trajectory of at least a part of the robot as the motion control of the robot. The robot control device according to claim 1 .

6. The control unit performs motion control of the robot based on the distance information measured before contacting the measurement object, and then modifies the motion control of the robot based on the tactile information measured after contacting the measurement object. The robot control device according to claim 1 .

7. the robot is configured such that at least a portion of the robot in which the distance measuring sensor or the tactile sensor is provided is movable; The control unit controls the operation of at least a part of a region where the distance measuring sensor or the tactile sensor is provided so that the measurement point of the distance measuring sensor or the measurement point of the tactile sensor is located at a location where the measurement of the measurement object by the distance measuring sensor or the tactile sensor is insufficient. The robot control device according to claim 1 .

8. The robot is configured to be capable of grasping an object as the measurement target, The control unit determines a gripping point taking into consideration gripping stability from among a plurality of gripping point candidate points based on the information of the measurement target recognized by the recognition unit, and controls the operation of the robot so that the object as the measurement target is gripped at the determined gripping point. The robot control device according to claim 1 .

9. In the robot, the tactile sensor is provided in a portion that frequently comes into contact with the measurement object, and the distance measuring sensor is provided in a portion that frequently comes into contact with the measurement object. The robot control device according to claim 1 .

10. In the robot, the tactile sensors and the distance measuring sensors are provided alternately. The robot control device according to claim 1 .

11. The robot has a convex portion and a concave portion, the tactile sensor is provided on the convex portion, and the distance measuring sensor is provided on the concave portion. The robot control device according to claim 1 .

12. The protrusion is made of a flexible material that deforms upon contact with the measurement object. The robot control device according to claim 11.

13. The recognition unit estimates a shape between the measurement points of the measurement object measured by the distance measuring sensor and the measurement points of the measurement object measured by the tactile sensor by interpolation. The robot control device according to claim 1 .

14. The recognition unit estimates the shape of the measurement target in a state where the measurement points are weighted so that the weighting of the measurement points by the tactile sensor is greater than the weighting of the measurement points by the distance measuring sensor. The robot control device according to claim 1 .

15. The recognition unit estimates a shape of the measurement object based on a probability distribution of the measurement points calculated based on the measurement points measured by the distance measuring sensor and the measurement points measured by the tactile sensor. The robot control device according to claim 1 .

16. The recognition unit estimates an overall shape of the measurement object based on partial observation information of the measurement object obtained by the distance measuring sensor and the tactile sensor. The robot control device according to claim 1 .

17. The recognition unit performs pattern matching between information on a plurality of template objects prepared in advance and partial observation information of the measurement object obtained by the distance measuring sensor and the tactile sensor, thereby estimating the overall shape of the measurement object. The robot control device according to claim 1 .

18. The recognition unit estimates the overall shape and position and orientation of the measurement object by performing pattern matching between prior information of the measurement object and partial observation information of the measurement object obtained by the distance measuring sensor and the tactile sensor. The robot control device according to claim 1 .

19. Recognizing the object to be measured based on distance information of the object to be measured measured by a distance measuring sensor provided on the robot and tactile information of the object to be measured measured by a tactile sensor provided on the robot; Controlling the operation of the robot based on the recognized information of the measurement target. Includes A method for controlling a robot.

20. A robot provided with a distance measuring sensor and a tactile sensor; a robot control device that controls the robot; Including, The robot control device includes: a recognition unit that recognizes the measurement target based on distance information of the measurement target measured by the distance measuring sensor provided on the robot and tactile information of the measurement target measured by the tactile sensor provided on the robot; a control unit that controls the operation of the robot based on the information of the measurement target recognized by the recognition unit; Equipped Robot system.

Images