Object contact state estimation system, object contact state estimation device, object contact state estimation method, and program

Abstract

To provide an object contact state estimation system, an object contact state estimation device, an object contact state estimation method, and a program that can precisely estimate the posture of an object to be operated.SOLUTION: An object contact state estimation system comprises: an end effector which can be operated for an object to be operated; an environment information acquisition part which acquires environment information as information representing the environment where the object to be operated is present; a measurement part which measures a vibration transmission value of the object to be operated when the object to be operated and the end effector come into contact or the object to be operated and the environment come into contact; and an estimation part which estimates the contact state of the object to be operated based upon the environment information and vibration transmission value.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to an object contact state estimation system, an object contact state estimation device, an object contact state estimation method, and a program. [Background technology]

[0002] Remote control systems for remotely controlling robots are being developed. For example, in the technology described in Patent Document 1, a robot is photographed from various directions by multiple cameras, and the images are displayed on multiple monitors, and an operator remotely controls the robot while viewing the multiple monitors. In addition, in the technology described in Patent Document 1, operation information regarding the operation of the robot arm during work and camera selection information are recorded. and stored as automatic switching information, and the operation information is position information of the tip of the robot arm, posture information of the robot arm, or a combination of position information and posture information. In the technology described in Patent Document 1, the captured image displayed on the monitor is automatically switched by sending the captured image from a camera selected from multiple cameras based on the automatic switching information to the monitor.

[0003] In addition, in the case of remote control of a robot, the robot generally performs a task on an object to be controlled based on a remote control instruction. In such a case, if the posture of the object to be controlled is not known, a process of estimating the posture of the object to be controlled has been performed using, for example, polygons (see, for example, Patent Document 2). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 6843051 [Patent Document 2] Japanese Patent Application Publication No. 2023-131029 Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the conventional technology, when estimating the orientation of an object, errors occur in estimating the orientation of the object because polygons are separated from each other or penetrate each other.

[0006] The present invention has been made in consideration of the above-mentioned problems, and aims to provide an object contact state estimation system, an object contact state estimation device, an object contact state estimation method, and a program that can accurately estimate the posture of an object to be operated. [Means for solving the problem]

[0007] (1) In order to achieve the above object, an object contact state estimation system according to one embodiment of the present invention is an object contact state estimation system including: an end effector operable with respect to a target object to be manipulated; an environmental information acquisition unit that acquires environmental information that is information indicating the environment in which the target object to be manipulated exists; a measurement unit that measures a vibration transmission value of the target object to be manipulated when the target object to be manipulated comes into contact with the end effector or when the target object to be manipulated comes into contact with the environment; and an estimation unit that estimates the contact state of the target object to be manipulated based on the environmental information and the vibration transmission value.

[0008] (2) The object contact state estimation system according to one aspect of (1) above may further include a control unit that generates a control command for the end effector based on the information obtained by the estimation unit.

[0009] (3) In the object contact state estimation system according to one aspect of (1) or (2), the end effector may include a vibrator, a first vibrator disposed on the pad side of the index fingertip at an angle α from the center of the spherical shape of the index fingertip, and a second vibrator disposed on the pad side of the cylindrical shape of the index fingertip, and the measuring unit may be configured such that the first measuring unit is disposed on the pad side of the index fingertip at an angle β from the center of the spherical shape of the thumb, and the second measuring unit is disposed on the cylindrical side closer to the pad side.

[0010] (4) In the object contact state estimation system according to one aspect of (3) above, the estimation unit may determine whether or not contact has occurred by calculating, for a predetermined time period during which vibration has been applied, changes in the spectral ratio and phase difference between the spectrum of a vibration waveform that is the vibration transmission value measured by the measurement unit and the spectrum of a vibration waveform that is the vibration transmission value input to the measurement unit.

[0011] (5) In the object contact state estimation system according to any one of the above (1) to (4), the estimation unit may include a contact state prediction model that classifies the hardness of the object to be operated based on the environmental information and the vibration transmission value, and outputs the contact state between the object to be operated and the end effector or the contact state between the object to be operated and the environment, and an optical flow value; and a weight adjustment model that receives as input the optical flow value, the probability of likelihood when classifying the hardness of the object to be operated, and the probability of likelihood when classifying the contact state between the object to be operated and the end effector or the contact state between the object to be operated and the environment, and outputs weights of the mesh penetration amounts of the two objects.

[0012] (6) In the object contact state estimation system according to one aspect of (5) above, the environmental information may be image data captured by a plurality of image capture devices, the environmental information acquisition unit may acquire detection values ​​detected by sensors provided in the end effector, the contact state prediction model may input the image data to an encoder, convolve the detection values ​​acquired from the sensors using a first network, convolve the measured vibration transmission values ​​using a second network, concatenate the output of the encoder, the result of the convolution process using the first network, and the result of the convolution process using the second network to form a multimodal representation, input the multimodal representation and the output of the encoder to a decoder to output the optical flow value, input the multimodal representation to a third network to output the hardness classification result, and input the multimodal representation to a fourth network to output the contact state prediction result.

[0013] (7) In the object contact state estimation system according to one aspect of (5) above, the weight adjustment model may be configured to create teaching data in which the penetration amount is adjusted based on a simulation, and to learn how to determine the weights through reinforcement learning.

[0014] (8) In order to achieve the above object, an object contact state estimation device according to one embodiment of the present invention is an object contact state estimation device that estimates the attitude of a target object relative to the target object by operating an end effector, and includes: an acquisition unit that acquires environmental information that is information indicating the environment in which the target object exists, and acquires a vibration transmission value of the target object when the target object comes into contact with the end effector or when the target object comes into contact with the environment; and an estimation unit that estimates the contact state of the target object based on the environmental information and the vibration transmission value.

[0015] (9) In order to achieve the above object, an object contact state estimation method according to one aspect of the present invention is a method for estimating the posture of a target object to be operated by an object contact state estimation device in which an acquisition unit operates an end effector to control the target object, the method acquiring environmental information that is information indicating the environment in which the target object to be operated exists, and acquiring a vibration transmission value of the target object to be operated when the target object to be operated comes into contact with the end effector or when the target object to be operated comes into contact with the environment, and an estimation unit estimating the contact state of the target object to be operated based on the environmental information and the vibration transmission value.

[0016] (10) In order to achieve the above object, a program according to one aspect of the present invention is a program in which an acquisition unit causes a computer of an object contact state estimation device that estimates the posture of a target object by using an object contact state estimation device that operates an end effector to control the target object, to acquire environmental information that is information indicating the environment in which the target object exists, acquire a vibration transmission value of the target object when the target object comes into contact with the end effector or when the target object comes into contact with the environment, and estimate the contact state of the target object based on the environmental information and the vibration transmission value. [Effects of the Invention]

[0017] According to the above (1) to (10), the orientation of the object to be operated can be estimated with high accuracy. [Brief explanation of the drawings]

[0018] [Figure 1] FIG. 2 is a side view of the end effector according to the embodiment. [Figure 2] 3A and 3B are diagrams illustrating examples of mounting positions of vibrators according to an embodiment. [Figure 3] 3A and 3B are diagrams illustrating examples of mounting positions of piezoelectric element sensors according to an embodiment. [Figure 4] FIG. 10 is a diagram showing an example of how the imaging device is attached. [Figure 5] FIG. 1 is a diagram illustrating an example of the configuration of an object contact state estimation system according to an embodiment. [Figure 6] FIG. 10 is a diagram showing an example of a contact pattern between two objects. [Figure 7] 10A and 10B are diagrams for explaining a method for detecting a contact state according to an embodiment. [Figure 8] FIG. 10 is a sequence diagram of an example of a processing procedure of the object contact state estimating device according to the embodiment. [Figure 9] FIG. 10 is a diagram for explaining processing of a contact state prediction model. [Figure 10] 10A and 10B are diagrams for explaining learning of weight adjustment using a weight adjustment model according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0019] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the drawings used in the following description, the scale of each component is appropriately changed so that each component can be recognized. In all the drawings for explaining the embodiments, the same reference numerals are used for components having the same functions, and repeated explanations will be omitted. Furthermore, in this application, "based on XX" means "based on at least XX," and includes cases where it is based on other elements in addition to XX. Furthermore, "based on XX" is not limited to cases where XX is used directly, but also includes cases where it is based on XX that has been calculated or processed. "XX" is any element (for example, any information).

[0020] (Summary) When using multiple image capture devices, there is a technology that uses graph theory, for example, to align the contours of images captured by the multiple image capture devices. Furthermore, in robotic tasks, the number of objects to be manipulated is not limited to one, but can be two or more. For example, when tightening a screw into a base with a screwdriver, the smallest objects to be manipulated are the screw and the screwdriver. In estimating the object pose when two objects are in contact, solutions such as non-contact or penetration may be estimated. However, it has been difficult to determine whether two objects are in contact based on captured images alone.

[0021] In the conventional object pose estimation method that matches the contours of a simulated object model to the contours of an object on an RGB image, pose estimation is performed based on constraints on contact between the hand polygon and the object polygon. With this conventional method, it is not possible to determine when contact will occur, and for what object the contact constraints should be applied. Another conventional technology proposed is a system that forcibly vibrates the shaft to indirectly detect the application of external force to the forceps of a surgical robot system. While this conventional technology can detect the application of external force through active vibration, it is difficult to identify the type of contact.

[0022] In contrast, in this embodiment, the type of contact of the held object is identified based on the image of a camera installed near the object and the results of actively vibrating the held object, and the weight of the polygon penetration amount constraint is changed according to the results. As a result, in this embodiment, not only can the applicability of constraint conditions be automated, but the degree and direction of penetration between object polygons can also be adjusted.

[0023] [Example of end effector configuration] Fig. 1 is a side view of the end effector of this embodiment. As shown in Fig. 1, the end effector 22 includes two or more fingers 23. A vibrator 24 is attached to a second finger 232 (for example, the pad of the index finger), a piezoelectric sensor 25 (measuring unit) is attached to a first finger 231 (for example, the pad side of the thumb), and an image capture device 26 (environmental information acquisition unit) is attached to the inside of the palm, for example.

[0024] 2 is a diagram showing an example of the mounting position of the vibrator according to this embodiment, in which the length direction of an extended finger is defined as the x-axis direction, and the thickness direction of the finger is defined as the y-axis direction. The finger of the robot 2 has a fingertip shape in which the part corresponding to the tip of the DIP joint (distal interphalangeal joint) of a human fingertip is made up of a cylinder g12 and a hemisphere g11. For example, two vibrators 24 are attached to the fingertip. The first vibrator 24a is placed on the finger pad side g15 at an angle α from the center of the spherical shape g11 of the index fingertip, and the second vibrator 24b is placed on the finger pad side g16 of the cylindrical shape g12. These two vibrators 24 are vibrated to shake the target object in contact with the fingertip. 2, other sensors include, for example, joint sensors, imaging devices, etc. The attachment positions shown in FIG. 2 are merely examples and are not limiting.

[0025] 3 is a diagram showing an example of the mounting position of the piezoelectric element sensor according to this embodiment, in which the length direction of an extended finger is defined as the x-axis direction, and the thickness direction of the finger is defined as the y-axis direction. The thumb of the robot 2 has a fingertip shape in which the part corresponding to the tip of the DIP joint (distal interphalangeal joint) of a human fingertip is made up of a cylinder g22 and a hemisphere g21. For example, two piezoelectric element sensors 25 are attached to the fingertip. The first piezoelectric element sensor 25a is arranged on the finger pad side g25 at an angle β when viewed from the center of the spherical shape g21 of the thumb tip, and the second piezoelectric element sensor 25b is arranged on the finger pad side g26 of the cylindrical side g22, for example. In addition, in Fig. 3, other sensors are, for example, joint sensors, imaging devices, etc. Furthermore, the attachment positions shown in Fig. 3 are just an example and are not limited to this.

[0026] Although an example in which there are two vibrators 24 in Figure 2 and two piezoelectric element sensors 25 in Figure 3 has been described, the number of each is not limited to this, and may be one of each, or three or more, and the number of vibrators 24 and piezoelectric element sensors 25 may be different. Furthermore, the vibrator 24 and the piezoelectric sensor 25 are preferably installed at positions where they can easily apply force to the target object. Moreover, the angle α and the angle β in FIG. 2 may be the same or different.

[0027] FIG. 4 is a diagram showing an example of how the imaging device is attached. Finger portion 231 corresponds to, for example, a human thumb. Finger portion 232 corresponds to, for example, a human index finger. Finger portion 233 corresponds to, for example, a human middle finger or ring finger. Finger portion 244 corresponds to, for example, a human ring finger or little finger. Each finger portion has a joint and a node. Base body 30 includes portions corresponding to the back and palm of a human hand. Camera 261 is installed at a position corresponding to the ball of a human's thumb, for example. Camera 261 may also be installed at a position corresponding to the outer side of the proximal joint of a human's thumb, not on the index finger side. In this way, camera 261 is installed between finger portion 231 and finger portion 234. Camera 262 is installed at a position corresponding to the ball of a human's foot, for example. Camera 262 may also be installed at a position corresponding to the index finger side of the proximal joint of a human's thumb. In this way, camera 262 is installed between finger portion 231 and finger portion 222. The camera 263 is installed, for example, at a position corresponding to the side of the thumb at the base of the four human fingers, or the side of the thumb and index finger at the thenar eminence. In this way, the camera 263 is installed between the finger portion 231 and the finger portion 232. Camera 264 is installed at a position corresponding to the side of the base of the little finger on the little finger side or the ring finger side, or the side of the hypothenar region of a human being. In this way, camera 264 is installed between finger portion 234 and finger portion 231. 4 are merely examples and are not limited to these. For details of the installation positions and imaging ranges, see, for example, Japanese Patent Application Laid-Open No. 2023-105585.

[0028] [Configuration example of object contact state estimation system] 5 is a diagram showing an example of the configuration of an object contact state estimation system according to this embodiment. As shown in FIG. 5, the object contact state estimation system 1 includes, for example, a robot 2, an object contact state estimation device 4, and an operation input unit 6. The robot 2 includes, for example, an arm 21, an end effector 22, a vibrator 24, a piezoelectric sensor 25, an imaging device 26, a sensor 27, an actuator 28, and a communication unit 29. The object contact state estimating device 4 includes, for example, a communication unit 41 (acquisition unit), a prediction unit 42 (estimation unit), an estimation unit 43, a control unit 44, and a storage unit 45.

[0029] The operation input unit 6 is a controller that allows the operator to remotely control the robot 2. The operation input unit 6 has multiple buttons, such as a data glove that the operator wears on his / her hand or a controller for a game console.

[0030] (robot) The robot 2 includes at least one end effector 22, and may include two or more end effectors 22. The robot 2 may also include a body, a base, a cart, etc. The robot 2 also includes a power source and the like (not shown). The robot 2 operates in accordance with the object contact state estimation device 4. The robot 2 and the object contact state estimation device 4 are connected to each other via a wired or wireless network.

[0031] The arm 21 is a mechanism for moving the end effector 22 and has a joint, to which a sensor 27 and an actuator 28 are attached. One end of the arm 21 is connected to the end effector 22 via a joint, and the other end is connected to a body, a base, or the like via a joint.

[0032] The end effector 22 has at least two fingers 23 (23-1, 23-2, ...). Each finger 23 has a joint, and a sensor 27 and an actuator 28 are attached to the joint.

[0033] The vibrator 24 vibrates in response to a vibration command from the object contact state estimation device 4. The vibrator 24 is, for example, an eccentric motor. The vibration conditions are, for example, when the fingertips of the index finger and thumb are in contact with the target object, at a vibration frequency that does not interfere with the human motion spectrum, and for a predetermined time that does not interfere with human motion.

[0034] The piezoelectric sensor 25 detects the vibration caused by the vibrator 24 .

[0035] The image capturing device 26 is, for example, an RGB image capturing device. As will be described later, the number of image capturing devices 26 may be plural.

[0036] The sensor 27 is an encoder, a torque sensor, etc. attached to each joint of the arm and the end effector.

[0037] The actuators 28 are attached to the joints of the arm and the end effector, respectively. The actuators 28 operate in response to operation instructions from the object contact state estimation device 4.

[0038] The communication unit 29 acquires vibration instructions, operation instructions, etc. from the object contact state estimation device 4. The communication unit 29 outputs the first detection value detected by the piezoelectric element sensor 25 and the second detection value detected by the sensor 27 to the object contact state estimation device 4.

[0039] (Object contact state estimation device) The object contact state estimation device 4 estimates the posture of the operation target object based on a first detection value acquired from the robot 2. The object contact state estimation device 4 controls the operation of the robot 2 based on the estimated posture of the operation target object, operation input information acquired from the operation input unit 6, and the first detection value and second detection value acquired from the robot 2. The object contact state estimation device 4 and the operation input unit 6 are connected to each other via a wired or wireless network.

[0040] The communication unit 41 acquires a second detection value from the sensor 27. The communication unit 41 acquires captured image data from the imaging device 26. The communication unit 41 acquires a first detection value from the piezoelectric element sensor 25. The communication unit 41 outputs a vibration instruction to the robot 2. The communication unit 41 outputs an operation instruction to the robot 2.

[0041] The prediction unit 42 includes a contact state prediction model 421 and a weight adjustment model 422 . The prediction unit 42 inputs the first detection value, the second detection value, and the image data acquired by the communication unit 41 into the trained contact state prediction model 421, and outputs a result of classifying the hardness of the target object and a contact prediction result indicating whether or not there is contact. The prediction unit 42 performs image processing, which will be described later, using the image data, the first detection value, and the second detection value. The prediction unit 42 inputs the probability (cost) of the likelihood of classifying the target object's hardness, the probability (cost) t of the likelihood of contact prediction indicating whether or not there is contact, and the optical flow value into the trained weight adjustment model 422 to adjust the weight, i.e., calculate the weight W of the mesh penetration cost. Note that optical flow is data that detects the movement of an object between frames and displays its velocity as a vector. Note that the use of this weight W means that when polygon data is generated using image data captured by multiple image capture devices 26, the generated data allows for the first object to penetrate (be embedded into) the second object.

[0042] The estimation unit 43 performs well-known image processing or the like on image data captured by each of the n (n is an integer from 1 to N) image capture devices 26, and estimates the initial values ​​(1, . . . , N) of the orientation of the target object. The estimation unit 43 performs global optimization (minimization of contour and weight W × polygon penetration) on the initial value of the estimated object posture using the mesh penetration cost weight W, and performs 6D object posture estimation by matching the contours of multiple objects. For details on 6D object posture estimation, see, for example, Reference 1.

[0043] Reference 1: Yuto Harada, Tatsuya Aoki, et al., "6D Pose Estimation of Objects from Multiple Viewpoints in Virtual Space," Proceedings of the 2021 National Conference of the Japanese Society for Artificial Intelligence, p.2J1GS8a04-2J1GS8a04, 2021

[0044] The control unit 44 generates an operation instruction based on the estimation result obtained by the estimation unit 43 and the operation input information. The control unit 44 determines whether or not the target object has been grasped by using the second detection value of the sensor 27. After grasping the target object, the control unit 44 generates a vibration instruction to vibrate the target object for a predetermined period of time.

[0045] The storage unit 45 stores programs, mathematical expressions, threshold values, predetermined values, identification information for identifying the robot 2, identification information for identifying the operation input unit 6, and the like, which are required for controlling each unit.

[0046] [Method for detecting contact state using vibration] FIG. 6 is a diagram showing an example of a contact pattern between two objects. The image g110 is an example of a state in which a first object g101 is in contact with a hard second object g111. The image g120 is an example of a state in which a first object g101 is brought into contact with a hard second object g121 and moved to the right side of the paper. The image g130 is an example of a state in which the first object g101 is brought into contact with a soft second object g131, causing the second object g131 to slightly dent. As such, there are several patterns for the contact state between two objects.

[0047] FIG. 7 is a diagram for explaining a contact state detection method according to this embodiment. The image indicated by the reference symbol g160 is an example of a state in which a first object g151 (e.g., a screwdriver) is not in contact with a second object g152 (e.g., a screw). In this case, when the end effector 22 grips the first object g151, vibration is transmitted to the first object g151 by excitation by the vibrator 24 (g161). However, because the first object g151 is not in contact with the second object g152, the vibration detected by the piezoelectric element sensor 25 has the same spectrum as the input vibration signal.

[0048] The image indicated by reference symbol g170 is an example of a state in which a first object g151 is in contact with a second object g152. In this case, when the end effector 22 is gripping the first object g151, vibration is transmitted to the first object g151 by excitation by the vibrator 24 (g171). Furthermore, because the first object g151 is in contact with the second object g152, the vibration detected by the piezoelectric element sensor 25 is different from the input vibration, as indicated by reference symbol g172.

[0049] For this reason, in this embodiment, for example, whether or not contact is occurring is determined by calculating the change in the spectral ratio and phase difference between the spectrum of the vibration waveform acquired from the piezoelectric element sensor 25 and the spectrum of the vibration waveform input to the vibrator 24 for a predetermined period of time during which vibration is applied.

[0050] [Processing procedure of the object contact state estimation device] FIG. 8 is a sequence diagram of an example of a processing procedure of the object contact state estimating device according to this embodiment.

[0051] The communication unit 41 acquires a detection value from the encoder (sensor 27) of the robot 2 (step S1). The prediction unit 42 extracts (crops) information at a specific time from the acquired detection value (step S2).

[0052] The communication unit 41 acquires image data from the camera 26 of the robot 2 (step S3). The prediction unit 42 performs well-known image processing (for example, binarization, feature extraction, clustering, contour detection, etc.) on the acquired image data to detect the target object (step S4). Note that by using the image data, the direction in which the object is moving can also be determined.

[0053] The control unit 44 vibrates the vibrator 24 for a predetermined time. The communication unit 41 acquires a piezoelectric element detection value (first detection value) from the piezoelectric element sensor 25 of the robot 2 (step S5). The prediction unit 42 converts the acquired piezoelectric element detection value into spectrum data in the frequency domain (step S6).

[0054] The prediction unit 42 inputs the time crop, the object detection result, and the data converted into the frequency domain into a contact state prediction model 421, predicts the hardness classification result (g201) of the target object and the contact state prediction result (g202) (for example, "1" for contact or "0" for no contact), and calculates and outputs the optical flow (g203) (step S7). In addition, hardness is classified into, for example, metallic hardness, hard rubber-like hardness, softness, etc.

[0055] The prediction unit 42 inputs the probability indicating the likelihood of hardness classification, the probability indicating the likelihood of contact state prediction, and the optical flow value into the weight adjustment model 422, and adjusts the weight W for each, i.e., calculates the weight W for the mesh penetration cost (step S8).

[0056] The estimation unit 43 performs well-known image processing on the image data captured by each of the n image capturing devices 26, and estimates the initial values ​​(1, . . . , N) (g204) of the orientation of the target object (step S9).

[0057] The estimation unit 43 performs global optimization on the initial values ​​of the estimated object posture using the mesh penetration cost weight W, and performs 6D object posture estimation by matching the contours of multiple objects (step S10). That is, by this process, when matching the contours of image data captured by multiple image capture devices 26, the polygon meshes of the two objects are matched by optimization.

[0058] The estimation unit 43 outputs an object posture estimation result indicating the estimated object posture (step S11). Then, the control unit 44 generates a grasping plan using the estimated object posture estimation result and the operation input information, generates an operation instruction, and controls the operation of the robot 2.

[0059] Here, an example of the configuration and processing of the contact state prediction model 421 will be described. FIG. 9 is a diagram for explaining the processing of the contact state prediction model. The image data is input to the trained encoder g310. Data of a specific time width is extracted from the detection values ​​of the piezoelectric element sensor 25 and the joint torque detected by the sensor 27, and convolution processing g302 (first network) is performed on the time series data at multiple resolutions. The detected value of the encoder (sensor 27) is input to, for example, a CNN (Convolutional Neural Network) g303 (second network), and convolution processing is performed by the CNN. The output of the encoder g301, convolutional g302, and CNN g303 is a compressed one-dimensional feature vector. The concatenation of these data forms a multimodal representation g304.

[0060] The multimodal representation g304 and the output of the encoder g301 are input to a trained decoder g305, which decodes the input and outputs optical flow values. The encoder g301 and decoder g305 are connected via a skip connection. The multimodal representation g304 is input to a trained network g306 (third network), which outputs a stiffness classification result. The multimodal representation g304 is input to a trained network g307 (fourth network), which outputs a contact state prediction result (for example, whether there is contact or not).

[0061] Next, learning of weight adjustment by weight adjustment model 422 will be described. FIG. 10 is a diagram for explaining learning of weight adjustment using the weight adjustment model according to this embodiment. The optical flow value, the probability of the likelihood of the hardness classification, and the probability of the likelihood of the contact classification are input to the trained weight adjustment model 422. Then, the weight adjustment model 422 outputs a weight W of the mesh penetration cost (the penetration function is, for example, the signed distance between two objects touching (penetrating) and the signed distance between two objects). The weight adjustment model 422 is trained, for example, by creating teaching data in which the penetration amount is adjusted based on a simulation, and then learning how to determine the weights through reinforcement learning.

[0062] In this way, in this embodiment, the weight adjustment model 422 is trained based on the teaching data of whether or not the object is in contact, the optical flow, and the amount of depression that the object makes in the target object at that time (the amount of penetration from the initial state). As a result, this embodiment makes it possible to appropriately predict the contact state of two objects.

[0063] As described above, in this embodiment, the type of contact of the held object is identified from the image of a camera installed near the object and the result of actively vibrating the held object, and the weight of the polygon penetration amount constraint is changed depending on the result, for example, to soft object or deformation direction. Note that the object contact state estimation device 4 detects contact between objects and their movement direction from the spectrum of the applied vibration pattern and the vibration pattern acquired from the piezoelectric element sensor 25, multiple object polygons in the image, and indirect torque. The contact state between two objects can be predicted appropriately.

[0064] For this reason, in this embodiment, a vibrator 24 and a piezoelectric sensor 25 are newly provided on the hand to detect whether the object to be grasped is in contact with another object. In this embodiment, contact between objects and their movement direction are detected from the spectrum of the applied vibration signal pattern and the received vibration signal pattern, the polygons of multiple objects in the image, and the indirect torque. As a result, according to this embodiment, it is possible to adjust the presence or absence of constraints and their weights based on the detected information.

[0065] In addition, in this embodiment, the type of contact of the grasped object is identified from the image of a camera installed near the object and the results of actively vibrating the grasped object, and the weight of the polygon penetration amount constraint is changed depending on the results (for example, for soft objects, deformation direction). As a result, according to this embodiment, not only can it be automated to determine whether or not constraint conditions are to be applied, but it can also adjust the degree and direction of penetration between object polygons.

[0066] In the above example, the first object and the second object are described as examples of two objects, but the second object may be, for example, the environment (wall, floor, table, etc.). Even in such a case, whether or not the first object is in contact with the environment (second object) can be estimated based on the result of measurement by the piezoelectric element sensor 25 after excitation by the vibrator 24.

[0067] Note that a program for realizing all or part of the functions of the object contact state estimation device 4 of the present invention may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be loaded into a computer system and executed to perform all or part of the processing performed by the object contact state estimation device 4. Note that the term "computer system" as used herein includes hardware such as an OS and peripheral devices. The term "computer system" also includes a WWW system equipped with a homepage provision environment (or display environment). The term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into computer systems. The term "computer-readable recording medium" also refers to devices that retain a program for a certain period of time, such as volatile memory (RAM) within a computer system that serves as a server or client when the program is transmitted via a network such as the Internet or a communication line such as a telephone line. Alternatively, some or all of these components may be realized by hardware (including circuitry) using LSI (Large Scale Integration) such as ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), GPU (Graphics Processing Unit), or SOC (System On Chip), or may be realized by a combination of software and hardware.

[0068] The program may also be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by transmission waves in the transmission medium. Here, the "transmission medium" that transmits the program refers to a medium that has the function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. The program may also be a program that realizes part of the above-mentioned functions. Furthermore, the program may be a so-called differential file (differential program) that can realize the above-mentioned functions in combination with a program already recorded in the computer system.

[0069] The above describes the form for carrying out the present invention using an embodiment, but the present invention is not limited to such an embodiment, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention. [Explanation of symbols]

[0070] 1...object contact state estimation system, 2...robot, 4...object contact state estimation device, 6...operation input unit 6, 21...arm, 22...end effector, 23, 23-1, 23-2, ...finger portion, 24...vibrator, 25...piezoelectric element sensor, 26...imaging device, 27...sensor, 28...actuator, 29...communication unit, 41...communication unit, 42...prediction unit, 43...estimation unit, 44...control unit, 45...storage unit

Claims

1. an end effector operable with respect to an object to be manipulated; an environment information acquisition unit that acquires environment information that indicates an environment in which the operation target object exists; a measurement unit that measures a vibration transmission value of the object to be manipulated when the object to be manipulated comes into contact with the end effector or when the object to be manipulated comes into contact with the environment; an estimation unit that estimates a contact state of the operation target object based on the environmental information and the vibration transmission value; An object contact state estimation system comprising:

2. a control unit that generates a control command for the end effector based on the information obtained by the estimation unit; The object contact state estimation system according to claim 1 , further comprising:

3. (vibrator, piezoelectric sensor) the end effector comprises a vibrator; The first oscillator is disposed on the finger pad side at an angle α from the center of the spherical shape of the index fingertip, and the second oscillator is disposed on the cylindrical finger pad side of the index fingertip, The measuring unit has a first measuring unit disposed on the finger pad side at an angle β when viewed from the center of the spherical shape of the fingertip of the thumb, and a second measuring unit disposed on the cylindrical side closer to the finger pad side. The object contact state estimation system according to claim 1 or 2.

4. The estimation unit Whether or not there is contact is determined by calculating a change in the spectral ratio and phase difference between the spectrum of the vibration waveform, which is the vibration transmission value measured by the measurement unit, and the spectrum of the vibration waveform, which is the vibration transmission value input to the measurement unit, for a predetermined time period during which vibration is applied. The object contact state estimation system according to claim 3 .

5. The estimation unit a contact state prediction model that classifies the hardness of the object to be manipulated based on the environmental information and the vibration transmission value, and outputs a contact state between the object to be manipulated and the end effector or a contact state between the object to be manipulated and the environment, and an optical flow value; a weight adjustment model that receives as input the optical flow value, a probability of certainty when classifying the hardness of the object to be manipulated, and a probability of certainty when classifying the contact state between the object to be manipulated and the end effector or the contact state between the object to be manipulated and the environment, and outputs weights of mesh penetration amounts of the two objects; The object contact state estimation system according to claim 1 or 2, comprising:

6. the environmental information is image data captured by a plurality of image capture devices, the environmental information acquisition unit acquires a detection value detected by a sensor included in the end effector; The contact state prediction model is inputting the image data into an encoder, convolving the detected value acquired from the sensor with a first network, and convolving the measured vibration transmission value with a second network; concatenating the output of the encoder, the result of the convolution processing by the first network, and the result of the convolution processing by the second network to form a multimodal representation; inputting the multimodal representation and the output of the encoder to a decoder, which outputs the optical flow values; inputting the multimodal representation to a third network, which outputs the stiffness classification result; inputting the multimodal representation to a fourth network, which outputs a prediction result of the contact state; The object contact state estimation system according to claim 5 .

7. The weight adjustment model creates teaching data in which the penetration amount is adjusted based on a simulation, and learns how to determine the weights through reinforcement learning. The object contact state estimation system according to claim 5 .

8. An object contact state estimation device that estimates an attitude of a target object relative to the target object by operating an end effector, an acquisition unit that acquires environmental information that indicates an environment in which the object to be manipulated exists, and acquires a vibration transmission value of the object to be manipulated when the object to be manipulated comes into contact with the end effector or when the object to be manipulated comes into contact with the environment; an estimation unit that estimates a contact state of the operation target object based on the environmental information and the vibration transmission value; An object contact state estimation device comprising:

9. A method for estimating a posture of a target object by an object contact state estimation device in which an acquisition unit operates an end effector to control the target object, the method comprising: acquiring environmental information that indicates an environment in which the object to be manipulated exists, and acquiring a vibration transmission value of the object to be manipulated when the object to be manipulated comes into contact with the end effector or when the object to be manipulated comes into contact with the environment; an estimation unit estimating a contact state of the operation target object based on the environmental information and the vibration transmission value; Method for estimating object contact state.

10. a computer of an object contact state estimation device that estimates the posture of a target object by using the object contact state estimation device that operates an end effector to control the target object; acquiring environmental information that indicates an environment in which the operation target object exists, and acquiring a vibration transmission value of the operation target object when the operation target object comes into contact with the end effector or when the operation target object comes into contact with the environment; a contact state of the operation target object is estimated based on the environmental information and the vibration transmission value; program.

Images