Contact target estimation device, method, and program

Abstract

One aspect of the present invention first detects, from image information captured by imaging a work area, a plurality of objects in the image frame, and defines a first interference region for each of the plurality of objects detected. Next, for each of the plurality of objects, it is determined, on the basis of the first interference region, operation information of the operator, and a second interference region set for a robot that moves according to the operator's operation, whether or not there is an interference between the robot and the object. Subsequently, from the plurality of objects for which it has been determined that there is an interference, the object that the robot is going to contact is selected, on the basis of information indicating positional relationships between the objects and the robot.

Description

Technical Field

[0001] One aspect of the present invention relates to a contact target estimation device, method, and program used in a system for remotely operating a target object using, for example, a robot.

Background Art

[0002] In recent years, various systems for remotely operating an object at a remote location have been developed by operating a robot arranged at the remote location by an operator via a network. In this type of system, smoother operation can be expected by previously estimating information regarding the object to be operated.

[0003] Therefore, for example, in Non-Patent Document 1 or Non-Patent Document 2, a technique has been proposed in which eye line information of an operator is acquired using a head-mounted display (HMD), and an object that the robot on the remote side is about to grasp is previously estimated based on the acquired eye line information, thereby reducing the delay in the operation of the robot.

Prior Art Documents

Non-Patent Documents

[0004]

Non-Patent Document 1

Non-Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, in the technologies described in Non-Patent Document 1 or 2, since the visual line information of the operator is required, an HMD and a camera for transmitting video are separately required. For this reason, there is a problem that the system configuration is likely to become complicated.

[0006] In addition, in the technologies described in Non-Patent Document 1 or 2, based on the video captured by the camera installed on the robot side, the position of the object is measured, and an object likely to be the contact target of the operation is estimated. For this reason, when a plurality of objects are included in the video, it is necessary to measure the positions of all of these plurality of objects and perform a process of comprehensively estimating the object to be the contact target, and a lot of time is required for the process. As a result, it becomes difficult to maintain the real-time performance of the control process.

[0007] This invention has been made paying attention to the above circumstances, and aims to provide a technology that prevents the complication of the system configuration and can estimate the contact target in a short time even when there are a plurality of objects that may be contact candidates, thereby improving the real-time performance of the process.

Means for Solving the Problems

[0008] In order to solve the above problems, one aspect of the contact target estimation device or estimation method according to this invention is a system for remotely operating an object existing in a work area at a remote location by operating a robot at the remote location according to an operation of an operator. An information processing device operating as a contact target estimation device includes a first processing unit or process, a second processing unit or process, and a third processing unit or process.

[0009] The first processing unit or process detects a plurality of objects shown in the video frame from the video information obtained by imaging the work area, and defines a first interference area for each of the plurality of detected objects. The second processing unit or process determines, for each of the plurality of objects, whether there is interference with the robot based on the first interference area, the operation information of the operator, and a second interference area set for the robot that operates according to the operation of this operator. The third processing unit or process selects, from among the plurality of objects determined to have interference, the object that the robot is about to contact based on information representing the positional relationship between the object and the robot.

[0010] According to one aspect of the present invention, based on video information obtained by imaging a work area including a plurality of objects and operation information of a robot by an operator, an object that the robot is about to contact is estimated. For this reason, for example, it becomes possible to estimate a target to be contacted without using the operator's line-of-sight information, thereby eliminating the need for additional devices such as an HMD and avoiding complication of the system.

[0011] In addition, an interference area is defined for each object detected from the video information, and an object that may interfere with the robot is selected based on the interference area of this object and the interference area of the robot arm. Then, from among the selected objects, for example, the object with the highest possibility of contact by the robot arm is selected. Therefore, even when there are a plurality of objects that are candidates for contact in the work area, it becomes possible to estimate the object with the highest possibility of being the contact target in a short time, thereby maintaining the real-time nature of the processing.

Advantages of the Invention

[0012] That is, according to one aspect of the present invention, it is possible to provide a technique that prevents complication of the system configuration and enables estimation of the object to be contacted in a short time even when there are a plurality of candidates for contact, thereby improving the real-time nature of the processing.

Brief Description of the Drawings

[0013]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

[0014] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0015] [One Embodiment] [Configuration Example] (1) System Figure 1 is a diagram showing an example of a remote control system according to an embodiment of the present invention.

[0016] In a remote control system according to an embodiment, an operator US remotely controls a robot arm RB disposed at a work site or the like at a remote location via a network NW, so as to perform a desired operation on an object TL to be operated on, such as a part, a tool, a device, or the like.

[0017] At the remote location, in addition to the robot arm RB and a robot control device RU that controls the operation thereof, a camera CM and a contact target estimation device EU are disposed. Among these, the camera CM is composed of, for example, an RGB-D camera, and images an area including the on-site object TL group and the robot arm RB and outputs video information of the area.

[0018] On the other hand, on the operation side, an operation-side terminal device OU and a monitor MT are provided, and further, the operator US is equipped with a motion sensor SS that detects, for example, the movement of the hand. The monitor MT is used to display the video information captured by the camera CM at the remote location.

[0019] The motion sensor SS is composed of, for example, an infrared sensor, and detects the movement of the hand of the operator US as a change in the position of a predetermined part of the hand. For example, the motion sensor SS detects the position of the center of the hand of the operator US, the positions of the joints of each finger, the positions of each fingertip, and the position of the wrist as three-dimensional coordinate information, respectively. Further, the motion sensor SS detects whether the operator US is operating with the left or right hand and whether the front or back side of the hand is used for the operation. Here, the front side of the hand indicates the back side of the hand, and the back side of the hand indicates the palm side of the hand, respectively.

[0020] The operation-side terminal device OU has a function of receiving video information transmitted from the contact target estimation device EU at a remote location via the network NW and displaying it on the monitor MT, and a function of transmitting sensor information representing the movement of the hand of the operator US detected by the motion sensor SS to the contact target estimation device EU at a remote location via the network NW. The sensor information includes, in addition to the three-dimensional coordinate information indicating the change in the position of each part of the hand of the operator US, information indicating whether it is the left or right hand, and information indicating the front-back orientation of the hand.

[0021] (2) Contact target estimation device EU FIG. 2 and FIG. 3 are block diagrams showing an example of the hardware configuration and software configuration of the contact target estimation device EU, respectively.

[0022] The contact target estimation device EU is an information processing device such as a personal computer or a server computer, and includes a control unit 1 using hardware processors such as a central processing unit (CPU) and a graphics processing unit (GPU). The control unit 1 is connected, via a bus 7, to a storage unit having a program storage unit 2 and a data storage unit 3, a camera interface (hereinafter, the interface is abbreviated as I / F) unit 4, a communication I / F unit 5, and a robot I / F unit 6. Note that the control unit 1 may be configured using a programmable logic device (PLD), a field programmable gate array (FPGA), or the like.

[0023] The camera I / F unit 4 is used to capture video information output from the camera CM. The communication I / F unit 5 transmits the video information captured by the camera CM to the operation-side terminal device OU via the network NW in accordance with the communication protocol defined by the network NW, and receives the sensor information transmitted from the operation-side terminal device OU via the network NW. The robot I / F unit 6 outputs information representing the selection result of the contact target to the robot control device RU.

[0024] The program storage unit 2 is configured by combining, for example, a non-volatile memory such as an SSD (Solid State Drive) that can be written and read at any time as a storage medium and a non-volatile memory such as a ROM (Read Only Memory). In addition to middleware such as an OS (Operating System), it stores application programs necessary for executing various control processes according to one embodiment. Hereinafter, the OS and each application program are collectively referred to as a program.

[0025] The data storage unit 3 is, for example, a combination of a non-volatile memory such as an SSD that can be written and read at any time as a storage medium and a volatile memory such as a RAM (Random Access Memory). As the main storage area necessary for implementing one embodiment, it includes a video information storage unit 31, a hand information storage unit 32, and an object information storage unit 33.

[0026] The video information storage unit 31 temporarily stores the video information acquired from the camera CM for object detection processing described later.

[0027] The hand information storage unit 32 is used to store sensor information representing the movement of the hand of the operator US detected by the motion sensor SS and the velocity vector of the hand movement generated based on this sensor information. In addition, information representing the interference area of the operating hand of the robot arm RB is also stored in the hand information storage unit 32.

[0028] The object information storage unit 33 is used to store information representing a plurality of objects detected from the video information of the camera CM in time series. The information representing an object includes an object ID uniquely assigned to the detected object, position and velocity, information representing the interference area of the object, a velocity vector, and information representing an interference determination result. Details of the object information will be described in the operation example.

[0029] The control unit 1 includes, as processing functions necessary for implementing one embodiment, a video information acquisition processing unit 11, a video information transmission processing unit 12, a hand information generation processing unit 13, an object detection processing unit 14, an interference area definition processing unit 15, an interference determination processing unit 16, a contact target selection processing unit 17, and an estimated information output processing unit 18. The above processing units 11 to 18 are all realized by causing a hardware processor of the control unit 1 to execute an application program stored in the program storage unit 2.

[0030] The video information acquisition processing unit 11 acquires the time-series video information output from the camera CM via the input / output I / F unit 4, and performs processing to temporarily store each acquired video information in the video information storage unit 31.

[0031] The video information is composed of a video frame including a bounding box and point cloud information. The bounding box displays the position of an object appearing in the video frame with a rectangular frame. For example, when the coordinates of the upper left end of the video frame are (0, 0), it is represented by the coordinates of the upper left end point and the lower right end point of the rectangular frame. The point cloud information is, for example, depth information, and represents the distance distribution for each pixel in the video frame from the camera CM.

[0032] The video information transmission processing unit 12 performs processing to transmit the video frame including the bounding box among the video information captured by the camera CM from the communication I / F unit 5 to the operation-side terminal device OU.

[0033] The hand information generation processing unit 13 receives the sensor information transmitted from the operation-side terminal device OU via the communication I / F unit 5, and calculates the velocity vector of the hand movement based on the received sensor information. Then, the hand information generation processing unit 13 stores the information representing the position of each part of the hand, the information indicating whether it is the left or right hand, and the information indicating whether the operation side is the front or back of the hand included in the received sensor information, and the calculated velocity vector in the hand information storage unit 32.

[0034] In addition, the hand information generation processing unit 13 also performs a process of storing information representing an interference region preset with reference to the center of the operating hand of the robot arm RB in the hand information storage unit 32. The specific setting method of the information representing the interference region will be described in the operation example.

[0035] The object detection processing unit 14 reads the video information from the video information storage unit 31 frame by frame, and detects a plurality of objects shown in the read video frame and information representing the operating hand of the robot arm RB. This detection is performed by using a bounding box. The object detection processing unit 14 outputs the video data and point cloud information of the region surrounded by the detected bounding box.

[0036] The interference region definition processing unit 15 performs a process of excluding objects that are outside the operating range of the robot arm RB for each object detected by the object detection processing unit 14. In addition, the interference region definition processing unit 15 performs a process of defining the interference region for each object that has not been excluded by the above exclusion process. An example of the above exclusion process and the process of defining the interference region will be described in the operation example.

[0037] The interference determination processing unit 16 determines the presence or absence of interference between each object and the operating hand of the robot arm RB corresponding to the hand information of the operator US based on the hand information of the operator US stored in the hand information storage unit 32 and the object information of a plurality of objects stored in the object information storage unit 33, using a prepared determination formula for each object. Then, interference determination information representing the determination result is added to the object information storage unit 33. An example of the interference determination process will also be described in the operation example.

[0038] The contact target selection processing unit 17 first reads the hand information of the operator US stored in the hand information storage unit 32 and the object information regarding the object determined to have interference by the interference determination processing unit 16. Subsequently, based on the read information, for each object determined to have interference, the contact target selection processing unit 17 determines, for each object, information indicating whether the palm (back of the hand side) or the back (palm side) of the operator US's hand faces the object, and a center capture degree which is an index indicating how much the operator US's hand captures the center of the object. Then, the contact target selection processing unit 17 performs score calculation based on the information indicating whether the palm or the back of the hand faces the object and the center capture degree, and selects an object with the highest possibility of contact by the operating hand of the robot arm RB based on the calculated cost.

[0039] The estimated information output processing unit 18 outputs the object information regarding the object selected by the contact target selection processing unit 17 from the robot I / F unit 6 to the robot control device RU as auxiliary information for the operation of the robot arm RB.

[0040] (Operation example) Next, an operation example of the contact target estimation device EU configured as described above will be described. FIG. 4 is a flowchart showing an example of the overall processing procedure and processing content executed by the control unit 1 of the contact target estimation device EU.

[0041] (1) Acquisition and transmission of video information The control unit 1 of the contact target estimation device EU monitors the start of remote operation in step S10. In this state, assuming that the operator US starts the system to remotely operate an object using the robot arm RB, then the control unit 1 of the contact target estimation device EU, under the control of the video information acquisition processing unit 11, acquires, in step S11, the video information of the work area captured by the camera CM via the camera I / F unit 4, and stores the acquired video information in the video information storage unit 31.

[0042] At this time, the imaging range of the camera CM is set to include a plurality of objects existing in the work area and the operating hand (grasping part) of the robot arm RB. For this reason, in the video information acquisition processing unit 11, video information including images of a plurality of objects existing in the work area and an image of the operating hand of the robot arm RB is obtained in the video frame. Further, since an RGB-D camera is used as the camera CM, video information including video data with bounding boxes and point cloud information is acquired.

[0043] When the above video information is acquired, the control unit 1 of the contact target estimation device EU reads only the video data among the video data and the point cloud information included in the video information stored in the video information storage unit 31 under the control of the video information transmission processing unit 12 in step S12. Then, the read video data is transmitted from the communication I / F unit 5 to the operation-side terminal device OU via the network NW.

[0044] As a result, on the operation side, the above video data is displayed on the monitor MT by the operation-side terminal device OU. Therefore, hereafter, the operator US can perform remote operations on the object while viewing the video of the work area displayed on the monitor MT.

[0045] (2) Acquisition of sensor information, generation and storage of hand information When the operator US moves their hand to remotely operate the robot arm RB, the control unit 1 of the contact target estimation device EU first receives, at a predetermined cycle via the communication I / F unit 5, sensor information representing the movement of the hand of the operator US detected by the motion sensor SS under the control of the hand information generation processing unit 13 in step S13. The sensor information includes, for example, three-dimensional coordinate information representing the position of the center of the hand of the operator US, the positions of the joints of each finger, the positions of each fingertip, and the position of the wrist, and information indicating whether it is the left or right hand.

[0046] Subsequently, every time sensor information is received, the hand information generation processing unit 13 calculates a velocity vector from the difference between the three-dimensional coordinate information representing the positions of the respective parts of the hand of the operator US included in the sensor information and the three-dimensional coordinate information representing the positions of the respective parts included in the sensor information received at the previous timing.

[0047] Then, the hand information generation processing unit 13 stores the three-dimensional coordinate information representing the positions of the respective parts included in the received sensor information, the information indicating whether it is the left or right hand, and the calculated velocity vector in the hand information storage unit 32. Further, the hand information generation processing unit 13 stores information representing the interference region set for the operating hand of the robot arm RB in the hand information storage unit 32. The information representing the interference region is set, for example, as a sphere representing a range with a diameter of 90 mm from the center of the palm of the operating hand of the robot arm RB. FIG. 9 shows an example of the hand information stored in the hand information storage unit 32.

[0048] (3) Detection of Object and Definition of Interference Region When the video information is acquired, the control unit 1 of the contact target estimation device EU executes the following processing for detecting an object from the video data for each video frame and defining an interference region for each detected object in step S15.

[0049] FIG. 5 is a flowchart showing an example of the processing procedure and processing content of a series of processes executed by the object detection processing unit 14 and the interference region definition processing unit 15 in step S15.

[0050] In step S20, the object detection processing unit 14 reads a video frame from the video information storage unit 31 and detects a plurality of objects shown in the read video frame and the operating hand of the robot arm RB based on the bounding boxes assigned to them. Then, in step S21, the object detection processing unit 14 assigns a unique object ID to each of the detected objects, and then outputs the video data and point cloud information of the region surrounded by the bounding box for each of the objects.

[0051] FIG. 8 shows an example of the detected objects TL1, TL2, TL3 and the bounding boxes B1, B2, B3 representing the image regions of the respective objects TL1, TL2, TL3. As shown in this figure, the shapes and sizes of the bounding boxes B1, B2, B3 change according to the shapes and sizes of the objects TL1, TL2, TL3.

[0052] The interference region definition processing unit 15 first selects one of the detected objects in step S22 and calculates the distance between this object and the operating hand of the robot arm RB. This distance is calculated based on the three-dimensional coordinate information of each of the object and the operating hand of the robot arm RB. Note that the three-dimensional coordinate information of the object and the operating hand of the robot arm RB is obtained from the coordinate information indicating the position of each bounding box and the point cloud information.

[0053] Subsequently, in step S23, the interference region definition processing unit 15 compares the calculated distance with a preset threshold value. Then, it determines that an object whose distance exceeds the threshold value is an object that cannot be operated by the robot arm RB, and excludes the object from the determination target.

[0054] Subsequently, in step S24, the interference region definition processing unit 15 performs a process of defining an interference region for each object that was not excluded by the above exclusion process, that is, each object whose distance is within the threshold value range. The definition of the interference region is performed using, for example, a sphere centered on the center position of the object. To calculate the diameter of the sphere indicating the interference region, first, the two ends of the object existing within the bounding box are searched. Subsequently, among the candidate points that are the two ends of the object, a straight line passing through the center position of the bounding box is connected, and the two points with the largest distance are selected, and the distance between the selected two points is set as the diameter of the sphere.

[0055] The interference area definition processing unit 15 finally stores, in the object information storage unit 33, the information representing the object obtained by the above series of processes at step S25. The information representing the object stored at this time includes the object ID, the position and velocity of the object, the sphere information representing the interference area of the object, and the velocity vector. FIG. 10 shows an example of the object information stored in the object information storage unit 33.

[0056] (4) Determination of whether interference occurs Next, under the control of the interference determination processing unit 16, the control unit 1 of the contact target estimation device EU executes, at step S16, a process of determining the presence or absence of interference between each detected object and the operating hand of the robot arm RB as follows.

[0057] FIG. 6 is a flowchart showing an example of the processing procedure and processing content of the determination process executed by the interference determination processing unit 16 at step S16.

[0058] First, at step S30, the interference determination processing unit 16 reads a list of object information from the object information storage unit 33 and reads hand information representing the movement of the hand operation of the operator US from the hand information storage unit 32. Then, based on the read object information and hand information, the interference determination processing unit 16 executes, at steps S31 to S36, a process of determining the presence or absence of interference using a determination formula prepared in advance for each object.

[0059] That is, the elements used in the interference determination are the position vectors and velocity vectors of the hand of the operator US and the object. The position vectors and velocity vectors of the object A and the hand H at a certain time t [seconds] are respectively

Equation

[0060] Similarly, the relative position and relative velocity between the hand of the operator US and the object are

Equation

[0061] When the current time is t0 seconds, the relative position between the hand of the operator US and the object at an arbitrary time t [seconds] is

Number

[0062] In this equation, in order to handle three-dimensional motion, when the above position vector and velocity vector are represented by three axes 1, 2, and 3 respectively,

Number

[0063] Solving Equation (1),

Number

[0064] Here, a, b, and c are respectively

Number

[0065] Also, the interference start time t1 and the interference end time t2, which are the solutions, are respectively

Number

[0066] In the above equation

Number

[0067] When the inside of the root of the above formula (2) is positive, the interference determination processing unit 16 determines that there is interference, and when it is negative, it determines that there is no interference. When it is determined that there is interference, in the above determination formula, the calculation is performed on the assumption of uniform linear motion between the interference start time t1 [seconds] and the interference end time t2 [seconds]. As a result, from the current position, speed, and interference time, the interference position is [Number] obtained as.

[0068] The interference determination processing unit 16 additionally stores in the object information storage unit 33, in association with the object ID, interference information including information indicating the determination result of the presence or absence of interference and the interference time and interference position obtained by the above calculation for the object to be determined.

[0069] Subsequently, in the same manner, the interference determination processing unit 16 performs the above series of interference determination processes for each of all the objects included in the object information list.

[0070] (5) Selection of the object to be contacted The control unit 1 of the contact target estimation device EU finally executes the process of selecting, under the control of the contact target selection processing unit 17, the object with the highest possibility of being the contact target of the operating hand of the robot arm RB in step S17 as follows.

[0071] FIG. 7 is a flowchart showing an example of the processing procedure and processing content of the selection process executed by the contact target selection processing unit 17.

[0072] That is, the contact target selection processing unit 17 first reads the hand information of the operator US from the hand information storage unit 32 and reads the object information related to the object determined to have interference from the object information storage unit 33 in step S41. Then, for each of the objects determined to have interference, the contact target selection processing unit 17 performs score calculation in steps S42 to S43.

[0073] For example, for each of the objects determined to have interference, the contact target selection processing unit 17 calculates information FB indicating whether the front or back side of the hand of the operator US faces the object, and a center capture degree CC which is an index indicating how much the hand of the operator US holds the center of the object.

[0074] Next, in step S44, the contact target selection processing unit 17 calculates a cost Cost indicating the possibility of contact based on the information FB indicating whether the front or back of the hand faces the object and the center capture degree CC. Cost = CC × FB and calculates it according to.

[0075] Here, the center capture degree CC is the distance when the interference regions of the hand of the operator US and the object are closest to each other.

Equation

[0076] That is, the center capture degree CC is calculated as the distance between the hand of the operator US and the object at the time (t1 + t2) / 2, which is the middle time between the start time t1 and the end time t2 of the period when interference occurs.

[0077] Also, the information FB indicating whether the front or back side of the hand of the operator US faces the object is a parameter set in advance such that the value when the palm of the hand faces the object is smaller than the value when the back of the hand faces the object. For example, "1.0" is set for the back of the hand and "1.2" is set for the front of the hand. This is because generally, the operation of contacting and gripping the object is performed on the palm of the hand of the operator US, that is, the palm side of the robot arm RB. Note that any of the values of the above parameters can be arbitrarily adjusted.

[0078] Then, the contact target selection processing unit 17 compares the above costs Cost between objects, and determines the object with the smallest value of the cost Cost as the object with the highest possibility of being gripped by the hand of the operator US, and selects it as the object to be the contact target. In the selection, as described above, one object with the highest possibility may be selected, or a predetermined number of objects ranked in descending order of possibility may be selected.

[0079] The control unit 1 of the contact target estimation device EU outputs, under the control of the estimation information output processing unit 18, the object information related to the selected object from the robot I / F unit 6 to the robot control device RU in step S18. The robot control device RU controls the operation of the robot arm RB in consideration of the object information.

[0080] (Function and effect) As described above, in one embodiment, in the contact target estimation device EU, each object shown in the video frame is detected from the video information of the work area imaged by the camera CM together with the hand part of the robot arm RB, and after excluding the objects at an inoperable distance by the robot arm RB among the detected objects, an interference area is defined for each of the remaining objects. On the other hand, in parallel, hand information representing the position of the hand of the operator US is generated from the sensor information obtained by the motion sensor SS worn by the operator US.

[0081] Next, for each of the above objects, based on the position vector and velocity vector of the object and the hand of the operator US, it is determined whether the interference area of the object interferes with the interference area of the operating hand of the robot arm RB. Then, for each object determined to interfere, a cost Cost is calculated based on the information FB indicating which side of the operator US's hand is facing and the center capture degree CC indicating how much the operator US's hand captures the center of the object, and based on the calculated cost Cost, the object with the highest possibility of being grasped by the hand of the robot arm RB among the above objects is selected, and the object information related to the selected object is used for robot control.

[0082] Therefore, according to one embodiment, it becomes possible to estimate an object that the robot arm RB is about to grip according to the operation of the operator US without using the line-of-sight information of the operator US. As a result, it is possible to avoid complicating the system by eliminating the need for additional devices such as an HMD.

[0083] In addition, an interference area is defined for each object detected from the video, an object that may interfere with the interference area set for the hand of the robot arm RB is identified, and the object with the highest possibility of being gripped by the robot arm RB is selected from the identified objects. For this reason, even when there are a plurality of objects that are candidates for the operation target in the work area, the operation target object can be estimated in a short time, and thus it is possible to maintain the real-time performance of the process.

[0084] Furthermore, when defining the interference area for each of the above objects, objects whose distance from each object to the robot arm RB exceeds a threshold value are excluded in advance, and the interference area is defined only for the objects that are not excluded. For this reason, the time required for the process of defining the interference area can be shortened, and thus it is possible to further enhance the real-time performance of the process.

[0085] [Other Embodiments] (1) In one embodiment, the case where the contact target estimation device EU is independently provided at a remote location is described as an example, but all or some of the functions of the contact target estimation device EU may be provided in the robot control device RU. Further, the contact target estimation device EU does not necessarily have to be arranged on the remote side and may be arranged on the operation side, or may be arranged on the Web or in the cloud. Also, in that case, the processing function of the contact target estimation device EU may be distributed to information processing devices arranged on the remote side and the operation side, respectively, or to a server device arranged on the Web or in the cloud.

[0086] (2) In one embodiment, the case of using an RGB-D camera was described as an example. However, a normal camera may also be used. In this case, by providing the contact target estimation device EU with a function of detecting an object image from the video information captured by the camera and a function of generating depth information from the detected object image, it is possible to estimate an object to be a contact target in the same manner as in one embodiment.

[0087] (3) As a means for detecting an object image, for example, a machine learning model that takes a video frame as an input and outputs information representing the class and position of an object shown in the video frame can be used. Further, as a means for generating depth information, for example, a machine learning model for detecting distance information that takes a video frame as an input and outputs depth information representing the distance distribution for each pixel of the image shown in the video frame from the camera CM can be used.

[0088] (4) In one embodiment, the case of detecting an object such as a tool, a part, or a device as a contact target was described as an example. However, as the contact target, other living organisms such as a person, an animal, or a plant may also be used.

[0089] (5) The number of cameras CM is not limited to one, and a plurality of cameras CM may be arranged, and the videos of these cameras CM may be selectively used or synthesized to detect an object. Further, an acceleration sensor may be used as a motion sensor for detecting the motion of the operator, and the part to be detected for motion is not limited to the hand and may be the foot, the waist, or the neck.

[0090] In addition, with respect to the functional configuration, processing procedure, processing content, etc. of the contact target estimation device, various modifications can be made without departing from the gist of the present invention.

[0091] The embodiments of the present invention have been described in detail above, but the foregoing description is merely an exemplification of the present invention in every aspect. Needless to say, various improvements and modifications can be made without departing from the scope of the present invention. That is, in practicing the present invention, a specific configuration according to the embodiment may be appropriately adopted.

[0092] In short, the present invention is not limited to the above-described embodiments as they are, and at the implementation stage, the components can be modified and embodied without departing from the gist thereof. Further, various inventions can be formed by appropriately combining a plurality of components disclosed in the above-described embodiments. For example, some components may be deleted from all the components shown in the embodiments. Furthermore, components from different embodiments may be appropriately combined.

Explanation of Signs

[0093] EU... Contact target estimation device CM... Camera RU... Robot control device RB... Robot arm US... Operator OU... Operation-side terminal device SS... Motion sensor MT... Monitor 1... Control unit 2... Program storage unit 3... Data storage unit 4... Camera I / F unit 5... Communication I / F unit 6... Robot I / F unit 7... Bus 11... Video information acquisition processing unit 12... Video information transmission processing unit 13... Hand information generation processing unit 14... Object detection processing unit 15... Interference area definition processing unit 16... Interference determination processing unit 17... Contact target selection processing unit 18... Estimation information output processing unit 31... Video information storage unit 32... Hand information storage unit 33... Object information storage unit

Claims

1. A contact target estimation device used in a system for remotely operating an object existing in a work area at a remote location by operating a robot at the remote location according to an operator's operation, a first processing unit that detects a plurality of the objects shown in a video frame of the video information from the video information obtained by imaging the work area, and defines a first interference region for each of the plurality of detected objects; a second processing unit that determines the presence or absence of interference between each of the plurality of objects and the robot based on the first interference region defined for the object, a second interference region set for the robot, and the operation information of the operator; a third processing unit that selects, from among the plurality of objects determined to have interference, the object that the robot is about to contact based on information representing the positional relationship between the object and the robot; A contact target estimation device comprising:

2. The first processing unit includes: a processing unit that excludes, from among the plurality of objects, an object whose distance from the robot exceeds a preset range; a processing unit that defines the first interference region for the object that was not subject to the exclusion; The contact target estimation device according to claim 1, comprising:

3. The second processing unit calculates the relative position between the object and the robot at an arbitrary time based on the position vectors and velocity vectors of the object and the robot, and determines the presence or absence of interference between the object and the robot based on the calculated relative position. The contact target estimation device according to claim 1.

4. The second processing unit calculates the relative position in a three-dimensional space using the three-axis position vector and the three-axis velocity vector. The contact target estimation device according to claim 3.

5. The third processing unit selects the object that the robot is about to contact based on the distance between the robot and the object when they are closest to each other. The contact target estimation device according to claim 1.

6. The third processing unit selects the object that the robot is about to contact by taking into account information indicating whether the object faces the front or back of the operator's hand in addition to the distance between the robot and the object when they are closest to each other. The contact target estimation device according to claim 5.

7. A contact target estimation method executed by an information processing apparatus used in a system for remotely operating an object existing in a work area at a remote location by operating a robot at the remote location according to an operator's operation, a first processing step of detecting a plurality of the objects shown in a video frame of the video information from the video information obtained by imaging the work area, and defining a first interference area for each of the plurality of detected objects; a second processing step of determining the presence or absence of interference between each of the plurality of objects and the robot based on the first interference area defined for the object, a second interference area set for the robot, and the operation information of the operator; a third processing step of selecting, from among the plurality of objects determined to have interference, the object that the robot is about to contact based on information representing the positional relationship between the object and the robot; A contact target estimation method comprising the above steps.

8. A program for causing at least one of the first processing unit, the second processing unit, and the third processing unit included in the contact target estimation apparatus according to any one of Claims 1 to 6 to be executed by a processor included in the contact target estimation apparatus.

Images