Information processing device, information processing method, and program

Abstract

To provide an information processing device, an information processing method, and a program capable of suppressing an increase in tact time due to measurement of a three-dimensional shape.SOLUTION: An information processing device includes a processor. The processor acquires an image obtained by an imaging device provided in a robot hand at a plurality of via points in which the robot hand passes during conveyance operation with respect to a first target object being a target of gripping by the robot hand, the image including a figure indicating a second target object being a target of three-dimensional measurement, and estimates a three-dimensional shape of the second target object on the basis of the acquired image.SELECTED DRAWING: Figure 8

Description

[Technical field]

[0001] The present disclosure relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] Recently, robot systems have been introduced in various fields, including the industrial field, to reduce manpower and improve work efficiency. For example, in the industrial field, the number of industrial robots installed worldwide is on the rise. However, when introducing robots, including industrial robots, a so-called teaching and playback method is often adopted, which may require peripheral dedicated equipment such as a positioning jig or a task to teach the robot how to operate. For this reason, the teaching and playback method is not suitable for a form in which the type or quantity of the target object (e.g., the product to be produced) varies (e.g., a variable-type variable-quantity production form). Therefore, an autonomous robot system that recognizes the shape of the target object and grasps it is being considered as a robot system that can handle a variable number of objects.

[0003] For example, Patent Document 1 describes a work removal system that includes a robot, a hand attached to the robot's endoscope for gripping a workpiece, a three-dimensional sensor attached to the robot's endoscope for acquiring position information of a three-dimensional point cloud of a partial area within a container, a work state calculation unit that calculates the position and posture of the workpiece based on the position information of the three-dimensional point cloud of a first partial area acquired by the three-dimensional sensor, a data acquisition position calculation unit that calculates the robot position of a second partial area for which position information is next acquired based on the position information of the three-dimensional point cloud of the first partial area acquired by the three-dimensional sensor, and a control unit that controls the robot and the hand based on the position and posture of the workpiece calculated by the work state calculation unit and the robot position of the second partial area calculated by the data acquisition position calculation unit.

[0004] Patent Document 2 describes a control device that measures the shape of a workpiece using a three-dimensional camera fixed to a robot hand, and performs calibration to convert the camera coordinates of the three-dimensional camera into the robot coordinates of the robot hand based on the measured shape data of the workpiece. The control device is equipped with a calibration execution unit that executes a normal calibration process in which, after each operation, the workpiece is measured at the stopping position of the robot hand using the three-dimensional camera to detect the workpiece coordinates based on the shape data obtained, the robot coordinates when the workpiece coordinates were obtained are obtained, and parameters for converting the position in the camera coordinates to a position on the robot coordinates are calculated based on the obtained multiple sets of workpiece coordinates and robot coordinates. The control device is equipped with a calibration execution unit that executes a high-precision calibration process in which a provisional rotation center is determined based on the relative position of the tool center of the robot hand and the three-dimensional camera obtained by the normal calibration process, and the robot hand is rotated multiple times around the provisional rotation center to detect the workpiece coordinates based on the shape data obtained by measuring the workpiece with the three-dimensional camera at each position, and the robot coordinates when the workpiece coordinates were obtained are obtained, and the final rotation center of the robot hand is determined based on the obtained multiple sets of workpiece coordinates and robot coordinates.

[0005] Patent Document 3 describes a three-dimensional measuring device that includes an image acquisition unit that acquires an image of a workpiece captured by an imaging device, an encoder value acquisition unit that acquires encoder values ​​of an encoder attached to each joint of a multi-joint robot, an imaging device attitude calculation unit that calculates the attitude of the imaging device based on the encoder values ​​acquired by the encoder value acquisition unit, and a three-dimensional shape calculation unit that calculates the three-dimensional shape of the workpiece based on the image of the workpiece acquired by the image acquisition unit and the attitude of the imaging device calculated by the imaging device attitude calculation unit.

[0006] Patent document 4 describes a robot vision system having a camera, characterized in that it comprises a light source that illuminates a workpiece, a memory unit that stores a reference position of the workpiece, an irradiation condition changing unit that changes the irradiation conditions of the light source, a shape information calculation unit that calculates the shape of the workpiece by referring to multiple images captured by the camera while changing the irradiation conditions of the light source, and a distance calculation unit that calculates the distance from the robot to the workpiece from the reference position and the calculated shape of the workpiece.

[0007] Patent document 5 describes a robot hand with hand visualization that includes a work imaging means for obtaining image data representing the shape and position of a work to be grasped by the robot hand, a work position and shape detection means for calculating data on the position and shape of the work based on the image data of the work captured by the work imaging means, a gripping position data generation means for generating gripping position data for grasping the work by the robot hand based on the position and shape data calculated by the work position and shape detection means, and a hand gripping position control means for changing the gripping position of the robot hand based on the gripping position data generated by the gripping position data generation means, and is characterized in that the robot hand can autonomously grasp a work at an optimal gripping position. [Prior art documents] [Patent documents]

[0008] [Patent Document 1] JP 2018-192592 A [Patent Document 2] JP 2022-122648 A [Patent Document 3] JP 2021-133458 A [Patent Document 4] JP 2023-10172 A [Patent Document 5] JP 2003-94367 A Summary of the Invention [Problem to be solved by the invention]

[0009] However, the techniques described in Patent Documents 1 to 5 still have room for improvement in measuring the three-dimensional shape of a target object in order to grasp the target object with a robot hand. Therefore, the present disclosure has been made in consideration of the above circumstances, and aims to provide an information processing device, an information processing method, and a program capable of suppressing an increase in takt time caused by measuring the three-dimensional shape. [Means for solving the problem]

[0010] A first aspect of the technology disclosed herein is an information processing device that includes a processor, the processor acquiring images obtained by imaging devices provided on the robot hand at multiple waypoints through which the robot hand passes during a transport operation of a first object to be grasped by the robot hand, the images including an image showing a second object to be measured in three dimensions, and estimating the three-dimensional shape of the second object based on the acquired images.

[0011] A second aspect of the technology disclosed herein is an information processing device according to the first aspect, in which the three-dimensional shape of the second target object is estimated by applying a volume intersection method to the results of extracting an image showing the second target object from a plurality of images obtained at each of a plurality of intermediate points.

[0012] A third aspect of the technology disclosed herein is an information processing device according to the first aspect, in which a transport operation is performed for each of a plurality of first target objects, and the plurality of intermediate points are updated based on the measurement results of the three-dimensional shape of the second target object in the immediately preceding transport operation.

[0013] A fourth aspect of the technology disclosed herein is an information processing device according to the first aspect, wherein the transport operation includes a first operation in which the robot hand moves to grasp the first target object, and a second operation in which the first target object grasped by the robot hand is transported to another location, and a waypoint in the second operation is set based on measurement results of the three-dimensional shape of the second target object during the first operation.

[0014] A fifth aspect of the technology disclosed herein is an information processing device according to the third or fourth aspect, in which the intermediate points are set based on candidate points for three-dimensional measurement obtained from measurement results and points to which the robot hand can move.

[0015] A sixth aspect of the technology disclosed herein is an information processing device according to the fifth aspect, in which the candidate point is set at a position where an axis along a third principal component obtained by performing principal component analysis on a point cloud consisting of a plurality of points corresponding to a three-dimensional shape, each point indicating a three-dimensional spatial coordinate, coincides with an optical axis of the imaging device.

[0016] A seventh aspect of the technology of the present disclosure is an information processing device according to the sixth aspect, in which the candidate points are set at positions in a direction along the third principal component such that the number of points in the point cloud included in the shooting range of the imaging device falls within a predetermined range.

[0017] An eighth aspect of the technique of the present disclosure is the information processing device according to the fifth aspect, in which the movable point is set using a spherical coordinate system having the center of gravity of the second target object in three-dimensional measurement as its origin.

[0018] A ninth aspect of the technology disclosed herein is an information processing device according to the eighth aspect, in which the intermediate point is a lattice point represented by a spherical coordinate system, and is the closest point to the candidate point and the movable point of the robot hand.

[0019] A tenth aspect of the technology of the present disclosure is an information processing method that includes acquiring images obtained by an imaging device provided on a robot hand at multiple waypoints passed by the robot hand during a transport operation of a first object to be grasped by the robot hand, the images including an image showing a second object to be measured in three dimensions, and estimating the three-dimensional shape of the second object based on the acquired images.

[0020] An eleventh aspect of the technology of the present disclosure is a program that causes a computer to execute processing including acquiring images obtained by an imaging device provided on a robot hand at multiple waypoints passed by the robot hand during a transport operation of a first object to be grasped by the robot hand, the images including an image showing a second object to be measured in three dimensions, and estimating the three-dimensional shape of the second object based on the acquired images. Effect of the Invention

[0021] According to the present disclosure, an information processing device, an information processing method, and a program are provided that are capable of suppressing an increase in takt time caused by measurement of a three-dimensional shape. [Brief description of the drawings]

[0022] [Figure 1] FIG. 1 is a conceptual diagram illustrating an example of a configuration of a robot arm system according to an embodiment. [Diagram 2] FIG. 2 is a block diagram showing an example of a hardware configuration of an electrical system of the control device according to the embodiment. [Diagram 3] 11 is a conceptual diagram showing an example of processing contents of an operation control unit in the processor according to the embodiment. FIG. [Figure 4] 10 is a conceptual diagram showing an example of processing contents of an acquisition unit and an image processing unit in the processor according to the embodiment. FIG. [Diagram 5] FIG. 13 is a conceptual diagram showing an example of processing contents of a three-dimensional measurement unit in the processor according to the embodiment. [Figure 6] FIG. 13 is a conceptual diagram showing an example of processing contents of a candidate point calculation unit in the processor according to the embodiment. [Figure 7] FIG. 13 is a conceptual diagram illustrating an example of processing content of a via point determination unit in the processor according to the embodiment. [Figure 8] 11 is a conceptual diagram showing an example of processing contents of an operation control unit in the processor according to the embodiment. FIG. [Figure 9] 10 is a flowchart illustrating an example of a flow of a control process of a robot arm according to the embodiment. [Figure 10] 11 is a flowchart illustrating an example of the flow of a waypoint calculation process according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0023] Hereinafter, embodiments of the technology of the present disclosure will be described with reference to the drawings.

[0024] FIG. 1 is a conceptual diagram showing an example of the configuration of a robot arm system 10. As an example, as shown in Fig. 1, a robot arm system 10 includes a robot arm 12 and a control device 15. The robot arm 12 is a device for transporting an object B. In the example shown in Fig. 1, a plurality of cylindrical objects B are placed on a platform A1, and the robot arm 12 transports the objects B on the platform A1 to a destination platform A2.

[0025] Here, the transport operation by the robot arm 12 includes an operation of picking up the object B to be transported and an operation of moving the grasped object B to a destination. In other words, the transport operation includes an operation of the robot hand 16 moving to grasp the object B and an operation of transporting the object B grasped by the robot hand 16 to another location.

[0026] The robot arm system 10 can be used, for example, in the transport, assembly, or processing of various parts, semi-finished products, finished products, or articles in the industrial field. The robot arm system 10 can also be used, for example, in the transport or delivery of personal items (e.g., cups, ornaments, etc.) in the nursing care field.

[0027] Note that, although a cylindrical object is shown here as an example of object B, this is merely one example. The shape of object B is not particularly limited as long as it can be grasped by the robot hand 16. The optical properties of object B are also not particularly limited, and object B may be transparent or opaque to visible light, or may have a gloss (for example, a metallic gloss or a resin surface gloss).

[0028] The control device 15 is a device capable of controlling the operation of the robot arm 12. The control device 15 may be a terminal (e.g., a personal computer or a tablet) separate from the robot arm 12, or may be incorporated in the robot arm 12.

[0029] The robot arm 12 includes an arm body 14, a robot hand 16, and a camera 18. The arm body 14 moves the robot hand 16 to a predetermined position under the control of a control device 15. The arm body 14 has a base end attached to a base C, and the robot hand 16 is attached to a tip end. In the arm body 14, a link portion 14A rotates via a plurality of joints 14B, thereby realizing the movement of the robot hand 16.

[0030] The robot hand 16 grasps the object B under the control of the control device 15. In the example shown in FIG. 1, the robot hand 16 is attached to the tip of the arm body 14 via the base end 16C. The robot hand 16 has a pair of fingers 16A and 16B extending from the base end 16C. The robot hand 16 grasps the object B by pinching the object B between the pair of fingers 16A and 16B. The robot hand 16 is an example of a "robot hand" according to the technology of the present disclosure.

[0031] The camera 18 is an imaging device capable of capturing an image of the object B. The camera 18 is attached to the base end 16C of the robot hand 16. Note that this is merely an example, and the camera 18 only needs to be fixed to the robot hand 16, and the fixing location is not particularly limited. The camera 18 is a so-called RGB-D camera that includes an imaging element capable of acquiring an RGB image and a distance measuring sensor. The camera 18 is capable of capturing an image in which distance information to a subject (i.e., depth) is embedded along with RGB values ​​for each pixel. The camera 18 is an example of an "imaging device" according to the technology of the present disclosure.

[0032] FIG. 2 is a block diagram showing an example of a hardware configuration of an electrical system of the control device 15. As shown in FIG. 2, as an example, the control device 15 includes a computer 20, a communication I / F (interface) 28, a reception device 30, and a display device 32. The computer 20 includes a processor 22, a storage 24, and a RAM (Random Access Memory) 26. The processor 22, the storage 24, the RAM 26, the communication I / F 28, the reception device 30, and the display device 32 are connected to a bus 34. The computer 20 is an example of a "computer" and an "information processing device" according to the technology of the present disclosure, and the processor 22 is an example of a "processor" according to the technology of the present disclosure.

[0033] A memory is connected to the processor 22. The memory includes a storage 24 and a RAM 26. The processor 22 includes, for example, a CPU (Central Processing Unit).

[0034] The storage 24 is a non-volatile storage device that stores various programs and various parameters. Examples of the storage 24 include flash memory (e.g., an EEPROM (Electrically Erasable and Programmable Read Only Memory) and an SSD (Solid State Drive)) and / or an HDD (Hard Disk Drive). Note that the flash memory and the HDD are merely examples, and at least one of a flash memory, an HDD magnetoresistive memory, and a ferroelectric memory may be used as the storage 24.

[0035] The RAM 26 is a memory that temporarily stores information, and is used as a work memory by the processor 22. The RAM 26 may be, for example, a dynamic random access memory (DRAM) or a static random access memory (SRAM).

[0036] The communication I / F 28 is connected to the robot arm 12. The communication I / F 28 is responsible for sending and receiving information to and from the robot arm 12. For example, the communication I / F 28 transmits information to the robot arm 12 in response to a request from the processor 22. The communication I / F 28 also receives information transmitted from the robot arm 12, and outputs the received information to the processor 22 via the bus 34.

[0037] The reception device 30 receives instructions from a user. The reception device 30 is, for example, a keyboard and a mouse. The display device 32 displays various information (for example, images and characters) under the control of the control device 15. Examples of the display device 32 include an EL (Electro-Luminescence) display and a liquid crystal display.

[0038] The storage 24 stores a motion control program 24A and a via point calculation program 24B. The motion control program 24A is a program for controlling the transport operation by the robot arm 12. The via point calculation program 24B is a program for calculating points through which the robot hand 16 passes during the transport operation. The via point calculation program 24B is an example of a "program" according to the technology of the present disclosure.

[0039] The processor 22 executes the operation control process by reading the operation control program 24A from the storage 24 and executing the read operation control program 24A on the RAM 26. The operation control process is realized by the processor 22 operating as the operation control unit 22A.

[0040] Moreover, the processor 22 executes a via point calculation process by reading out a via point calculation program 24B from the storage 24 and executing the read via point calculation program 24B on the RAM 26. The via point calculation process is realized by the processor 22 operating as an acquisition unit 22B, an image processing unit 22C, a three-dimensional measurement unit 22D, a candidate point calculation unit 22E, and a via point determination unit 22F.

[0041] FIG. 3 is a conceptual diagram showing an example of the processing contents of the operation control section 22A in the processor 22. As shown in FIG. As an example, as shown in Fig. 3, the operation control unit 22A outputs an operation signal 36 to the robot arm 12. The operation signal 36 is a signal indicating a waypoint in the transport operation of the robot arm 12 and an object B to be transported. In the following, of the transport operations by the robot arm 12, an operation to pick up the object B1 will be described as an example. In this case, the operation control unit 22A outputs an operation signal 36A to the robot arm 12 indicating a waypoint in the operation to pick up the object B1. The object B1 is an example of a "first target object" according to the technology of the present disclosure.

[0042] The robot arm 12 starts an operation to pick up the object B based on the operation signal 36A. As the robot arm 12 operates, the robot hand 16 passes through the way points specified in the operation signal 36A. The camera 18 captures images at each way point. In the example shown in Fig. 3, four way points D1 to D4 are passed.

[0043] Here, the waypoints D1 to D4 are set in advance. Although details will be described later, the waypoints D1 to D4 are determined based on the three-dimensional measurement result obtained by the operation of moving the object B in the previous transport operation. Note that, if the previous transport operation has not been performed (i.e., in the case of the first transport operation), a separate three-dimensional measurement may be performed to determine the waypoints before starting the transport operation. The object B1 to be transported is also appropriately set based on a predetermined algorithm. For example, the object B1, whose three-dimensional shape dimensions are obtained with an accuracy high enough to be grasped by the robot hand 16 as a result of the three-dimensional measurement in the previous transport operation, may be set as the transport object.

[0044] FIG. 4 is a conceptual diagram showing an example of the processing contents of the acquisition unit 22B and the image processing unit 22C in the processor 22. As shown in FIG. As an example, as shown in FIG. 4, the camera 18 outputs a captured image group 38 acquired at each of a plurality of via points D1 to D4 to the processor 22 of the control device 15. In the processor 22, the acquisition unit 22B acquires the captured image group 38. The captured image group 38 includes captured images 40A to 40D acquired at each of a plurality of via points D1 to D4. That is, the captured images 40A to 40D are images captured at the via points D1 to D4, respectively. In the following description, when there is no need to distinguish between the captured images 40A to 40B, they are also simply referred to as "captured images 40". The captured image 40 is an image including an image showing an object B2 to be subjected to three-dimensional measurement. For convenience of explanation, an example in which the captured image 40 includes one object B2 will be described below, but the object of the three-dimensional measurement may be a plurality of objects B. The object B2 is an example of a "second target object" according to the technology of the present disclosure.

[0045] The acquisition unit 22B outputs the acquired captured image group 38 to the image processing unit 22C. The image processing unit 22C executes an object detection process on the captured image group 38 to extract an image region 42 indicating the object B2 in the captured image 40. Specifically, the image processing unit 22C acquires the trained model for object extraction 24C from the storage 24.

[0046] The trained model for object extraction 24C realizes the function of extracting object B2 by, for example, performing machine learning using training data on a neural network. The training data is, for example, a data set obtained from past test results, in which images including object B2 are used as example data and information capable of identifying an image area showing object B2 is used as answer data.

[0047] Then, the image processing unit 22C inputs the captured image 40 to the trained model for object extraction 24C. The trained model for object extraction 24C outputs extraction region information 44 according to the input captured image 40. The extraction region information 44 is information capable of identifying an image region 42 showing the object B2 in the captured image 40. The image processing unit 22C acquires the extraction region information 44 output from the trained model for object extraction 24C. In this manner, the image processing unit 22C extracts the image region 42 showing the object B2 from the captured image 40.

[0048] Although the example of extracting an image region using an AI (Artificial Intelligence) method has been given here, this is merely one example. For example, an image region may be extracted using a pattern matching method.

[0049] In addition, although an example in which image processing is performed on the captured image group 38 consisting of a plurality of still images has been given here, the technology of the present disclosure is not limited to this. For example, an example in which image processing is performed on several frames (e.g., 2 to 3 frames) extracted from a moving image captured by the camera 18 may be used.

[0050] In addition, although an example in which image processing is performed collectively on a plurality of captured images 40 has been given here, the technology of the present disclosure is not limited to this. For example, image processing may be performed in parallel each time a captured image 40 is obtained.

[0051] FIG. 5 is a conceptual diagram showing an example of the processing contents of the three-dimensional measurement unit 22D in the processor 22. As shown in FIG. 5, the three-dimensional measuring unit 22D acquires imaging position information 46 from the acquiring unit 22B. The imaging position information 46 is information indicating the position where the captured image 40 was captured (for example, the three-dimensional coordinates of the optical center of the camera 18 when the captured image 40 was captured). The imaging position information 46 is, for example, attached to the captured image 40. In addition, the three-dimensional measuring unit 22D acquires extraction region information 44 from the image processing unit 22C.

[0052] The three-dimensional measuring unit 22D executes a three-dimensional measurement process of the object B2 using the imaging position information 46 and the extraction area information 44. Specifically, the three-dimensional measuring unit 22D estimates the three-dimensional shape of the object B2 using a volume intersection method for an image area 42 showing the object B2 observed from a plurality of imaging positions. The volume intersection method (SfS; Shape from Silhouette) is a method for estimating the shape of a target object from a silhouette obtained by observing the target object at a plurality of imaging positions. The three-dimensional measuring unit 22D calculates a cone-shaped space F1 virtually extended from the imaging position E1 toward the image area 42 in the captured image 40A. Here, the imaging position E1 is the position where the captured image 40A is captured. The three-dimensional measuring unit 22D also calculates a cone-shaped space F2 virtually extended from the imaging position E2 toward the image area 42 in the captured image 40B. In the example shown in FIG. 5, an example is given in which a three-dimensional shape is estimated from two imaging positions, but it goes without saying that two or more imaging positions may be used.

[0053] Then, the three-dimensional measurement unit 22D estimates the three-dimensional shape G contained in both the space F1 and the space F2 as the shape of the object B2. As a result of the three-dimensional measurement process, object shape information 48 indicating the three-dimensional shape G of the object B2 is obtained. In this manner, the three-dimensional measurement unit 22D estimates the three-dimensional shape of the object B2 based on the captured images 40. More specifically, the three-dimensional measurement unit 22D estimates the three-dimensional shape of the object B2 by using a volume intersection method on image regions 42 in the multiple captured images 40.

[0054] FIG. 6 is a conceptual diagram showing an example of the processing contents of the candidate point calculation unit 22E in the processor 22. As shown in FIG. 6 as an example, candidate point calculation section 22E acquires object shape information 48 from three-dimensional measurement section 22D. First, candidate point calculation section 22E converts three-dimensional shape G indicated by object shape information 48 into a point group H. Point group H is a collection of points corresponding to three-dimensional shape G, and each point has position coordinates in three-dimensional space. Furthermore, candidate point calculation section 22E performs principal component analysis on point group H. As a result, a first principal component σ1, a second principal component σ2, and a third principal component σ3 are obtained for the distribution in three-dimensional space of each point constituting point group H.

[0055] Here, the first principal component σ1 and the second principal component σ2 obtained by the principal component analysis are the direction with the largest variation and the direction with the second largest variation, respectively, in the three-dimensional spatial distribution of the points included in the point group H. The first principal component σ1 and the second principal component σ2 are orthogonal to each other and form a plane. The third principal component σ3 is a direction orthogonal to the first principal component σ1 and the second principal component σ2, and the direction along the third principal component σ3 is a direction in which the variation of the points is smaller than that of the first principal component σ1 and the second principal component σ2. In other words, the direction along the third principal component σ3 can be said to be a direction in which there is little information regarding the distribution of the points in the point group H.

[0056] Then, the candidate point calculation unit 22E sets a candidate point I for three-dimensional measurement on the axis L extending in a direction along the third principal component σ3. In other words, the candidate point calculation unit 22E sets the candidate point I at a position where the axis L along the third principal component σ3 coincides with the optical axis of the camera 18. The candidate point I is a point that is a candidate for the position of the camera 18 during three-dimensional measurement.

[0057] Furthermore, candidate point calculation unit 22E sets candidate point I at a position where the number of points in point group H included in the imaging range (i.e., the angle of view) of camera 18 falls within a preset range. When the number of points in the imaging range is equal to or greater than the lower limit of the preset range (i.e., when camera 18 is sufficiently far from point group H), the silhouette of the target object can be contained within the angle of view. This improves the accuracy of three-dimensional shape measurement. On the other hand, when the number of points in the imaging range is equal to or less than the upper limit of the preset range (i.e., when camera 18 is at an appropriate distance from point group H), spatial resolution is ensured, and measurement of the three-dimensional shape with sufficient accuracy is achieved.

[0058] The candidate point calculation unit 22E calculates an imaging position that satisfies the above-mentioned condition as a candidate point I for the next transport operation (here, an operation to move the object B1). Then, the candidate point calculation unit 22E outputs candidate point information 50 that is information indicating the candidate point I.

[0059] FIG. 7 is a conceptual diagram showing an example of the processing contents of the via point determination unit 22F in the processor 22. As shown in FIG. As an example, as shown in FIG. 7, the via point determination unit 22F acquires point group information 52 and candidate point information 50 from the candidate point calculation unit 22E. The point group information 52 is information indicating the position coordinates of each of a plurality of points included in the point group H. First, the via point determination unit 22F calculates points to which the robot hand 16 can move. Specifically, the via point determination unit 22F sets points to which the robot hand 16 can move using a spherical coordinate system with the center of gravity of the point group H as the origin. More specifically, the via point determination unit 22F derives the center of gravity J of the point group H. Then, the via point determination unit 22F generates lattice points in the spherical coordinate system with the center of gravity J of the point group H as the origin. In generating the lattice points, the via point determination unit 22F generates a point shifted by the first angle difference Δθ and a point shifted by the second angle difference Δφ with respect to the coordinates of the candidate point I, thereby generating the lattice points. Similarly, for a point obtained by shifting the distance r of candidate point I from the origin by the difference Δr, a point shifted by the first angle difference Δθ and a point shifted by the second angle difference Δφ are generated to generate lattice points.

[0060] Then, the via point determination unit 22F searches for points to which the robot hand 16 can move from among the generated lattice points. Specifically, the via point determination unit 22F derives points to which the robot hand 16 can move from among the lattice points based on the length of the link portion 14A of the arm body 14 and the rotational angle of the joint 14B, etc. As a result, movable point information 54, which is information indicating points to which the robot hand 16 can move, is obtained.

[0061] The via point determination unit 22F determines a via point in the next transport operation based on the candidate point information 50 and the movable point information 54. Specifically, the via point determination unit 22F determines a point that is closest to the point to which the robot hand 16 can move and the candidate point I as the via point. For example, the via point determination unit 22F calculates the via point by using a bisection method for the candidate point I and the movable point. Specifically, the coordinates of the candidate point I and the movable point are set as initial values, and a point that is closest to both, obtained by performing a predetermined number of iterative calculations, is determined as the via point. The via point determination unit 22F outputs via point information 56 that is information indicating the determined via point.

[0062] Although the bisection method is used to determine the waypoints, this is merely an example. Waypoints may be determined using other numerical analysis methods.

[0063] FIG. 8 is a conceptual diagram showing an example of the processing contents of the operation control section 22A in the processor 22. As shown in FIG. As an example, as shown in FIG. 8, the operation control unit 22A acquires via point information 56 from the via point determination unit 22F. The operation control unit 22A generates an operation signal 36B based on the via point information 56. The operation signal 36B is a signal indicating a via point in the operation of going to place the object B1. Then, the operation control unit 22A outputs the operation signal 36B to the robot arm 12. The robot arm 12 starts the operation of going to place the object B1 on the platform A2 based on the operation signal 36B. With the operation of the robot arm 12, the robot hand 16 passes through the via points specified in the operation signal 36B. Also, the camera 18 captures images at each via point. In the example shown in FIG. 8, an example of passing through three via points K1 to K3 is shown. In this way, the via points in the operation of going to place the object B1 are set based on the measurement results of the three-dimensional shape of the object B2 during the operation of going to pick up the object B1.

[0064] After the robot arm 12 finishes the transport operation of object B1, the operation control unit 22A starts the transport operation for the next object B (e.g., object B3). In the operation of picking up object B3 in this transport operation, the waypoint of the robot arm 12 is calculated based on the captured image 40 obtained in the operation of placing object B1 in the previous transport operation. In this way, multiple waypoints in a certain transport operation are set based on the measurement results of the three-dimensional shape of object B in the previous transport operation.

[0065] Next, the control process (hereinafter also simply referred to as "control process") of the robot arm 12 in the robot arm system 10 according to this embodiment will be described with reference to Figs. 9 and 10. Fig. 9 is a flowchart showing an example of the control process. Fig. 10 is a flowchart showing an example of the way point calculation process. The flow of the way point calculation process shown in Fig. 10 is an example of an "information processing method" according to the technology of the present disclosure.

[0066] 9 as an example, in the control process, first, in step ST14, the operation control section 22A determines an object B1 to be transported from among a plurality of objects B. After the process of step ST14 is executed, the control process proceeds to step ST16.

[0067] In step ST16, the operation control section 22A causes the robot arm 12 to start an operation to pick up the object B1. After the process of step ST16 is executed, the control process proceeds to step ST18.

[0068] In step ST18, the operation control section 22A causes the camera 18 to capture an image of the object B2 to be captured at each waypoint in the operation of going to pick up the object B1. After the process of step ST18 is executed, the control process proceeds to step ST20.

[0069] In step ST20, the acquisition unit 22B acquires the captured image 40 and the imaging position information 46 obtained by the camera 18. After the process of step ST20 is executed, the control process proceeds to step ST100.

[0070] In step ST100, a waypoint calculation process is executed. After the process of step ST100 is executed, the control process proceeds to step ST22.

[0071] In step ST22, the operation control unit 22A causes the robot arm 12 to execute an operation to place the object B1 so as to pass through the via point obtained as a result of the via point calculation process in step ST100. After the process of step ST22 is executed, the control process proceeds to step ST24.

[0072] In step ST24, the operation control section 22A causes the camera 18 to capture an image of the object B2 to be captured at each waypoint in the operation of placing the object B1. After the process of step ST24 is executed, the control process proceeds to step ST26.

[0073] In step ST26, the acquisition unit 22B acquires the captured image 40 and the imaging position information 46 obtained by the camera 18. After the process of step ST26 is executed, the control process proceeds to step ST100A.

[0074] In step ST100A, a via point calculation process is executed. After the process of step ST100A is executed, the control process proceeds to step ST28.

[0075] In step ST28, the operation control unit 22A determines whether or not a condition for terminating the control process (hereinafter referred to as the "termination condition") has been satisfied. An example of the termination condition is that an instruction to terminate the transport operation has been input via the reception device 30. If it is determined in step ST28 that the termination condition has been satisfied, the determination is affirmative, and the control process is terminated. If it is determined in step ST28 that the termination condition has not been satisfied, the determination is negative, and the control process returns to step ST14.

[0076] 10, in the way point calculation process in steps ST100 and ST100A, in step ST101, the image processing unit 22C extracts an image region 42 indicating an object B2 by executing an object extraction process on the captured image 40 acquired by the acquisition unit 22B. After the process of step ST101 is executed, the way point calculation process proceeds to step ST103.

[0077] In step ST103, the three-dimensional measuring unit 22D executes a three-dimensional measuring process of the object B2 by a volume intersection method based on the extraction result obtained in the image processing unit 22C. After the process of step ST103 is executed, the via point calculation process proceeds to step ST105.

[0078] In step ST105, candidate point calculation section 22E converts the three-dimensional measurement results obtained in three-dimensional measurement section 22D into a point group H. After the process of step ST105 is executed, the via point calculation process proceeds to step ST107.

[0079] In step ST107, the candidate point calculation section 22E performs a principal component analysis on the point group H obtained in step ST105. After the process of step ST107 is executed, the via point calculation process proceeds to step ST109.

[0080] In step ST109, the candidate point calculation unit 22E sets the optical axis of the camera 18 in a direction along the third principal component σ3 of the principal component analysis result obtained in step ST107. After the process of step ST109 is executed, the via point calculation process proceeds to step ST111.

[0081] In step ST111, the candidate point calculation unit 22E sets a candidate point I at a position on the optical axis set in step ST109 where the number of points within the angle of view falls within a predetermined range. After the process of step ST111 is executed, the via point calculation process proceeds to step ST113.

[0082] In step ST113, the via point determination unit 22F generates lattice points in a spherical coordinate system having the origin at the center of gravity of the point group H. After the process of step ST113 is executed, the via point calculation process proceeds to step ST115.

[0083] In step ST115, the via point determination unit 22F searches for a point to which the robot hand 16 can move from among the lattice points obtained in step ST113. After the process of step ST115 is executed, the via point calculation process proceeds to step ST117.

[0084] In step ST117, the via point determination unit 22F determines the via points based on the movable points and the candidate points. After step ST117 is executed, the via point calculation process ends.

[0085] As described above, in the robot arm system 10 according to this embodiment, the acquisition unit 22B of the control device 15 acquires the captured image 40 obtained by the camera 18 provided on the robot hand 16. The captured image 40 is an image obtained by capturing images at a plurality of waypoints through which the robot hand 16 passes during the transport operation of the robot hand 16 for the object B1. The captured image 40 includes an image showing the object B2. The three-dimensional measurement unit 22D estimates the three-dimensional shape of the object B2 based on the acquired captured image 40. This eliminates the need to move the robot hand 16 to a plurality of measurement points to measure the three-dimensional shape, separate from the transport operation, and suppresses an increase in takt time caused by measuring the three-dimensional shape.

[0086] Furthermore, in the robot arm system 10 according to this embodiment, the three-dimensional shape of the object B2 is estimated in the three-dimensional measurement unit 22D by applying a volume intersection method to the result of extracting an image area 42 showing the object B2 from a plurality of captured images 40 obtained at each of a plurality of waypoints. This makes it possible to shorten the time required for three-dimensional measurement even when estimating the three-dimensional shape based on the captured images 40 obtained by the camera 18 provided on the robot hand 16.

[0087] For example, compared to a configuration in which a stereo camera for three-dimensional measurement is provided separately from the robot arm 12, the configuration of the robot arm system 10 can be expected to be simplified and the cost reduced.

[0088] Also, for example, even if the object B2 has various optical characteristics, it is possible to estimate the three-dimensional shape accurately. Specifically, if the object B2 has optical characteristics that are opaque to visible light, it is difficult to grasp the external shape by three-dimensional measurement using a stereo camera. Also, if the object B2 has gloss, the shining position of the object B2 (i.e., the position where the intensity of the reflected light on the object B2 is the highest) changes depending on the angle at which the object B2 is viewed by each of the two cameras. Also, each of the two cameras adjusts its sensitivity according to the shining position of the object B2. This makes it difficult to accurately obtain the parallax between the stereo cameras, making it difficult to perform accurate three-dimensional measurement, or the stereo camera system becomes expensive. On the other hand, in this configuration, the three-dimensional shape of the object B2 is estimated by using a volume intersection method on the result of extracting an image area 42 showing the object B2 from the captured image 40. As a result, even if the object B2 has various optical characteristics, it is possible to perform three-dimensional measurement at low cost and at high speed.

[0089] Moreover, in the robot arm system 10 according to this embodiment, the transport operation is performed for each of the multiple objects B, and the multiple waypoints are set based on the measurement results of the three-dimensional shape of the object B in the immediately preceding transport operation. As a result, it is possible to make the robot hand 16 pass through waypoints that are more suitable for measurement.

[0090] Furthermore, in the robot arm system 10 according to this embodiment, the transport operation by the robot arm 12 includes an operation in which the robot hand 16 moves to grasp the object B1 (i.e., an operation to pick up the object B1), and an operation in which the object B1 grasped by the robot hand 16 is transported to another location (i.e., an operation to place the object B1). The waypoints in the operation to place the object B1 are set based on the measurement results of the three-dimensional shape of the object B2 during the operation to pick up the object. As a result, it is possible to make the robot hand 16 pass through waypoints that are more suitable for measurement.

[0091] Moreover, in the robot arm system 10 according to this embodiment, the waypoints are set based on candidate points I for three-dimensional measurement obtained from the measurement results and movable points of the robot hand 16. This realizes the setting of waypoints that are suitable for three-dimensional measurement and do not restrict the movement of the robot hand 16.

[0092] In addition, in the robot arm system 10 according to this embodiment, the candidate point calculation unit 22E performs a principal component analysis on a point group H consisting of a plurality of points corresponding to the three-dimensional shape of the object B1. Then, the candidate point calculation unit 22E sets a candidate point I at a position where an axis along the third principal component σ3 obtained by the principal component analysis coincides with the optical axis L of the camera 18. In this way, an index indicating the variation of the point group H is quantitatively obtained by the principal component analysis. Furthermore, the optical axis L is made to coincide with the direction along the third principal component σ3, which is a direction perpendicular to the plane consisting of the first principal component σ1 and the second principal component σ2, which is a direction in which the variation is large. This allows the camera 18 to be directed perpendicular to the direction in which the point group H is distributed. As a result, a measurement point more suitable for three-dimensional measurement can be set.

[0093] Furthermore, in the robot arm system 10 according to this embodiment, in the candidate point calculation unit 22E, the candidate point I is set at a position where the number of points in the point group H included in the shooting range of the camera 18 is within a preset range in the direction along the third principal component σ3. As a result, the candidate point I for three-dimensional measurement is set at a position where the number of points included in the shooting range of the camera 18 is equal to or greater than a threshold value and where spatial resolution is ensured, so that a measurement point more suitable for three-dimensional measurement can be set.

[0094] In the robot arm system 10 according to this embodiment, the via point determination unit 22F sets the via points using a spherical reference system with the center of gravity of the object B2 as the origin in three-dimensional measurement. This makes it easy to calculate the via points with the object B2 at the center.

[0095] In the robot arm system 10 according to this embodiment, the via point determining unit 22F determines the via point as a lattice point defined by a spherical coordinate system, and is the closest point to the candidate point and the movable point of the robot hand 16. This allows the setting of a via point that is suitable for three-dimensional measurement and does not restrict the movement of the robot hand 16.

[0096] In the above embodiment, an example in which three-dimensional measurement is performed during both the action of picking up the object B1 and the action of placing the object B1 has been described, but the technology of the present disclosure is not limited to this. For example, three-dimensional measurement may be performed during either the action of picking up the object B1 or the action of placing the object B1.

[0097] In the above embodiment, the robot hand 16 transports the object B from the platform A1 to the platform A2, but the technology of the present disclosure is not limited to this. For example, the object B may be picked up from a state where it is stacked inside a box and transported to another location, or the object B may flow on a conveyer belt and be picked up from there. In addition, for example, the object B may be assembled to another part at the transport destination, or may be handed over to another robot hand or a worker.

[0098] In the above embodiment, an example in which the motion control program 24A and the way point calculation program 24B (hereinafter also simply referred to as "various programs") are stored in the storage 24 has been described, but the technology of the present disclosure is not limited to this. For example, the various programs may be stored in a portable non-transient storage medium such as an SSD or a USB (Universal Serial Bus) memory. The various programs stored in the non-transient storage medium are installed in the control device 15. The processor 22 executes the motion control process and the way point calculation process (hereinafter also simply referred to as "various processes") according to the various programs. In addition, the motion control program 24A and the way point calculation program 24B may be stored as one program in the storage 24 or the non-transient storage medium.

[0099] In addition, various programs may be stored in a storage device such as another computer or server connected to the control device 15 via a network, and the various programs may be downloaded and installed in the control device 15 upon request from the control device 15.

[0100] It is not necessary to store all of the various programs in a storage device of another computer or server device connected to the control device 15, or in the storage 24; only a part of the various programs may be stored therein.

[0101] The hardware resources that execute various processes can be various processors, as shown below. An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource that executes various processes by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, which is a processor having a circuit configuration that is specially designed to execute a specific process, such as an FPGA, PLD, or ASIC. Each processor has a built-in or connected memory, and each processor executes various processes by using the memory.

[0102] The hardware resources that execute various processes may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same type or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resources that execute various processes may be a single processor.

[0103] As an example of a configuration using one processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes various processes. Second, there is a configuration using a processor that realizes the functions of the entire system including multiple hardware resources that execute various processes in a single IC chip, as typified by SoC. In this way, various processes are realized using one or more of the above-mentioned various processors as hardware resources.

[0104] Furthermore, more specifically, the hardware structure of these various processors can be an electric circuit that combines circuit elements such as semiconductor elements. Also, the various processes described above are merely examples. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the order of processes may be changed, without departing from the spirit of the invention.

[0105] The above description and illustrations are detailed descriptions of the parts related to the technology of the present disclosure, and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, function, action, and effect is an example of the configuration, function, action, and effect of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above description and illustrations, within the scope of the gist of the technology of the present disclosure. In addition, in order to avoid confusion and to facilitate understanding of the parts related to the technology of the present disclosure, the above description and illustrations omit explanations of technical common sense that do not require explanation in order to enable the implementation of the technology of the present disclosure.

[0106] In this specification, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. In addition, in this specification, the same idea as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."

[0107] All publications, patent applications, and standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or standard was specifically and individually indicated to be incorporated by reference.

[0108] The following supplementary notes are further disclosed regarding the above embodiment.

[0109] <Appendix 1> A processor is provided. The processor is acquiring images obtained by an imaging device provided on the robot hand at a plurality of waypoints through which the robot hand passes during a transport operation of a first object to be grasped by the robot hand, the images including an image of a second object to be subjected to three-dimensional measurement; A three-dimensional shape of the second target object is estimated based on the acquired image. Information processing device. <Appendix 2> The three-dimensional shape of the second object is estimated by applying a volume intersection method to the results of extracting an image showing the second object from the multiple images obtained at each of the multiple via points. 2. An information processing device according to claim 1. <Appendix 3> the conveying operation is performed for each of the plurality of first target objects; The plurality of waypoints are set based on a measurement result of the three-dimensional shape of the second target object in the immediately preceding transport operation. 3. An information processing device according to claim 1 or 2. <Appendix 4> the transport operation includes a first operation in which the robot hand moves to grasp a first target object, and a second operation in which the first target object grasped by the robot hand is transported to another location; The waypoint in the second operation is set based on a measurement result of a three-dimensional shape of the second target object during the first operation. 4. An information processing device according to any one of claims 1 to 3. <Appendix 5> The via points are set based on the candidate points for the three-dimensional measurement obtained from the measurement results and the movable points of the robot hand. 5. The information processing device according to claim 3 or 4. <Appendix 6> The candidate point is set at a position where an axis along a third principal component obtained by performing principal component analysis on a point group consisting of a plurality of points corresponding to the three-dimensional shape, each point indicating a three-dimensional space coordinate, coincides with an optical axis of the imaging device. 6. The information processing device according to claim 5. <Appendix 7> The candidate point is set at a position where the number of points in the point cloud included in the imaging range of the imaging device falls within a preset range in a direction along the third principal component. 7. The information processing device according to claim 6. <Appendix 8> The movable point is set using a spherical coordinate system having the center of gravity of the second target object in the three-dimensional measurement as its origin. 6. The information processing device according to claim 5. <Appendix 9> The via point is a lattice point defined by the spherical coordinate system, and is the closest point between the candidate point and the movable point of the robot hand. 9. The information processing device according to claim 8. [Explanation of symbols]

[0110] 10 Robot Arm System 12 Robot Arm 14 Arm body 14A Link part 14B Joints 15 Control device 16 Robot Hand 16A,16B Finger 16C Proximal end 18 Camera 20. Computers 22 processors 22A Operation control section 22B Acquisition Department 22C Image processing unit 22D 3D Measurement Section 22E Candidate point calculation section 22F Route point determination section 24 Storage 24A Motion Control Program 24B Waypoint calculation program 24C Pre-trained model for object detection 28 Communication I / F 30 Reception device 32 Display device 34 Bus 36,36A,36B operating signal 38 Images 40, 40A, 40B Captured images 42 Image Area 44 Extraction area information 46 Imaging location information 48 Object shape information 50 Candidate point information 52 Point Cloud Information 54 Movable point information 56 Waypoint Information I candidate point A1, A2 units B,B1,B2 Object C. Base D1, D2, D3, D4 waypoints E1, E2 imaging positions F1,F2 space G Three-dimensional shape H point cloud I candidate point J center of gravity K1, K2, K3 waypoints L axis

Claims

1. A processor is provided. The processor, acquiring images obtained by an imaging device provided on the robot hand at a plurality of waypoints through which the robot hand passes during a transport operation of a first object to be grasped by the robot hand, the images including an image of a second object to be measured in three dimensions; A three-dimensional shape of the second target object is estimated based on the acquired image. Information processing device.

2. The three-dimensional shape of the second target object is estimated by applying a volume intersection method to a result of extracting an image showing the second target object from the plurality of images obtained at each of the plurality of via points. The information processing device according to claim 1 .

3. the conveying operation is performed for each of the plurality of first target objects, The plurality of waypoints are set based on a measurement result of the three-dimensional shape of the second target object in the immediately preceding transport operation. The information processing device according to claim 1 .

4. the transport operation includes a first operation in which the robot hand moves to grasp a first target object, and a second operation in which the first target object grasped by the robot hand is transported to another location, The waypoint in the second operation is set based on a measurement result of a three-dimensional shape of the second target object during the first operation. The information processing device according to claim 1 .

5. The via points are set based on the candidate points for the three-dimensional measurement obtained from the measurement results and the movable points of the robot hand.

5. The information processing device according to claim 3.

6. The candidate point is set at a position where an axis along a third principal component obtained by performing principal component analysis on a point group consisting of a plurality of points corresponding to the three-dimensional shape, each point indicating a three-dimensional spatial coordinate, coincides with the optical axis of the imaging device. The information processing device according to claim 5 .

7. The candidate point is set at a position where the number of points in the point cloud included in the imaging range of the imaging device falls within a preset range in a direction along the third principal component. The information processing device according to claim 6.

8. The movable point is set using a spherical coordinate system having a center of gravity of the second target object in the three-dimensional measurement as its origin. The information processing device according to claim 5 .

9. The via point is a lattice point defined by the spherical coordinate system, and is a point that is closest to the candidate point and the movable point of the robot hand. The information processing device according to claim 8.

10. acquiring images obtained by an imaging device provided on the robot hand at a plurality of waypoints through which the robot hand passes during a transport operation of a first object to be grasped by the robot hand, the images including an image of a second object to be measured in three dimensions; and and estimating a three-dimensional shape of the second target object based on the acquired image. Information processing methods.

11. On the computer, acquiring images obtained by an imaging device provided on the robot hand at a plurality of waypoints through which the robot hand passes during a transport operation of a first object to be grasped by the robot hand, the images including an image of a second object to be measured in three dimensions; and Estimating a three-dimensional shape of the second target object based on the acquired image. A program that executes a process including:

Images