METHOD FOR CREATING LEARNING DATA SET, DEVICE FOR CREATING LEARNING DATA, TRAINING DEVICE, AND ESTIMATION DEVICE
By employing fluorescent markers and machine learning, the method addresses the challenge of estimating three-dimensional shapes of flexible objects, improving robot operation accuracy and versatility in human environments.
Patent Information
- Application Number
- JP2020039132
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2019-09-10
- Filing Date
- 2020-03-06
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2040-03-06
AI Technical Summary
Conventional methods face difficulties in estimating the three-dimensional shape of flexible objects like clothing from image information, necessitating improved object recognition technologies.
A method involving the use of fluorescent markers that emit light under specific illumination conditions, combined with machine learning, to create learning data sets for robot motion control, allowing accurate estimation of gripping positions on flexible objects.
Enables precise robot operations on flexible objects by enhancing object recognition accuracy and simplifying the creation of learning data sets, facilitating cost-effective deployment in human environments.
Smart Images

Figure 0007595421000001 
Figure 0007595421000002 
Figure 0007595421000003
Abstract
Description
[Technical field]
[0001] The present disclosure relates to a method for creating a learning data set, a learning data creation device, a training device, And, estimation equipment Place Regarding. [Background technology]
[0002] Research is being conducted to enable robots to recognize situations and select actions based on image information.
[0003] For example, a technology has been proposed in which a robot equipped with dual arms can pick up a point on a piece of randomly placed clothing, estimate its three-dimensional shape, and repeatedly shift the grip of the clothing between the two arms to find a suitable gripping point for folding (Non-Patent Document 1). [Prior art documents] [Non-patent literature]
[0004] [Non-Patent Document 1] Y. Kita, F. Kanehiro, T. Ueshiba, N. Kita: "Strategy for Folding Clothing on the Basis of Deformable Models", ICIAR 2014, p. 442-452, 2014. Summary of the Invention [Problem to be solved by the invention]
[0005] However, it is difficult to estimate the 3D shape of soft objects such as clothing from image information using conventional methods. Therefore, a new object recognition technology based on image information is required. [Means for solving the problem]
[0006] A method for creating a learning dataset according to an aspect of an embodiment of the present invention includes: lightcapturing an image of the object to obtain a first image; The first lighting condition light is different from the first lighting condition light. Second Lighting Condition light The object is photographed to obtain a second image, and the object includes: The above First lighting condition light A sign portion recognizable in The marker portion is provided at a gripping position of the object. In addition, a method for creating a training dataset according to another aspect of an embodiment of the present invention includes photographing an object under a first lighting condition light to obtain a first image, photographing the object under a second lighting condition light different from the first lighting condition light to obtain a second image, a marker portion that can be recognized under the first lighting condition light is mixed into or applied to the object, the marker portion is a fluorescent material mixed into or applied to the object, and the object is contained in a container. [Brief description of the drawings]
[0007] [Figure 1] 1 is a functional block diagram of a motion control device for a robot according to a first embodiment. FIG. [Diagram 2] FIG. 2 is a hardware configuration diagram of a control device. [Diagram 3] 4 is a flowchart of a learning data acquisition process in the first embodiment. [Figure 4] FIG. 2 is a diagram showing a first example of a captured image of an object in the first embodiment. [Diagram 5] FIG. 4 is a diagram showing a second example of a captured image of an object in the first embodiment. [Figure 6] 4 is a flowchart of robot gripping control in the first embodiment. [Figure 7] FIG. 11 is a functional block diagram of a motion control device for a robot according to a second embodiment. [Figure 8] 13 is a flowchart of robot gripping control in the second embodiment. [Figure 9] FIG. 11 is a functional block diagram of a motion control device for a robot according to a third embodiment. [Figure 10] 13 is a flowchart of a learning data acquisition process in the third embodiment. [Figure 11] FIG. 13 is a diagram showing an example of a captured image of an object in the third embodiment. [Figure 12] 13 is a flowchart of robot gripping control in the third embodiment. [Figure 13] FIG. 13 is a functional block diagram of a motion control device for a robot according to a fourth embodiment. [Figure 14]13 is a flowchart of robot operation control in the fourth embodiment. [Figure 15] FIG. 13 is a configuration diagram of a main part of a robot motion control device according to a modified example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0008] Hereinafter, an embodiment will be described with reference to the accompanying drawings. In order to facilitate understanding of the description, the same components in each drawing are denoted by the same reference numerals as much as possible, and duplicated description will be omitted.
[0009] [First embodiment] The first embodiment will be described with reference to Figures 1 to 6. Figure 1 is a functional block diagram of a robot motion control device 1 according to the first embodiment.
[0010] The robot motion control device 1 shown in Fig. 1 is a device that controls the motion of the robot with respect to an object 20. Information from a light-emitting part that is provided on the object 20 and emits light when irradiated with ultraviolet light (an example of invisible light) is used as an index for determining the robot's motion. Note that this "light emission" only needs to be recognizable by the robot's visual system, and does not necessarily have to be visible to people around the robot.
[0011] Particularly in the first embodiment, the robot's action is a grasping action in which an object 20 such as clothing or a mug is grasped by a dual-arm multi-degree-of-freedom robot arm 5. Also, in the first embodiment, a fluorescent marker 6 is used as the light-emitting unit, which is provided at the grasping position of the object 20 or indicates information on the grasping position. The fluorescent marker 6 in this embodiment is invisible to wavelengths in the visible light range, and is formed of a material that emits light at wavelengths in the visible light range when receiving invisible light, for example, ultraviolet light. For this reason, the fluorescent marker 6 can have an appearance that is unlikely to impede the discrimination of the texture of the object 20, such as transparent or nearly transparent white, when observed with visible light, and does not bother the user.
[0012] Note that the fluorescent marker 6 does not have to be invisible (transparent, white, etc.) to wavelengths in the visible light range depending on the conditions of use, such as the conditions of the training data set to be created. Also, the invisible light is not limited to being invisible, and may be anything that can change the appearance of the fluorescent marker 6. In this case, the visible light in this disclosure can be read as light of a first wavelength, invisible light, or ultraviolet light, and can be read as light of a second wavelength.
[0013] That is, the fluorescent marker can be one that visually changes when irradiated with the first light and when irradiated with the second light. Visual changes include, for example, color changes and changes in contrast with parts other than the fluorescent marker. Specifically, if a phosphor that absorbs blue light and emits yellow light is used as the fluorescent substance contained in the fluorescent marker, the fluorescent marker appears pale yellow under visible light of the first wavelength and appears white or the like under emission of the second wavelength, which is blue light, and the fluorescent marker visually changes.
[0014] In addition, in the present disclosure, "fluorescence" includes fluorescence and phosphorescence.
[0015] 1, when the object 20 is clothing, the task of folding the clothing in a predetermined procedure from a state in which the clothing is grasped by a grasping action can also be included in the actions executed by the robot. In this case, for example, multiple types of fluorescent markers 6 that emit light in different colors are provided on the clothing 20, and the position at which the robot arm grasps the clothing 20 can be changed and operated in an order set by the emission color information, thereby executing the folding action in a predetermined procedure.
[0016] The robot's motion control device 1 can determine the robot's behavior based on image information of the object 20 captured by the camera 4. Machine learning can be used to determine this behavior. As shown in FIG. 1, the object 20 used during learning has a fluorescent marker 6 attached at the position where it should be grasped by the robot (grasping position), and machine learning is performed using a captured image of this object. However, when actually controlling the robot's motion after learning is complete, the object 20 does not need to have the fluorescent marker 6. This is because by using a trained model, the gripping position taught by the fluorescent marker 6 can be estimated from image information of the object 20 alone.
[0017] As shown in FIG. 1, a robot motion control device 1 includes a visible light 2, an ultraviolet light 3, a camera 4, a robot arm 5, and a control device 10.
[0018] The visible light 2 (an example of a visible light irradiating section) irradiates the object 20 with visible light. The ultraviolet light 3 (an example of an invisible light irradiating section) irradiates the object 20 with invisible light. The fluorescent marker 6 provided on the object 20 can be made to emit light by irradiation with ultraviolet light by the ultraviolet light 3. Note that the ultraviolet light 3 may be configured to irradiate light of a wavelength other than ultraviolet light, so long as it is invisible light that can make the fluorescent marker 6 emit light. Furthermore, the light emitted by the ultraviolet light is not limited to being completely invisible to humans or cameras, and may include some wavelengths in the visible light range.
[0019] The camera 4 (an example of an imaging unit) captures an image of an object 20.
[0020] In this embodiment, the robot arm 5 performs a gripping operation of the object 20. The robot arm 5 is a robot arm with multiple degrees of freedom, such as seven degrees of freedom, and can move the hand position to any position and direction. In addition, a robot hand capable of gripping the object 20 is provided at the tip of the robot arm 5 as an end effector, for example. The robot arm 5 can operate each part in response to an operation command output from the control device 10. The robot arm 5 may be a double-armed or single-armed robot, but a double-armed robot is preferable when the object 20 is a flexible object and is to be manipulated as in this embodiment.
[0021] The control device 10 controls the overall operation of the robot. The control device 10 of the present embodiment includes an irradiation control unit 11, an image acquisition unit 12, a learning data creation unit 13, a learning unit 14, a grip position estimation unit 15, and a robot control unit 16 for controlling the robot operation.
[0022] The irradiation control unit 11 controls the irradiation timing of the visible light 2 and the ultraviolet light 3. Specifically, the irradiation control unit 11 can switch between irradiation of the object 20 with visible light by the visible light light 2 and irradiation of the object 20 with ultraviolet light by the ultraviolet light light 3 at a predetermined timing.
[0023] The image acquisition unit 12 acquires an image of the object 20 captured by the camera 4. The image acquisition unit 12 of this embodiment receives information on the switching timing between the visible light 2 and the ultraviolet light 3 from the irradiation control unit 11. Based on this switching timing, the image acquisition unit 12 alternately acquires a visible light image 31 (see FIGS. 5 and 6) of the object 20 being irradiated with visible light, which is captured by the camera 4 while the irradiation control unit 11 is switching the irradiation, and an ultraviolet light image (invisible light image) (see FIGS. 5 and 6) of the object 20 being irradiated with ultraviolet light and the fluorescent markers 6 emitting light.
[0024] The learning data creation unit 13 acquires, as a set of learning data, the visible light image 31 and the ultraviolet light image 32 acquired by the image acquisition unit 12. The learning data creation unit 13 creates a predetermined number of sets of learning data and outputs them to the learning unit .
[0025] The learning unit 14 has a model having a specific input-output relationship, and changes the input-output relationship of the model by machine learning. As the model, for example, a supervised learning model such as a multilayer neural network or a genetic algorithm, or a reinforcement learning model can be implemented. In addition, as a machine learning method for the model, deep learning, random forest regression, kernel ridge regression, etc. can be used. For example, when the model is a supervised learning model, the learning unit 14 executes machine learning of the model using the learning data created by the learning data creation unit 13 so that the input-output relationship of the model approaches the input-output relationship of the learning data. The learning unit 14 outputs the learned model that has completed the machine learning and obtained the input-output relationship based on the learning data to the grip position estimation unit 15. Note that the terms "learning" and "learned model" may also be written as "training", "generation", "trained model", or "generative model".
[0026] The grip position estimation unit 15 (estimation unit) estimates information about the light-emitting unit from the visible light image 31 captured by the camera 4, using a trained model that has been machine-learned by the learning unit 14. Specifically, the grip position estimation unit 15 estimates the position where the fluorescent marker 6 should be in the object 20 from the visible light image 31, and estimates the grip position of the object 20.
[0027] The robot control unit 16 controls the operation of the robot arm 5 with respect to the target object 20 in accordance with the information of the light-emitting unit estimated by the gripping position estimation unit 15. Specifically, when the target object 20 is clothing, the robot control unit 16 controls the operation of the robot arm 5 to grip the gripping position of the clothing 20 estimated by the gripping position estimation unit 15 and execute the task of folding the clothing 20.
[0028] In the first embodiment, the operation time of each element of the control device 10 can be divided into two stages: "learning time" when the learning data creation unit 13 and the learning unit 14 perform machine learning of the model, and "estimation time" when the grip position estimation unit 15 and the robot control unit 16 estimate the grip position of the garment 20 using the learned model and perform the gripping and folding operations by the robot arm 5. As shown in Fig. 1, during learning, an object 20 with a fluorescent marker 6 added to the grip position is used to acquire learning data of the grip position, while during estimation, an object 20 with no fluorescent marker 6 added can be used.
[0029] 1 and the like are merely divided for the sake of convenience in order to explain the multiple functions of the control device 10, and the multiple functional blocks may be a single configuration, or a single functional block may be further divided into multiple configurations. For example, when the output of the trained model is the control amount of the robot arm 5 instead of the gripping position, it is not necessary to explicitly estimate the gripping position, and therefore the gripping position estimation unit 15 can be included in the robot control unit 16.
[0030] In the first embodiment, the learning data creation unit 13, the learning unit 14, and the gripping position estimation unit 15 constitute a "detection unit that detects the light-emitting unit based on the ultraviolet light image 32 captured by the camera 4 while the irradiation control unit 11 is switching the irradiation."
[0031] Fig. 2 is a hardware configuration diagram of the control device 10. As shown in Fig. 2, the control device 10 can be physically configured as a computer system including a CPU (Central Processing Unit) 101, a RAM (Random Access Memory) 102 and a ROM (Read Only Memory) 103 which are main storage devices, an input device 104 such as a keyboard and a mouse which are input devices, an output device 105 such as a display, a communication module 106 which is a data transmission / reception device such as a network card, an auxiliary storage device 107, and the like.
[0032] 1 are realized by loading predetermined computer software onto hardware such as the CPU 101 and the RAM 102, operating the communication module 106, the input device 104, and the output device 105 under the control of the CPU 101, and reading and writing data from and to the RAM 102 and the auxiliary storage device 107. That is, by executing the program of this embodiment on a computer, the control device 10 functions as the irradiation control unit 11, the image acquisition unit 12, the learning data creation unit 13, the learning unit 14 (learned model), the grip position estimation unit 15, and the robot control unit 16 in FIG.
[0033] Each function of the control device 10 may be a circuit configured with an analog circuit, a digital circuit, or an analog-digital mixed circuit. Also, a control circuit for controlling each function may be provided. Each circuit may be implemented using an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or the like.
[0034] At least a part of each function of the control device 10 may be configured with hardware, or may be configured with software, and may be implemented by a CPU or the like through information processing of the software. In the case of being configured with software, the control device 10 and a program for realizing at least a part of its functions may be stored in a storage medium such as a flexible disk or a CD-ROM, and may be read and executed by a computer. The storage medium is not limited to a removable medium such as a magnetic disk or an optical disk, and may be a fixed storage medium such as a hard disk device or a memory. In other words, the information processing by the software may be specifically implemented using hardware resources. Furthermore, the processing by the software may be implemented in a circuit such as an FPGA, and executed by hardware such as a processor. The execution of the job may be performed using an accelerator such as a GPU (Graphics Processing Unit).
[0035] For example, the computer can be the device of the above embodiment by reading out the dedicated software stored in a computer-readable storage medium. The type of storage medium is not particularly limited. The computer can be the device of the above embodiment by installing the dedicated software downloaded via a communication network. In this way, the information processing by the software is specifically implemented using the hardware resources. Note that the hardware such as the memory, the processor, and the computer may each be provided in one or more units.
[0036] The control device 10, the visible light 2, the ultraviolet light 3, and the camera 4 can be provided on the body of a robot equipped with a robot arm 5. Alternatively, the control device 10, the visible light 2, the ultraviolet light 3, and the camera 4 may be installed in a location physically separated from the robot equipped with the robot arm 5 and connected to the robot so as to be able to communicate with the robot. Alternatively, in the control device 10, a learning data creation unit 13 and a learning unit 14 used during learning may be provided separately from the robot, and a grip position estimation unit 15 and a robot control unit 16 used during estimation may be provided within the robot.
[0037] In addition, the control device 10 may not be a single unit, and some of the functions of the control device 10 may be installed separately in physically separate locations. For example, the irradiation control unit 11 of the control device 10 may be provided inside or near the visible light light 2 or the ultraviolet light 3.
[0038] Fig. 3 is a flowchart of a learning data acquisition process in the first embodiment. The process of the flowchart in Fig. 3 is executed before machine learning of the model by the learning unit 14. Note that supervised learning will be exemplified here as a model learning method.
[0039] In step S101, an object 20 (training object) with a fluorescent marker 6 is placed within the imaging range of the camera 4.
[0040] In step S102, the irradiation control unit 11 controls the visible light 2 to be turned on and turns off the ultraviolet light 3, so that the target object 20 is irradiated with visible light.
[0041] In step S103, the image acquiring unit 12 acquires a visible light image 31 (training visible light image) in a state in which the target object 20 is irradiated with visible light.
[0042] In step S104, the irradiation control unit 11 controls the visible light 2 to be turned off and controls the ultraviolet light 3 to be turned on, so that the target object 20 is irradiated with ultraviolet light.
[0043] In step S105, the image acquisition unit 12 acquires an ultraviolet light image 32 (training invisible light image) in a state in which the object 20 is irradiated with ultraviolet light.
[0044] In step S106, the learning data creation unit 13 stores the visible light image 31 acquired in step S103 and the ultraviolet light image 32 acquired in step S105 as a pair of learning data.
[0045] In step S107, the learning data creation unit 13 judges whether the number of pieces of learning data stored is equal to or greater than a predetermined amount. If the number of pieces of learning data is less than the predetermined amount (No in step S107), the process proceeds to step S108, where the viewpoint of the camera 4 and the shape of the object 20 are changed, and the process returns to step S102 to continue acquiring learning data. The viewpoint of the camera 4 can be changed, for example, by providing a mechanism for the camera 4 to automatically move around the mounting base, and the camera 4 rotates while capturing an image around the object 20 placed on the mounting base. The shape of the object 20 can be changed, for example, by changing the grounding portion of the object 20 placed on the mounting base, such as by swapping the top and bottom of the cup, when the object 20 is a rigid body such as a cup. When the object 20 is a flexible object such as clothing, the position of the fluorescent marker 6 on the object 20 can be changed, such as by rolling or folding the clothing.
[0046] Fig. 4 is a diagram showing a first example of a captured image of the object 20 in the first embodiment. Fig. 5 is a diagram showing a second example of a captured image of the object 20 in the first embodiment. In Figs. 4 and 5, (A) is an example of a visible light image 31, and (B) is an example of an ultraviolet light image 32, and (A) and (B) are grouped together as a pair of learning data.
[0047] In the example of Fig. 4, the object 20, a T-shirt, is neatly spread out and placed. As shown in Fig. 4(A), the fluorescent marker 6 does not emit light in visible light, so the fluorescent marker 6 does not appear in the visible light image 31. On the other hand, as shown in Fig. 4(B), the fluorescent marker 6 emits light in the ultraviolet light image 32, and the emission of light at both shoulders of the T-shirt is clearly visible.
[0048] On the other hand, in the example of FIG. 5, the T-shirt of the object 20 is crumpled up. In step S108, the shape of the object 20 may be changed from the state shown in FIG. 4 to the state shown in FIG. 5. In this example, as in the example of FIG. 4, the fluorescent marker 6 is not reflected in the visible light image 31 as shown in FIG. 5(A), and the fluorescent marker 6 is emitted in the ultraviolet light image 32 as shown in FIG. 5(B). This allows the positions of both shoulders, which are the target of the gripping position, to be specified. In this way, by using the positions of the fluorescent marker 6 when the T-shirt of the object 20 has various shapes, that is, the information on the positions of both shoulders, as learning data, it is possible to create a trained model that can appropriately determine a specific position of a specific part of the object that is suitable for manipulation by a robot, even for an object to which the fluorescent marker 6 is not attached. This improves the estimation ability of the trained model, and improves the accuracy of the gripping operation of the robot.
[0049] In addition, by using the fluorescent marker 6, which is invisible under visible light, as the marker, it is possible to easily create a set of images using one object, including an image with the marker displayed and an image without the marker displayed, i.e., an image that can be considered equivalent to an image in which the marker is not applied to the object in the image data. This makes it easy to create a learning dataset.
[0050] On the other hand, if the predetermined number of pieces of learning data or more have been stored (Yes in step S107), this control flow ends. After that, the learning unit 14 performs machine learning of the model using the stored learning data.
[0051] Fig. 6 is a flowchart of robot gripping control in the first embodiment. The processing of the flowchart in Fig. 6 is executed after machine learning of the model by the learning unit 14.
[0052] In step S201, the object 20 (operation object) without the fluorescent marker 6 is placed within the imaging range of the camera 4.
[0053] In step S202, the irradiation control unit 11 controls the visible light 2 to be turned on and turns off the ultraviolet light 3, so that the target object 20 is irradiated with visible light.
[0054] In step S203, the image acquisition unit 12 acquires a visible light image 31 in a state in which visible light is irradiated onto the target object 20. Note that the camera 4 that captures the visible light image 31 acquired in this step may be a different camera from the camera 4 that captured the images in steps S103 and S105 of the learning data acquisition process in FIG.
[0055] In step S204, the visible light image is input to the trained model by the grip position estimation unit 15, and the grip position of the target object 20 is estimated. As described with reference to Fig. 5 and Fig. 6, the input / output relationship of the model is trained so that the trained model outputs the marker position displayed in the ultraviolet light image 32 in response to an input of a visible light image 31 in which the fluorescent marker 6 is not displayed, and therefore, when a visible light image 31 in which the fluorescent marker 6 is not displayed is input to the trained model, the position of the fluorescent marker 6 in this visible light image 31, i.e., the grip position of the target object 20, is output.
[0056] In step S205, the robot control unit 16 performs control so that the robot arm 5 grasps the grasping position estimated in step S204.
[0057] In step S206, the robot control unit 16 executes the robot arm operation after gripping the object 20. For example, if the object 20 is clothing, such an operation may be a task of folding the clothing. If the object 20 is tableware such as a mug, such an operation may be a task of putting the gripped tableware away in a predetermined position.
[0058] Note that multiple types of fluorescent markers 6 may be used as affixed to the object 20. For example, when performing a folding task, multiple types of fluorescent markers 6 that emit light in different colors may be attached to the garment 20 in step S101 of the flowchart in FIG. 3, and the folding order may be taught by the emitted color. In this case, in step S206 of the flowchart in FIG. 6, the gripping position of the garment 20 by the robot arm is changed sequentially in the order taught by the emitted color of the fluorescent markers 6, and the folding operation can be performed in a predetermined procedure. With this configuration, more complicated operations can be taught by the simple task of attaching the fluorescent markers 6 to the object 20. In addition, the multiple types of fluorescent markers 6 may have different shapes and methods of attachment. For example, the first fluorescent marker can be a solid line, and the second fluorescent marker can be a dashed line.
[0059] In the robot motion control device 1 of the first embodiment, a fluorescent marker 6 is attached to the grasping position of an object 20 such as clothing, and an ultraviolet light image 32 in which this fluorescent marker 6 is illuminated is used as learning data for machine learning, making it easy to estimate the grasping position.
[0060] In addition, the robot motion control device 1 of the first embodiment can easily generate learning data by switching between capturing the visible light image 31 and the ultraviolet light image 32 of the object 20 with the fluorescent marker 6. This makes it easier to prepare for machine learning. It also improves the estimation accuracy of the trained model. When controlling the robot motion using the trained model, the gripping position of the object 20 can be estimated only from the visible light image 31 of the object 20 without the fluorescent marker 6, so that the gripping operation of the object 20 by the robot arm 5 can be easily and appropriately performed. As a result, the robot motion control device 1 of the first embodiment can provide a new object recognition technology that is simpler and more accurate based on image information.
[0061] Furthermore, once the learning unit 14 has completed machine learning of the model, in the scene where the robot is actually made to perform a grasping operation (the above-mentioned "estimation time"), it is possible to estimate the grasping position without attaching a fluorescent marker 6 to the object 20, so there is no need to prepare a special object 20 for the robot's operation. This makes it possible to introduce the robot at low cost, for example, into human living environments, improving the versatility of the robot. These advantages are particularly useful for the spread of personal robots used in applications such as providing various types of support for housework in human living environments.
[0062] [Second embodiment] The second embodiment will be described with reference to Fig. 7 and Fig. 8. Fig. 7 is a functional block diagram of a robot movement control device 1A according to the second embodiment. As shown in Fig. 7, the robot movement control device 1A of the second embodiment differs from the first embodiment in that it does not include a learning data creation unit 13 and a learning unit 14.
[0063] Moreover, in the robot motion control device 1A of the second embodiment, the robot 5 grasps the object 20, to which a fluorescent marker 6 has been added, after recognizing the fluorescent marker 6. That is, the grasping position estimation unit 15A refers to the ultraviolet light image 32 and specifies the grasping position of the object 20 from the position of the fluorescent marker 6 shown in the ultraviolet light image 32.
[0064] In the second embodiment, the gripping position estimating unit 15A constitutes "a detection unit that detects a light emitting unit based on an ultraviolet light image 32 captured by the camera 4."
[0065] FIG. 8 is a flowchart of the robot gripping control in the second embodiment.
[0066] In step S301, an object 20 with a fluorescent marker 6 is placed in the imaging range of the camera 4.
[0067] In step S302, the irradiation control unit 11 turns off the ultraviolet light 3 and irradiates the target object 20 with visible light.
[0068] In step S303, the image acquisition unit 12 acquires a visible light image 31 of the object 20 being irradiated with visible light.
[0069] In step S304, the irradiation control unit 11 controls the turning on of the ultraviolet light 3, and the target object 20 is irradiated with ultraviolet light.
[0070] In step S305, the image acquisition unit 12 acquires an ultraviolet light image 32 of the object 20 being irradiated with ultraviolet light.
[0071] In step S306, the gripping position estimation unit 15A processes the ultraviolet light image 32. For example, the gripping position estimation unit 15A processes the ultraviolet light image 32 (see FIG. 4 and FIG. 5(B)) so that the image is two-level grayscale and the marker portion is displayed in white and the other portion is displayed in black, so that the portion in which the light of the fluorescent marker 6 is reflected can be emphasized.
[0072] In step S307, the gripping position estimating unit 15A detects a marker position from the processed ultraviolet light image, and estimates the detected marker position as the gripping position of the object 20.
[0073] In step S308, the robot control unit 16 performs control so that the robot arm 5 grasps the grasping position estimated in step S307.
[0074] In step S309, the robot control unit 16 executes the robot arm operation (such as a task of folding clothes, a task of putting away the mug, etc.) after gripping the object 20.
[0075] The robot motion control device 1A of the second embodiment can directly detect the position of the fluorescent marker 6 in the object 20 from only the ultraviolet light image 32 of the object 20 bearing the fluorescent marker 6 without prior machine learning, and can therefore detect the grasping position of the object 20 easily and with high accuracy.
[0076] In addition, since the fluorescent marker 6 is transparent under visible light, even if the object 20 with the fluorescent marker 6 is placed in a human living environment, it does not get in the way of a human's visual recognition of the object 20, and does not impair the design or aesthetics of the object 20. Furthermore, if the ultraviolet light 3 is turned on at a high frequency (e.g., 30 Hz or higher) that is not visible to humans, the ultraviolet light image 32 for detecting the gripping position can be acquired without the human being being aware of it. Therefore, even in an environment where the robot and the human are close to each other, such as inside a house, the behavior of the robot can be controlled with high precision without affecting the human behavior, and the affinity between humans and robots can be improved.
[0077] In this embodiment, the visible light light 2 may be turned off when the ultraviolet light 3 is turned on. This allows an ultraviolet image 32 in which the fluorescent marker 6 can be clearly recognized to be acquired. On the other hand, if the fluorescent marker 6 can be appropriately recognized, the visible light light 2 may be turned on when the ultraviolet image 32 is acquired. In addition, the ultraviolet image 32 may be acquired in a similar manner even in an environment in which shooting using only ultraviolet light is not possible due to the presence of a visible light source that is difficult or impossible for the operation control device 1A to control, such as sunlight or room lighting. In this case, the ultraviolet image 32 may be an image acquired under visible light with the fluorescent marker 6 acquired under ultraviolet light displayed thereon.
[0078] [Third embodiment] The third embodiment will be described with reference to Figs. 9 to 12. Fig. 9 is a functional block diagram of a robot motion control device 1B according to the third embodiment. As shown in Fig. 9, in the robot motion control device 1B of the third embodiment, the object 21 is an amorphous object such as a liquid or powder or granular material, a gas, or the like, the light-emitting portion (fluorescent marker) is a fluorescent material 7 mixed in the object 21 or applied to the surface of the object 21, a learning system learns the relationship between a visible light image 31 and the external shape of the object 21 due to the emission of the fluorescent material 7 in an ultraviolet light image 32, and a learned model estimates the type, shape, position, and the like of the object 21 from the visible light image 31.
[0079] The control device 10B includes an object recognition unit 17.
[0080] The object recognition unit 17 (estimation unit) estimates information about the light-emitting unit from the visible light image 31 captured by the camera 4, using a trained model. Specifically, the object recognition unit 17 estimates the outer shape of the object 21 from the visible light image 31 captured by the camera 4, using a trained model that has been machine-learned by the learning unit 14. Furthermore, the type of the object 21 may be estimated. For example, if the outer shape of the object 21 moves from above to below and accumulates in a predetermined area, the object recognition unit 17 can recognize that the object 21 is an object with an indeterminate shape, such as a liquid or powder.
[0081] In the third embodiment, the process can be divided into two stages: "learning" in which the learning data creation unit 13 and the learning unit 14 perform machine learning of the model, and "estimation" in which the object recognition unit 17 and the robot control unit 16 use the learned model to perform object recognition of the target object 21 and robot arm operation according to the object recognition. As shown in Fig. 9, in learning, in order to obtain learning data of the external shape of the target object 21, the target object 21 mixed with, for example, a powdered or liquid fluorescent material 7 can be used, while in estimation, the target object 21 without the fluorescent material 7 can be used.
[0082] In the third embodiment, the learning data creation unit 13, the learning unit 14, and the object recognition unit 17 constitute a "detection unit that detects the light-emitting unit based on the ultraviolet light image 32 captured by the camera 4 while the irradiation control unit 11 is switching the irradiation."
[0083] Fig. 10 is a flowchart of a learning data acquisition process in the third embodiment. The process of the flowchart in Fig. 10 is executed before the learning unit 14 performs machine learning of the model.
[0084] In step S401, the object 21 (training object) having the fluorescent material 7 mixed therein is placed within the imaging range of the camera 4.
[0085] In step S402, the irradiation control unit 11 controls the visible light 2 to be turned on and turns off the ultraviolet light 3, so that the target object 21 is irradiated with visible light.
[0086] In step S403, the image acquisition unit 12 acquires a visible light image 31 of the object 21 in a state where visible light is irradiated thereon.
[0087] In step S404, the irradiation control unit 11 controls the visible light 2 to be turned off and controls the ultraviolet light 3 to be turned on, so that the target object 21 is irradiated with ultraviolet light.
[0088] In step S405, the image acquisition unit 12 acquires an ultraviolet light image 32 of the object 21 in a state where the object 21 is irradiated with ultraviolet light.
[0089] In step S406, the learning data creation unit 13 stores the visible light image 31 acquired in step S403 and the ultraviolet light image 32 acquired in step S405 as a pair of learning data.
[0090] In step S407, the learning data creation unit 13 judges whether the number of pieces of learning data stored is equal to or greater than a predetermined amount. If the number of pieces of learning data is less than the predetermined amount (No in step S407), the process proceeds to step S408, where the viewpoint of the camera 4 and the shape of the object 21 are changed, and the process returns to step S402 to continue acquiring learning data. The viewpoint of the camera 4 can be changed, for example, by providing a mechanism for the camera 4 to automatically move around the mounting base, and the camera 4 rotates while capturing an image around the object 21 placed on the mounting base. In addition, the shape of the object 21 can be changed, for example, in the case where the object 21 is water, by increasing or decreasing the amount of water in a plastic bottle, or by increasing the speed at which the water pours from the plastic bottle into a glass to change the shape of the water flowing down.
[0091] Fig. 11 is a diagram showing an example of a captured image of the object 21 in the third embodiment. In Fig. 11, (A) is an example of a visible light image 31, and (B) is an example of an ultraviolet light image 32, and (A) and (B) are collected as a pair of learning data. In the example of Fig. 11, as shown in Fig. 11(A), the fluorescent material 7 does not emit light in visible light, so the fluorescent material 7 is not captured in the visible light image 31 and only the transparent water of the object 21 is captured. On the other hand, as shown in Fig. 11(B), in the ultraviolet light image 32, it is clear that the fluorescent material 7 is emitting light along the shape of the water of the object 21.
[0092] If the predetermined number of pieces of learning data or more have been stored (Yes in step S407), this control flow ends. After that, the learning unit 14 performs machine learning of the model using the stored learning data.
[0093] Fig. 12 is a flowchart of robot gripping control in the third embodiment. The processing of the flowchart in Fig. 12 is executed after machine learning of the model by the learning unit 14.
[0094] In step S501, the object 21 (operation object) not containing the fluorescent material 7 is placed within the imaging range of the camera 4.
[0095] In step S502, the irradiation control unit 11 controls the visible light 2 to be turned on and turns off the ultraviolet light 3, so that the target object 21 is irradiated with visible light.
[0096] In step S503, the image acquisition unit 12 acquires a visible light image 31 of the object 21 in a state where visible light is irradiated thereon.
[0097] In step S504, the object recognition unit 17 inputs the visible light image into the trained model and estimates the object shape of the target object 21. As described with reference to Fig. 11, the input / output relationship of the model is trained so that the shape of the fluorescent material 7 displayed in the ultraviolet light image 32 is output in response to an input of a visible light image 31 in which the fluorescent material 7 is not displayed in the trained model, and therefore, when a visible light image 31 in which the fluorescent material 7 is not displayed is input in the trained model, the shape of the fluorescent material 7 in this visible light image 31, i.e., the outer shape of the target object 21, is output.
[0098] In step S505, object recognition based on the object shape estimated in step S504 is performed by the object recognition unit 17. In the example shown in Fig. 11, the object recognition unit 17 can recognize the remaining amount of water in the plastic bottle and the amount of water that has flowed out.
[0099] In step S506, the robot control unit 16 executes a robot arm operation according to the object recognition result in step S505. For example, if the amount of water remaining in the plastic bottle is equal to or greater than a predetermined amount, an operation of moving the plastic bottle containing the water as the target object 21 to a refrigerator can be executed, and if the amount remaining is equal to or less than the predetermined amount, an operation of throwing the plastic bottle in a trash can can be executed.
[0100] In conventional object recognition technology, it was difficult to recognize transparent objects such as water and glass from images. In particular, in the case of liquids, the shape changes successively when segmenting, so creating training data for segmentation itself was costly. However, by using the method of the third embodiment, for example, a visible light image 31 and an ultraviolet light image 32 captured by switching between visible light and ultraviolet light at high speed, it is possible to easily grasp the position of a transparent object and create annotation data for segmentation. By using a learning model trained based on these data, it is expected that the recognition accuracy of transparent objects will be improved.
[0101] Similarly, for non-transparent objects, data for semantic segmentation can be easily created by applying markers (fluorescent material 7) to parts of the object, or by applying multiple types of markers to different parts that need to be recognized.
[0102] In other words, the robot motion control device 1B of the third embodiment targets objects 21 that are mainly transparent or do not have a stable shape, such as liquids, powders, and gases, and are difficult to identify using general image recognition. Even for such objects 21, learning data can be easily generated by switching between a visible light image 31 and an ultraviolet light image 32 of the object 21 mixed with the fluorescent material 7, making it easier to prepare for machine learning and improving the estimation accuracy of the trained model. In addition, for example, by attaching a marker (fluorescent material 7) to an object 21 whose shape is variable or which can be separated into multiple parts, such as food ingredients, it becomes possible to recognize the object as the same object based on the marker even after the object is processed, such as by cutting.
[0103] Specific examples of the object 21 in the third embodiment include water splashes, oil splashes, finely chopped food, powdered medicine on a spoon, steam, etc. As with the object 21, other objects that are difficult to identify using general image recognition include transparent objects such as glass or transparent resin containers, fences, wire mesh, etc., where the background is visible through the transparent objects and the edges are difficult to detect, objects whose shapes change irregularly such as slime, objects with mirror finish such as window glass and mirrors, and objects that reflect light such as the surface of water and corridors. The fluorescent material 7 can be applied to the object 21 by, for example, applying it to the surface of the object 21, or by mixing it into the object 21 when it is difficult to apply it to the surface of the object 21.
[0104] [Fourth embodiment] The fourth embodiment will be described with reference to Fig. 13 and Fig. 14. Fig. 13 is a functional block diagram of a robot movement control device 1C according to the fourth embodiment. As shown in Fig. 13, the robot movement control device 1C of the fourth embodiment does not include a learning system (learning data creation unit 13, learning unit 14), and includes, as a light-emitting unit, a fluorescent information unit 8 that emits light when irradiated with ultraviolet light and includes various information related to an object 22.
[0105] The control device 10C has an information reading unit 18 as an element replacing the grip position estimating unit 15 of the first embodiment.
[0106] The information reading unit 18 reads various pieces of information about the object 22 based on the fluorescent information section 8 in the ultraviolet light image 32. The fluorescent information section 8 is, for example, a two-dimensional barcode that emits light when irradiated with invisible light, as shown in FIG.
[0107] Robot control unit 16 controls the operation of robot arm 5 according to various information related to object 22 read by information reading unit 18. For example, in the example of Fig. 13, object 22 is a doorknob, and fluorescent information unit 8 includes information that the door to which this doorknob is attached is a push-open type, and robot control unit 16 causes robot arm 5 to grasp doorknob 22 and push the door open according to the information from fluorescent information unit 8.
[0108] In the fourth embodiment, the information reading unit 18 constitutes "a detection unit that detects a light emitting unit based on an ultraviolet light image 32 captured by the camera 4 while the irradiation control unit 11 is switching the irradiation."
[0109] FIG. 14 is a flowchart of robot operation control in the fourth embodiment.
[0110] In step S601, an object 22 with a fluorescent information section 8 (fluorescent two-dimensional barcode) attached thereto is placed within the imaging range of the camera 4.
[0111] In step S602, the irradiation control unit 11 controls the visible light 2 to be turned on and turns off the ultraviolet light 3, so that the object 22 is irradiated with visible light.
[0112] In step S603, the image acquisition unit 12 acquires a visible light image 31 of the object 22 being irradiated with visible light.
[0113] In step S604, the irradiation control unit 11 controls the turning on of the ultraviolet light 3, and the target object 22 is irradiated with ultraviolet light.
[0114] In step S605, the image acquisition unit 12 acquires an ultraviolet light image 32 of the object 22 being irradiated with ultraviolet light.
[0115] In step S606, information reading unit 18 processes ultraviolet light image 32. Information reading unit 18 processes ultraviolet light image 32, for example, by converting the image into two gradations so that the portion of ultraviolet light image 32 in which the light of fluorescent information section 8 is captured is displayed in white and the other portions are displayed in black, so that the portion of ultraviolet light image 32 in which the light of fluorescent information section 8 is captured can be emphasized.
[0116] In step S607, information reading unit 18 detects the two-dimensional barcode in fluorescence information unit 8 from the processed ultraviolet light image.
[0117] In step S608, the information reading unit 18 reads information about the object 22 from the detected two-dimensional barcode.
[0118] In step S609, the robot control unit 16 executes a robot arm operation according to the information of the object 22 read in step S608. For example, if the object 22 is a doorknob and the information read from the fluorescent information unit 8 is that the door to which this doorknob is attached is a push-open type, the robot control unit 16 causes the robot arm 5 to execute a task of gripping the doorknob 22 and pushing the door open.
[0119] In the robot movement control device 1C of the fourth embodiment, the information of the fluorescent information section 8 in the object 22 is directly read only from the ultraviolet light image 32 of the object 22 having the fluorescent information section 8 attached thereto, so that the robot can easily obtain the information necessary to determine the robot's movement.
[0120] Furthermore, since the fluorescent information unit 8 is transparent under visible light, even if the object 22 with the fluorescent information unit 8 is placed in, for example, a human living environment, it does not get in the way of a human's visual recognition of the object 22, and does not impair the design or aesthetics of the object 20. Furthermore, if the visible light light 2 and the ultraviolet light 3 are switched at a high frequency (for example, 30 Hz or higher) that is not visible to humans, the ultraviolet light image 32 for obtaining information on the fluorescent information unit 8 can be acquired without the human being being aware of it. Therefore, even in an environment where the robot and the human are close to each other, such as inside a house, the behavior of the robot can be controlled with high precision without affecting the human behavior, and the affinity between humans and robots can be improved.
[0121] [Variations] A modification of the above embodiment will be described with reference to Fig. 15. Fig. 15 is a configuration diagram of the main parts of a robot motion control device 1D according to the modification. Fig. 15 illustrates an imaging system of the motion control device 1D, such as the camera 4 and the light source 24, and omits illustration of elements similar to those of the above embodiment. The robot motion control device 1D according to the modification includes a control device 10D similar to the control devices 10, 10A, 10B, and 10C of the above embodiments, and can perform robot behavior learning and behavior planning similar to those of the above embodiments.
[0122] In the modified example, the "light-emitting portion (such as the fluorescent marker 6) that is recognizable under invisible light" used in the above embodiment is replaced with a "reflective portion including a retroreflective material."
[0123] Retroreflective materials are materials that cause the phenomenon of retroreflection, in which light is reflected in the same direction as the light was incident. Generally, there are three types of "reflection": diffuse reflection, specular reflection (regular reflection), and retroreflection. Diffuse reflection is the most common type of reflection, and occurs when light hits a material with an uneven surface and the reflected light is diffused in various directions. Specular reflection occurs when light hits a material with a smooth surface, such as a mirror, and the reflected light is reflected in one direction according to the principle that the angle of incidence = angle of reflection. In contrast, retroreflection is a phenomenon in which light is reflected in the same direction as the incident light by a special material.
[0124] In a modified example, the retroreflection of this retroreflective material is used to create a marker 9 (reflective portion) that disappears and appears depending on the position of the light source 24 as seen by the camera 4.
[0125] The procedure for performing behavior learning and behavior planning of a robot using the marker 9 including a retroreflective material is, for example, as follows. (1) Apply retroreflective paint to the position on the object 23 where you want to attach the marker 9. (2) When the object 23 is viewed from the camera 4 under normal ambient light (such as sunlight or visible light), it appears normal due to diffuse reflection from the original material of the object 23 (i.e., the light emitted by the marker 9 is not visible). (3) On the other hand, when light is emitted from light source 24 from near camera 4 in the same direction as the camera 4 is looking, the marker portion reflects the light toward camera 4 by retroreflection, and the reflected light appears stronger than the other (diffusely reflected) portions. (4) The light returning from this retroreflection can be observed by camera 4, and the marker portion can be extracted, for example, using a brightness threshold.
[0126] In the modified example, as shown in Fig. 15, a light source 24 is provided that irradiates light in the same direction as the field of view of the camera 4. The camera 4 can capture an image including reflected light from the marker 9 only when the light source 24 is turned on. In the modified example, the control device 1D can also perform model learning and behavior planning for the robot by using an image including reflected light from the marker 9 when the light source 24 is turned on (reflected light image) as in the above embodiment, and by identifying a gripping point. Incidentally, irradiating light in the same direction as the field of view of the camera 4 means irradiating light in the same direction to an extent that the reflected light of the light can be captured by the camera.
[0127] That is, the "marker 9 (reflective section) including a retroreflective material" in the modified example corresponds to the "light-emitting section" in the embodiment, and the higher-level concept of both is also expressed as the "sign section." The "environment in which light is irradiated in the same direction as the field of view of the camera 4" in the modified example corresponds to the "under invisible light" in the embodiment, and the higher-level concept of both is also expressed as the "first lighting condition." The "under ambient light" in the modified example corresponds to the "under visible light" in the embodiment, and the higher-level concept of both is also expressed as the "second lighting condition." The "light irradiated in the same direction as the field of view of the camera 4" in the modified example corresponds to the "invisible light" in the embodiment, and the higher-level concept of both is also expressed as the "first lighting condition light." The light source 24 in the modified example corresponds to the ultraviolet light 3 in the embodiment, and the higher-level concept of both is also expressed as the "first lighting condition light irradiating section." The "image including reflected light from marker 9 when light source is turned on (reflected light image)" of the modified example corresponds to the "invisible light image" of the embodiment, and the higher-level concept of both is also referred to as the "first image." Furthermore, the "normal ambient light when light source 24 is turned off" of the modified example corresponds to the "visible light" of the embodiment, and the higher-level concept of both is also referred to as the "second lighting condition light." The "image not including reflected light from marker 9 when light source 24 is turned off (ambient light image)" of the modified example corresponds to the "visible light image" of the embodiment, and the higher-level concept of both is also referred to as the "second image."
[0128] The method of using a retroreflective material in the light-emitting portion as in the modified example does not require limiting normal ambient light (for example, by darkening it or filtering out light of a specific wavelength), compared to the method of using a fluorescent material in the light-emitting portion as in the above embodiment, and therefore makes it easier to obtain an "image including reflected light from the marker 9 when the light source 24 is turned on" to be used for model learning, etc. For this reason, it is particularly useful in outdoor implementation where it is difficult to control ambient light, and it is possible to expand the environment in which this embodiment can be implemented, thereby improving versatility.
[0129] The present embodiment has been described above with reference to specific examples. However, the present disclosure is not limited to these specific examples. Any design changes made by a person skilled in the art to these specific examples are also included within the scope of the present disclosure as long as they have the features of the present disclosure. The elements of each of the above-mentioned specific examples and their arrangements, conditions, shapes, etc. are not limited to those exemplified and can be changed as appropriate. The elements of each of the above-mentioned specific examples can be appropriately combined as long as no technical contradiction occurs. Furthermore, the present disclosure includes any combination of the features of the above-mentioned multiple embodiments.
[0130] In the above embodiment, the configuration of irradiating ultraviolet light to obtain an image emitted by the light-emitting unit is exemplified, but a configuration of irradiating invisible light other than ultraviolet light may be used if the light emission of the light-emitting unit can be made stronger than when visible light is irradiated. In addition, invisible light refers to light rays with wavelengths other than visible light. Also, for example, a configuration may be used in which, under visible light or in a state in which visible light is continuously irradiated, invisible light that can make the light emission of the light-emitting unit stronger than under visible light is switched on and off to provide a first illumination condition and a second illumination condition. Then, data regarding the light-emitting unit can be created by taking the difference between the images obtained when the invisible light is turned off and when it is turned on.
[0131] Furthermore, the sign is not limited to one that uses the retroreflective material or fluorescent material described above.The first illumination condition and the second illumination condition can also be changed as appropriate.
[0132] In the above embodiment, the configuration in which the imaging unit such as the camera 4 is fixed during acquisition of the first image and the second image is exemplified, but the imaging unit may move during acquisition of the first image and the second image. In this case, if the first image acquired under the first lighting condition and the second image acquired under the second lighting condition are shifted more than expected, and learning data is created without any change, there is a risk that parts other than those where the sign unit is not actually provided will be processed as sign units. In such a case, it is preferable to process the images so that the positions of the light-emitting units in the two images are appropriately overlapped, for example, so that they can be considered to have been taken from the same position. Examples of image processing include translation, rotation, and changing the shape of the image. This processing can be performed using, for example, a trained machine learning model or a conversion by matching from image features without machine learning.
[0133] In the above embodiment, a grasping operation in which the dual-arm robot arm 5 grasps the target object 20, 21, 22 is exemplified as an example of the robot operation controlled by the robot operation control device 1, 1A, 1B, 1C, but an operation other than the grasping operation may be applied. For example, the present invention also includes the action of the robot using an element other than the robot arm 5, such as a method other than the grasping operation, for example, manipulation of an object by suction or the like, or setting the position of a light emitting part as a goal and moving toward a goal point detected by the robot.
[0134] In the above embodiment, the robot has been described as an example, but the present disclosure is not limited thereto. The present disclosure may be realized as a system. The robot in the above embodiment is an example of a system.
[0135] In the above embodiment, the creation of the learning data set and the learning of the learning model using the learning data set are performed consecutively, but the present invention is not limited to this.
[0136] In this specification, "transparent" does not necessarily mean completely transparent, but also includes transparency that is difficult for humans or imaging units such as cameras to recognize.
[0137] In addition to the above description, the following items are disclosed. (Appendix 1) capturing an image of an object having a recognizable sign portion provided thereon under a first illumination condition; acquiring a first image in which the sign portion can be recognized and a second image obtained by capturing the object under a second illumination condition; How to create datasets for machine learning. (Appendix 2) A method for generating a learning model in which a first image in which a sign attached to an object is recognizable under a first lighting condition and a second image in which the object is captured under a second lighting condition are used as input to estimate information about the sign. (Appendix 3) an imaging unit configured to image an object having a sign portion that is recognizable under a first illumination condition; a second illumination condition light irradiating unit that irradiates the object with light under a second illumination condition; a first illumination condition light irradiating unit that irradiates the object with light under a first illumination condition; an illumination control unit that switches between illumination of the light under a second illumination condition onto the object by the light illumination unit under the second illumination condition and illumination of the light under a first illumination condition onto the object by the light illumination unit under the first illumination condition at a predetermined timing; and a learning data creation unit that acquires a second image in a state where the object is irradiated with light under the second lighting condition, and a first image in a state where the object is irradiated with light under the first lighting condition and the marker unit is recognizable. (Appendix 4) a trained model that is trained using as learning data a second training image in which a training object is captured under a second lighting condition and a first training image in which the training object is illuminated under a first lighting condition and a marker portion provided on the training object is in a state that can be recognized under the first lighting condition; an estimation unit that estimates information about the sign unit from a second image captured by an imaging unit using the trained model; An inference device comprising: (Appendix 5) An inference device according to claim 4; a robot control unit that creates a robot action plan in response to the information of the sign unit estimated by the estimation unit, and controls an operation of the robot based on the action plan; A robot motion control device comprising: (Appendix 6) an imaging unit configured to image an object having a sign portion that is recognizable under a first illumination condition; a first illumination condition light irradiating unit that irradiates the object with light under a first illumination condition; a detection unit that detects the marker based on a first image in a state in which the object is irradiated with the light under the first illumination condition; a robot control unit that creates a robot action plan based on information about the sign unit detected by the detection unit; A robot motion control device comprising: (Appendix 7) A trained model that is trained using a set of learning data that includes a second image of an object captured under second lighting conditions and a first image of a sign provided on the object captured under first lighting conditions in which the sign is recognizable under the first lighting conditions. (Appendix 8) A robot, The trained model described in Appendix 7; and a control unit that controls an operation of the robot based on a second image in a state in which the operation target object is irradiated with light under a second illumination condition; A robot equipped with (Appendix 9) The identification unit is a light-emitting unit that can be recognized under invisible light, 2. A method for creating a machine learning dataset as described in Appendix 1, wherein the second lighting condition is under visible light and the first lighting condition is under invisible light in which the light-emitting unit emits light. (Appendix 10) 2. The method for creating a machine learning dataset described in Appendix 1, wherein the sign portion is a reflective portion including a retroreflective material, the second lighting condition is ambient light, and the first lighting condition is a condition under which an imaging unit can capture an image of the light reflected by the reflective portion. (Appendix 11) An image of an object having a light-emitting portion that can be recognized under invisible light is captured; acquiring a visible light image of the object captured under visible light and an invisible light image of the object captured under visible light from the light emitting unit; A method for creating a machine learning dataset as described in Appendix 1. (Appendix 12) 12. The method for creating a machine learning dataset described in claim 11, wherein irradiation of the object with visible light by a visible light irradiation unit for acquiring the visible light image and irradiation of the object with invisible light by an invisible light irradiation unit for acquiring the invisible light image are switched at a predetermined timing, and the visible light image and the invisible light image are acquired by a single imaging unit. (Appendix 13) Using a visible light image obtained by capturing an object under visible light and an invisible light image in which a light-emitting unit provided on the object that emits light under invisible light is visible, information on the light-emitting unit is estimated using the visible light image as an input. A method for generating a learning model as described in Appendix 2. (Appendix 14) an imaging unit that images an object having a light-emitting unit that can be recognized under invisible light; A visible light irradiation unit that irradiates the object with visible light; an invisible light irradiating unit that irradiates the object with invisible light; an irradiation control unit that switches between irradiation of the object with visible light by the visible light irradiation unit and irradiation of the object with invisible light by the invisible light irradiation unit at a predetermined timing; a learning data creation unit that acquires a visible light image in a state where the visible light is irradiated onto the object, and an invisible light image in a state where the invisible light is irradiated onto the object and the light emitting unit is emitting light, 4. A learning data creation device as described in appendix 3. (Appendix 15) a trained model that is trained using as learning data a training visible light image in which a training object is captured under visible light and a training invisible light image in which the training object is irradiated under invisible light and a light-emitting unit that is recognizable under invisible light and is provided on the training object is emitting light; an estimation unit that estimates information about the light-emitting unit from a visible light image captured by an imaging unit using the trained model; Equipped 5. The inference apparatus of claim 4. (Appendix 16) a trained model that is trained using as learning data a training visible light image in which a training object is captured under visible light and a training invisible light image in which the training object is irradiated under invisible light and a light-emitting unit that is recognizable under invisible light and is provided on the training object is emitting light; an estimation unit that estimates information about the light-emitting unit from a visible light image captured by an imaging unit using the trained model; and a reasoning device having a robot control unit that creates a robot action plan in response to the information of the light-emitting unit estimated by the estimation unit, and controls an operation of the robot based on the action plan; Equipped 6. A motion control device for a robot according to claim 5. (Appendix 17) The light emitting unit is a fluorescent marker that is installed at a gripping position of the training object, the estimation unit is a grip position estimation unit that estimates a position of the fluorescent marker from a visible light image captured by the imaging unit by using the trained model to estimate a grip position of an operation target object, the robot control unit controls an operation of the robot so as to grasp the grasp position of the operation target object estimated by the grasp position estimation unit. 17. An apparatus for controlling the motion of a robot according to claim 16. (Appendix 18) the operation object is clothing, the robot control unit controls the robot to grasp the garment at the grasping position estimated by the grasping position estimation unit and execute a task of folding the garment. 18. An apparatus for controlling the motion of a robot according to claim 17. (Appendix 19) The fluorescent markers are provided on the clothing in a plurality of types that emit light in different colors; The grip position estimation unit estimates position information and emission color information of the fluorescent marker, the robot control unit changes the gripping position and performs the folding operation in an order set by the light emission color information. 19. An apparatus for controlling the motion of a robot as described in appendix 18. (Appendix 20) The light-emitting unit is a fluorescent material that is mixed into the training object or applied to a surface of the training object, the estimation unit has an object recognition unit that estimates a type of an operation target object from a visible light image by using the trained model; the robot control unit controls an operation of the robot in accordance with the type of the operation target object estimated by the object recognition unit. 20. A motion control device for a robot according to any one of claims 16 to 19. (Appendix 21) The training object is a liquid, a powder, or a gas; The fluorescent material is mixed into the training object when the learning data is acquired. 21. A motion control device for a robot according to claim 20. (Appendix 22) an imaging unit that images an object having a light-emitting unit that can be recognized under invisible light; an invisible light irradiating unit that irradiates the object with invisible light; a detection unit that detects the light emitting unit based on an invisible light image in a state in which the invisible light is irradiated onto the object; a robot control unit that creates a robot behavior plan based on information about the light-emitting unit detected by the detection unit; Equipped 7. A motion control device for a robot according to claim 6. (Appendix 23) The robot grasps and manipulates the object using a robot arm, The light emitting unit is a fluorescent marker that indicates a gripping position of the object, the detection unit has a gripping position estimation unit that calculates a gripping position of the target object based on a position of the fluorescent marker in the invisible light image, the robot control unit controls an operation of the robot arm so as to grasp the target object at the grasp position calculated by the grasp position estimation unit. 23. A motion control device for a robot according to claim 22. (Appendix 24) The light emitting unit is a fluorescent information unit that emits light by irradiation with invisible light and includes various information related to the object, the detection unit has an information reading unit that reads various pieces of information about the object based on the fluorescent information portion in the invisible light image, The robot control unit controls an operation of the robot in accordance with various information related to the object read by the information reading unit. 24. A motion control device for a robot according to claim 22 or 23. (Appendix 25) The fluorescent information portion is a two-dimensional barcode that emits light when irradiated with invisible light. 25. An apparatus for controlling the motion of a robot as described in appended claim 24. (Appendix 26) The device is trained using learning data that includes a set of learning data, which is a visible light image of an object captured under visible light and an invisible light image of an object captured under invisible light in which a light-emitting portion that is provided on the object and emits light under invisible light is visible. The trained model described in Appendix 7. (Appendix 27) 27. The trained model of claim 26, capable of estimating information about the light-emitting unit based on an input of a visible light image of an object to be operated. (Appendix 28) A robot, a trained model that is trained using learning data in which a set of learning data is a visible light image of an object captured under visible light and an invisible light image of an object captured under invisible light in which a light-emitting unit that is provided on the object and emits light under invisible light is made visible; a control unit that controls an operation of the robot based on a visible light image of a state in which the visible light is irradiated onto an operation target object; Equipped 10. The robot described in appended claim 8. (Appendix 29) An image of an object is captured in an environment where light is irradiated in the same direction as the field of view of the imaging unit, the object having a reflective portion including a retroreflective material that reflects light back to the imaging unit; acquiring an ambient light image obtained by capturing an image of the object under ambient light, and a reflected light image including the reflected light from the reflecting portion; A method for creating a machine learning dataset as described in Appendix 1. (Appendix 30) 30. A method for creating a machine learning dataset as described in Appendix 29, wherein a light source for irradiating the object with light to obtain the reflected light image and a light source for obtaining the ambient light image are switched on and off at a predetermined timing, and the reflected light image and the ambient light image are obtained by a single imaging unit. (Appendix 31) Using an ambient light image obtained by capturing an object under ambient light and a reflected light image in which a reflective portion provided on the object and including a retroreflective material that reflects light back to the imaging unit in an environment where light is irradiated in the same direction as the field of view of the imaging unit is visible, information on the reflective portion is estimated using the ambient light image as an input. A method for generating a learning model as described in Appendix 2. (Appendix 32) An imaging unit that captures an image of an object; A reflecting portion provided on the object and including a retroreflective material that reflects light back to the imaging portion in an environment in which light is irradiated in the same direction as the field of view of the imaging portion; A light source that irradiates the object with the light; an illumination control unit that switches the light source on and off at a predetermined timing; a learning data creation unit that acquires a reflected light image in a state in which the light source is turned on and the object is irradiated with light from the light source in the same direction as the field of view of the imaging unit, causing the reflector to emit light, and an ambient light image in a state in which the light source is turned off and the object is irradiated with ambient light, 4. A learning data creation device as described in appendix 3. (Appendix 33) A trained model is trained using as learning data a training ambient light image in which a training object is captured under ambient light, and a training reflected light image in a state in which a reflective portion, which is provided on the training object and includes a retroreflective material that reflects reflected light back to the imaging portion in an environment in which light is irradiated in the same direction as the field of view of the imaging portion, is emitting light; an estimation unit that estimates information about the reflecting portion from an ambient light image captured by the imaging unit using the trained model; Equipped 5. The inference apparatus of claim 4. (Appendix 34) A trained model is trained using as learning data a training ambient light image in which a training object is captured under ambient light, and a training reflected light image in a state in which a reflective portion, which is provided on the training object and includes a retroreflective material that reflects reflected light back to the imaging portion in an environment in which light is irradiated in the same direction as the field of view of the imaging portion, is emitting light; an estimation unit that estimates information about the reflecting portion from an ambient light image captured by the imaging unit using the trained model; and a robot control unit that creates a robot action plan in response to the information of the reflector estimated by the estimation unit, and controls an operation of the robot based on the action plan; Equipped 6. A motion control device for a robot according to claim 5. (Appendix 35) An imaging unit that captures an image of an object; A reflecting portion provided on the object and including a retroreflective material that reflects light back to the imaging portion in an environment in which light is irradiated in the same direction as the field of view of the imaging portion; a detection unit that detects the reflection portion based on a reflected light image in a state in which the light is irradiated onto the object; a robot control unit that creates a robot action plan based on information about the reflecting unit detected by the detection unit; Equipped 7. A motion control device for a robot according to claim 6. (Appendix 36) The device is trained using a set of learning data, which is an ambient light image of an object captured under ambient light and a reflected light image captured under an environment in which a reflective portion including a retroreflective material that reflects reflected light back to the imaging unit under an environment in which light is irradiated in the same direction as the field of view of the imaging unit and that is provided on the object, is visible. The trained model described in Appendix 7. (Appendix 37) A robot, a trained model trained with learning data including a set of learning data including an ambient light image of an object captured under ambient light and a reflected light image captured under an environment in which a reflective portion provided on the object and including a retroreflective material that reflects reflected light back to the imaging portion under an environment in which light is irradiated in the same direction as the field of view of the imaging portion is visible; A control unit that controls an operation of the robot based on an ambient light image obtained by capturing an object to be operated under ambient light; Equipped 10. The robot described in appended claim 8. [Explanation of symbols]
[0138] 1, 1A, 1B, 1C, 1D Robot motion control device 2 Visible light light (visible light irradiation unit, second lighting condition light irradiation unit) 3. Ultraviolet light (invisible light irradiation unit, first lighting condition light irradiation unit) 4 Camera (imaging unit) 5. Robot Arm 6 Fluorescent marker (light-emitting part, marking part) 7 Fluorescent material (light-emitting part, sign part) 8 Fluorescence information section (light-emitting section, label section) 9 Markers (reflective parts, sign parts) 10, 10A, 10B, 10C, 10D Control device 11 Irradiation control unit 12 Image acquisition section 13. Learning data creation unit (detection unit) 14 Learning section (detection section) 15, 15A Grasp position estimation unit (estimation unit, detection unit) 16 Robot control unit 17 Object recognition unit (estimation unit, detection unit) 18 Information reading unit (estimation unit, detection unit) 20, 21, 22, 23 Object 31 Visible light image (second image) 32 Ultraviolet light image (invisible light image, 1st image) 24 light source (first illumination condition light irradiation unit)
Claims
1. capturing an image of an object using light under a first illumination condition to obtain a first image; acquiring a second image by photographing the object under a second illumination condition light different from the first illumination condition light; the object is provided with a mark portion recognizable under the first illumination condition light, The marker is provided at a gripping position of the object. How to create a training dataset.
2. capturing an image of an object using light under a first illumination condition to obtain a first image; acquiring a second image by photographing the object under a second illumination condition light different from the first illumination condition light; A mark portion recognizable under the first illumination condition light is mixed into or applied to the object, The marking portion is a fluorescent material mixed into or applied to the object, The object is contained in a container. How to create a training dataset.
3. The sign portion is a light-emitting portion that can be recognized by invisible light, The method for creating a training dataset according to claim 1 or 2, wherein the light under the second illumination condition is visible light, and the light under the first illumination condition is invisible light emitted by the light-emitting unit.
4. The method for creating a training dataset according to claim 1 , wherein the marker portion is a fluorescent marker.
5. The method for creating a training dataset according to claim 1 or 4, wherein the target object includes a plurality of types of the marker portions that emit light in different colors.
6. The method for creating a training dataset according to claim 2 , wherein the target object is an irregular object.
7. The method for creating a training dataset according to claim 2 , wherein the object is a liquid, a powder or a gas.
8. The method for creating a training dataset according to claim 2 , wherein the object is water.
9. The method for creating a training dataset according to claim 2 , wherein the object is transparent.
10. The method for creating a training dataset according to claim 2 , wherein the container is a cup.
11. 2. The method for creating a training dataset according to claim 1, wherein the sign portion is a reflective portion including a retroreflective material, the second lighting condition light is ambient light, and the first lighting condition light is a condition under which reflected light from the reflective portion can be captured by an imaging unit.
12. an imaging unit configured to image an object having a sign portion that is recognizable under light of a first illumination condition; a second illumination condition light irradiating unit that irradiates the object with light under a second illumination condition; a first illumination condition light irradiating unit that irradiates the object with light under a first illumination condition different from the light under the second illumination condition; an illumination control unit that switches between illumination of the light under the second illumination condition by the second illumination condition light irradiating unit onto the object and illumination of the light under the first illumination condition by the first illumination condition light irradiating unit onto the object at a predetermined timing; a learning data creation unit that acquires a second image in a state where the object is irradiated with the light under the second illumination condition, and a first image in a state where the object is irradiated with the light under the first illumination condition and the marker unit is recognizable, A learning data creation device, wherein the marker unit is provided at a gripping position of the object.
13. an imaging unit that images an object having a fluorescent marking portion that is recognizable under a first illumination condition light mixed therein or coated thereon; a second illumination condition light irradiating unit that irradiates the object with light under a second illumination condition; a first illumination condition light irradiating unit that irradiates the object with light under a first illumination condition different from the light under the second illumination condition; an illumination control unit that switches between illumination of the light under the second illumination condition by the second illumination condition light irradiating unit onto the object and illumination of the light under the first illumination condition by the first illumination condition light irradiating unit onto the object at a predetermined timing; a learning data creation unit that acquires a second image in a state where the object is irradiated with the light under the second illumination condition and a first image in a state where the object is irradiated with the light under the first illumination condition and the marker unit is recognizable, The object is contained in a container,
14. A training device that is trained to estimate the marker portion as the gripping position of the training object using as learning data a second training image in which a training object is captured under a second lighting condition light and a first training image in which the training object is illuminated under a first lighting condition light different from the second lighting condition light and in which a marker portion provided on the training object is in a state that can be recognized under the first lighting condition light.
15. A training device that is trained to infer information about a training object using as learning data a second training image in which a training object contained in a container is captured under a second lighting condition light, and a first training image in which the training object is irradiated with a first lighting condition light different from the second lighting condition light and a marking portion made of a fluorescent material mixed into or applied to the training object is in a state that can be recognized under the first lighting condition light.
16. The sign portion is a light-emitting portion that can be recognized by invisible light, 16. The training device of claim 14 or 15, wherein the second lighting condition light is visible light and the first lighting condition light is invisible light.
17. The training device of claim 14 , wherein the marking portion is a fluorescent marker.
18. The training device according to claim 14 or 17, wherein the training object includes a plurality of types of the sign portions that emit light in different colors.
19. The training device of claim 15 , wherein the training object is an amorphous object.
20. The training device according to claim 15 , wherein the training object is a liquid, a powder or a gas.
21. 21. Training device according to claim 15, 19 or 20, wherein the training object is transparent.
22. an imaging unit configured to image an object having a marker unit that is recognizable under a first illumination condition light that is invisible light; a first illumination condition light irradiating unit that irradiates the object with light under a first illumination condition; an estimation unit that estimates a gripping position of the object based on a first image in a state in which the object is irradiated with the light under the first illumination condition and the marker unit is recognizable; An estimation device comprising:
23. The estimation device according to claim 22 , wherein the marking portion is a fluorescent marker.
24. The estimation device according to claim 22 or 23, wherein the first image is processed, and the gripping position of the object is estimated based on the processed first image.
Citation Information
Patent Citations
Method of detecting mark
JP2008146432A
Machine learning device, inspection device, and machine learning method
JP2019015654A
Speckle contrast analysis using machine learning for visualizing flow
US20190065905A1
Cited By
Data generation apparatus and control method
US20240155210A1
