Machine learning method, machine learning program, and information processing apparatus

Abstract

To provide a machine learning method which can accurately recognize a mask even when an obstacle has an influence on a contour of the mask.SOLUTION: In the machine learning method, a mask image in which only an object is displayed is acquired, and a plurality of composite images in which dummy silhouette images are arranged so as to partially overlap the object in the mask image and a contour of the object is partially concealed are generated under random conditions, and a plurality of training datasets consisting of combinations of the mask image and the composite images are inputted to perform training.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present disclosure relates to a machine learning method, a machine learning program, and an information processing device. [Background technology]

[0002] A method for recognizing objects by image processing has been developed for picking up objects by a robot end effector. Patent Document 1 discloses a method for using a 3D camera to acquire a 2D trichromatic (RGB) image of a box and a 2D depth map image. The method uses an image segmentation process that uses a simplified Mask R-CNN (convolutional neural network) executable by a CPU to predict which pixels in the RGB image are associated with each box. The pixels associated with each box are given a unique label that is linked to define a mask for the box. The method then uses the segmentation image to identify a position for picking up the box. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2023-81310 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, when attempting to generate a model using such a mask, there is a problem in that if a robot arm or a person covers the workpiece, the contour of the mask cannot be accurately captured. Therefore, the objective of the present disclosure is to provide a machine learning method that can accurately recognize a mask even if an obstacle affects the contour of the mask. [Means for solving the problem]

[0005] The machine learning method of the present disclosure includes: Obtain a mask image showing only the object, A dummy is placed so as to overlap a part of the object on the mask image, and a plurality of composite images in which the contour of the object is partially hidden are generated under random conditions; The object of the present invention is to provide a machine learning method that learns by inputting multiple training data sets each composed of a combination of the mask image and the synthetic image.

[0006] The above configuration provides a machine learning method that can accurately recognize a mask even if an obstacle affects the mask contour.

[0007] The machine learning method of the present disclosure includes: The synthetic image is generated by generating a dummy silhouette image and then arranging the dummy silhouette image on the object under random conditions.

[0008] With the above configuration, an image of an object blocked by an obstacle is synthesized.

[0009] The machine learning method of the present disclosure includes: The synthetic image is generated by randomly placing the dummy between the camera and the object, and generating a segmentation mask image that shows only the object.

[0010] With the above configuration, an image of an object blocked by an obstacle is synthesized.

[0011] The machine learning program of the present disclosure is Obtain a mask image showing only the object, A dummy is placed so as to overlap a part of the object on the mask image, and a plurality of composite images are generated under random conditions using an image in which a contour of the object is partially hidden; The machine learning program causes an information processing device to execute learning by inputting multiple training data sets each composed of a combination of the mask image and the composite image.

[0012] The above configuration provides a machine learning program that can accurately recognize a mask even if an obstacle affects the mask's contour.

[0013] The information processing device of the present disclosure includes: a mask image acquisition unit for acquiring a mask image in which only the object is displayed; a composite image generating unit that generates a plurality of composite images under random conditions of an image in which a dummy is arranged so as to overlap a part of the object on the mask image and a contour of the object is partially hidden; A storage unit that stores a trained machine learning machine that learns by inputting a plurality of training data sets each composed of a combination of the mask image and the composite image; The information processing device includes: a calculation unit that outputs the posture of the object by inputting an image including the object to the trained machine learning machine read from the memory unit.

[0014] The above configuration provides an information processing device that can accurately recognize a mask even if an obstacle affects the contour of the mask. Effect of the Invention

[0015] The present disclosure provides a machine learning method that can accurately recognize a mask even when obstacles affect the mask contour. [Brief description of the drawings]

[0016] [Figure 1] 1 is a block diagram showing a configuration of an information processing device according to an embodiment; [Diagram 2] 1 is a flowchart of a machine learning method according to an embodiment. [Diagram 3] 1 is a first flowchart of a synthetic image generating method according to the first and second embodiments. [Figure 4] 6 is a second flowchart of the synthetic image generating method according to the first embodiment. [Diagram 5]13 is a second flowchart of the synthetic image generating method according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] Embodiment Hereinafter, an embodiment of the present invention will be described with reference to the drawings. However, the invention according to the claims is not limited to the following embodiment. Furthermore, not all of the configurations described in the embodiment are necessarily essential as means for solving the problem. For clarity of explanation, the following description and drawings are omitted and simplified as appropriate. In each drawing, the same elements are given the same reference numerals, and repeated explanations are omitted as necessary.

[0018] (Description of Information Processing Device According to an Embodiment) Fig. 1 is a block diagram showing a configuration of an information processing device according to an embodiment. The information processing device according to the embodiment will be described with reference to Fig. 1. The information processing device 100 is an information processing device for image recognition to cause an end effector of a robot to acquire an object.

[0019] As shown in FIG. 1, an information processing device 100 according to the embodiment includes a mask image acquisition unit 101, a composite image generation unit 102, a storage unit 103, and a calculation unit 104.

[0020] The information processing device 100 physically includes a processor such as a CPU (Central Processing Unit) that executes a program and performs processing, and a memory that stores the program. Each unit of the information processing device is realized by the processor, the memory, and the like. The information processing device 100 may be configured as one device or multiple devices. The information processing device 100 may be a cloud server that distributes and processes some or all of its functions.

[0021] The mask image acquisition unit 101 acquires a mask image in which only the object is displayed. The mask image is a binary image of the same size as the captured image, in which pixels to be processed are colored white and other pixels are colored black.

[0022] The composite image generating unit 102 generates a plurality of composite images under random conditions, in which dummies are arranged so as to overlap a part of the object in the mask image, and the contour of the object is partially hidden. A dummy is an object such as a person or an object that may be present between the camera and the object. The random conditions refer to a state in which the size, aspect ratio, angle, and position of the dummy are variously changed with respect to the object.

[0023] The storage unit 103 stores a trained machine learning machine that learns by inputting a plurality of training data sets each composed of a combination of a mask image and a composite image. The training data set is a data set for machine learning. The trained machine learning machine uses, for example, a machine learning machine trained by a neural network.

[0024] The calculation unit 104 outputs the posture of the object by inputting a captured image including the object to the trained machine learning machine read from the storage unit. The captured image including the object is an image of the object that is actually captured. By outputting the posture of the object, the end effector of the robot can acquire the object.

[0025] The above configuration provides an information processing device that can accurately recognize a mask even if an obstacle affects the contour of the mask.

[0026] (Description of Machine Learning Method According to the Embodiment) 2 is a flowchart of the machine learning method according to the embodiment. The machine learning method according to the embodiment will be described with reference to FIG.

[0027] 2, first, the information processing device 100 obtains a mask image in which only an object is displayed (step S201). Next, the information processing device 100 randomly generates a plurality of composite images in which a dummy silhouette image is arranged so as to overlap a part of the object on the mask image, and in which the contour of the object is partially hidden (step S202). Next, the information processing device 100 learns by inputting a plurality of training data sets constituted by combinations of the mask image and the composite image (step S203).

[0028] The above configuration provides a machine learning method that can accurately recognize a mask even if an obstacle affects the contour of the mask. The machine learning method is executed by an information processing device using a machine learning program.

[0029] (Description of a synthetic image generating method according to the first and second embodiments) Fig. 3 is a first flowchart of the composite image generating method according to the first and second embodiments. Fig. 4 is a second flowchart of the composite image generating method according to the first embodiment. Fig. 5 is a second flowchart of the composite image generating method according to the second embodiment. The composite image generating method according to the first embodiment will be described with reference to Figs. 3 to 5.

[0030] As shown in Fig. 3, first, a camera model and a 3D model of the object are generated in the simulator (step S301). The camera is not a real one, but one on the simulator is used. The object is also not a real one, but one on the simulator is used.

[0031] Next, an image captured by the camera is generated (step S302). The camera model, the 3D model of the object, the surrounding objects such as the floor, and the ambient light are randomly changed. For example, the 3D model of the object and the surrounding objects such as the floor are variously changed in position, attitude, size, and texture to generate an image.

[0032] Next, a segmentation mask image showing only the object is generated based on the positional relationship between the camera position in the simulator and the object (step S303). A segmentation mask image with a black background and a white object is generated.

[0033] From here, a method for generating a composite image according to the first embodiment will be described. Next, the contour coordinates (u, v) of the object in the camera coordinate system are extracted from the segmentation mask image (step S304). The contour coordinates are the coordinates in the camera coordinate system of the gray line shown in the lower diagram of Fig. 3. The camera coordinate system is a three-dimensional coordinate system based on the position of the camera.

[0034] 4, the number of dummies to be placed on the mask image is randomly determined (step S401). Next, dummies are collected in advance from a library or downloaded 2D shapes or image data, and their color values ​​are all set to 0 to make them black (step S402).

[0035] Finally, a dummy silhouette image is created by randomly specifying the contour coordinates, offset (u,v) from the contour coordinates, rotation angle, and size, and placed on the object to generate a composite image (step S403). The silhouette image is a binary image in which the dummy image is colored gray.

[0036] In this way, multiple synthetic images can be generated, which are necessary for machine learning.

[0037] Hereinafter, a method for generating a composite image according to the second embodiment will be described. After step S303, the number of dummies to be placed on the mask image is randomly determined (step S501), as shown in FIG.

[0038] Next, dummies are placed in front of the camera and the object, with their position offset, rotation angle, and size changed randomly (step S502). The amount of offset for the position is determined based on the position and size of the object.

[0039] Finally, a segmentation mask image that captures only the object is generated based on the positional relationship between the camera position in the simulator and the object, and a composite image is generated (step S503). For example, objects that are not labeled by the dummy segmentation are not displayed on the image.

[0040] In this way, multiple synthetic images can be generated, which are necessary for machine learning.

[0041] In addition, a part or all of the processing in the information processing device 100 described above can be realized as a computer program. Such a program can be stored using various types of non-transitory computer-readable media and supplied to a computer. The non-transitory computer-readable medium includes various types of tangible recording media. Examples of the non-transitory computer-readable medium include magnetic recording media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / Ws, and semiconductor memories (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, and RAMs (Random Access Memory)). The program may also be supplied to a computer by various types of temporary computer-readable media. Examples of the temporary computer-readable medium include electrical signals, optical signals, and electromagnetic waves. The temporary computer-readable medium can supply the program to a computer via a wired communication path such as an electric wire and an optical fiber, or via a wireless communication path.

[0042] The present invention is not limited to the above-described embodiment, and can be modified as appropriate without departing from the spirit and scope of the present invention. [Explanation of symbols]

[0043] 100 Information processing device, 101 Mask image acquisition unit, 102 Composite image generation unit, 103 Storage unit, 104 Calculation unit

Claims

1. Obtain a mask image showing only the object, A dummy is placed so as to overlap a part of the object on the mask image, and a plurality of composite images in which the contour of the object is partially hidden are generated under random conditions; A machine learning method that learns by inputting multiple training datasets each composed of a combination of the mask image and the synthetic image.

2. The machine learning method according to claim 1 , wherein the synthetic image is generated by generating a dummy silhouette image and then placing the dummy silhouette image on the image of the object under random conditions.

3. The machine learning method according to claim 1 , wherein the synthetic image is generated by randomly placing the dummy between a camera and the object and generating a segmentation mask image that captures only the object.

4. Obtain a mask image showing only the object, A dummy is placed so as to overlap a part of the object on the mask image, and a plurality of composite images are generated under random conditions using an image in which a contour of the object is partially hidden; A machine learning program that causes an information processing device to learn by inputting multiple training data sets each composed of a combination of the mask image and the composite image.

5. a mask image acquisition unit for acquiring a mask image in which only the object is displayed; a composite image generating unit that generates a plurality of composite images under random conditions of an image in which a dummy is arranged so as to overlap a part of the object on the mask image and a contour of the object is partially hidden; A storage unit that stores a trained machine learning machine that learns by inputting a plurality of training data sets each composed of a combination of the mask image and the composite image; and a calculation unit that outputs the posture of the object by inputting a captured image including the object to the trained machine learning machine read from the memory unit.

Images