Method for creating training data and system for creating training data

Abstract

To enhance the accuracy of training data for an AI that identifies an article to be picked up on the basis of a captured image.SOLUTION: The method includes: an imaging step of acquiring an image of an article M; an image processing step of applying image processing to the image acquired in the imaging step; and a storing step of storing both an extracted image, obtained by extracting a region corresponding to the article from the captured image in the imaging step, and the processed image in the image processing step. In the image processing step, a projective transformation is applied to the extracted image to generate a plurality of transformed images, and training data is produced for an AI that identifies an article capable of being picked up from a plurality of stacked articles on the basis of the transformed images.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a method for creating learning data for an AI that identifies an item based on a captured image. [Background technology]

[0002] In recent years, AI has been used to identify randomly placed items so that robots and other devices can pick them up. Accurate AI-based picking requires the preparation of a large amount of training data (image data) for learning. However, creating training data by actually arranging multiple items requires a great deal of time and effort. Furthermore, when arranging items, there is a risk of bias in the arrangement depending on the worker. To address this issue, a workpiece information processing device has been proposed that performs at least one of translation, rotation, reduction, and enlargement processing on a basic image of a workpiece (item) to create training data (learning data) (Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-057250 Summary of the Invention [Problem to be solved by the invention]

[0004] However, because the image captured by the camera is a central projection image, if the objects are thick and overlapping and tilted up or down, a discrepancy will occur between the training data (learning data) image created by performing at least one of parallel translation, rotation, reduction, and enlargement on the basic image of the work (object) and the image of the object captured by the camera during the actual pick-up process. As a result, the training data (learning data) created in Patent Document 1 becomes low-accuracy learning data, and the pick-up accuracy decreases.

[0005] The present invention aims to improve the accuracy of learning data for AI that identifies items to be picked up based on captured images. [Means for solving the problem]

[0006] The first invention of the present invention is a method for creating learning data for an AI that identifies items that can be picked up from multiple stacked items, and includes an imaging step of capturing an image of the item, an image processing step of performing image processing on the image captured in the imaging step, and a storage step of storing an extracted image obtained by extracting an area corresponding to the item from the image captured in the imaging step and the image processed in the image processing step, wherein the image processing step performs projective transformation on the extracted image to create multiple transformed images, and creates learning data based on the transformed images.

[0007] The second invention of the present invention is a learning data creation method according to the first invention, characterized in that the image processing step generates a simulated overlap pattern image showing the overlapping state of items by overlapping the converted images.

[0008] The third invention of the present invention is a method for creating learning data according to the second invention, characterized in that the image processing step selects an optimal image from the plurality of converted images according to the degree of overlap of the items to be placed.

[0009] The fourth invention of the present invention is a learning data creation system for identifying items that can be picked up from multiple items stacked on top of each other, and is characterized in that it comprises an imaging means for capturing an image of the item, an image processing means for performing image processing on the image of the item captured by the imaging means, and a storage means for storing the image captured by the imaging means and the image processed by the image processing means, and the image processing means performs projective transformation on the extracted image of the item captured by the imaging means to create multiple transformed images, and creates learning data based on the transformed images. [Effects of the Invention]

[0010] According to the present invention, it is possible to improve the accuracy of learning data for an AI that identifies an item to be picked up based on a captured image. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a schematic side view showing the configuration of a training data creation system that executes a training data creation method according to an embodiment of the present invention. [Figure 2] FIG. 10 is a diagram showing an example of a simulated overlap pattern image. DETAILED DESCRIPTION OF THE INVENTION

[0012] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. Fig. 1 is a schematic side view showing the configuration of a training data creation system that executes a training data creation method according to one embodiment of the present invention.

[0013] The learning data creation system 10 of this embodiment is a system that captures images of randomly placed items and creates learning data for a device that picks up the items one by one using a robot or the like based on AI image processing. In the learning data creation system 10, an item M to be identified is placed on a stage 12 and an image of the item is captured by a camera (imaging means) 14 positioned approximately directly above the stage 12. The image of the item M captured by the camera 14 is sent to a learning data creation device 16, which creates learning data based on the captured image of the item M. The item M to be identified is, for example, a bag-shaped item that contains contents and has a predetermined thickness, and the bag is, for example, rectangular in plan view.

[0014] The learning data creation device 16 includes an image processing device (image processing means) 18 and a storage device (storage means) 20. In the learning data creation process (learning data creation method) of this embodiment, first, an image of the item M captured by the camera 14 is stored in the storage device 20 as a sample image (step S1: imaging step, storage step). The sample images are, for example, captured one or more images of the front and back of the item M placed flat on the stage 12 and stored in the storage device 20.

[0015] The image processing device 18 reads out each sample image stored in the storage device 20, and extracts an image (extracted image) of an area corresponding to the article M from each sample image (step S2). Next, each extracted image is subjected to projective transformation (projective transformation corresponding to multiple orientations) using multiple pre-set patterns (step S3: image processing step), and the multiple transformed images obtained for each extracted image are stored in the storage device 20 as transformed images (step S4: storage step). Note that the multiple projective transformation patterns are set by controlling the values ​​of projective transformation parameters, and these are set corresponding to the expected tilted orientations of the article M.

[0016] The image processing device 18 further creates a pattern (simulated overlapping pattern image) simulating a state in which multiple items M are randomly overlapped, using the extracted image and the converted image stored in the storage device 20, and stores the image in the storage device 20 (step S5). At this time, the item M located at the top of the simulated overlapping pattern image is linked with an identifier that indicates that it can be picked up, and the other items M (items M that are overlapped by other items M) are linked with identifiers that indicate that they cannot be picked up, and these are stored in the storage device 20.

[0017] Examples of simulated overlapping pattern images are shown in Figures 2(a) and 2(b). In Figure 2(a), a simulated overlapping pattern image is created by sequentially overlaying two converted images T1 and T2 on a randomly placed extracted image E1. In Figure 2(b), a simulated overlapping pattern image is created by sequentially overlaying two converted images T3 and T4 on a randomly placed extracted image E2. In each simulated overlapping pattern image, the topmost converted images T2 and T4 (shown in gray) are identified as corresponding to items that can be picked up, while the extracted images E1 and E2 and the converted images T1 and T3 are identified as corresponding to items that cannot be picked up.

[0018] The above-mentioned simulated overlap pattern image creation process (step S5) is repeated a predetermined number of times (e.g., 1000 times) until a predetermined number (e.g., 1000) of different patterns are created, and the data of each created pattern is stored in the memory device 20 as learning data.

[0019] In the simulated overlap pattern image creation process (step S5), the simulated overlap pattern image is created, for example, in accordance with the following procedure.

[0020] First, multiple extracted images are arranged within a predetermined image area with their positions and angles randomly set (step S5-1). Next, assuming that one extracted image will be randomly overlaid on each extracted image, optimal transformed images (projectively transformed images) of the extracted images to be overlaid are selected from storage device 20 according to the expected overlapping conditions (overlapping position, area, thickness of the items, etc.) (step S5-2). The selected transformed images are then arranged on the randomly arranged extracted images to create a simulated overlapping pattern image (step S5-3). The created simulated overlapping pattern image and information (identifiers) linking each extracted image and transformed image in the simulated overlapping pattern image to whether it is pickable or non-pickable are stored in storage device 20 (step S5-4). Note that it is not necessary to overlay transformed images on all of the multiple extracted images arranged within the predetermined image area; a simulated overlapping pattern image may be created in which extracted images are arranged without transformed images overlaid. In this case, extracted images arranged without transformed images overlaid are identified as corresponding to pickable items.

[0021] The above process (steps S5-1 to S5-4) is repeated a predetermined number of times (for example, 1000 times) to create and store a predetermined number of simulated overlap pattern images (for example, 1000 images) with different overlap patterns, at which point the learning data creation process of this embodiment is completed.

[0022] As described above, the training data creation method of the training data creation system of this embodiment allows images that are closer to images actually captured by a camera to be used as training data, thereby improving the pick-up accuracy of a device that uses AI to pick up overlapping items. Furthermore, by using converted images generated by the training data creation system, training data can be created in a short time.

[0023] In the learning data creation system of this embodiment, converted images corresponding to various postures (patterns) of the article are created in advance in step S3. However, instead of step S3, in step S5-3, a simulated overlapping pattern image may be created in which a converted image with its tilt adjusted is arranged by performing projective transformation each time depending on the degree of overlap with the extracted image (overlapping position, area, thickness of the article, etc.).

[0024] In addition, in this embodiment, the simulated overlap pattern image is created by overlaying the converted image on the extracted image, but the simulated overlap pattern image may also be created by overlaying only the converted image that has been projectively transformed. Furthermore, in this embodiment, the optimal converted image is overlaid taking into consideration the degree of image overlap (overlapping position, area, thickness of the object, etc.), but it is also possible to create a simulated overlap pattern image by overlaying without taking these factors into consideration. [Explanation of symbols]

[0025] 10 Learning data creation system 14 Camera (imaging means) 16 Learning data creation device 18 Image processing device (image processing means) 20 Storage device (storage means)

Claims

1. A method for creating learning data for an AI that determines which items can be picked up from among multiple items stacked together, comprising: an imaging step of capturing an image of the article; an image processing step for performing image processing on the image captured in the imaging step; a storage step for storing an extracted image obtained by extracting an area corresponding to the article from the image captured in the imaging step and an image processed in the image processing step, The method for creating learning data is characterized in that the image processing step performs projective transformation on the extracted image to create a plurality of transformed images, and creates learning data based on the transformed images.

2. 2. The method for creating learning data according to claim 1, wherein said image processing step generates a simulated overlap pattern image showing an overlapping state of items by arranging said converted images in an overlapping manner.

3. 3. The learning data creation method according to claim 2, wherein the image processing step selects an optimal image from the plurality of converted images depending on the degree of overlap of the objects to be placed.

4. A system for creating learning data for identifying an item that can be picked up from a plurality of stacked items, comprising: The system comprises an imaging means for capturing an image of an article, an image processing means for performing image processing on the image of the article captured by the imaging means, and a storage means for storing the image captured by the imaging means and the image processed by the image processing means, A learning data creation system characterized in that the image processing means performs projective transformation on the extracted image of the item captured by the imaging means to create a plurality of transformed images, and creates learning data based on the transformed images.

Images