Learning model generation device, information processing device, learning model generation method, and information processing method

By generating a learning model for detecting image information of multiple objects, the problem of difficulty in detecting unknown shape objects in the prior art is solved, and accurate shape detection of unknown shape objects is achieved.

CN119072720APending Publication Date: 2024-12-03MITSUBISHI ELECTRIC CORP
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
CN202280094719.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2022-04-15
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

When the conventional position detection device detects an object with an unknown shape, it is difficult to accurately detect the shape of the object.

Method used

By obtaining object image information from multiple shooting points, using the first learning model to calculate the accuracy of the shape and the preset class, generate learning data related to multiple objects, and then generate a second learning model for the use of multiple objects Image information detects the shape of the object.

Benefits of technology

Even if the object has an unknown shape, its shape can be accurately detected, which improves the reliability of position detection.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119072720A_ABST
    Figure CN119072720A_ABST
Patent Text Reader

Abstract

A learning model generation device (1000) is provided with: an image information acquisition unit (300) that acquires image information of an object obtained by imaging the object from a plurality of imaging points; a first estimation unit (101) that estimates using a first learning model that calculates, on the basis of the image information acquired by the image information acquisition unit, the accuracy at which the shape of the object conforms to each of a plurality of preset classes; a determination unit (102) that determines, on the basis of the estimation result obtained by the first estimation unit, a class to which the shape of the object conforms; a learning data generation unit (103) that generates learning data including image information relating to the plurality of objects on the basis of the determination results obtained by the determination unit and the image information of the objects; and a learning model generation unit (104) that generates, on the basis of the learning data generated by the learning data generation unit, a second learning model for detecting the shape of the object from image information including the plurality of objects.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to a learning model generating device, an information processing device, a learning model generating method and an information processing method. Background Art

[0002] Conventionally, there is a disclosed position detection device that detects the position of an object based on a learning model learned in advance using 3D data of a plurality of simple shapes and an image of the object captured by a camera (see Patent Document 1). When the position detection device detects the shape of an object based on image information including a plurality of objects as detection targets, the position of the object is detected based on the reliability of which of the plurality of simple shape models is matched.

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application Publication No. 2020-77231 Summary of the invention

[0006] Problems to be solved by the invention

[0007] However, in the case of detecting the shape of an object based on image information including multiple objects as the object, for example, the position detection device described in Patent Document 1 is sometimes unable to detect the shape of the object based on image information when the object as the detection object is an object with an unknown shape that is significantly different from any model in a simple shape model.

[0008] The purpose of the present invention is to solve the above-mentioned problems and to provide a learning model generating device, an information processing device, a learning model generating method and an information processing method which can detect the shape of an object even if the object has an unknown shape based on image information containing multiple objects.

[0009] Means for solving problems

[0010] The learning model generating device of the present invention is characterized in that it comprises: an image information acquiring unit, which acquires image information of an object obtained by photographing the object from multiple shooting points; a first inference unit, which performs inference through a first learning model, wherein the first learning model calculates the accuracy of the shape of the object conforming to each of multiple preset classes based on the image information acquired by the image information acquiring unit; a determination unit, which determines the class to which the shape of the object conforms based on the inference result obtained by the first inference unit; a learning data generating unit, which generates learning data containing image information related to multiple objects based on the determination result obtained by the determination unit and the image information of the object; and a learning model generating unit, which generates a second learning model for detecting the shape of an object based on the image information containing multiple objects based on the learning data generated by the learning data generating unit.

[0011] Effects of the Invention

[0012] According to the present invention, there is a learning model generating unit, which generates a second learning model for detecting the shape of an object based on image information containing multiple objects based on learning data generated by a learning data generating unit according to image information obtained by photographing the object from multiple shooting points. Therefore, even if the object as the object is an object with an unknown shape, the shape of the object can be detected based on the image information containing multiple objects. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 This is a block diagram showing a schematic configuration of an information processing device according to Embodiment 1.

[0014] Figure 2 This is a block diagram showing a schematic configuration of a target object detection unit according to the first embodiment.

[0015] Figure 3 This is a block diagram showing an example of the hardware configuration of the information processing device according to the first embodiment.

[0016] Figure 4 This is a flowchart showing an example of a process of generating a second learning model performed by the information processing device according to the first embodiment.

[0017] Figure 5 This is a flowchart showing an example of a process of selecting an object based on image information performed by the information processing device according to the first embodiment. DETAILED DESCRIPTION

[0018] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0019] Implementation method 1.

[0020] Figure 1is a block diagram showing a schematic structure of an information processing device 1000 according to Embodiment 1. Figure 2 1 is a block diagram showing a schematic structure of the target object detection unit 200 of Embodiment 1. The information processing device 1000 is a device that detects and selects any object from the multiple objects based on image information obtained from the camera CA when multiple objects are photographed using the camera CA. For example, the information processing device 1000 is a device that, when multiple commodities in a randomly stacked state are moved one by one by a robot arm, selects the best commodity from the multiple commodities in the stacked state as the commodity to be moved based on image information including the multiple commodities in the stacked state obtained from the camera CA, and outputs the selected result to a control device (not shown) that controls the robot arm. In addition, the information processing device 1000 constitutes a learning model generation device in Embodiment 1.

[0021] like Figure 1 and Figure 2 As shown, the information processing device 1000 includes an image information acquisition unit 300, a learning unit 100, and a target object detection unit 200. When the camera CA captures one or more objects from multiple shooting points (viewpoints), the image information acquisition unit 300 acquires multiple image information obtained by observing the object from multiple shooting points. For example, when multiple cameras CA arranged at different positions capture one or more objects from multiple shooting points respectively, the image information acquisition unit 300 acquires multiple image information obtained by observing the object from multiple shooting points. In addition, the image information acquisition unit 300 can also be configured to acquire multiple image information obtained by observing the object from multiple shooting points when one camera CA captures one or more objects while moving between multiple shooting points (viewpoints).

[0022] The learning unit 100 generates a learning model based on the image information acquired by the image information acquisition unit 300, and outputs the generated learning model to the target object detection unit 200. The details of the learning unit 100 will be described later. The target object detection unit 200 detects multiple objects included in the image information based on the image information including multiple objects acquired by the image information acquisition unit 300 and the second learning model acquired from the learning unit 100, and selects any object from the detected multiple objects. For example, the target object detection unit 200 selects an object suitable as an object to be moved by the robot arm from the multiple objects included in the acquired image information. The details of the target object detection unit 200 will be described later.

[0023] Figure 3This is a block diagram showing an example of the hardware configuration of the information processing device 1000 according to Embodiment 1. For example, the information processing device 1000 includes a CPU (Central Processing Unit) 1001 , a main storage device 1002 , and an auxiliary storage device 1003 including a HDD (Hard Disk Drive) or the like.

[0024] The main storage device 1002 includes, for example, a ROM (Read Only Memory) 1002A and a RAM (Random Access Memory) 1002B. In addition, the main storage device 1002 may include, in addition to the ROM 1002A and the RAM 1002B, or in place of the ROM 1002A and the RAM 1002B, a nonvolatile or volatile semiconductor memory such as a flash memory, an EPROM (Erasable Programmable Read Only Memory), an EEPROM (Electrically Erasable Programmable Read-Only Memory), a magnetic disk, a floppy disk, an optical disk, a compact disk, a mini disk, a DVD (Digital Versatile Disc), or the like.

[0025] In addition, for example, the information processing device 1000 has a communication interface 1004A, an input interface 1004B, a display interface 1004C, and an output interface 1004D as interfaces for inputting and outputting information with external devices. For example, the communication interface 1004A inputs and outputs information with an external storage device 1006 that is provided outside the information processing device 1000 and stores various information. In addition, for example, the input interface 1004B receives input of information from an input device 1007 that is provided outside the information processing device 1000. In addition, for example, the display interface 1004C outputs the processing results of the CPU 1001, the information stored in the main storage device 1002 and the auxiliary storage device 1003, etc. to a display device 1008 such as a liquid crystal display that is provided outside the information processing device 1000. In addition, for example, the output interface 1004D outputs the processing results of the CPU 1001, the information stored in the main storage device 1002 and the auxiliary storage device 1003, etc. to an output device 1009. The output device 1009 is a device provided outside the information processing device 1000 , such as a control unit of a robot arm or a server.

[0026] Each function of the information processing device 1000 configured in this way is realized by the CPU 1001 executing a program composed of software or firmware or a combination of software and firmware stored in the main storage device 1002 and the auxiliary storage device 1003. In addition, the information processing device 1000 may also include a dedicated processing circuit having a system LSI (Large-Scale Integration) or the like instead of the above-mentioned CPU 1001, the main storage device 1002, and the auxiliary storage device 1003.

[0027] Next, refer to Figure 1 The details of the learning unit 100 are described below. Figure 1 As shown, the learning unit 100 includes a first inference unit 101 , a shape determination unit 102 , a learning data generation unit 103 , a learning model generation unit 104 , a first learning model storage unit 105 , a learning data storage unit 106 , and a second learning model storage unit 107 .

[0028] The first inference unit 101 infers the shape and surface texture of the object contained in the image information based on the image information acquired by the image information acquisition unit 300. For example, the first inference unit 101 includes a plurality of inference units 101a for inferring the shape and surface texture of the object for each of a plurality of shooting points at which the camera CA captures the object, and each inference unit 101a infers the shape and surface texture of the object for each of a plurality of image information corresponding to the plurality of shooting points. The first inference unit 101 acquires the first learning model pre-stored in the first learning model storage unit 105 from the first learning model storage unit 105, and performs inference based on the first learning model and the image information acquired by the image information acquisition unit 300.

[0029] The first learning model is a learned model obtained by performing machine learning using multiple CGs (Computer Graphics) representing objects that are different in at least one of shape and surface texture as learning data, and calculates the accuracy (inference rate) of the shape and surface texture of the object contained in the input image information and each of the shape categories of multiple pre-set shape categories (category, multiple classes) by inference. For example, the CG used as the learning data of the first learning model is generated by deforming image information obtained by photographing multiple objects that are different in at least one of shape and surface texture, and extracting features related to the shape and surface texture of each of the deformed image information of the multiple objects. In addition, for example, the first learning model is composed of NN (Neural Networks). The first inference unit 101 outputs the accuracy of conformity with each of the multiple shape categories to the shape determination unit 102 for each of the multiple image information obtained by the image information acquisition unit 300.

[0030] The shape determination unit 102 determines which of the plurality of shape categories preset as the shape categories of the first learning model the shape of the object included in the image information acquired by the image information acquisition unit 300 matches based on the information input from the first estimation unit 101 as the determination result obtained by the first estimation unit 101, and outputs the determination result to the learning data generation unit 103. In addition, when the shape determination unit 102 determines that the shape of the object included in the image information acquired by the image information acquisition unit 300 does not match any of the plurality of shape categories preset as the shape categories of the first learning model based on the information input from the first estimation unit 101, the shape determination unit 102 outputs the determination result indicating that the shape does not match any of the shape categories to the learning data generation unit 103. In addition, the shape determination unit 102 constitutes the determination unit in the first embodiment.

[0031] The learning data generation unit 103 generates data including image information related to a plurality of objects based on the determination result obtained by the shape determination unit 102 and the image information obtained by the image information acquisition unit 300. For example, the learning data generation unit 103 generates a CG in which a plurality of objects are randomly arranged based on the determination result obtained by the shape determination unit 102 and the image information obtained by the image information acquisition unit 300, and stores the CG in the learning data storage unit 106. For example, the learning data generation unit 103 generates image information in which an object included in the image information obtained by the image information acquisition unit 300 is deformed, copied, and overlapped, and generates a CG in which features related to the shape and surface texture of the object are extracted from the image information. In addition, the data generated by the learning data generation unit 103 only needs to be data including image information related to a plurality of objects, for example, it may be a CG in which a plurality of objects are arranged, it may be a CG in which a plurality of objects are overlapped, or it may be data including image information of a plurality of objects.

[0032] The learning model generation unit 104 obtains the information of the CG generated by the learning data generation unit 103 from the learning data storage unit 106, and uses the information as learning data to generate a second learning model for detecting the shape of an object based on image information containing multiple objects. For example, the learning model generation unit 104 can use the CG data of a state in which multiple objects are randomly arranged generated by the learning data generation unit 103 as learning data to generate a second learning model for detecting the shapes of each of the multiple objects in the data based on the image information of the state in which multiple objects are randomly arranged, or can generate a second learning model for detecting the shape of a specific object in the data based on the image information of the state in which multiple objects are randomly arranged. For example, the second learning model is a learned model for detecting multiple objects individually through instance segmentation based on image information of a state in which multiple objects are arranged. In addition, for example, the second learning model is composed of Mask R-CNN (Region Based Convolutional Neural Networks), and detects the shape and surface texture of an object by inference through instance segmentation. The learning model generation unit 104 causes the second learning model storage unit 107 to store the generated second learning model.

[0033] In addition, the learning model generation unit 104 outputs information representing an arbitrary second learning model stored in the second learning model storage unit 107 to the target object detection unit 200 based on the determination result of the shape determination unit 102. The second learning model storage unit 107 stores a plurality of second learning models corresponding to each of the plurality of shape categories of the first learning model for detecting an object from image information in a state in which a plurality of objects are randomly arranged. Similar to the first learning model described above, these plurality of second learning models are also learned models that use information on CG in a state in which a plurality of objects are randomly arranged as learning data. For example, the learning model generation unit 104 outputs information representing a second learning model corresponding to a shape category corresponding to the determination result obtained by the shape determination unit 102 to the target object detection unit 200.

[0034] Next, refer to Figure 4 , the process of generating the second learning model performed by the information processing device 1000 is described. Figure 4 FIG. 1 is a flowchart showing an example of a process of generating a second learning model performed by the information processing device 1000 according to the first embodiment. Figure 4 As shown, for example, the information processing device 1000 first obtains image information when a target object is photographed by multiple cameras CA through the image information acquisition unit 300 (step ST11). In this process, the information processing device 1000 obtains multiple image information when the camera CA photographs an object from multiple shooting points.

[0035] After performing the processing of step ST11, the information processing device 1000 calculates the accuracy of the shape and surface texture of the object contained in the image information and each shape category by inference based on the first learning model stored in the first learning model storage unit 105 and the image information obtained by the image information acquisition unit 300 (step ST12). In this process, the information processing device 1000 detects the shape and surface texture of the object based on the image information of the object captured by the camera CA, and calculates the accuracy of the object and each shape category.

[0036] After performing the processing of step ST12, the information processing device 1000 determines the shape category to which the object contained in the image information acquired by the image information acquisition unit 300 corresponds based on the accuracy calculated for each shape category (steps ST13 and ST14). For example, the information processing device 1000 determines, through the shape determination unit 102, the shape category corresponding to the highest accuracy among the accuracies calculated for each shape category as the shape category to which the object corresponds. In addition, for example, when any accuracy calculated for each shape category is less than a predetermined threshold value set in advance, the information processing device 1000 determines, through the shape determination unit 102, that the object corresponds to a new shape category that does not correspond to any shape category, that is, it is an object that does not exist in the shape category pre-set in the first learning model.

[0037] In the processing of steps ST13 and ST14, when the object contained in the image information acquired by the image information acquisition unit 300 is an object existing in a shape category pre-set in the first learning model ("Yes" in step ST14), that is, when it is an object that conforms to any shape category among the shape categories pre-set in the first learning model, the information processing device 1000 selects a second learning model corresponding to the shape category from the second learning models pre-stored in the second learning model storage unit 107, outputs information representing the selected second learning model to the object detection unit 200 (step ST18), and ends the processing.

[0038] In the processing of steps ST13 and ST14, when the object included in the image information obtained by the image information obtaining unit 300 is not an object of the shape category pre-set in the first learning model ("No" in step ST14), that is, when it is an object that does not match any of the shape categories pre-set in the first learning model, the information processing device 1000 generates new learning data through the learning data generating unit 103 based on the image information obtained by the image information obtaining unit 300 (step ST15). In this processing, the information processing device 1000 generates learning data for generating a new second learning model that is not stored in the second learning model storage unit 107 through the learning data generating unit 103 based on the fact that the shape determining unit 102 determines that the object included in the image information obtained by the image information obtaining unit 300 does not match any of the shape categories of the plurality of shape categories of the first learning model.

[0039] After performing the processing of step ST15, the information processing device 1000 generates a new second learning model based on the learning data generated in the processing of step ST15 (step ST16). In this process, even if it is an unknown object with a shape and surface texture that are not preset, the information processing device 1000 generates a second learning model that can select a specific object from the image information representing a state in which a plurality of the objects are arranged in a disorderly manner. In addition, the information processing device 1000 can also generate the second learning model by performing transfer learning based on the acquired image information through the learning model generation unit 104.

[0040] After performing the processing of step ST16, the information processing device 1000 stores the generated second learning model in the second learning model storage unit 107 (step ST17). After performing the processing of step ST17, the information processing device 1000 outputs information indicating the newly generated second learning model to the target object detection unit 200 (step ST18), and ends the processing.

[0041] Next, refer to Figure 2 The details of the target object detection unit 200 are described below. Figure 2 As shown, the target object detection unit 200 includes a second estimation unit 201 , a candidate selection unit 202 , a feature quantity calculation unit 203 , and a target object selection unit 204 .

[0042] The second inference unit 201 detects the shape and surface texture of the object included in the image information by inference based on the second learning model obtained from the second learning model storage unit 107 and the image information obtained by the image information acquisition unit 300. For example, the target object detection unit 200 acquires an arbitrary second learning model from the second learning model storage unit 107 based on the information indicating the arbitrary second learning model obtained from the learning model generation unit 104, and detects the object included in the image information by inference based on a plurality of image information obtained by photographing the shape and surface texture of the object obtained by the image information acquisition unit 300 from a plurality of photographing points. In other words, the target object detection unit 200 detects the object included in the image information by inference based on the image information obtained by the image information acquisition unit 300 and the second learning model corresponding to the shape category of the object included in the image information.

[0043] For example, the second inference unit 201 includes a plurality of inference units 201a for inferring the shape and surface texture of the object for each of a plurality of shooting points at which the camera CA shoots the object, and each inference unit 201a infers the shape and surface texture of the object for each of a plurality of image information corresponding to the plurality of shooting points. The second inference unit 201 outputs the shape and surface texture of the object detected by the inference and the accuracy of the inference to the candidate selection unit 202 as an inference result.

[0044] Furthermore, the candidate selection unit 202 selects one object suitable as an object to be moved by the robot arm from among the multiple objects included in the image information obtained by the image information acquisition unit 300, based on the inference result obtained by the second inference unit 201, as the output candidate object of the target object detection unit 200. For example, when the image information obtained by the image information acquisition unit 300 includes multiple objects, the candidate selection unit 202 selects any one object from among the multiple objects included in the image information based on the accuracy of the inference performed by the second inference unit 201. In other words, when the image information obtained by the image information acquisition unit 300 includes multiple objects, the candidate selection unit 202 selects any one segmentation from among the multiple segments of the objects included in the image information based on the accuracy of the inference.

[0045] Furthermore, the candidate selection unit 202 may select only one object when only one object included in the image information acquired by the image information acquisition unit 300 is detected. Furthermore, the candidate selection unit 202 may select two or more objects from among the multiple objects when multiple objects are included in the image information acquired by the image information acquisition unit 300. Furthermore, the candidate selection unit 202 may be configured to not perform object selection processing on specific image information among the multiple image information acquired by the image information acquisition unit 300 when an object included in the specific image information is not detected.

[0046] For example, the candidate selection unit 202 includes a plurality of selection units 202a for selecting an arbitrary object for each of the inference results of the inference units 201a, and each selection unit 202a selects an arbitrary object for each of the plurality of image information corresponding to the plurality of shooting points. The candidate selection unit 202 outputs the selection result to the feature quantity calculation unit 203.

[0047] The feature quantity calculation unit 203 calculates the feature quantity of the object selected by the candidate selection unit 202. For example, the feature quantity calculation unit 203 has a plurality of calculation units 203a corresponding to each selection unit 202a, and calculates the feature quantity for each object corresponding to the selection results of the plurality of selection units 202a. In other words, the feature quantity calculation unit 203 calculates the feature quantity of the object selected by the selection unit 202a for each of the plurality of image information acquired by the image information acquisition unit 300. For example, the feature quantity calculation unit 203 calculates feature quantities such as the position, posture, apparent area (division area), and area of ​​the bounding box of the object selected by the candidate selection unit 202. The position of the object calculated by the feature quantity calculation unit 203 may be the apparent center of gravity position (center of gravity position of the division) of the object, may be the center position of the bounding box, or may be a specific position of the bounding box, for example, the position of a specific corner such as the position of the upper right corner. In addition, the feature quantity calculation unit 203 constitutes the calculation unit in the first embodiment.

[0048] In addition, the candidate selection unit 202 may select any object from among the objects included in the image information based on the feature quantity of the object calculated by the feature quantity calculation unit 203. For example, the candidate selection unit 202 may select any one object whose segmentation of the detected object is not split into a plurality of segments by the segmentation of other objects from among the plurality of objects included in the image information obtained by the image information acquisition unit 300 based on the inference result of the second inference unit 201 and the feature quantity calculated by the feature quantity calculation unit 203. In addition, for example, the candidate selection unit 202 may select any one object from among the plurality of objects included in the image information obtained by the image information acquisition unit 300 based on the segmentation area of ​​the detected object. Specifically, the candidate selection unit 202 may select the object whose segmentation area of ​​the detected object is the largest from among the plurality of objects included in the image information obtained by the image information acquisition unit 300.

[0049] In addition, for example, the candidate selection unit 202 may select any one of the objects included in the image information acquired by the image information acquisition unit 300 based on the segmented area of ​​the detected object and the area of ​​the bounding box of the object. Specifically, the candidate selection unit 202 may select the object whose segmented area relative to the area of ​​the bounding box of the detected object is the largest from the multiple objects included in the image information acquired by the image information acquisition unit 300. In addition, for example, the candidate selection unit 202 may select the object whose product (As^2 / Ab) of the segmented area (As) of the object and the segmented area (As) of the object relative to the area (Ab) of the bounding box of the detected object is the largest from the multiple objects included in the image information acquired by the image information acquisition unit 300. In addition, the segmented area of ​​the object relative to the area of ​​the bounding box of the object and the product of the segmented area of ​​the object and the segmented area of ​​the object relative to the area of ​​the bounding box of the object may be calculated by the feature amount calculation unit 203, or may be calculated by the candidate selection unit 202 based on the segmented area of ​​the object and the area of ​​the bounding box calculated by the feature amount calculation unit 203.

[0050] In this way, the candidate selection unit 202 selects any object from the objects selected by the candidate selection unit 202 based on the calculation result of the feature quantity calculation unit 203, thereby, for example, being able to detect an object with a high priority among multiple objects that can be moved by a robot arm, thereby improving the operability when moving the object by the robot arm.

[0051] In addition, for example, the candidate selection unit 202 may obtain information related to the position of the end effector (not shown) of the robot arm, and select an object from the plurality of objects included in the image information obtained by the image information acquisition unit 300 based on the distance between the position of the end effector and the center position of the detected object (or the center position of the bounding box). Specifically, the candidate selection unit 202 may select the object with the smallest distance between the position of the end effector and the center position of the bounding box of the detected object from the plurality of objects included in the image information obtained by the image information acquisition unit 300. In addition, in such a case, the candidate selection unit 202 constitutes the position information acquisition unit in the first embodiment.

[0052] The object selection unit 204 selects and outputs information related to any object included in the image information acquired by the image information acquisition unit 300 based on the feature amount of the object calculated by the feature amount calculation unit 203. For example, the object selection unit 204 selects any one object from the plurality of objects included in the image information acquired by the image information acquisition unit 300 based on the feature amount calculated by the feature amount calculation unit 203 for each of the plurality of image information acquired by the image information acquisition unit 300. Alternatively, the object selection unit 204 may select any one object from the plurality of objects included in the image information acquired by the image information acquisition unit 300 based on the feature amount calculated by the feature amount calculation unit 203 for each of the plurality of image information acquired by the image information acquisition unit 300 and the accuracy of the estimation by the second estimation unit 201. The object selection unit 204 outputs information related to the position of the selected object. In addition, the object selection unit 204 constitutes a selection unit in the first embodiment.

[0053] Next, refer to Figure 5 The process of selecting an object based on image information performed by the information processing apparatus 1000 will be described. Figure 5 This is a flowchart showing an example of a process of selecting an object based on image information performed by the information processing device 1000 according to Embodiment 1. For example, when a user moves a plurality of identical objects arranged in a disorderly manner one by one using a robot arm, first, the user photographs one of the objects from a plurality of photographing points using a camera CA, inputs the image information obtained by the camera CA to the information processing device 1000, and causes the learning unit 100 to select a second learning model of a shape category corresponding to the photographed object.

[0054] Next, the user causes the camera CA to photograph the plurality of objects randomly arranged from a plurality of photographing points, and inputs a plurality of image information obtained by the camera CA to the information processing device 1000. For example, the user causes a plurality of cameras CA arranged at a plurality of photographing points to photograph the plurality of objects randomly arranged in a container, and inputs a plurality of image information obtained by the plurality of cameras CA to the information processing device 1000. Thus, the information processing device 1000 obtains a plurality of image information including a plurality of objects (step ST21).

[0055] After performing the processing of step ST21, the information processing device 1000 performs inference based on the obtained multiple image information and the selected second learning model (step ST22). In this process, the information processing device 1000 performs instance segmentation using the second learning model, and detects each of the multiple objects included in each image information by inference.

[0056] After performing the processing of step ST22, the information processing device 1000 selects, for example, any object as a candidate for being held and moved by the robot arm from among the objects contained in the image information for each of the plurality of image information obtained based on the inference result of step ST21 (step ST23). In addition, the movement of the object by the robot arm is not limited to being held by the end effector, and for example, the object may be attracted by the end effector with negative pressure, by magnetic force, or by hooking a part of the object by the end effector.

[0057] After performing the process of step ST23 , the information processing device 1000 calculates the feature amount of the object selected in the process of step ST23 for each of the acquired plurality of image information (step ST24 ).

[0058] After performing the process of step ST24, the information processing device 1000 selects any one object based on the feature amount calculated for each of the plurality of image information obtained (step ST25). Through this process, one object moved by the robot arm is selected from the plurality of objects arranged randomly.

[0059] After performing the processing of step ST25, the information processing device 1000 outputs information related to the selected object (step ST26). In this processing, the information processing device 1000 outputs information required for the robot arm to move any one of the multiple objects to the control unit of the robot arm (not shown). For example, the information processing device 1000 outputs information related to the position, such as the coordinates of the selected object.

[0060] As described above, the information processing device 1000 of embodiment 1 has a learning model generating unit 104, which generates a second learning model for detecting the shape of an object based on image information including multiple objects based on the learning data generated by the learning data generating unit 103 according to image information obtained by photographing the object from multiple photographing points. Therefore, even if the object as the object is an object with an unknown shape, the shape of the object can be detected based on the image information including multiple objects.

[0061] In addition, the information processing device 1000 of embodiment 1 has: a second inference unit 201, which performs inference through a second learning model, and the second learning model is used to detect the shapes of each of the multiple objects based on image information obtained by photographing the multiple objects from multiple shooting points; and an object selection unit 204, which selects any object from the multiple objects based on the inference result of the second inference unit 201. Therefore, even if the object as the object is an object with an unknown shape, the shape of the object can be detected based on the image information containing the multiple objects.

[0062] In the first embodiment, the first inference unit 101 and the second inference unit 201 are configured to detect the shape and surface texture of the object contained in the image information by inference based on the image information acquired by the image information acquisition unit 300, but the present invention is not limited thereto. The shape inference unit and the object detection unit only need to be configured to detect at least the shape of the object by inference.

[0063] Furthermore, the present invention can modify or combine any constituent elements of the above-described embodiments or omit any constituent elements of the embodiments within the scope of the invention.

[0064] Industrial Applicability

[0065] The learning model generation device, information processing device, learning model generation method, and information processing method of the present invention can be used, for example, in an operation of moving any one of a plurality of commodities using a robot arm.

[0066] Hereinafter, various aspects of the present invention will be collectively described as supplementary notes.

[0067] (Note 1)

[0068] A learning model generation device, characterized in that the learning model generation device has:

[0069] an image information acquisition unit that acquires image information of an object obtained by photographing the object from a plurality of photographing points;

[0070] a first inference unit that performs inference using a first learning model, wherein the first learning model calculates the accuracy of whether the shape of the object matches each of a plurality of preset classes based on the image information acquired by the image information acquisition unit;

[0071] a determination unit that determines the class to which the shape of the object corresponds based on the inference result obtained by the first inference unit;

[0072] a learning data generating unit that generates learning data including image information related to a plurality of the objects based on the determination result obtained by the determining unit and the image information of the objects; and

[0073] A learning model generating unit generates a second learning model for detecting the shape of the object from image information including the plurality of objects based on the learning data generated by the learning data generating unit.

[0074] (Note 2)

[0075] The learning model generating device according to Supplement 1 is characterized in that

[0076] The learning data generating unit generates learning data including image information related to the plurality of objects based on the fact that the determining unit determines that the shape of the object does not conform to any of the plurality of classes.

[0077] (Note 3)

[0078] An information processing device, characterized in that the information processing device has:

[0079] an image information acquisition unit that captures a plurality of objects from a plurality of shooting points and acquires image information of the plurality of objects;

[0080] a second inference unit that performs inference using a second learning model, wherein the second learning model is used to detect the shape of each of the plurality of objects based on the image information acquired by the image information acquisition unit; and

[0081] A selection unit selects any object from among the plurality of objects based on the estimation result obtained by the second estimation unit.

[0082] (Note 4)

[0083] The information processing device according to Supplement 3 is characterized in that

[0084] The information processing device includes a calculation unit that calculates the position of each of the plurality of objects based on the estimation result obtained by the second estimation unit.

[0085] The selection unit selects any object among the plurality of objects based on the calculation result obtained by the calculation unit.

[0086] (Note 5)

[0087] The information processing device according to Supplement 3 or 4, characterized in that

[0088] The information processing device includes a calculation unit that calculates the divided area of ​​each of the plurality of objects based on the estimation result obtained by the second estimation unit.

[0089] The selection unit selects any object among the plurality of objects based on the calculation result obtained by the calculation unit.

[0090] (Note 6)

[0091] The information processing device according to Supplement 4 or 5 is characterized in that

[0092] The information processing device includes a position information acquisition unit that acquires information related to the position of an end effector of a robot arm that can move any of the plurality of objects selected by the selection unit.

[0093] The selection unit selects any one of the plurality of objects based on the calculation result obtained by the calculation unit and the information obtained by the position information acquisition unit.

[0094] (Note 7)

[0095] A learning model generation method is performed by a device having an image information acquisition unit, a first inference unit, a determination unit, a learning data generation unit, and a learning model generation unit, wherein the learning model generation method comprises the following steps:

[0096] The image information acquisition unit captures an object from a plurality of capturing points to acquire image information of the object;

[0097] The first inference unit performs inference by using a first learning model, wherein the first learning model calculates the accuracy of the shape of the object conforming to each of a plurality of preset classes based on the image information acquired by the image information acquisition unit;

[0098] The determination unit determines the class to which the shape of the object corresponds based on the inference result obtained by the first inference unit;

[0099] The learning data generating unit generates learning data including image information related to a plurality of the objects based on the determination result obtained by the determining unit and the image information of the objects; and

[0100] The learning model generation unit generates a second learning model for detecting the shape of the object from image information including the plurality of objects, based on the learning data generated by the learning data generation unit.

[0101] (Note 8)

[0102] An information processing method is performed by a device having an image information acquisition unit, a second inference unit, and a selection unit, wherein the information processing method comprises the following steps:

[0103] The image information acquisition unit captures a plurality of objects from a plurality of capturing points to acquire image information of the plurality of objects;

[0104] The second inference unit performs inference using a second learning model, wherein the second learning model is used to detect the shape of each of the plurality of objects based on the image information acquired by the image information acquisition unit; and

[0105] The selection unit selects any object from among the plurality of objects based on the estimation result obtained by the second estimation unit.

[0106] Description of symbols

[0107] 100: learning unit; 101: first inference unit; 102: shape determination unit (determination unit); 103: learning data generation unit; 201: second inference unit; 202: candidate selection unit (position information acquisition unit); 203: feature value calculation unit (calculation unit); 204: target object selection unit (selection unit); 300: image information acquisition unit; 1000: information processing device (learning model generation device).

Claims

1. A learning model generation device, characterized in that: The learning model generating device has: an image information acquisition unit that acquires image information of an object obtained by photographing the object from a plurality of photographing points; a first inference unit that performs inference using a first learning model, wherein the first learning model calculates the accuracy of whether the shape of the object matches each of a plurality of preset classes based on the image information acquired by the image information acquisition unit; a determination unit that determines the class to which the shape of the object corresponds based on the inference result obtained by the first inference unit; a learning data generating unit that generates learning data including image information related to a plurality of the objects based on the determination result obtained by the determining unit and the image information of the objects; and A learning model generating unit generates a second learning model for detecting the shape of the object from image information including the plurality of objects based on the learning data generated by the learning data generating unit.

2. The learning model generation device according to claim 1, characterized in that: The learning data generating unit generates learning data including image information related to the plurality of objects based on the fact that the determining unit determines that the shape of the object does not conform to any of the plurality of classes.

3. An information processing device, characterized in that: The information processing device has: an image information acquisition unit that captures a plurality of objects from a plurality of shooting points and acquires image information of the plurality of objects; a second inference unit that performs inference using a second learning model, wherein the second learning model is used to detect the shape of each of the plurality of objects based on the image information acquired by the image information acquisition unit; and A selection unit selects any object from among the plurality of objects based on the estimation result obtained by the second estimation unit.

4. The information processing device according to claim 3, characterized in that: The information processing device includes a calculation unit that calculates the position of each of the plurality of objects based on the estimation result obtained by the second estimation unit. The selection unit selects any object among the plurality of objects based on the calculation result obtained by the calculation unit.

5. The information processing device according to claim 3, characterized in that: The information processing device includes a calculation unit that calculates the divided area of ​​each of the plurality of objects based on the estimation result obtained by the second estimation unit. The selection unit selects any object among the plurality of objects based on the calculation result obtained by the calculation unit.

6. The information processing device according to claim 4 or 5, characterized in that: The information processing device includes a position information acquisition unit that acquires information related to the position of an end effector of a robot arm that can move any of the plurality of objects selected by the selection unit. The selection unit selects any one of the plurality of objects based on the calculation result obtained by the calculation unit and the information obtained by the position information acquisition unit.

7. A learning model generation method, the learning model generation method being performed by a device having an image information acquisition unit, a first inference unit, a determination unit, a learning data generation unit, and a learning model generation unit, characterized in that: The learning model generation method has the following steps: The image information acquisition unit captures an object from a plurality of capturing points to acquire image information of the object; The first inference unit performs inference by using a first learning model, wherein the first learning model calculates the accuracy of whether the shape of the object matches each of a plurality of preset classes based on the image information acquired by the image information acquisition unit; The determination unit determines the class to which the shape of the object corresponds based on the inference result obtained by the first inference unit; The learning data generating unit generates learning data including image information related to a plurality of the objects based on the determination result obtained by the determining unit and the image information of the objects; and The learning model generation unit generates a second learning model for detecting the shape of the object from image information including the plurality of objects, based on the learning data generated by the learning data generation unit.

8. An information processing method, the information processing method being performed by a device having an image information acquisition unit, a second inference unit, and a selection unit, characterized in that: The information processing method comprises the following steps: The image information acquisition unit captures a plurality of objects from a plurality of capturing points to acquire image information of the plurality of objects; The second inference unit performs inference using a second learning model, wherein the second learning model is used to detect the shape of each of the plurality of objects based on the image information acquired by the image information acquisition unit; and The selection unit selects any object from among the plurality of objects based on the estimation result obtained by the second estimation unit.

Citation Information

Patent Citations

  • Position detection program, position detection method and position detection device

    JP2020077231A