Method and device for creating learning data set

Abstract

A method and apparatus for effectively creating a learning dataset for machine learning is provided with a plurality of objects as objects. Object information is associated with a visual mark, and a learning dataset generation fixture consisting of marks fixed on the base portion is provided with an area as a guide for configuring the object, and an image group of the entire object including the marks is obtained in a state where the area is used as a guide to configure the object, and object information associated with the visual mark is obtained from the obtained image group, and a reconstructed image group is generated by hiding the area corresponding to the visual mark or the learning dataset generation fixture from the image group, and a bounding box is set for the reconstructed image group based on the obtained object information, and the estimated posture information and position information of the object, the object information, and the information related to the bounding box are associated with the photographic image to generate a learning dataset for object recognition and position and posture estimation of the object.

Description

Technical Field

[0001] The present invention relates to an automated method for generating a learning data set in object recognition and position and posture estimation based on machine learning. Background Art

[0002] Traditionally, robots equipped with artificial intelligence (hereinafter referred to as "AI") have been used as a measure to automate operations in factories, etc. In recent years, with the development of machine learning and deep learning (deep learning), the development of AI using machine learning and other methods has been rapidly progressing in production systems such as factories as a measure to achieve full factory automation.

[0003] There is a demand for robot work automation in all industries, among which the food industry and logistics industry are expected areas of future development, and there is a great demand for robot work automation.

[0004] However, since many products handled in the food industry or logistics industry are flexible and their shapes change in complex ways when handled, there is a problem that it is difficult to operate with the manipulator installed in the robot. In addition, the food industry or logistics industry used to handle a small number of products, mainly focusing on mass production of a small number of varieties, but now it is not only that, but also requires small production of many varieties or variable production of variants, and it is also difficult to correctly and quickly recognize many varieties of goods. In other words, it is necessary to identify many varieties of goods in a short time and accurately perform sorting work such as removing assorted gifts or defective products.

[0005] For these reasons, factory automation through robots has not yet been fully developed in the food industry or in the logistics industry.

[0006] Until now, in order to collect learning data about products, it was necessary to capture an image of the target object, manually identify the object from the acquired photographic image, draw a bounding box, etc. In addition, the data input of the position and posture of the object was also performed manually.

[0007] Therefore, a method of automating manual work by attaching a marker to a subject and photographing the subject can be considered.

[0008] However, in the method of photographing by attaching a marker to an object, since the positional relationship between the target object and the marker or the number of markers is not fully discussed, there is a problem that the marker is reflected in the target object or is hidden by the target object.

[0009] That is, if the marker is reflected in the target object or bounding box, not only can high-quality learning data not be obtained, but the marker may also be learned as a feature of the object. In addition, if the marker is blocked by the target object, there is a problem of reduced accuracy in object recognition.

[0010] Regarding object recognition technology, there is a known database construction system that automatically collects learning data using object recognition technology around vehicles (see Patent Document 1). This is about using the detection results of a sensor as teacher data and performing object recognition from another sensor output. For this purpose, learning data with teachers is automatically collected for machine learning.

[0011] However, the database construction system disclosed in Patent Document 1 is a technology that uses multiple types of sensors to perform object recognition, and requires a sensor that has been trained to detect.

[0012] In addition, there is known an image generation method capable of generating a training image for machine learning of image recognition from an image with unknown photographic conditions (see Patent Document 2). This method generates a third new training image from two images held for generating training images for image recognition.

[0013] However, in the image generation method disclosed in the above-mentioned Patent Document 2, since an image different from the actual environment is generated, there is a problem that a high-quality learning data set cannot be created.

[0014] In addition, there is a known method of using an RGB camera to estimate the position and posture of an object, which is a method using a three-dimensional model of the object (see non-patent document 1). However, in the method disclosed in the above non-patent document 1, since a three-dimensional model of the object must be available in advance, in order to generate a learning image that is different from the actual environment, there is still a problem that a high-quality learning data set cannot be created.

[0015] Regarding the problem that the marker is captured and reflected in the figure image, and the marker is learned as a feature of the object, a method of hiding the marker from the photographic image can be considered.

[0016] In this regard, a technique is known in which, during the process of video photography, the scenery actually photographed remains intact and a moving object to be hidden is hidden in real time (see Patent Document 3). This technique is to detect a dynamic object in a moving image, calculate the moving direction and speed, and predictively obtain a background image of the dynamic object when it moves in the calculated moving direction and speed. This predictively obtained background image is used to hide the dynamic object in real time.

[0017] However, there is a problem that the moving body cannot be hidden unless it is the moving body disclosed in the above-mentioned Patent Document 3.

[0018] [Patent Document 1] Japanese Patent Application Publication No. 2017-102838

[0019] [Patent Document 2] Japanese Patent Application Publication No. 2017-45441

[0020] [Patent Document 3] Japanese Patent Application Publication No. 2014-96661

[0021] [Non-patent document 1] Wadim Kehl et al., “SSD-6D: Making RGB-Based 3D Detection and 6D Pose Estimation Great Again”, ICCV 2017, pp.1521-1529 Summary of the invention

[0022] Technical problem to be solved by the invention

[0023] In view of this situation, the present invention is directed to robot control used in factory automation systems, taking multiple objects as objects, not requiring multiple sensors and three-dimensional models of the objects, but using photographic images of the actual environment, with the purpose of providing a method for efficiently creating a learning data set for machine learning and a method for hiding a visual marker group.

[0024] Solutions to the problem

[0025] In order to solve the above-mentioned problem, the method for creating a learning data set of the present invention is to associate the object information of the object with the visual mark, and through the base portion set as the guide area of ​​the configuration position of the object, use the learning data set generation fixture composed of the mark fixed above the base portion to obtain an image group of the entire object including the mark in a state where the object is configured with the area as a guide. Then, the object information associated with the visual mark is obtained from the obtained image group, and the reconstructed image group is generated by hiding the visual mark or the area corresponding to the learning data set generation fixture from the image group, and the bounding box of the object is set for the reconstructed image group based on the obtained object information. Then, the posture information and position information of the object estimated from the photographic image, the object information and the information related to the bounding box are associated with the photographic image to generate a learning data set for object recognition and position and posture estimation of the object.

[0026] Here, visual markers refer to markers that can measure posture and position with high accuracy during the robot's autonomous operation. By using visual markers, not only can the object information of the object be obtained from the photographic image, but also the posture information and position information can be obtained at the same time, which can effectively generate a learning data set.

[0027] By setting the visual markers above the base and around the area, the object can be prevented from being obscured by the visual markers. Preferably, a plurality of visual markers are fixed to the base in a predetermined positional relationship with the object. By setting a plurality of visual markers, for example, even if a certain visual marker is obscured by the object due to the shooting angle, the object can be identified by detecting other visual markers. In addition, by detecting two or more visual markers at the same time, higher accuracy object recognition can be achieved.

[0028] The image group can be obtained by photographing the object while it is moving, etc., with a single camera, or by using multiple cameras. In either case, by collecting a large number of images of the object taken from different angles, they can be used as learning data for machine learning.

[0029] In the photographic images used to generate the learning data set, no processing other than the hiding processing of the jig area for generating the learning data set described later is performed, so it is possible to use photographic images similar to those obtained in the actual environment using the learned model for machine learning. Therefore, the accuracy of object recognition and position and posture estimation can be improved.

[0030] In addition, the object information of the object refers to basic information such as the name and size of the object, and this information is stored in advance in the computer when the learning data set is generated.

[0031] In addition, a bounding box refers to a display area in which an object is surrounded and framed in an image, etc., and is a rectangular area, etc. When setting a bounding box for an object for an acquired image group, the center of gravity position of the object is acquired and set as the position information of the object so that the bounding box surrounds the entire object as much as possible.

[0032] The learning data set creation method of the present invention preferably performs a hiding process on the learning data set from the image group corresponding to the area of ​​the learning data set generation fixture, and the hiding process includes the following steps 1) to 3).

[0033] 1) Detect the region corresponding to the jig generated by the learning dataset from the image set (detection step).

[0034] 2) By detecting the object region from the region including the imaged object extracted in the detection step, only the region corresponding to the learning data set generation jig is extracted (extraction step).

[0035] 3) The image group in the area extracted in the extraction step is overlaid as a background image or a noise image (overlay step).

[0036] Here, the extraction step in the above-mentioned hiding process may detect the object region by using the shape information that has been approximated in advance, or may detect the object region by using a learning model that has been learned in advance.

[0037] By hiding the area corresponding to the learning dataset generation jig, it is possible to prevent the computer from recognizing the base portion or visual mark that constitutes the learning dataset generation jig as part of the object during machine learning.

[0038] In the covering step, preferably, the image in the area extracted in the extraction step is covered as a noise image. By covering as a noise image, the recognition rate of various types of objects can be improved.

[0039] In the learning data set creation method of the present invention, the base portion is preferably a circular, oval or rectangular plate, but may also be an L-shaped plate formed by bending and extending the plate. This is because, for example, in the case of a circular plate, multi-viewpoint images can be easily acquired by rotating the plate and reshooting.

[0040] In the learning data set creation method of the present invention, when the base portion is circular, it is preferred to arrange the visual mark around the area in an upward direction with the center point of the area. Since the area has a guiding function when arranging the object, the object is usually arranged at the center of the area. By arranging each visual mark with the center point facing upward, the structure of the visual mark is easily detected even when the object is photographed from various angles. Therefore, a guide with the center point as the main purpose can also be further arranged in the center of the area.

[0041] In the learning data set creation method of the present invention, the bounding box is preferably a two-dimensional bounding box or a three-dimensional bounding box. For example, when three visual markers are set, by detecting multiple markers simultaneously, not only can higher-precision object recognition be achieved, but also the estimation accuracy of the position and posture of the object can be improved, so a three-dimensional bounding box suitable for the object can be set.

[0042] Regarding the visual markers used in the learning dataset creation method of the present invention, two-dimensional graphic markers or three-dimensional markers including AR (Augmented Reality) markers can be used. AR markers refer to graphic images used as identifiers for specifying the location where additional information is displayed in an augmented reality system. In addition, regarding three-dimensional markers, for example, spherical markers used in optical motion capture systems can be used. When these spherical markers are used to detect positions and postures, more than three spherical markers are fixed on the base portion of the learning dataset generation jig.

[0043] In the area in the learning dataset creation method of the present invention, the identification ID of the learning dataset generation jig may also be displayed.

[0044] By displaying the identification ID of the learning dataset generation jig in the area, the management of the learning dataset generation jig can be simplified. The identification ID is preferably displayed in the center of an area that is easily hidden when configuring an object in order to prevent it from being reflected in the acquired image.

[0045] In the learning data set creation method of the present invention, a direction adjustment guide for adjusting the direction of the object can also be set in the area. By setting the direction adjustment guide, it is easier to manage the object information on the database, thereby improving convenience.

[0046] The position information in the learning data set creation method of the present invention can use the position information of any point of the object. For any point, the position information of the point in the reference posture of the object is registered in advance. In this case, the position information is the difference information between the position of the point of the object configured using the area as a guide and the position of the point in the reference posture in the reference coordinate system.

[0047] In addition, when the position information in the learning data set creation method of the present invention is the center of gravity position information, the center of gravity position information is the difference information between the center of gravity position of the object configured with the base as a guide and the center of gravity position of the reference posture in the reference coordinate system.

[0048] Here, the camera coordinate system can be used instead of the reference coordinate system to represent the posture information and position information of the object. For example, when a fixed camera is used for shooting, the three-dimensional coordinates in the camera coordinate system of the camera are used to represent the posture information and position information of the object. Then, the three-dimensional coordinates of the camera coordinate system are transformed into three-dimensional coordinates in the real space, and the three-dimensional coordinates in the real space are transmitted to the robot, so that the robot can correctly capture the object.

[0049] The posture information in the learning data set creation method of the present invention is, specifically, information calculated using the photographic image of the visual marker and the relative positional relationship between the visual marker and the object, and the difference information between the posture of the object configured with the area as a guide and the reference posture in the predefined reference coordinate system. By estimating the posture of the object using the photographic image of the visual marker and the relative positional relationship between the visual marker and the object, high-precision posture estimation can be performed.

[0050] Regarding the reference coordinate system, for example, by using a visual marker pre-set on a learning data set generation jig as a reference marker, it can be defined by configuring it at a position and posture that is desired to be used as a reference. In this way, data related to the reference position and reference posture of the reference marker is stored in a database, and in the learning data set creation stage, the reference coordinate system of the actual space is used, and the difference in position and posture in the reference coordinate system is associated with the image as information on position and posture.

[0051] Even when there is no reference mark indicating the reference coordinate system in the photographic image, the reference coordinate system can be identified from the photographic image. For example, in the case of a fixed camera, if the reference coordinate system can be set by the reference mark in the photographic image before zooming, even when zooming is performed and becomes invisible, the position and posture of the reference mark outside the photographic image can be calculated as long as the zoom magnification is known.

[0052] On the other hand, when multiple cameras are fixed, if the positional relationship between the cameras is known in advance, even if there is no reference mark indicating the reference coordinate system in the photographic image, the position and posture of the reference mark outside the photographed image can be calculated. In addition, even if the reference mark is configured at a position that can only be seen by one camera, the reference coordinate system can be calculated by using the image captured in the state of the temporarily configured reference mark in a place where the visible camera and the invisible camera can simultaneously shoot. In addition, even if the camera is in dynamic movement, as long as the positional relationship between the position of the camera when the reference coordinate system was first captured and the camera at the moving destination is known, the reference coordinate system can be calculated by calculation.

[0053] The image group obtained in the learning data set creation method of the present invention can be obtained by placing the object on the learning data set generation fixture and collecting images while conveying the object through a conveying device, or by placing the object on the learning data set generation fixture and collecting images while rotating it through a rotating device, or by placing the object on the learning data set generation fixture and collecting images while moving it through a moving device. Multi-view image groups can be easily obtained by these image acquisition methods. Any of these image acquisition methods can be used in combination. By using a conveying device, a rotating device or a moving device, even if only one camera is used as an image shooting device, a multi-view image group can be easily obtained, which facilitates the learning data set generation work and can also shorten the time.

[0054] Here, the conveying device is a device that conveys an object using a conveying path as a track. Specifically, when the object is arranged on the learning data set generation fixture, it is preferred to use a conveyor belt to convey the object. In addition, regarding the rotating device, for example, when the object is arranged on the learning data set generation fixture, a rotating mechanism that can rotate in a horizontal plane is used. In addition, regarding the moving device, for example, when the object is arranged on the learning data set generation fixture, a direct-acting mechanism that enables the learning data set generation fixture to move on a horizontal linear track is used. In addition, a rotating-moving mechanism that combines a rotating mechanism and a direct-acting mechanism and moves while rotating can also be used. These rotating mechanisms, direct-acting mechanisms or rotating-moving mechanisms can be arranged on a conveyor belt, and a multi-viewpoint image group can be obtained with a single camera.

[0055] Furthermore, in the sense of acquiring a multi-viewpoint image group, a device for automatically or manually applying deformation to an object may be provided. By providing a deformation applying device, a wider image group can be acquired. Objects that are considered to be effective for deformation, such as bags, are considered to be objects to be deformed. Therefore, such a device for deforming an object includes, for example, a method of automatically deforming using a robot arm and a manipulator, a method of manually deforming by a human, and a method of preparing a base that can deform a posture and using the base to perform deformation.

[0056] The learning data set of the present invention is created according to the learning data set creation method of the present invention. In addition, the learning data set generation jig of the present invention is used in the learning data set creation method of the present invention, and as described above, it is composed of an area that serves as a guide for the configuration position of the object, and a visual mark fixed on the base portion, and the visual mark is associated with object information such as the object name of the object.

[0057] In the object recognition and position and posture estimation method of the present invention, machine learning is performed using the learning data set created by the learning data set creation method of the present invention, the learned model is obtained, and the object is detected by image recognition from the newly captured image to create a bounding box for recognition. Then, a partial image is generated with the bounding box for recognition cut out, and the information (e.g., the name of the object, etc.) and the posture of the detected object are estimated based on the generated partial image using the learned model. In addition, the position, width, and height of the bounding box for recognition are obtained, and the learned model is used to estimate the position of the detected object in the three-dimensional coordinates of the actual space.

[0058] In addition, the position of the identification bounding box that frames the entire detected object may be the center of gravity of the bounding box. In addition, the position in the three-dimensional coordinates of the real space related to the detected object may also be the center of gravity of the object.

[0059] In the object recognition and position and posture estimation method of the present invention, the recognition bounding box may be a two-dimensional bounding box or a three-dimensional bounding box.

[0060] In the object recognition and position and posture estimation method of the present invention, the learned model specifically includes at least one of a classifier or a regressor for estimating the posture or position of the object. The estimated position may be the center of gravity position of the object.

[0061] In the object recognition and position pose estimation method of the present invention, the captured image is a photographic image of multiple objects, a bounding box for identification is created for each detected object, and the names of all detected objects and the estimated poses and positions are calculated as three-dimensional coordinates in the actual space.

[0062] Thus, for each detected object, the estimated posture and position in the three-dimensional coordinates of the real space can be communicated to the robot of the factory automation system. In addition, by estimating and calculating the position and center of gravity in the three-dimensional coordinates of the real space for each detected object, the robot can accurately capture the object.

[0063] The photographic images in the object recognition and position and posture estimation method of the present invention are preferably photographic images taken in an environment that is the same as or similar to the generation environment of the learning data set. By using photographic images taken in an environment that is the same as or similar to the generation environment of the learning data set, the accuracy of object recognition and position and posture estimation can be improved.

[0064] The learning dataset creation program of the present invention is a program for causing a computer to execute the following steps a) and b) which are a part of the processing of the learning dataset creation method of the present invention.

[0065] a) A step of setting a bounding box of an object for an image group.

[0066] b) A step of associating the pose information and position information, object information and bounding box related information of the object estimated from the photographic image with the photographic image to generate a learning data set.

[0067] The object recognition and position and posture estimation program of the present invention is used to enable a computer to execute the following c) to e) processing, and the program is a part of the object recognition and position and posture estimation method of the present invention.

[0068] c) A step of detecting the name of the object through image recognition based on the newly captured photographic image and creating a bounding box for recognition.

[0069] d) A step of generating a partial image from which the recognition bounding box is cut out, and estimating the posture of the detected object using the learned model based on the generated partial image.

[0070] e) Get the position, width and height of the bounding box used for recognition, and use the learned model to estimate the position of the detected object in 3D coordinates in real space.

[0071] Furthermore, the position of the recognition bounding box that frames the entire detected object may be the position of the center of gravity of the framed bounding box. Furthermore, the position of the object detected in the three-dimensional coordinates of the real space may be the position of the center of gravity of the object.

[0072] It is also possible that the position in the three-dimensional coordinates of the real space associated with the detected object is the position of the center of gravity of the object.

[0073] The learning data set creation device of the present invention includes the following parts 1) to 4).

[0074] 1) A learning data set generation jig is composed of a base portion having an area provided therein as a guide for the placement position of an object, and a visual marker associated with object information of the object fixed to the base portion.

[0075] 2) A camera device that obtains a group of images of the entire object including a visual marker, in a state where the object is arranged in an area as a guide.

[0076] 3) A device for obtaining object information associated with the visual marker from the acquired image set.

[0077] 4) A device for generating a reconstructed set of images by hiding the areas corresponding to the visual markers or learning dataset generation fixtures in the set of images.

[0078] 5) A device for setting a bounding box of an object for the acquired image group.

[0079] 6) A device for generating a learning data set for object recognition and position and posture estimation of an object by associating the pose information and position information, object information and bounding box-related information of the object estimated from the photographic image with the photographic image.

[0080] The learning data set creation device of the present invention is provided with a unit for generating a reconstructed image group by performing hiding processing on the area corresponding to the visual mark of the learning data set or the learning data set generation fixture. The hiding unit preferably includes a detection unit for detecting the area corresponding to the learning data set generation fixture from the image group, and an extraction unit for detecting the object area by extracting the area including the photographed object by the detection unit, and only extracting the area corresponding to the learning data set generation fixture, and a covering unit for covering the area extracted by the extraction unit as a background image or a noise image in the image group. The extraction unit here can detect the object area using the previously approximated shape information, or can detect the object area using the previously learned learning model.

[0081] In addition, in order to prevent the learning dataset generation jig from being photographed, the jig may have transparency in addition to the visual mark. If the part other than the visual mark of the learning dataset generation jig has transparency, it is possible to prevent the learning dataset generation jig from obstructing the object in the photographic image, thereby improving the accuracy of object recognition.

[0082] In the learning data set creation device of the present invention, it is preferred that the visual markers constituting the learning data set generation fixture are preferably arranged around the area with the center point of the area as the upward direction. The visual marker is a two-dimensional graphic marker or a three-dimensional marker including an AR marker. In addition, the base portion may be a circular, oval or rectangular plate, or an L-shaped plate formed by bending and extending the plate. The bounding box may be a two-dimensional bounding box or a three-dimensional bounding box. The identification ID of the learning data set generation fixture may also be displayed in the area. In addition, a direction adjustment guide for adjusting the direction of the object may also be provided in the area.

[0083] An industrial robot that performs machine learning using a learning data set created by the learning data set creation method according to the present invention, an industrial robot that uses the object recognition and position posture estimation method of the present invention, and an industrial robot that is installed with the object recognition and position posture estimation program of the present invention can be used in factory automation systems for packaging or processing operations in the food industry or logistics industry.

[0084] Effects of the Invention

[0085] According to the present invention, a learning data set for machine learning can be effectively created by using photographic images of the actual environment with multiple objects as the object, without requiring multiple sensors and 3D models of the objects, and can perform object recognition and position and posture estimation with high accuracy, and can be effectively used for robot control in factory automation systems. In addition, according to the present invention, since the visual marker group can be hidden, it has the effect of being able to create a higher quality learning data set. BRIEF DESCRIPTION OF THE DRAWINGS

[0086]

Figure 1

[0087]

Figure 2

[0088]

Figure 3

[0089]

Figure 4

[0090]

Figure 5

[0091]

Figure 6

[0092]

Figure 7

[0093]

Figure 8

[0094]

Fig. 9

[0095]

Fig.10

[0096]

Fig.11

[0097]

Fig.12

[0098]

Fig.13

[0099]

Fig.14

[0100]

Fig.15

[0101]

Fig.16

[0102]

Fig.17

[0103]

Fig.18

[0104]

Fig.19

[0105]

Fig. 20

[0106]

Fig.21

[0107]

Fig. 22

[0108]

Fig.23

[0109]

Fig.24

[0110]

Fig.25

[0111]

Fig.26

[0112]

Fig. 27

[0113] Hereinafter, an example of an embodiment of the present invention will be described in detail with reference to the drawings. In addition, the scope of the present invention is not limited to the following embodiments or illustrations, and other changes and modifications are possible.

[0114] [Example 1]

[0115] Figure 1 The schematic flow of the creation process of the learning data set and the schematic flow of the object recognition and position and posture estimation process are shown. Figure 1As shown, in the learning data set creation stage, first, an AR marker (two-dimensional graphic marker) is used as a visual marker to associate the object information of the object with the AR marker (step S01). Next, a learning data set generation fixture using the associated two-dimensional graphic marker is prepared (step S02). The area of ​​the learning data set generation fixture is used as a guide to configure the object (step S03). In the state where the object is configured, a multi-view image group of the object is obtained (step S04). The two-dimensional graphic marker is detected and identified for the obtained image group (step S05). A bounding box that frames the entire object is set for the obtained image group (step S06). The estimated object posture information and center of gravity position information, object information, and bounding box related information are associated with the image (step S07). For the obtained image group, the learning data set generation fixture including the AR marker is hidden (step S08). Then, the multi-view image group is registered in the database as a learning data set (step S09).

[0116] In addition, in the setting of the bounding box in step S06, the entire object can be framed as much as possible, and even if some parts are not framed, they can be used as a learning data set. Even if the object is blocked by other objects, it is also possible to frame the visible range that is not blocked.

[0117] Next, using the registered learning data set, a learned model is obtained through machine learning such as deep learning (step S10).

[0118] Then, in the object recognition and position and posture estimation stage, a photographic image is first acquired (step S11), the object in the photographic image is detected by image recognition, and a bounding box for recognition is set (step S12). In addition, the name of the object can be obtained when the object is detected and recognized. The partial image of the recognition bounding box of the object is intercepted to estimate the posture and center of gravity position in a predefined reference coordinate system (three-dimensional coordinates in real space) (step S13).

[0119] The reference coordinate system is defined by placing visual markers at the positions and postures that you want to use as references.

[0120] Figure 2 The functional block diagram of the device for creating a learning data set, the device for object recognition and the device for position and posture estimation is shown. Figure 2As shown, in the learning data set creation device 10, an image of an object is captured using the photography unit 20 and the learning data set generation jig 30. The captured image is operated by the computer 40 of the learning data set creation device 10 as the AR marker recognition unit 40a, and the AR marker is recognized. In advance, the AR marker is associated with the object information of the object to be recognized in the database 80, and the computer 40 operates as the object recognition unit 40b to recognize the object information such as the object name of the object. Based on the information obtained from the AR marker recognition unit 40a and the object recognition unit 40b, the computer 40 operates as the bounding box setting unit 40c to set the bounding box. In addition, the computer 40 is operated as the object posture estimation unit 40d to estimate the posture of the object. In addition, the computer 40 is operated as the center of gravity position estimation unit 40e to estimate the center of gravity position of the object. The information about the acquired object or the information about the posture and the center of gravity position is assigned to the set bounding box by the computer 40 operating as the bounding box assignment unit 40f.

[0121] The learning data set generation jig 30 in the image is hidden from the photographic image by operating the computer 40 as the learning data set generation jig area hiding unit 40g. The learning data set generation jig area hiding unit 40g, from the acquired image,

[0122] The detection unit 40h cuts the area corresponding to the learning data set generation jig according to the shape of the learning data set generation jig, the extraction unit 40i further extracts the area corresponding to the generation jig only through the area extracted by the detection unit 40h, and the hiding unit 40j covers the area extracted by the extraction unit with a background image or a noise image. The image hidden by the learning data set generation jig area hiding unit 40g is stored in the database 80 as a learning data set.

[0123] The learning data set is used to create a classifier 9a and a regressor 9b through machine learning of the model generation computer 9, and a learned model is obtained.

[0124] The generated learned model is used in the object recognition and position and posture estimation device 11. In the object recognition and position and posture estimation device 11, an object is photographed by the photographing unit 21. The obtained photographic image is operated by the computer 41 as an object recognition unit 41a to detect the object contained in the image, and obtain object information such as the object name. For the detected object, the computer 41 is operated as a setting unit 41b for the recognition bounding box, thereby setting the recognition bounding box. When the recognition bounding box is set, the posture and center of gravity position are estimated for each recognition bounding box by operating the computer 41 as a learned model 41C.

[0125] In addition, the imaging unit 20 and the imaging unit 21, the computer 40 and the computer 41, or the storage unit of the database 80 and the storage unit of the database 81 may be configured to use the same device.

[0126] (About the creation of learning datasets)

[0127] Figure 3 The system configuration diagram for creating a learning dataset is shown in Figure 2. Figure 3 As shown, when creating a learning data set, a single camera 2, a plate 14, a reference mark 33, a conveyor 12 for moving the plate 14 and the position of the object 5 on the plate 14, and a computer 4 are used.

[0128] The plate 14 is composed of an AR marker 3 and a base 17, and the AR marker 3 is fixed to the base 17. Here, an area not shown is provided on the base 17, and an object 5 to be identified is arranged in the area. The object 5 is photographed by a camera 2 provided above. In addition, the plate 14 can be individually manufactured according to the shape of the object 5, or a plate of a predetermined size can be used.

[0129] The reference mark 33 is arranged outside the conveyor belt 12 and is arranged at a fixed position. The reference mark 33 is connected to the base through a bracket to prevent the reference mark 33 from being blocked by the object 5 during photography. Therefore, the method of setting the reference mark 33 is not limited to such a method, and the same method as the plate 14 can be used. By arranging the reference mark 33 at a position and posture desired as a reference, a reference coordinate system is defined.

[0130] The computer 4 is provided with a database 8 that stores information on the position and posture of a group of images captured by the camera 2 , the AR marker 3 , and the object 5 .

[0131] Figure 4 The flowchart of learning dataset generation is shown in Figure 2. Figure 4As shown, first, the object 5 arranged on the base portion 17 is photographed by the camera 2, and a photographic image is input into the computer 4 (step S101). In the computer 4, the AR marker contained in the acquired image is detected by image analysis (step S102). In the object information associated with the AR marker registered in the database 8, information such as the general name for identifying the object is registered. Next, the object information associated with the AR marker is obtained from the database 8 (step S103). In the information associated with the AR marker registered in the database 8, information related to the size (dimensions) of the object is registered. Based on the posture information of the AR marker, the posture of the object is estimated (step S104). Furthermore, the center of gravity position of the object is calculated (step S105). Here, the center of gravity position of the object is the center of gravity position calculated based on the shape of the object and the configuration on the learning data set generation fixture, and is calculated based on the position (x, y, z) information of the center of gravity of the object in the reference coordinate system. In addition, regarding the calculation of the center of gravity position, in addition to the height of the object and the configuration position and configuration posture on the learning data set generated fixture, the height of the fixture and the configuration position and posture of the AR marker on the fixture are also obtained as object information in advance, which can also be used to calculate the center of gravity position of the object.

[0132] Then, a bounding box is set to frame the object (step S106), and the object information, posture, center of gravity position, and the set bounding box information (position in the image, width, height) are associated with the image (step S107). If there are unprocessed AR markers in the image, the unprocessed AR markers are processed in the same way. If all AR markers in the image are processed (step S108), an image linked to the object information, posture and center of gravity position information of the object and the bounding box information is output (step S109). The image is taken multiple times, and after all images are processed (step S110), the plate containing the marker is hidden from the photographic image (step S111). In this way, a learning data set is created.

[0133] Here, refer to Figures 5 to 7 The learning data set generation jig of Example 1 is described. Figure 5 The diagram shows a top view of the fixture generated by the learning data set of Example 1. Figure 5 As shown, the plate 14a is composed of a base portion 17 and AR markers (3a to 31), and a guide area 18 for placing the object 5 is provided on the base portion 17. The AR markers (3a to 31) are attached to the base portion 17 so as to surround the area 18. Here, the description is made using the plate 14a as an example, and the plates (14b, 14c) described later also have the same structure.

[0134] Figure 6These are photographic effect diagrams of the objects of Example 1, showing the cases where the object (1) is cylindrical and (2) is rectangular.

[0135] The object 5 is not only placed in the area 18, but also fixed to the area 18. In addition, the area 18 is a guide for arranging the object 5, so Figure 6 (1) As shown in the object 5a, it is preferable to arrange the bottom surface of the object so as to be contained in the area 18, but it is not necessarily limited to this. Figure 6 The object 5 b shown in (2) may be arranged so as to partially protrude from the region 18 .

[0136] exist Figure 6 In (2), the AR markers (3b, 3h) are blocked due to the part of the object 5b that is placed beyond the image. However, it is not necessary to recognize all the AR markers (3a to 3l) in one shot. It is sufficient to read at least one AR marker.

[0137] like Figure 6 As shown in (1), when the object 5a is arranged on the board 14a, the information about the board 14a and the information about the object 5a are pre-registered in the database 8. That is, the type, shape, number, position and posture of the AR markers pre-set in the form of a plate, the name of the object corresponding to the AR marker, the long side (mm), short side (mm) and height (mm), and the posture (deg) of the object in the reference coordinate system are all registered in the database 8. Table 1 below shows an example of a database table. In Table 1, the object code is used as an identification code of the object. In addition, the marker code is the identification ID of the board 14.

[0138]

Table 1

[0139]

[0140] As shown in Table 1 above, with respect to object code 1, the mark code registered in the database is 3, the object's long side is 100 mm, the object's short side is 50 mm, the object's height is 100 mm, the object's center of gravity position in the reference coordinate system is x 20, y 30, z -150, and the object's posture in the reference coordinate system is 0 (deg). In addition, with respect to object code 2, the mark code registered is 2, the object's long side is 200 mm, the object's short side is 150 mm, the object's height is 10 mm, the object's center of gravity position in the reference coordinate system is x 10, y 20, z -30, and the object's posture in the reference coordinate system is 0 (deg). Regarding object code 3, the registered mark code is 1, the object's long side is 300 mm, the object's short side is 250 mm, the object's height is 50 mm, the object's center of gravity in the reference coordinate system is x 12, y 28, z -80, and the object's posture in the reference coordinate system is 10 (deg). Similarly, each data is registered for each object.

[0141] Therefore, when the AR marker 3 and the object 5a are photographed by the camera 2 in a state where the relative position and posture relationship between the AR marker (3a-3l) and the object 5a is fixed, the computer 4 detects one of the AR markers (3a-3l) to obtain information such as the type of the AR marker, which is one of the AR markers (3a-3l), the position and posture of the AR marker, and the name of the corresponding object 5a. The detected AR marker may be one, but if two or more AR markers (3a-3l) can be detected, information related to the object 5a can be obtained more reliably.

[0142] In addition, the posture of the object 5a in the reference coordinate system is estimated based on the relative position and posture relationship between the AR markers (3a to 3l) and the object 5a. In addition, the center of gravity position of the object 5 in the reference coordinate system is estimated based on the relative position relationship between the AR markers (3a to 3l) and the object 5a and the sizes of the object 5a and the plate 14.

[0143] Using the data on the size of the object registered in advance in the database 8, a bounding box that frames the object 5a is set based on the data of the estimated position or posture.

[0144] For an object 5a, by changing the angle and performing multiple photography, multiple output images of graphics can be obtained. Figure 6As shown in (1), even when the relative position and posture relationship between the AR marker (3a-3l) and the object 5a is fixed, the image obtained when shooting from the arrow 19a is different from the image obtained when shooting from the arrow 19b or the arrow 19c. Regarding the shooting method, in this embodiment, on the basis of fixing one camera 2, multiple shots are taken while moving the plate 14 and the object 5a. In addition, one camera 2 can be moved to shoot while the plate 14 and the object 5a are fixed, or multiple cameras can be used to shoot from different angles.

[0145] Figure 7 FIG. 1 is a diagram illustrating the configuration of an AR marker according to Example 1. Figure 7 In the figure, for the sake of convenience of explanation, not all AR markers (3a to 31) are displayed, and only AR markers (3a, 3d, 3g, and 3j) are displayed.

[0146] like Figure 7 As shown, the AR marker 3a is set to be in the direction from the center point P2 of the AR marker 3a toward the center point P1 of the base part 17, that is, the upward direction is the Y-axis direction, the right direction is the X-axis direction, and the front direction is the Z-axis direction.

[0147] The same is true for the AR markers (3d, 3g, 3j). That is, the direction from the center point P3 of the AR marker 3d to the center point P1 of the base 17, the direction from the center point P4 of the AR marker 3g to the center point P1 of the base 17, and the direction from the center point P5 of the AR marker 3j to the center point P1 of the base 17 are set as the Y-axis directions.

[0148] Regarding the positions and postures of the AR markers (3a, 3d, 3g, 3j), for example, in the case of the AR marker 3a, the corner (C 21 ~C 24 ) is used as the reference for estimation. Similarly, in the case of AR marker 3d, the position coordinates of the corner (C 31 ~C 34 ) position coordinates, and in the case of AR marker 3g, the corner (C 41 ~C 44 ) position coordinates, and in the case of AR marker 3j, the corner (C 51 ~C 54 ) is used as the reference for estimation.

[0149] By setting the AR markers (3a, 3d, 3g, 3j) so that the direction from each center point to the center point P1 of the base 17 becomes the Y-axis direction, the object 5 is arranged around the center point P1 of the base 17. When a multi-viewpoint image group is obtained in a fixed state, no matter from which angle the image is taken, the direction of the object 5 can be easily estimated based on the direction of the AR markers (3a to 3l). However, for this purpose, it is necessary to arrange the object 5 at the center of the base 17.

[0150] Therefore, if Figure 5 As shown, a position adjustment guide 18a is provided in the region 18. The position adjustment guide 18a is provided substantially in the center of the region 18, and facilitates position adjustment when the object 5 is arranged.

[0151] If the object 5 arranged on the area 18 is small, it is arranged so as to be accommodated in the position adjustment guide 18a. If the object 5 is large, it is arranged so that the center of the object 5 can abut against the position adjustment guide 18a.

[0152] In addition, a guide 18b for orientation adjustment is provided in the area 18. The guide 18b for orientation adjustment is displayed in the shape of an arrow, and the front face of the object 5 is arranged in the direction of the arrow. Even without the guide 18b for orientation adjustment, a learning data set can be created, but if the guide 18b for orientation adjustment is provided, the orientation of the object 5 is fixed to be consistent with the arrow, and the acquired images can be easily managed, and a high-quality learning data set can be created.

[0153] As described above, the benchmark set when configuring the object 5 in the area 18 is set in order to more effectively perform machine learning such as deep learning, so it is not necessary to strictly follow the configuration. For example, even if it is configured at a slight tilt, the data can be used in subsequent machine learning.

[0154] In addition, for the sake of convenience, Figure 5 and Figure 6 The area 18 is indicated by a dotted line, but is not actually displayed. The area 18 serves as a guide to estimate the position of the object from the position of the AR markers (3a to 31) attached to the base 17. However, as another embodiment, it may be displayed by a dotted line or the like.

[0155] In addition, the identification ID of the board may be displayed in the area 18. For example, if a number that can identify the board is displayed near the center of the area 18 where the orientation adjustment guide 18a is provided, the possibility of being captured in the photographic image is reduced, and the board can be easily managed, thereby improving convenience.

[0156] Figure 8A schematic diagram showing images used to create a learning data set in Example 1. Figure 8 The image 7a shown in FIG. Figure 2 An image taken by the camera 2 shown in FIG. Figure 8 As shown in FIG. 7a, a conveyor belt 12, objects (5a to 5c), plates (14a to 14c), and boundary boxes (6a to 6c) are shown. Here, object 5a is a food packaged in a barrel-shaped container. In addition, object 5b is a food packaged in a paper container. Although the container is arranged in an upright state, it can also be arranged in a Figure 6 (2) is arranged in a flat state. The object 5c is a soft resin container packaging food. Here, the object used is food, and objects other than food can also be used.

[0157] Arrow 19d indicates the flow direction of the conveyor belt 12, and image 7a is taken when the conveyor belt 12 flows in the direction of arrow 19d. In addition, in image 7a, although objects other than the above are not shown, it is also possible to configure other devices in the background for shooting. When the object (5a~5c) flows through the conveyor belt 12 once, multiple shots are taken. In addition, since the relative position and posture of the AR marker (3a~3l) and the object (5a~5c) move in a fixed state, the position or posture of the AR marker (3a~3l) will change.

[0158] In addition, the same object can be photographed by setting the camera 2 in various directions. It is possible to photograph multiple times when the object flows once on the conveyor belt 12, or to photograph the same object by setting the camera 2 in various directions, so that images of various patterns can be obtained, and it is possible to generate a higher-precision model through subsequent machine learning such as deep learning.

[0159] When shooting in the above manner, Figure 8 In the image 7a shown, image data is obtained, and the type and position and posture of the AR markers (3a to 3l) are first detected for each board (14a to 14c) from each image. Regarding the AR markers (3a to 3l), the name, long side (mm), short side (mm), height (mm) and posture (deg) of the object in the reference coordinate system are registered in advance for each board (14a to 14c), and the position or posture of each AR marker (3a to 3l) is also registered. Therefore, if at least one of the AR markers (3a to 3l) is detected, the name, long side (mm), short side (mm), height (mm) and posture (deg) of the object in the reference coordinate system can be obtained. Furthermore, if two or more AR markers are detected, the object can be identified with high accuracy.

[0160] Regarding the object 5a, among the AR markers (3a to 3l) provided on the plate 14a, the AR markers (3f to 3h) are not captured in the image 7a, but the AR markers (3a to 3e, 3i to 3l) are captured.

[0161] Regarding the object 5b, among the AR markers (3a to 3l) provided on the plate 14b, the AR marker 3g is not captured in the image 7a, but the AR markers (3a to 3f, 3h to 3l) are captured.

[0162] Regarding object 5c, among the AR markers (3a~3l) set on plate 14a, AR markers (3f~3h) are not captured in image 7a, and AR markers (3j, 3k) are only partially captured, while AR markers (3a~3e, 3i, 3l) are captured.

[0163] Therefore, for the object 5b, the name of the object 5b is obtained by detecting the AR markers (3a to 3f, 3h to 3l) provided on the plate 14b. Similarly, the name of the object 5c is obtained by detecting the AR markers (3a to 3e, 3i, 3l) provided on the plate 14c, and the name of the object 5a is obtained by detecting the AR markers (3a to 3e, 3i to 3l) provided on the plate 14a.

[0164] The position and posture information of the objects (5a to 5c) is estimated based on the position and posture information defined in the AR markers (3a to 31) on each of the plates (14a to 14c).

[0165] Specifically, first, the posture of the object 5b is calculated based on the posture information defined in the AR markers (3a to 3f, 3h to 3l) set on the board 14b. Similarly, the posture of the object 5c is calculated based on the posture information defined in the AR markers (3a to 3e, 3i, 3l) set on the board 14c, and the posture of the object 5a is calculated based on the posture information defined in the AR markers (3a to 3e, 3i to 3l) set on the board 14a.

[0166] Next, the center of gravity position of the object 5b is calculated based on the position information defined in the AR markers (3a to 3f, 3h to 3l) provided on the plate 14b and the information on the relative positional relationship between the AR markers (3a to 3f, 3h to 3l) provided on the plate 14b and the object 5b. Similarly, the center of gravity position of the object 5c is calculated based on the position information defined in the AR markers (3a to 3e, 3i, 3l) provided on the plate 14c and the information on the relative positional relationship between the AR markers (3a to 3e, 3i, 3l) provided on the plate 14c and the object 5c, and the center of gravity position of the object 5a is calculated based on the position information defined in the AR markers (3a to 3e, 3i to 3l) provided on the plate 14a and the information on the relative positional relationship between the AR markers (3a to 3e, 3i to 3l) provided on the plate 14a and the object 5a.

[0167] Using the above calculated data and data related to the size of the object (5a-5c), a bounding box (6a-6c) is generated to frame the object (5a-5c). Here, for example, the bounding boxes (6a, 6c) that frame the object (5a, 5c) overlap, but since the bounding boxes only store coordinate data, there is no problem even if they overlap.

[0168] As mentioned above, reference will be made to Fig. 9 A method of setting a bounding box (6a to 6c) for an object (5a to 5c) is described. Fig. 9 This is a diagram illustrating the setting of a two-dimensional bounding box. Fig. 9 In order to set Figure 8 The following description will be given by taking the bounding box 6c set in the object 5c in the image 7a as an example. Fig.12 The configuration of the object 5c in the figure is different, and the object 5c is in a state of being slightly tilted to the left.

[0169] [Formula 1]

[0170] x'=xoosθ+ysinθ

[0171] y'=yoosθ+xsinθ+hsinφ

[0172] In addition, if Fig. 9 As shown, when setting the boundary box 6c, in order to set the above-mentioned offset, the area set is larger than the boundary box calculated according to the above-mentioned formula 1.

[0173] exist Figure 8 In the example of the image 7a, however, multiple images can be obtained as in the images (7b to 7f) by repeatedly photographing while the conveyor belt 12 is running. The images obtained are not limited to the images (7a to 7f), and more images can be obtained.

[0174] (About hiding the board containing the AR marker)

[0175] If a data group in which the state of the AR marker 3 or the base portion 17 in the board 14 is captured in an image as learning data is used as a learning data set for machine learning, for example, the model generation computer 9 may learn the AR marker 3 itself. If the model generation computer 9 learns the AR marker 3 itself, the quality of the learning data set is reduced, so it is better to delete it from the learning data such as the AR marker.

[0176] Here, we will refer to Figure 10 to Figure 13 Describes a method for hiding a board containing an AR marker. Fig.10 The hidden flow chart of the learning dataset generation fixture region is shown. In addition, Figure 11 to Figure 13 An illustration showing the hidden learning dataset generation fixture region.

[0177] first, Fig.11 (1) shows the acquired image before hiding. Fig.11 As shown in (1), the objects (5a to 5c), the plates (14a to 14c) and the conveyor belt 12 are displayed in the image 7a. Fig.10 As shown, the area of ​​the board is detected from the image 7a (step S41).

[0178] Fig.11 (2) shows the image obtained after cutting. Fig.11 As shown in (2), in the image 7a, only the portion in the shape of the plates (14a to 14c) is cut out, and objects (5a to 5c) other than the shape of the conveyor belt 12 and the plates (14a to 14c) are deleted.

[0179] Next, any one of the shapes of the cut plates (14a to 14c) is selected (step S42). Fig.12 (1) From Fig.11 (2) A diagram of the selected plate 14c in the image 7a is shown.

[0180] Here, using the approximate shape information in advance, only the area corresponding to the learning data set generation jig is further extracted from the area extracted by the detection unit 40h (step S43). Fig. 27 This is an illustration of how to hide an object. (1) shows the object, and (2) shows its approximate shape. Fig. 27 The object 5e shown in (1) is a plastic bottle, for example, Fig. 27 As shown in (2), a combined shape of a cylindrical shape 39a and a conical trapezoidal shape 39b can be used as a shape approximating the object 5e.

[0181] In addition, instead of using approximate shape information in advance, a model based on a previously learned Gaussian mixture distribution (GMM) can be used, such as extracting a plate area, to use a previously learned learning model to extract only the area corresponding to the jig generated from the learning data set.

[0182] Fig.12 (2) shows the image after extracting the plate area. Fig.12 (2) As shown in Fig.12 In the image shown in (1), the portions including the objects 5a and 5c are deleted, and only the plate 14c remains.

[0183] When there is an unprocessed plate, the unprocessed plate is selected (step S42) and the same process as plate 14c is performed. In this embodiment, the same process is performed on plates (14a, 14b). After all plates are processed (step S44), all extracted areas are covered with a noise image or a background image (step S45).

[0184] Fig.13 (1) is the image after the concealment process of Example 1. In this example, the area corresponding to the plate (14a to 14c) is covered with a noise image. Fig.13 (2) Covered with a background image. For convenience, the area corresponding to the plates (14a to 14c) is indicated by a dotted line, but is not actually displayed.

[0185] like Fig.13 As shown in (2), performing the overlay on the background image may be considered a more natural hiding process. However, Fig.13 The noise image shown in (1) is updated with a constant probability distribution, and a different noise image is generated each time, so it has the advantage of being difficult to learn through machine learning. Therefore, it is suitable for solving the problem of learning a board or AR marker.

[0186] Fig.14 The image image after the fixture area is hidden in the learning dataset is shown. Fig.14 As shown, the conveyor belt 12, the objects (5a to 5c), the hidden parts (15a to 15c), and the bounding boxes (6a to 6c) are displayed in the image 7a.

[0187] Although the bounding boxes (6a to 6c) are set on the objects (5a to 5c), respectively, as described above, these processes are not actually performed on the images.

[0188] In addition, regarding Figure 8 The plates (14a to 14c) shown are subjected to a noise image concealment process and become concealed portions (15a to 15c).

[0189] Here, although images (7a to 7f) are illustrated, more images are actually captured. Thus, by inserting bounding boxes (6a to 6c) and accumulating images such as image 7a in which the plates (14a to 14c) are hidden, a learning data set is created.

[0190] (About machine learning)

[0191] After the learning data set is generated, when the learning data set is used for specific robot control, it is necessary to use in advance the deep learning of the learning data set and other indirect machine learning. Therefore, the computer 4 uses the learning data set stored in the database 8 to perform deep learning on the object to be estimated and obtain the learned model.

[0192] Fig.15 Figure 2 shows a flow chart of the learned model generation. Fig.15 As shown, first, a learning data set is input (step S21). Based on the input learning data set, deep learning is performed (step S22). In this embodiment, deep learning is performed using TensorFolow (registered trademark), an artificial intelligence software library developed by Google (registered trademark) and disclosed as an open source. The learned model obtained by deep learning is output (step S23).

[0193] (About object recognition and position and posture estimation)

[0194] Fig.16 The system structure diagram of the robot control system during object recognition and position and posture estimation is shown. Fig.16 As shown, during object recognition and position and posture estimation, the robot control system 1 is composed of a camera 2 , a computer 4 and a robot 13 .

[0195] The object 5 to be recognized is placed on the conveyor 12. The computer 4 is provided with a database 8, and stores images captured by the camera 2. The robot 13 is provided with a mechanical arm 13a, and is configured to be able to grasp the object with the mechanical arm 13a.

[0196] Fig.17 The flowchart of object recognition and position and posture estimation is shown. Fig.17As shown, the conveyor belt 12 is photographed by the camera 2, and a photographic image is input (step S301). Through image recognition, the object is detected and the position in the image coordinate system is detected (step S302). Here, the object information (name, etc.) is obtained. In addition, a recognition bounding box is set (step S303). Select any one of the set recognition bounding boxes (step S304). The inside of the recognition bounding box is intercepted from the photographic image (step S305). Then, the posture of the object is estimated using the classifier of the learned model obtained by machine learning such as deep learning (step S306). Next, the width and height of the recognition bounding box are obtained (step S307), and the center of gravity position of the object in the three-dimensional coordinates of the actual space is estimated by the regressor of the learned model (step S308). It is determined whether there is an unselected recognition bounding box in the recognition bounding box (step S309). If there is an unselected recognition bounding box, one is selected at random. If all recognition bounding boxes have been selected, the names, postures and center of gravity positions in the three-dimensional coordinates of the actual space of all detected objects are output (step S310).

[0197] Fig.18 An example of an image used for object recognition and position and pose estimation is shown. Fig.18 As shown, the conveyor belt 12, the objects (5a to 5c) and the identification boundary boxes (16a to 16c) are shown on the image 70. The image 70 is obtained by photographing the object (5a to 5c) arranged on the conveyor belt 12 and flowing in the direction indicated by the arrow 19d by the camera 2.

[0198] Before using the learned model, deep learning using the learning dataset has been able to recognize objects and estimate their position and posture.

[0199] exist Fig.18 In the display, only the identification bounding boxes (16a-16c) are displayed to frame the objects (5a-5c), but the names, postures, and information related to the center of gravity of the objects (5a-5c) have been estimated in the computer 4, and by pre-planning the grasping motion of the robot 13, the robot 13 classifies the objects (5a-5c) based on the information estimated by the computer 4. Specifically, for example, like assorted gifts, it is possible to arrange the products one by one from a predetermined box containing a plurality of types of products, or to remove defective products from a large number of products flowing in a flow channel.

[0200] [Example 2]

[0201] Fig.19 FIG. 4 shows an image schematic diagram for creating a learning data set for the second embodiment. Fig.19As shown, the conveyor belt 12, objects (5a to 5c), and bounding boxes (6d to 6f) are displayed in the image 7a.

[0202] In this embodiment, a three-dimensional bounding box ( 6 d ˜ 6 f ) is set, which can be achieved by setting a plurality of AR markers 3 .

[0203] That is, in this embodiment, the learning data set is created using the boards (14a to 14c), but Figure 7 As shown, the center point P1 of the circular plate 14a is taken as the origin. Figure 5 As shown, the object 5 is arranged using the position adjustment guide 18a and the like so that the center of the bottom of the object becomes the origin.

[0204] The height, shape, width, and depth of the object 5 are stored in advance as object information data as attribute information of the object. Figure 1 In the database 8 shown, a three-dimensional bounding box (rectangular parallelepiped) can be drawn from the center point P1 of the circular plate 14a and the position (camera sight angle) of the camera 2. In fact, even without drawing, the camera coordinates of the eight corners of the rectangular parallelepiped can be known.

[0205] [Example 3]

[0206] Fig. 20 : is a plan view of the jig generated by the learning data set of Example 3, and the number of AR markers displayed is 8 for (1) and 3 for (2). Fig. 20 As shown in (1), eight AR marks (3a to 3h) are provided on the circular base portion 17 of the plate 140a. Fig. 20 As shown in (2), three AR marks (3a to 3c) are provided on the circular base portion 17 of the plate 140b.

[0207] Here, unlike the plates (14a to 14c) of Example 1, 12 AR markers (3a to 31) may not be necessarily provided. Considering the type, shape, manufacturing cost, etc. of the object 5, a structure with fewer AR markers may be provided. However, when capturing an image, it is preferred to capture two or more AR markers. Because, as described above, by capturing two or more AR markers, the recognition accuracy of the object can be improved, and the setting of a three-dimensional bounding box can also be realized.

[0208] [Example 4]

[0209] Fig.21 1 is a plan view of the learning data set generation jig of Example 4, where (1) shows a rectangular plate and (2) shows a square plate. Fig.21 As shown in (1), the plate 140c has 12 AR marks (3a to 31) on the rectangular base portion 170. Fig.21 As shown in (2), the plate 140d has 12 AR markers (3a to 31) on a square base portion 171. The objects 5 can be arranged in the areas (180, 181), respectively.

[0210] like Fig.21 As shown in (1) and (2), the shape of the base portion is not necessarily circular like the plates (14a to 14c) in Example 1, and may be rectangular or other shapes in consideration of the type and shape of the object 5. By designing the shape of the base portion or the type and number of AR markers according to the type and shape of the object 5, a higher quality learning data set can be created.

[0211] In addition, if Fig.21 As shown in (1), the AR markers (3c, 3e, 3i, 3k) are arranged obliquely relative to the rectangular shape of the plate 140c. Such an arrangement is not necessary as long as the AR markers surround the area and the corners of all AR markers are known. Therefore, for example, Fig.21 (2) Arrangement AR markers (3a to 31) as shown.

[0212] [Example 5]

[0213] Fig. 22 1 is an explanatory diagram regarding the object and the shooting direction. On the plate 140a shown in the third embodiment, (1) is a left side view of the object 5a, and (2) is a left side view of the object 5d. Fig. 22 As shown in (1), if the object placed on the plate 140a is a cylindrical object like the object 5a, even if Figure 3 The camera 2 shown in the figure is directed obliquely upward as shown by arrow 19e, and can also shoot any one of the upper surface and the side surface of the object 5a. However, if the object arranged on the plate 140a is an inverse cone trapezoidal shape like the object 5d, then Figure 3 When the imaging direction of the camera 2 is obliquely upward as shown in the arrow 19e, it may be difficult to image the side surface of the object 5a.

[0214] For example, in the case of an inverted cone-shaped object such as the object 5d, it is useful to photograph the object 5d from the side as shown by the arrow 19f, but if photographing from the side, there is a problem that it is difficult to photograph the AR markers (3a to 3h) set on the plate 140a.

[0215] Therefore, instead of using a flat plate, a plate processed into an L-shape may be used. Fig.23 1 is the appearance diagram of the learning data set generation fixture of Example 5, (1) is the front view, and (2) is the left view. Fig.23As shown in (1), the L-shaped plate 22 is composed of a bottom 22a, a side portion 22b and an AR mark (3a to 3r). The bottom 22a and the side portion 22b are integrally molded with resin. The bottom 22a is provided with an AR mark (3a to 3h), and the side portion 22b is provided with an AR mark (3i to 3r). The configuration of the AR mark (3a to 3h) on the bottom 22a is the same as the AR mark (3a to 3h) of the plate 140a.

[0216] like Fig.23 As shown in (1) and (2), when the object 5d is photographed from the direction of arrow 19f, the object 5d can be identified by detecting the AR markers (3i to 3r) provided on the side surface portion 22b. In addition, when the object 5d is photographed from the direction of arrow 19e, the object 5d can be identified by detecting any one of the AR markers (3a to 3h) provided on the bottom 22a or the AR markers (3i to 3r) provided on the side surface portion 22b.

[0217] In this way, the learning data set generation jig can be flexibly designed according to the shape of the object 5.

[0218] [Example 6]

[0219] Reference Fig.24 The conveying device and the rotational movement mechanism are described. Fig.24 The conveyor 120 is shown to horizontally move the conveyor belt 120a. The conveyor 120 is a conveying device that horizontally moves the objects arranged on the conveyor belt 120a.

[0220] A plate 14 serving as a learning data set generation jig is placed on the conveyor belt 120a, and the conveyor belt 120a is moved horizontally (in the direction indicated by arrow 19g) with the object 5 configured as a guide in an area (not shown) set on the base portion 17, so that a multi-viewpoint image group of the entire object 5 including the AR marker 3 can be effectively obtained.

[0221] Furthermore, if Fig.24 As shown, a plate 14 of a learning data set generation jig is mounted on a device that combines a rotating mechanism 120b and a direct-acting mechanism 120c, and the device is placed on a conveyor belt 120a, so that the conveyor belt 120a moves horizontally, and a multi-viewpoint image group can be obtained more efficiently. That is, the conveyor belt 120a moves in the direction indicated by the arrow 19g, while the rotating mechanism 120b rotates horizontally in the direction indicated by the arrow 19h, and the direct-acting mechanism 120c moves horizontally in the direction indicated by the arrow 19i, so that a multi-viewpoint image of the entire object including the mark is effectively obtained. As a result, as long as the object 5 and the plate 14 pass through the conveyor belt 120a once, they can be photographed from all angles, and a learning data set can be effectively created in a short time.

[0222] In addition, the moving direction shown in the arrow 19g or the rotation direction shown in the arrow 19h can also be the reverse direction, or can be the positive direction and the reverse direction alternately working. In addition, the moving direction shown in the arrow 19i is a reciprocating movement, or can only move in one direction.

[0223] [Example 7]

[0224] Will refer to Figure 4 A learning dataset creation program for causing a computer to execute a portion of the processing of a learning dataset creation method is described. The learning dataset creation program includes: 1) a step of setting a bounding box of an object for a multi-view image group; 2) associating the pose information and center of gravity position information, object information, and bounding box-related information of the object estimated from the captured image with the captured image, and causing the computer to execute the step of generating a learning dataset.

[0225] In step 1) above, the AR marker contained in the acquired image is detected by image analysis (step S102), the object information associated with the AR marker is acquired (step S103), and the posture of the object is estimated based on the posture information of the AR marker (step S104). The center of gravity position of the object is calculated (step S105), and a bounding box is set to frame the object (step S106).

[0226] In the above step 2), the object information, posture, center of gravity position and the set bounding box information (position, width, height in the image) are associated with the image (step S107). When there are unprocessed AR markers in the image, the unprocessed AR markers are processed in the same way. When all AR markers in the image are processed (step S108), an image associated with the object information, posture, center of gravity position information and bounding box information of the object is output (step S109), and all captured images are processed (step S110).

[0227] After processing of all images is completed, the plate containing the mark is hidden from the captured image (step S111) to create a learning dataset.

[0228] Regarding the step of hiding the plate including the mark from the photographic image after the step 2) above (step S111), refer to Fig.10 Provide explanation.

[0229] First, the area of ​​the board is detected from the acquired image (step S41). Next, any one board is selected from the shapes of the cut boards (step S42). The area of ​​the board is extracted using approximate shape information in advance (step S43). When there is an unprocessed board, the unprocessed board is selected (step S42) and the same processing is performed. After the processing of all boards is completed (step S44), all the extracted areas are covered with a noise image or a background image (step S45).

[0230] In addition, the object recognition and position and posture estimation program for causing a computer to execute part of the processing of the object recognition and position and posture estimation method will be referred to as Fig.17 The object recognition and position and posture estimation program causes the computer to perform the following steps: a) detecting the name of the object from a newly captured photographic image by image recognition and generating a bounding box for recognition; b) generating a partial image of a cutout bounding box for recognition and estimating the posture of the detected object using a learned model based on the generated partial image; c) obtaining the center of gravity position, width and height of the bounding box for recognition and estimating the center of gravity position of the detected object using a learned model.

[0231] In the above step a), the object is detected by image recognition, the position in the image coordinate system is detected (step S302), the object information (name, etc.) is obtained, and a bounding box for recognition is set (step S303).

[0232] In the above step b), any one of the set recognition bounding boxes is selected (step S304), the inside of the recognition bounding box is captured from the photographic image (step S305), and the posture of the object is estimated by a classifier of a learned model obtained by machine learning such as deep learning (step S306).

[0233] In the above step c), the width and height of the bounding box for recognition are obtained (step S307), and the center of gravity position of the object in the three-dimensional coordinates of the real space is estimated by the regressor of the learned model (step S308).

[0234] [Example 8]

[0235] In this embodiment, the learning dataset generation time, object recognition accuracy, and position and posture estimation accuracy are compared in the case of a single visual marker (hereinafter referred to as a single marker), the case of multiple visual markers (hereinafter referred to as multiple markers), and the manual case.

[0236] (Evaluation results of learning dataset generation time)

[0237] Table 2 below shows the results of measuring the learning data set generation time in the case of single label, multiple labels, and manual operation. 500 photographic images for comparison were taken for learning in the case of single label, multiple labels, and manual operation.

[0238] Fig.25 FIG. 2 is an explanatory diagram of the photographing method in the comparative experiment related to Example 8. Fig.25 As shown, in order to prevent data deviation, 38 points of 3 vertical × 8 horizontal were set in the image, and each point was rotated in units of 45 degrees and photographed. Manual annotation was performed by two people. Here, annotation in this specification refers not only to the setting of the boundary box of the photographic image, but also includes the association of object information.

[0239]

Table 2

[0240]

[0241] As shown in Table 2 above, the photography time for a single marker was 75 minutes, the time required for annotation was 32.2 minutes, and the total time was 112.2 minutes. The photography time for multiple markers was 436 minutes, the time required for annotation was 0 minutes, and the total time was 436 minutes. In addition, the photography time for manual work was 80 minutes, the time required for annotation was 886 minutes, and the total time was 966 minutes.

[0242] As mentioned above, the generation of a learning dataset with a single label has successfully improved the efficiency by 88.4% compared to the generation of a learning dataset manually. In addition, the efficiency of generating a learning dataset with multiple labels has increased by 54.9% compared to the generation of a learning dataset manually.

[0243] (Comparative experimental results on object recognition)

[0244] Tables 3 to 5 below show the results of object recognition accuracy measurements for single markers, multiple markers, and manual work. As experimental conditions, the "F value", "Precision", and "Recall" were measured for bottle A, bottle B, bag A, bag B, can A, and can B. The bottles here are plastic bottles.

[0245] Here, "Recall" is the reproduction rate, which represents the ratio of the actual positive value to the predicted positive value. It is a numerical representation of how much of what must be found is found. The minimum value is 0%, 100% or the closer to 100%, the better. In addition, the "Precision" accuracy is an indicator of the accuracy of the determined thing, and is calculated based on TP / (TP+FP) (TP: true positive class, FP: false positive class). The minimum value of accuracy is also 0%, and the closer to 100%, the better. In addition, the "F value" is one of the evaluation scales for prediction results in machine learning, and represents the unified average value of accuracy and reproduction rate. Specifically, the F value is calculated by 2×B×A / (A+B), where the value of the reproduction rate is "A" and the value of the accuracy is "B". The minimum value of the F value is 0%, and the closer to 100%, the better.

[0246]

Table 3

[0247] F value (%) Accuracy (%) Reproduction rate (%) Bottle A 100 100 100 Bottle B 98.5 99 98 Bag A 78.9 83 75.3 Bag B 83.1 84.4 81.8 Jar A 82.1 91 74.7 Jar B 81 94.4 70.8 average 87.3 92 83.4

[0248]

Table 4

[0249] F value (%) Accuracy (%) Reproduction rate (%) Bottle A 98 99 97 Bottle B 93.8 96.8 91 Bag A 93.6 98.9 88.9 Bag B 93.2 96.7 90 Jar A 97.4 100 95 Jar B 98.5 99 98 average 95.8 98.4 93.3

[0250]

Table 5

[0251] F value (%) Accuracy (%) Reproduction rate (%) Bottle A 99 98 100 Bottle B 98.5 98 99 Bag A 71.9 88.2 60.6 Bag B 79.8 70.7 91.6 Jar A 81.8 78.6 85.3 Jar B 76.8 92.7 65.6 average 84.6 87.7 83.7

[0252] As shown in Tables 3 to 5 above, the average F value of a single marker is 87.3%, the average F value of multiple markers is 95.8%, and the average F value when manual is used is 84.6%. It can be confirmed that the recognition performance of both single markers and multiple markers is higher than that of manual work.

[0253] In addition, if a single marker and multiple markers are compared, as shown in Tables 3 and 4, it can be confirmed that the F value, accuracy, and reproducibility show that the recognition accuracy of multiple markers is higher in the four objects of bag A, bag B, can A, and can B.

[0254] On the other hand, a single marker has a higher recognition accuracy for plastic bottles. That is, the F value, accuracy, and reproduction rate of bottle A in the case of a single marker are all 100%, and the F value of bottle B in the case of a single marker is 98.5%, the accuracy is 99%, and the reproduction rate is 98%, which confirms that the recognition accuracy is higher than that of the case of multiple markers. However, even in the case of using multiple markers, the F value of bottle A is 98%, the accuracy is 99%, and the reproduction rate is 97%, and the F value of bottle B is 93.8%, the accuracy is 96.8%, and the reproduction rate is 91%, showing high values, which are almost the same as the case of a single marker.

[0255] Therefore, it can be considered that the method using multiple labels is useful as a learning dataset generation fixture for recognizing a wider variety of target objects with high accuracy.

[0256] (Comparative experimental results on position and attitude estimation)

[0257] Fig.26 The figure shows the distribution diagram of the comparative experimental results on position and attitude estimation. Fig.26 (1) is a distribution diagram showing the position estimation error in single marker, multiple markers and artificial situations. Fig.26 In the figure, bottle A, bottle B, bag A, bag B, can A, and can B are compared, and the values ​​of the distribution graph are rounded off to the second decimal place.

[0258] like Fig.26 As shown in (1), regarding the manual position estimation error, bottle A is 28.0 mm, bottle B is 27.0 mm, bag A is 23.6 mm, bag B is 29.4 mm, can A is 13.3 mm, and can B is 15.2 mm. It can be confirmed that they are less than any of the cases with single marking or multiple markings.

[0259] In this regard, when single-marked, bottle A is 28.3mm, bottle B is 28.4mm, bag A is 46.8%, bag B is 51.1%mm, can A is 27.7mm, and can B is 32.1%mm. When multiple-marked, bottle A is 33.2%mm, bottle B is 35.1mm, bag A is 30.7mm, bag B is 39.9mm, can A is 17.3mm, and can B is 17.7mm.

[0260] Therefore, regarding bottle A and bottle B, it can be seen that in the case of a single mark, the error is almost the same as in the case of manual. In addition, in the case of bottle A and bottle B with multiple marks, it is confirmed that the error is slightly larger than that of a single mark, but the difference is small, and in addition, regarding bag A, bag B, can A, and can B, it is confirmed that the error can be reduced compared to the case of a single mark.

[0261] Fig.26 (2) is a distribution diagram showing the accuracy of posture estimation in the single-marker, multiple-marker and manual cases. Similar to the position estimation, the comparison is made for bottle A, bottle B, bag A, bag B, can A and can B.

[0262] like Fig.26As shown in (2), the accuracy of manually estimated pose is 73.4% for bottle A, 37.4% for bottle B, 86.1% for bag A, 89.9% for bag B, 71.9% for can A, and 79.1% for can B, which are lower than the cases with single or multiple markers.

[0263] In addition, the accuracy of pose estimation in the case of a single marker is 24.0% for bottle A, 14.3% for bottle B, 20.5% for bag A, 14.8% for bag B, 15.5% for can A, and 16.2% for can B.

[0264] In this regard, the accuracy of pose estimation under multiple markers is 48.5% for bottle A, 19.8% for bottle B, 69.3% for bag A, 77.5% for bag B, 67.4% for can A, and 76.5% for can B. It can be confirmed that the accuracy of bottle A, bottle B, bag A, bag B, can A, and can B are all higher than the accuracy of a single marker.

[0265] (Summary of experimental results)

[0266] In summary, when a single marker is used, the efficiency of data collection time is improved by 88.4%, and it is confirmed that sufficient performance is obtained in object recognition. On the other hand, when multiple markers are used, the efficiency of data collection time can be improved by 54.9%, and sufficient performance can be obtained not only in object recognition but also in position and pose estimation.

[0267] [Example 9]

[0268] (About the effect of hiding processing in a single tag)

[0269] In this embodiment, the object recognition accuracy is compared when a single mark is hidden by covering the background image in automatic collection and when a single mark is hidden by covering the noise image in automatic collection. As for the target objects, bottle A, bottle B, bag A, bag B, can A, and can B are compared as in Example 8. Table 6 below shows the comparison results of the F value related to the object recognition accuracy when the background is hidden by manual hiding and when the noise is hidden.

[0270]

Table 6

[0271]

[0272] As shown in Table 6 above, the F value of bottle A when it is manually hidden is 99%, while the F value when it is hidden as the background is 97.1%, which is lower than the F value when it is manually processed. On the contrary, when the noise hiding process is performed, the F value is 100%, which is higher than the F value when it is manually processed.

[0273] Regarding bottle B, the F value when it was manually hidden was 98.5%, while it was 98% when it was background hidden, which was lower than the F value when it was manually hidden. When it was hidden as noise, the F value was 98.5%, and the same recognition rate was obtained as when it was manually hidden.

[0274] In contrast, for bag A, the F value when manually hidden was 71.9%, but the F value when hidden as background was 81.8%, and the F value when hidden as noise was 78.9%. In either case, the recognition rate was higher than that of manual operation.

[0275] Regarding bag B, the F value was 79.8% when it was manually hidden, but it was 85.9% when it was hidden as background, and 83.1% when it was hidden as noise. In either case, a higher recognition rate was obtained than when it was manually hidden.

[0276] Regarding jar A, the F value when it was manually hidden was 81.8%, while the F value when it was background hidden was 51.9%, which was lower than the F value when it was manually hidden. The F value when it was noise hidden was 82.1%, which was higher than the recognition rate when it was manually hidden.

[0277] Regarding jar B, the F value when it was manually hidden was 76.8%, while the F value when it was background hidden was 55.3%, which was lower than the F value when it was manually hidden. The F value when it was noise hidden was 81%, which was higher than the recognition rate when it was manually hidden.

[0278] In summary, when a single mark was hidden with the same color as the background, only bag A and bag B achieved a higher recognition rate than the manual process, but when a single mark was hidden as noise, the recognition rate was equal to or higher than the manual process for all of bottle A, bottle B, bag A, bag B, can A, and can B. Therefore, when a single mark was hidden as noise, it was found that the recognition rate of various objects was improved.

[0279] Industrial Applicability

[0280] The present invention is useful as a factory automation system for industrial robots. That is, an industrial robot that performs machine learning using a learning data set created by the learning data set creation method of the present invention, an industrial robot that uses the object recognition and position and posture estimation method of the present invention, or an industrial robot that is installed with the object recognition and position and posture estimation program of the present invention can be used in an automation system for factories such as sorting operations or processing operations in the food industry or logistics industry. In addition, it can also be used in anti-theft systems for buildings, etc.

[0281] Explanation of symbols in the figure

[0282] 1 Robot control system

[0283] 2 Camera

[0284] 3, 3a~3r AR markers

[0285] 4, 40, 41 Computer

[0286] 5, 5a~5e Object

[0287] 6a~6c Bounding box

[0288] 7a~7f, 70 images

[0289] 8, 80, 81 Database

[0290] 9 Model generation computer

[0291] 9a Classifier

[0292] 9b Regressor

[0293] 10 Learning Dataset Creation Device

[0294] 11 Object recognition and position and posture estimation unit

[0295] 12. Conveyor Belt

[0296] 13 Robot

[0297] 13a Robotic Arm

[0298] 14, 14a~14c, 140a~140d Plate

[0299] 15a~15c Hidden part

[0300] 16a~16f Bounding box for recognition

[0301] 17, 170, 171 Base

[0302] Area 18, 180, 181

[0303] 18a Position adjustment guide

[0304] 18b Direction adjustment guide

[0305] 19a~19i Arrow

[0306] 20, 21 Photography Unit

[0307] 22 L-shaped plate

[0308] 22a Bottom

[0309] 22b Side view

[0310] 30 Learning Dataset Generation Fixture

[0311] 33 Fiducial Marker (AR Marker)

[0312] 36 Pedestal part

[0313] 37 Pillar

[0314] 38 points

[0315] 39a Cylindrical shape

[0316] 39b Conic trapezoid

[0317] 40a AR Marker Recognition Unit

[0318] 40b Object recognition unit

[0319] 40c Bounding box setting unit

[0320] 40d Attitude Estimation Unit

[0321] 40e Center of gravity estimation unit

[0322] 40f Bounding Box Allocation Unit

[0323] 40g Learning Dataset Generates Hidden Units of the Fixture Region

[0324] 40h detection unit

[0325] 40i Extraction Unit

[0326] 40j Hidden Units

[0327] 41 Object Recognition Unit

[0328] 41b Bounding box setting unit for recognition

[0329] 41c Learned Model

[0330] 120 Conveyor

[0331] 120a Conveyor belt

[0332] 120b Rotating mechanism

[0333] 120c direct acting mechanism

[0334] C Corner

[0335] P Center Point

Claims

1. A method for creating a learning data set, characterized in that: Associating the object information of the object with the visual tag, A base portion is provided with an area serving as a guide for the configuration position of an object, and a jig is generated by a learning data set consisting of the visual markers fixed to the base portion, and an image group of the entire object including the visual markers is obtained in a state where the area is used as a guide for configuring the object, acquiring the object information associated with the visual marker from the acquired image set, Generating a reconstructed set of images by hiding the regions corresponding to the visual markers or the learning dataset generation fixture from the set of images, Based on the acquired object information, a bounding box of the object is set for the reconstructed image group, The pose information and position information of the object estimated from the photographic image, the object information, and the information related to the bounding box are associated with the photographic image to generate a learning data set for object recognition and position and pose estimation of the object.

2. The method for creating a learning data set according to claim 1, characterized in that: The plurality of visual markers are fixed to the base portion in a predetermined positional relationship with the object.

3. The method for creating a learning data set according to claim 1, characterized in that: The concealment process includes: a detection step of detecting, from the image group, an area corresponding to the learning data set generation fixture; and an extraction step of detecting an object region from the region containing the photographed object extracted in the detection step, and extracting only a region corresponding to the learning data set generation fixture; and The covering step is a step of covering the image group in the area extracted by the extraction step as a background image or a noise image.

4. The method for creating a learning data set according to claim 3, characterized in that: The extracting step detects the object region in advance using approximate shape information.

5. The method for creating a learning data set according to claim 3, characterized in that: The extraction step detects the object region using a previously learned model.

6. The method for creating a learning data set according to claim 1, characterized in that: The base portion is a circular, oval or rectangular plate.

7. The method for creating a learning data set according to claim 1, characterized in that: When the base portion is circular, the visual mark is arranged so as to surround the area, with the center direction of the area being a reference direction.

8. The method for creating a learning data set according to claim 6, characterized in that: The base portion is an L-shaped plate formed by bending and extending the plate.

9. The method for creating a learning data set according to claim 1, characterized in that: The bounding box is a three-dimensional bounding box with the area as a reference position.

10. The method for creating a learning data set according to claim 1, characterized in that: The visual marker is a two-dimensional graphic marker or a three-dimensional marker including an AR (Augmented Reality) marker.

11. The method for creating a learning data set according to claim 1, characterized in that: In the area, the identification ID of the learning data set generating jig is displayed.

12. The method for creating a learning data set according to claim 1, characterized in that: In the region, a direction adjustment guide for adjusting the direction of the object is provided.

13. The method for creating a learning data set according to claim 1, characterized in that: The position information is difference information between the position of the object arranged using the area as a guide and the position of a reference posture in a predefined reference coordinate system.

14. The method for creating a learning data set according to claim 1, characterized in that: The posture information is information calculated using a photographic image of the visual marker and a relative positional relationship between the visual marker and the object, and is differential information between the posture of the object configured with the area as a guide and a reference posture in a predefined reference coordinate system.

15. The method for creating a learning data set according to claim 1, characterized in that: The image group is obtained by at least one of the following methods: photographing the object while it is configured on the learning data set generation fixture and transported by a transport device, photographing the object while it is configured on the learning data set generation fixture and rotated by a rotating device, and photographing the object while it is configured on the learning data set generation fixture and moved by a moving device.

16. A method for object recognition and position and posture estimation, characterized in that: Using the learning data set described in claim 1 to perform machine learning to obtain a learned model, Based on the newly captured photographic image, the name of the object is detected through image recognition and a bounding box is generated for recognition. generating a partial image in which the recognition bounding box is cut out, and estimating the posture of the object detected based on the generated partial image using the learned model, The position, width and height of the recognition bounding box are obtained, and the position of the detected object in the three-dimensional coordinates of the real space is estimated using the learned model.

17. The object recognition and position and posture estimation method according to claim 16, characterized in that: The recognition bounding box is a two-dimensional bounding box or a three-dimensional bounding box.

18. The object recognition and position and posture estimation method according to claim 16, characterized in that: The learned model includes at least a classifier or a regressor for estimating the pose or position of the object.

19. The object recognition and position and posture estimation method according to claim 16, characterized in that: The photographic image is an image in which a plurality of objects are photographed. Create a bounding box for each detected object. The names of all detected objects, as well as their estimated poses and positions, are calculated as 3D coordinates in real space.

20. The object recognition and position and posture estimation method according to claim 16, characterized in that: The photographic image is an image captured in an environment that is the same as or similar to an environment in which the learning dataset is created.

21. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store computer instructions, and when the computer-readable storage medium is executed on a computer, the computer executes the learning data set creation method according to claim 1.

22. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store computer instructions, and when the computer-readable storage medium is executed on a computer, the computer executes the object recognition and position and posture estimation method according to claim 16.

23. A learning data set creation device, characterized in that: include: A learning data set generation jig including a base portion provided with a guide area serving as a configuration position of an object, and a visual marker fixed to the base portion and associated with object information of the object; A photographing unit that obtains a group of images of the entire object including the visual mark in a state where the object is arranged with the area as a guide; means for acquiring object information associated with the visual marker from the acquired set of images; means for generating a reconstructed set of images by hiding regions corresponding to the visual markers or the learning dataset generation fixture from the set of images; A unit for setting a bounding box of an object for the acquired image group; A unit that associates posture information and position information of an object estimated from a photographic image, the object information, and information related to the bounding box with the photographic image to generate a learning data set for object recognition and position and posture estimation of the object.

24. The learning data set creation device according to claim 23, characterized in that: The plurality of visual markers are fixed to the base portion in a predetermined positional relationship with the object.

Images