Learning database building system, control device used in said system, and learning database building method

Abstract

To enable building a learning database with which the accuracy of recognizing a target object can be improved.SOLUTION: The present invention comprises: a 3D camera 24 capable of capturing an image of a target object 40; an imaging robot 20 capable of changing the position and attitude of the 3D camera 24; and a control device 10 capable of acquiring information of the image of the target object 40 captured by the 3D camera 24. The control device 10 generates three-dimensional information of the target object 40 by controlling the imaging robot 20 and carrying out three-dimensional imaging to capture the image of the target object 40 from a plurality of directions by the 3D camera 24, automatically generates annotation information for the image of the target object 40 captured by the 3D camera 24 on the basis of three-dimensional information, and builds a learning database in which a plurality of pieces of data from each of the annotation information and the three-dimension physical information of the target object 40 relative to the 3D camera 24 is adopted as learning data.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a learning database construction system, a control device used in the system, and a learning database construction method. [Background technology]

[0002] Conventionally, when constructing a database of learning data (also called teacher data) used for AI machine learning in image recognition technology, it is desirable to recognize the target object (hereinafter referred to as the target object) with high accuracy. Furthermore, constructing this learning database requires a lot of time and effort.

[0003] Patent Documents 1 and 2 are known as disclosures of techniques for photographing a target object and building a learning database.

[0004] Patent Document 1 discloses that in the construction stage of a learning data set (corresponding to a learning database), object information of a target object is associated with an AR marker (two-dimensional pattern marker) that is a marker for detecting position and orientation. Patent Document 1 also discloses that the AR marker is associated with object information such as the object name of the target object in a database, and a computer of a learning data set creation device operates as an AR marker recognition means to recognize the AR marker and acquire the object information of the target object.

[0005] Patent Document 2 also discloses a dataset storage unit that acquires from the simulator and stores multiple sets of learning datasets comprising three-dimensional coordinate data of the robot hand on the simulator when the robot hand gripping a workpiece undergoes a gripping operation and two-dimensional image data of the workpiece captured from a specified angle of view by a two-dimensional imaging device ID on the simulator, and constructs a learning model that infers the three-dimensional coordinates of the robot hand in the real world from the two-dimensional image of the workpiece captured by the two-dimensional imaging device ID from the same angle of view as the specified angle of view. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Patent No. 6474179 [Patent Document 2] JP 2020-82322 A Summary of the Invention [Problem to be solved by the invention]

[0007] However, since the technique of Patent Document 1 uses a two-dimensional camera, there is a risk that the recognition accuracy of the target object will be significantly reduced if the color of the background and the color of the target object are similar.In addition, since an AR marker is used, if the posture of the target object is such that the AR marker cannot be recognized, the target object information cannot be acquired. Therefore, the posture of the target object from which object information can be acquired is limited, making it difficult to build a highly accurate learning database.

[0008] In addition, since the technique in Patent Document 2 uses simulated images, it is unable to fully express the characteristics of real target objects, and there is a limit to how much the recognition accuracy of the target objects can be improved, making it difficult to build a highly accurate learning database.

[0009] The present invention has been made in consideration of the above points, and its purpose is to provide a learning database construction system capable of constructing a learning database that can improve the recognition accuracy of a target object (for example, the recognition accuracy of the three-dimensional posture of a target object or the recognition accuracy of a specific three-dimensional position on the target object), as well as a control device and a learning database construction method for use in the system. [Means for solving the problem]

[0010] The solution of the present invention for achieving the above-mentioned object comprises a 3D camera capable of photographing a target object, a movable body capable of changing the position and attitude of the 3D camera, and a control device capable of acquiring information on images of the target object photographed by the 3D camera, wherein the control device is characterized by comprising a 3D information generation unit that generates 3D information of the target object by controlling the movable body to perform 3D photographing of the target object from multiple directions using the 3D camera, an annotation information generation unit that automatically generates annotation information based on the 3D information for an image of the target object photographed by the 3D camera or a camera other than the 3D camera, and a learning database construction unit that constructs a learning database using multiple data of the annotation information and relative 3D physical information of the target object with respect to the camera used when automatically generating the annotation information as learning data.

[0011] Due to this specification, since the learning database is constructed with a plurality of data of annotation information and relative three-dimensional physical information of the target object with respect to the camera used when the annotation information is automatically generated as learning data, it is possible to construct a learning database that can improve the recognition accuracy of the target object (for example, the recognition accuracy of the three-dimensional posture of the target object and the recognition accuracy of a specific position on the target object in three dimensions). For example, when performing a specific process on the target object, the three-dimensional physical information can be obtained by simply acquiring a two-dimensional image of the target object, and the recognition accuracy of the target object can be improved based on this three-dimensional physical information, making it possible to perform the specific process on the target object with high accuracy.

[0012] The three-dimensional physical information is relative three-dimensional posture information of the target object with respect to the camera used when the annotation information is automatically generated.

[0013] This makes it possible to recognize the relative 3D posture of a target object simply by acquiring a 2D image of the target object. For example, when a target object is to be grasped by a robot, it becomes possible to optimize the grasping position.

[0014] The three-dimensional physical information may be information on a specific three-dimensional position on the target object relative to the camera used when automatically generating the annotation information.

[0015] This makes it possible to recognize a specific three-dimensional position on a target object simply by acquiring a two-dimensional image of the target object. For example, when processing a specific position on a target object using a robot, it becomes possible to identify the processing position with high accuracy.

[0016] A control device used in the learning database construction system is also within the scope of the technical idea of ​​the present invention. That is, a control device capable of acquiring information on an image of a target object captured by a three-dimensional camera supported on a movable body and capable of changing the position and attitude by operating the movable body, the control device including: a three-dimensional information generation unit that generates three-dimensional information of the target object by controlling the movable body to perform three-dimensional photography of the target object from multiple directions using the three-dimensional camera; an annotation information generation unit that automatically generates annotation information based on the three-dimensional information for an image of the target object captured by the three-dimensional camera or a camera other than the three-dimensional camera; and a learning database construction unit that constructs a learning database using multiple data of the annotation information and relative three-dimensional physical information of the target object with respect to the camera used when automatically generating the annotation information as learning data.

[0017] The learning database construction method implemented in the learning database construction system is also within the scope of the technical idea of ​​the present invention. That is, the method includes a 3D information generating step of generating 3D information of a target object by controlling a movable body supporting a 3D camera to perform 3D photography of the target object from multiple directions using the 3D camera, an annotation information generating step of automatically generating annotation information based on the 3D information for an image of the target object captured by the 3D camera or a camera other than the 3D camera, and a learning database construction step of constructing a learning database using multiple data of the annotation information and relative 3D physical information of the target object with respect to the camera used when automatically generating the annotation information as learning data.

[0018] Even with these specific requirements, as mentioned above, it is possible to construct a learning database that can improve the recognition accuracy of the target object (for example, the recognition accuracy of the three-dimensional posture of the target object or the recognition accuracy of a specific three-dimensional position on the target object). Effect of the Invention

[0019] In the present invention, a movable body is controlled to perform 3D photography in which the target object is photographed from multiple directions by a 3D camera, thereby generating 3D information of the target object. Furthermore, annotation information is automatically generated based on the 3D information for an image of the target object photographed by the 3D camera or a camera other than the 3D camera. A learning database is then constructed using multiple pieces of data, including the annotation information and the relative 3D physical information of the target object with respect to the camera used when automatically generating the annotation information, as learning data. This makes it possible to construct a learning database that can improve the recognition accuracy of the target object (for example, the recognition accuracy of the 3D posture of the target object or the recognition accuracy of a specific position on the target object in 3D). [Brief description of the drawings]

[0020] [Figure 1] 1 is a schematic diagram showing the overall configuration of a learning database construction system according to a first embodiment. [Diagram 2] 1 is a block diagram showing a configuration of a control device according to a first embodiment. [Diagram 3] 1 is a block diagram showing a configuration of an imaging robot according to a first embodiment. [Figure 4] FIG. 4 is a flowchart showing the procedure of a preparation process according to the first embodiment. [Diagram 5] FIG. 3 is a diagram showing an example of point cloud data of a target object according to the first embodiment. [Figure 6] FIG. 2 is a flowchart showing the procedure of deep learning processing according to the first embodiment. [Figure 7] FIG. 2 is an image diagram showing a plurality of examples of bounding boxes assigned to an image of a target object according to the first embodiment. [Figure 8] 4A to 4C are diagrams for explaining an operation of acquiring a three-dimensional posture of a target object with respect to a 3D camera according to the first embodiment. [Figure 9] FIG. 23 is a flowchart showing the procedure of a preparation process according to the seventh embodiment. [Figure 10] 13A to 13C are diagrams for explaining an inference operation of a specific position of a target object according to the ninth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0021] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Each embodiment described below will be described as an application of the present invention to construct a learning database (teacher database) used when a robot (movable body) performs a predetermined process on a target object (for example, gripping the target object or processing a specific position on the target object). In the following description, the same parts and components are given the same reference numerals. Since their names and functions are also the same, detailed description thereof will not be repeated.

[0022] <First embodiment> In this embodiment, a case will be described in which the present invention is applied to construct a learning database used for estimating the three-dimensional posture of a target object relative to a 3D camera (three-dimensional camera).

[0023] -Overall configuration and operation of the learning database construction system- First, the overall configuration of a learning database construction system 1 according to this embodiment will be described with reference to Fig. 1. The learning database construction system 1 includes, as main components, a control device 10, an image capturing robot 20, and a mounting device 30.

[0024] The control device 10 is realized by a server, a computer, etc., and executes various processes such as acquiring information on images captured by the 3D camera 24 mounted on the photographing robot 20 by transmitting and receiving various data to and from the photographing robot 20 and the mounting device 30 via a wired LAN or wireless LAN. The processes executed by the control device 10 will be described in detail later.

[0025] The photographic robot 20 is composed of an articulated robot equipped with an arm section 27 and a working unit 28 attached to the tip of the arm section 27, and performs various tasks such as moving the arm section 27, working unit 28, etc. to various positions or tilting them into various postures based on command information transmitted from the control device 10 or according to its own judgment processing (processing by the CPU 21 described later).

[0026] The mounting device 30 has a mounting table 31 on which a target object 40 to be subjected to deep learning or annotation is mounted. The mounting table 31 can be rotated and tilted.

[0027] The control device 10 operates the arm unit 27 of the photographing robot 20 to change the position and attitude of the 3D camera 24, thereby photographing the target object 40 placed on the mounting table 31 from various angles and automatically annotating the target object 40, automatically adding a bounding box to the photographed image of the target object 40, or performing segmentation of the target object 40 from the photographed image, and even acquiring the three-dimensional attitude of the target object 40 relative to the 3D camera 24 as learning data (details will be described later).

[0028] In this way, the learning database construction system 1 according to the present embodiment enables deep learning with reduced effort on the part of the operator. The configuration and operation of each part of the learning database construction system 1 will be described in detail below.

[0029] -Control device configuration- One aspect of the configuration of the control device 10, which is a component of the learning database construction system 1 according to this embodiment, will be described with reference to Fig. 2. The control device 10 includes, as main components, a CPU (Central Processing Unit) 11, a memory 12, an operation unit 13, and a communication interface 14.

[0030] The CPU 11 controls each part of the control device 10 by executing a program stored in the memory 12. For example, the CPU 11 executes a program stored in the memory 12 and performs various processes described later by referring to various data.

[0031] The CPU 11 includes a three-dimensional information generating unit 11a, an annotation information generating unit 11b, and a learning database constructing unit 11c as functional units realized by the programs. The functions of each of these units will be outlined below.

[0032] The three-dimensional information generator 11a has a function of generating three-dimensional information of the target object 40 by controlling the photographing robot 20 (particularly the arm unit 27 of the photographing robot 20) to perform three-dimensional photographing of the target object 40 from multiple directions with the 3D camera 24. Specifically, the three-dimensional information generator 11a has a function of acquiring three-dimensional information (three-dimensional image information) of only the target object 40, and has a function of acquiring three-dimensional photographs of the entire 360-degree circumference of the target object 40 by moving and rotating the arm unit 27 of the photographing robot 20, and creating three-dimensional point cloud data of the target object 40 from the acquired RGB+depth map for 360 degrees.

[0033] The annotation information generating unit 11b has a function of automatically generating annotation information based on the three-dimensional information for an image of the target object 40 captured by the 3D camera 24. Specifically, the annotation information generating unit 11b has a function of automatically generating a bounding box that approximately circumscribes the target object 40 (or a contour line of the target object 40 by segmentation) from three-dimensional point cloud data of the target object 40 and information on the shooting direction when the target object 40 is captured by the 3D camera 24 (the shooting direction is determined by the attitude of the 3D camera 24 according to the attitude of the arm unit 27).

[0034] The learning database construction unit 11c constructs a learning database using multiple pieces of data, each of which is the annotation information and information on the three-dimensional posture of the target object 40 relative to the 3D camera 24 (three-dimensional physical information in the present invention), as learning data. Specifically, the learning database construction unit 11c has a function of photographing the target object 40 from multiple directions, thereby learning (acquiring learning data) information on the bounding box (or the contour line of the target object 40 by segmentation) assigned to the target object 40 corresponding to each direction and information on the three-dimensional posture of the target object 40, and constructing a learning database using these learning data.

[0035] The memory 12 is realized by various RAMs, various ROMs, etc., and may be included in the control device 10, may be detachable from various interfaces of the control device 10, or may be a recording medium of another device accessible from the control device 10. The memory 12 stores programs executed by the CPU 11, data generated by execution of programs by the CPU 11, input data, other databases used in this embodiment, etc.

[0036] The operation unit 13 receives commands from a user, an administrator, or the like, and inputs the commands to the CPU 11 .

[0037] The communication interface 14 transmits data from the CPU 11 to the photographing robot 20 via wired LAN or wireless LAN, and conversely, receives data from the photographing robot 20 (including three-dimensional image information from the 3D camera 24) and passes it on to the CPU 11.

[0038] -Configuration of the shooting robot- Next, one embodiment of the configuration of the photography robot 20, which is a component of the learning database construction system 1, will be described with reference to Fig. 3. The photography robot 20 includes, as main components, a CPU 21, a memory 22, an operation unit 23, a 3D camera 24, a light 25, a communication interface 26, an arm unit 27, and a working unit 28.

[0039] The CPU 21 controls each part of the photographing robot 20 by executing a program stored in the memory 22. For example, the CPU 21 controls the posture of the arm unit 27 by adjusting the rotation angle of a motor provided in each joint of the photographing robot 20.

[0040] The memory 22 is realized by various RAMs, various ROMs, etc., and stores various programs, data generated by execution of programs by the CPU 21, operation instructions given from the control device 10, data input via the operation unit 23, etc.

[0041] The operation unit 23 is composed of buttons, switches, etc., and receives various commands from a user and inputs the commands to the CPU 21.

[0042] The 3D camera 24 is realized by an RGB-D camera or the like. The 3D camera 24 can obtain the distance to each part of the captured image by using, for example, two cameras. The 3D camera 24 performs three-dimensional shooting or normal two-dimensional shooting based on an instruction from the CPU 21.

[0043] The light 25 irradiates light in front of the 3D camera 24 in accordance with an instruction from the CPU 21 .

[0044] The communication interface 26 transmits and receives data to and from other devices such as the control device 10 via the Internet, a carrier network, a router, etc. For example, the communication interface 26 receives an operation command from the control device 10 and passes it to the CPU 21.

[0045] The arm unit 27 controls the position and attitude of the 3D camera 24 attached to the arm unit 27 and controls the position and attitude of the working unit 28 according to instructions from the CPU 21. The 3D camera 24 is fixed to the arm unit 27. The attitude of the arm unit 27 can be identified by monitoring the rotation angle position of each joint of the shooting robot 20. Therefore, by recognizing the attitude of the arm unit 27, the position and attitude of the 3D camera 24 can be grasped. For example, by recognizing the attitude of the arm unit 27, it is possible to grasp the coordinate position (hereinafter, referred to as the X-axis coordinate position) in a direction along the optical axis of the lens of the 3D camera 24 (hereinafter, referred to as the X-axis direction: see FIG. 8), the coordinate position (hereinafter, referred to as the Y-axis coordinate position) in a direction perpendicular to the X-axis direction and extending horizontally (hereinafter, referred to as the Y-axis direction), and the coordinate position (hereinafter, referred to as the Z-axis coordinate position) in a direction extending in a direction perpendicular to each of the X-axis direction and the Y-axis direction.

[0046] The working unit 28 corresponds to a human hand attached to the tip of the arm unit 27, and performs grasping operations etc., and performs various operations to grasp the target object 40 or change the position or orientation of the target object 40 according to instructions from the CPU 21.

[0047] -Information processing for control devices- Next, the information processing of the control device 10 will be described in detail with reference to Fig. 4. As a preparatory process for deep learning, the CPU 11 of the control device 10 executes the process shown in Fig. 4 according to the program in the memory 12. The process shown in Fig. 4 is a process performed by the three-dimensional information generating unit 11a.

[0048] First, the CPU 11 accepts 3D CAD data such as 3D shape and position information of the imaging environment (table, stage, robot itself, etc.) and registers it in the memory 12 (environment information registration) (step ST1). The imaging environment here refers to objects other than the target object 40 in the image captured by the 3D camera 24 (objects to be subtracted from the captured image in future image processing), such as the mounting device 30, the wall surface in the background, and the working unit 28 when the working unit 28 is within the field of view of the 3D camera 24.

[0049] The CPU 11 causes the 3D camera 24 attached to the arm unit 27 of the photographing robot 20 to photograph the target object 40, and acquires an RGB+depth map (step ST2). As described above, since the 3D camera 24 is fixed to the arm unit 27 of the photographing robot 20, the CPU 11 can calculate the posture information of the 3D camera 24 (the aforementioned X-axis coordinate position, Y-axis coordinate position, Z-axis coordinate position, etc.) from the posture information of the photographing robot 20 and the arm unit 27.

[0050] The CPU 11 obtains 3D information (information on the 3D image) of only the target object 40 by subtracting the data of the surrounding objects (data on the shooting environment) registered in step ST1 from the RGB+depth map captured in step ST2 (object data extraction in step ST3).

[0051] The CPU 11 moves or rotates the arm 27 of the photography robot 20, or rotates or tilts the mounting table 31 (changing the photography direction in step ST4), and photographs from a different angle (3D photography in step ST2). That is, the CPU 11 repeats the processes in steps ST2 to ST5 until 3D photography of the entire 360 ​​degrees around the target object 40 is completed (until YES is determined in step ST5).

[0052] The CPU 11 creates three-dimensional point cloud data of the target object 40 from the RGB+depth map for 360 degrees of the target object 40 (3D point cloud data creation in step ST6). Specifically, as shown in FIG 5, three-dimensional solid point cloud data is created from the three-dimensional captured image of the target object 40.

[0053] Furthermore, based on the point cloud created in step ST6, it is preferable that the CPU 11 moves the arm unit 27 to take additional photographs with the 3D camera 24 so that areas where the point cloud is lacking or there is noise can be photographed in more detail, and then re-synthesizes the three-dimensional point cloud (data complementation processing in step ST7).

[0054] Since three-dimensional point cloud data is created in this manner, this created three-dimensional point cloud data corresponds to the three-dimensional information referred to in this invention, and the process of creating the three-dimensional point cloud data corresponds to the three-dimensional information generation process referred to in this invention.

[0055] After the preparation process shown in Fig. 4 described above, the CPU 11 of the control device 10 subsequently executes the process shown in Fig. 6 as deep learning processing in accordance with the program in the memory 12. The process shown in Fig. 6 is processing performed by the annotation information generating unit 11b and the learning database constructing unit 11c.

[0056] The CPU 11 causes the 3D camera 24 attached to the arm unit 27 of the photographing robot 20 to photograph the target object 40 in two dimensions (step ST11).

[0057] The CPU 11 calculates the appearance of the target object 40 and automatically creates annotation information based on the position information and posture information of the shooting robot 20 and the arm unit 27, the position information and posture information of the target object 40, and the three-dimensional point cloud data of the target object 40 (step ST12).

[0058] Specifically, in this embodiment, as shown in, for example, Figs. 7(a) to (c), the CPU 11 automatically generates a bounding box that approximately circumscribes the target object 40 from the three-dimensional point cloud data of the target object 40 and the shooting direction as annotation information. Fig. 7(a) shows a bounding box added when the target object 40 is shot from the front, and Fig. 7(b) shows a bounding box added when the target object 40 is shot from the oblique front. Figs. 7(a) and 7(b) show a case where a bounding box surrounded by horizontal and vertical lines is added. This bounding box specifies the coordinate positions of four points that are the intersections of the horizontal and vertical lines, and in this embodiment, the coordinate positions of these four points become learning data.

[0059] 7(c) shows a rotated bounding box added when the target object 40 is photographed from diagonally behind. The rotated bounding box is obtained by removing the restriction that the straight lines constituting the bounding box must be horizontal and vertical lines when setting a minimum circumscribing rectangle for the target object 40, and it is possible to set a minimum circumscribing rectangle for the target object 40 in an area smaller than a bounding box defined by horizontal and vertical lines. When this rotated bounding box is added, the coordinate positions of the four points that are the intersections of the straight lines constituting the rotated bounding box become learning data.

[0060] Furthermore, instead of a bounding box, the annotation information may be an automatically generated contour line of the target object 40 (a boundary line separating the target object 40 from the background) obtained by performing segmentation. In this case, the boundary line separating the target object 40 from the background becomes learning data.

[0061] The above-mentioned processes in steps ST11 and ST12 correspond to the annotation information generating step performed by the annotation information generating unit 11b.

[0062] In step ST13, in addition to the information acquired in step ST12, the three-dimensional posture of the target object 40 as seen from the 3D camera 24 (relative to the 3D camera 24) in the posture of the 3D camera 24 at the time of shooting is acquired from the posture of the arm unit 27 of the shooting robot 20 at the time of shooting (corresponding to the posture of the 3D camera 24) and the three-dimensional posture information of the target object 40 acquired in the preparation process (acquired in step ST2). This three-dimensional posture is also the learning data referred to in the present invention. In general, as shown in FIG. 8, the three-dimensional posture of the target object 40 relative to the 3D camera 24 can be acquired from the deviation amount in each of the X-axis direction (horizontal direction), Y-axis direction (horizontal direction perpendicular to the X-axis direction), and Z-axis direction (vertical direction) of the lens of the 3D camera 24 at the installation position of the target object 40.

[0063] Thereafter, in step ST14, the CPU 11 moves and rotates the arm unit 27 of the photographing robot 20 (changing the photographing direction in step ST14) to photograph from a different angle (photographing in step ST11). That is, the CPU 11 repeats the processes from step ST11 to step ST15 until two-dimensional photographing of the entire 360 ​​degrees around the target object 40 is completed (until YES is determined in step ST15). As a result, information on the bounding box (or the contour line of the target object 40 by segmentation) assigned to the target object 40 corresponding to each of the multiple directions and information on the three-dimensional posture of the target object 40 (the three-dimensional posture of the target object 40 relative to the 3D camera 24) are learned (acquired as learning data), and a learning database is constructed from these learning data.

[0064] By constructing such a learning database, it becomes possible to estimate the three-dimensional posture of the target object 40 (to generate an inference model) simply by acquiring one two-dimensional image by the 3D camera 24. For example, when the target object 40 is grasped by the working unit 28 of the photographing robot 20, it becomes possible to estimate the three-dimensional posture of the target object 40 by simply acquiring one two-dimensional image by the 3D camera 24 while referring to the learning database (for example, by extracting learning data that matches the acquired two-dimensional image), and to move the working unit 28 to an optimal grasping position.

[0065] The above-mentioned processing in steps ST13 to ST15 corresponds to a learning database construction step performed by the learning database construction unit 11c.

[0066] -Effects of the embodiment- As described above, in this embodiment, the constructed learning database is constructed with a plurality of data of annotation information and relative three-dimensional posture information of the target object 40 with respect to the 3D camera 24 as learning data. Therefore, it is possible to construct a learning database that can improve the recognition accuracy of the target object 40 (recognition accuracy of the three-dimensional posture of the target object 40). For example, when the target object 40 is grasped by the working unit 28 of the photographing robot 20, the relative three-dimensional posture information of the target object 40 can be obtained by simply acquiring a two-dimensional image of the target object 40, and the recognition accuracy of the target object 40 can be improved based on this three-dimensional posture information, and when grasping the target object 40, it becomes possible to grasp the target object 40 at an optimal position (for example, near the center of gravity of the target object 40).

[0067] Other embodiments of the present invention will now be described.

[0068] <Second embodiment> In addition to the above-described embodiment, it is preferable that the CPU 11 uses the recognition result using deep learning of the target object 40 based on the annotation information that was previously automatically created, and thereafter calculates the similarity between the annotation information of the target object 40 calculated in step ST11 from the captured image including the target object 40 and the information recognized by deep learning, and if the similarity is large, performs annotation processing with greater emphasis on angles that are close to a similar angle.

[0069] <Third embodiment> In addition to the above-described embodiment, lights may be provided on the photographing robot 20, the mounting device 30, the ceiling, the wall surface, etc. Then, in step ST14, the CPU 11 moves or rotates the arm unit 27 of the photographing robot 20, turns the light on and off, changes the light intensity, or changes the color of the light to photograph from a different angle. That is, the CPU 11 repeats the processes of steps ST11 to ST15 until two-dimensional photographing of various light conditions for 360 degrees around the target object 40 is completed (until YES is determined in step ST15).

[0070] <Fourth embodiment> In addition to the above-described embodiment, the direction and posture of the target object 40 may be changed by the working unit 28 mounted on the image capturing robot 20. In this case, since the three-dimensional shape of the target object 40 changes, information on the changed direction and posture of the target object 40 is linked to the three-dimensional shape of the target object 40 at that time, and stored separately in the memory 12 for each posture of the target object 40.

[0071] When performing deep learning processing (processing shown in the flowchart of Figure 6), the CPU 11 reads out the orientation and posture of the target object 40 stored in the memory 12, and has the working unit 28 of the imaging robot 20 set the orientation and posture of the target object 40 to the registered state so that the target object 40 has the correct posture, and then performs the deep learning processing.

[0072] <Fifth embodiment> In addition to the above-described embodiment, in step ST11, the CPU 11 causes the photography robot 20 to perform two-dimensional photography, but the photography robot 20 may also perform three-dimensional photography. Then, annotation information may be added to each piece of three-dimensional photography data based on the 3D point cloud data.

[0073] Sixth embodiment In the embodiment described above, the 3D camera 24 used in the preparation process shown in Fig. 4 is also used to capture images for the deep learning process shown in Fig. 6. However, the present invention is not limited to this, and a camera other than the 3D camera 24 used in the preparation process shown in Fig. 4 may be used to capture images for the deep learning process shown in Fig. 6.

[0074] Seventh embodiment In the above-described embodiment, three-dimensional CAD data such as three-dimensional shape and position information of the imaging environment (table, stage, robot itself, etc.) is received and registered in memory 12, but the three-dimensional information of the imaging environment may also be acquired in a similar manner to that of the target object 40.

[0075] Referring to FIG. 9, in this embodiment, before placing the target object 40, three-dimensional information of the shooting environment is acquired in a manner similar to the target object acquisition method shown in FIG. 4, and if the environmental information is not registered (if NO is determined in step ST8), the acquired three-dimensional information is registered in memory 12 as environmental data (step ST9).

[0076] After that, in the same manner as in the first embodiment, CPU 11 repeats the processes from step ST2 to step ST5 until three-dimensional imaging of the entire 360-degree circumference of target object 40 is completed (until YES is determined in step ST5).

[0077] <Eighth embodiment> Some or all of the roles of the devices such as the control device 10 and the photographing robot 20 of the learning database construction system 1 of the embodiment described above may be performed by other devices. For example, some of the roles of the control device 10 may be performed by the photographing robot 20, multiple personal computers, or multiple servers on the cloud.

[0078] <Ninth embodiment> The first embodiment described above is intended to construct a learning database used to estimate the three-dimensional orientation of the target object 40 relative to the 3D camera 24. Instead, in the present embodiment, the present invention is applied to construct a learning database used to grasp a specific position on the target object 40 and to perform processing such as processing on the specific position.

[0079] FIG. 10 is a diagram for explaining an inference operation of a specific position of the target object 40 according to this embodiment. The position P1 of the target object 40 in FIG. 10 is a position to be grasped by the working unit 28. For example, it is a position near the center of gravity of the target object 40. In this embodiment, the learning data acquired by the learning database construction unit 11c described above is learned as a position (position P1) to be grasped by the working unit 28 on the target object 40. That is, in step ST13 of the deep learning process described using FIG. 6, a specific position (for example, position P1) of the target object 40 as seen from the 3D camera 24 in the attitude of the 3D camera 24 at the time of shooting (relative to the 3D camera 24) is acquired from the attitude of the arm unit 27 of the shooting robot 20 at the time of shooting (corresponding to the attitude of the 3D camera 24) and the three-dimensional attitude information of the target object 40 acquired in the preparation process described above. In this way, by photographing the target object 40 from multiple directions, information on the bounding box (or the contour of the target object 40 by segmentation) assigned to the target object 40 corresponding to each direction and information on specific positions of the target object 40 are learned (learning data is acquired), and a learning database is constructed from this learning data. This makes it possible to estimate the position on the target object 40 that should be grasped by the working unit 28 by simply acquiring one two-dimensional image.

[0080] In this case, the camera robot 20 may also be made to learn the TCP (Tool Center Point) of the camera robot 20. That is, the operation of the arm unit 27 of the camera robot 20 is controlled based on this TCP (e.g., position P2 in FIG. 10), and when the target object 40 is grasped, the working unit 28 is controlled based on the position to be grasped (position P1).

[0081] Furthermore, the configuration of this embodiment is not limited to gripping the target object 40, and can also be used to process a specific position on the target object 40. For example, when processing (e.g., welding or other processing) the position P3 of the target object 40 in Fig. 10, the learning data acquired by the learning database constructing unit 11c is learned as the processing position on the target object 40, and the processing position on the target object 40 can be estimated by simply acquiring one two-dimensional image.

[0082] <Other embodiments> The present invention is not limited to the above-described embodiments, and all modifications and applications within the scope of the claims and equivalents thereto are possible.

[0083] For example, in each of the above embodiments, the present invention has been described as being applied to construct a learning database used when the photographing robot 20 grasps the target object 40 or processes a specific position on the target object 40. The present invention is not limited to this, and may be used to construct a learning database used for other purposes. For example, a learning database may be constructed to be used in a system that recognizes the posture of the target object 40 transported on a conveyor when it reaches a predetermined position, notifies an operator of the target object 40 that is not in the predetermined posture, or corrects the posture using the photographing robot 20.

[0084] In addition, in each of the above-described embodiments, the placement device 30 is included as a component of the learning database construction system 1, but the placement device 30 is not an essential component of the present invention. For example, the learning database may be constructed by photographing a target object 40 placed on a floor surface. [Industrial Applicability]

[0085] The present invention is applicable to a learning database construction system for improving the recognition accuracy of a target object. [Explanation of symbols]

[0086] 1. Learning database construction system 10 Control device 11a 3D information generation section 11b Annotation information generation unit 11c Learning Database Construction Department 20. Photographic robot (movable) 24 3D Camera (3D Camera) 40 Target Object

Claims

1. A three-dimensional camera capable of photographing a target object; a movable body capable of changing the position and attitude of the three-dimensional camera; a control device capable of acquiring information on an image of the target object captured by the three-dimensional camera, The control device includes: a three-dimensional information generating unit that generates three-dimensional information of the target object by controlling the movable body to perform three-dimensional photography in which the target object is photographed from a plurality of directions by the three-dimensional camera; an annotation information generating unit that automatically generates annotation information based on three-dimensional information for an image of the target object captured by the three-dimensional camera or a camera other than the three-dimensional camera; and a learning database construction unit that constructs a learning database using multiple pieces of data, each of which is the annotation information and the relative three-dimensional physical information of the target object with respect to the camera used when automatically generating the annotation information, as learning data.

2. 2. The learning database construction system according to claim 1, A learning database construction system, characterized in that the three-dimensional physical information is relative three-dimensional posture information of the target object with respect to the camera used when automatically generating the annotation information.

3. 2. The learning database construction system according to claim 1, A learning database construction system, characterized in that the three-dimensional physical information is information on a specific three-dimensional position of the target object relative to the camera used when automatically generating the annotation information.

4. A control device capable of acquiring information on an image of a target object captured by a three-dimensional camera supported on a movable body and capable of changing a position and attitude by operating the movable body, comprising: a three-dimensional information generating unit that generates three-dimensional information of the target object by controlling the movable body to perform three-dimensional photography in which the target object is photographed from a plurality of directions by the three-dimensional camera; an annotation information generating unit that automatically generates annotation information based on three-dimensional information for an image of the target object captured by the three-dimensional camera or a camera other than the three-dimensional camera; and a learning database construction unit that constructs a learning database using multiple pieces of data, each of which is the annotation information and the relative three-dimensional physical information of the target object with respect to the camera used when automatically generating the annotation information, as learning data.

5. a three-dimensional information generating step of generating three-dimensional information of a target object by controlling a movable body supporting a three-dimensional camera to perform three-dimensional photography in which the target object is photographed from a plurality of directions by the three-dimensional camera; an annotation information generating step of automatically generating annotation information based on three-dimensional information for an image of the target object captured by the three-dimensional camera or a camera other than the three-dimensional camera; and a learning database construction step of constructing a learning database using as learning data multiple data of the annotation information and relative three-dimensional physical information of the target object with respect to the camera used when automatically generating the annotation information.

Images