Training database construction system, control device used in said system, and training database construction method

Abstract

[PROBLEM] To enable construction of a training database capable of improving target recognition accuracy. [SOLUTION] This invention comprises: a 3D camera 24 capable of imaging a target 40; an imaging robot 20 capable of changing the position and attitude of the 3D camera 24; and a control device 10 capable of acquiring information about an image of the target 40 captured by the 3D camera 24. The control device 10: generates three-dimensional information about the target 40 by controlling the imaging robot 20 to perform three-dimensional imaging for imaging the target 40 from a plurality of directions using the 3D camera 24; automatically generates annotation information on the basis of the three-dimensional information for an image obtained by capturing the target 40 using the 3D camera 24; and constructs a training database in which a plurality of pieces of data of both the annotation information and three-dimensional physical information about the target 40 relative to the 3D camera 24 are used as training data.

Description

Learning database construction system, control device used in the system, and learning database construction method

[0001] The present invention relates to a learning database construction system, a control device used in the system, and a learning database construction method.

[0002] Conventionally, when constructing a database of training data (also called teacher data) used in AI machine learning in image recognition technology, it is desirable to recognize target objects (hereinafter referred to as target objects) with high accuracy. Furthermore, constructing this training database has required a lot of time and effort.

[0003] Japanese Patent Application Laid-Open No. 2003-122299 and Japanese Patent Application Laid-Open No. 2003-122299 disclose techniques for photographing a target object and building a learning database.

[0004] Patent Literature 1 discloses that, in the stage of constructing a training dataset (corresponding to a training database), object information of a target object is associated with an AR marker (two-dimensional pattern marker), which is a marker for detecting position and orientation. Patent Literature 1 also discloses that, in a database, the AR marker is associated with object information such as the object name of the target object, and a computer of a training dataset creation device operates as AR marker recognition means to recognize the AR marker and acquire object information of the target object.

[0005] Patent Document 2 also discloses a dataset storage unit that acquires from the simulator and stores multiple sets of learning datasets comprising three-dimensional coordinate data of the robot hand on the simulator when the robot hand gripping a workpiece goes through a gripping operation and successfully grips the workpiece, and two-dimensional image data of the workpiece captured from a predetermined angle of view by a two-dimensional imaging device ID on the simulator, and constructs a learning model that infers the three-dimensional coordinates of the robot hand in the real world from two-dimensional images of the workpiece captured from the same angle of view as the predetermined angle of view by the two-dimensional imaging device ID.

[0006] Japanese Patent No. 6474179 Japanese Patent Application Laid-Open No. 2020-82322

[0007] However, because the technique of Patent Document 1 uses a two-dimensional camera, there is a risk that the recognition accuracy of the target object will be significantly reduced if the color of the background and the color of the target object are similar. Also, because the technique uses an AR marker, if the posture of the target object is such that the AR marker cannot be recognized, it is not possible to acquire information about the target object. This limits the posture of the target object from which object information can be acquired, making it difficult to build a highly accurate learning database.

[0008] Furthermore, since the technology in Patent Document 2 uses simulated images, it is unable to fully express the characteristics of real target objects, and there is a limit to how much improvement can be made in the recognition accuracy of target objects, making it difficult to build a highly accurate learning database.

[0009] The present invention has been made in consideration of the above points, and its purpose is to provide a learning database construction system capable of constructing a learning database that can improve the recognition accuracy of a target object (for example, the recognition accuracy of the three-dimensional posture of the target object or the recognition accuracy of a specific three-dimensional position on the target object), as well as a control device used in the system and a learning database construction method.

[0010] The solution of the present invention for achieving the above-mentioned object comprises a three-dimensional camera capable of photographing a target object, a movable body capable of changing the position and attitude of the three-dimensional camera, and a control device capable of acquiring information on images of the target object photographed by the three-dimensional camera, wherein the control device comprises a three-dimensional information generation unit that generates three-dimensional information of the target object by controlling the movable body to perform three-dimensional photography in which the target object is photographed from multiple directions using the three-dimensional camera, an annotation information generation unit that automatically generates annotation information based on the three-dimensional information for images of the target object photographed by the three-dimensional camera or a camera other than the three-dimensional camera, and a learning database construction unit that constructs a learning database using multiple data sets of the annotation information and three-dimensional physical information of the target object relative to the camera used when automatically generating the annotation information as learning data.

[0011] With this specification, the learning database is constructed using multiple pieces of learning data, each of which is annotation information and three-dimensional physical information of the target object relative to the camera used when automatically generating the annotation information, making it possible to construct a learning database that can improve the recognition accuracy of the target object (for example, the recognition accuracy of the three-dimensional posture of the target object and the recognition accuracy of a specific three-dimensional position on the target object). For example, when performing specific processing on a target object, the three-dimensional physical information can be obtained simply by acquiring a two-dimensional image of the target object, and the recognition accuracy of the target object can be improved based on this three-dimensional physical information, making it possible to perform the specific processing on the target object with high accuracy.

[0012] The three-dimensional physical information is relative three-dimensional posture information of the target object with respect to the camera used when the annotation information is automatically generated.

[0013] This allows the relative three-dimensional orientation of a target object to be recognized simply by acquiring a two-dimensional image of the target object. For example, when a target object is to be grasped by a robot, the grasping position can be optimized.

[0014] The three-dimensional physical information may be information about a specific three-dimensional position on the target object relative to the camera used when the annotation information is automatically generated.

[0015] This allows a specific three-dimensional position on a target object to be recognized simply by acquiring a two-dimensional image of the target object. For example, when processing a specific position on a target object using a robot, the processing position can be identified with high accuracy.

[0016] A control device used in the learning database construction system also falls within the scope of the technical idea of ​​the present invention. That is, a control device capable of acquiring information on images of a target object photographed by a three-dimensional camera supported on a movable body and whose position and orientation can be changed by actuation of the movable body, the control device comprising: a three-dimensional information generation unit that generates three-dimensional information of the target object by controlling the movable body to perform three-dimensional photography of the target object from multiple directions with the three-dimensional camera, an annotation information generation unit that automatically generates annotation information based on the three-dimensional information for images of the target object photographed by the three-dimensional camera or a camera other than the three-dimensional camera, and a learning database construction unit that constructs a learning database using, as learning data, multiple data sets of the annotation information and three-dimensional physical information of the target object relative to the camera used when automatically generating the annotation information.

[0017] The technical idea of ​​the present invention also encompasses a learning database construction method implemented in the learning database construction system, which includes a three-dimensional information generation step of generating three-dimensional information of a target object by controlling a movable body supporting a three-dimensional camera to perform three-dimensional photography of the target object from multiple directions with the three-dimensional camera, an annotation information generation step of automatically generating annotation information based on the three-dimensional information for an image of the target object photographed by the three-dimensional camera or a camera other than the three-dimensional camera, and a learning database construction step of constructing a learning database using, as training data, multiple data sets of the annotation information and three-dimensional physical information of the target object relative to the camera used when automatically generating the annotation information.

[0018] Even with these specific features, as mentioned above, it is possible to construct a learning database that can improve the recognition accuracy of the target object (for example, the recognition accuracy of the three-dimensional posture of the target object or the recognition accuracy of a specific three-dimensional position on the target object).

[0019] In the present invention, three-dimensional information of a target object is generated by controlling a movable body to perform three-dimensional photography in which the target object is photographed from multiple directions using a three-dimensional camera. Furthermore, annotation information is automatically generated based on the three-dimensional information for images of the target object photographed using the three-dimensional camera or a camera other than the three-dimensional camera. A training database is then constructed using multiple pieces of training data, including the annotation information and three-dimensional physical information of the target object relative to the camera used to automatically generate the annotation information. This makes it possible to construct a training database that can improve the recognition accuracy of the target object (e.g., the recognition accuracy of the three-dimensional orientation of the target object or the recognition accuracy of a specific three-dimensional position on the target object).

[0020] FIG. 12 is a schematic diagram showing the overall configuration of a learning database creation system according to the first embodiment. FIG. 13 is a block diagram showing the configuration of a control device according to the first embodiment. FIG. 14 is a block diagram showing the configuration of a photography robot according to the first embodiment. FIG. 15 is a flowchart showing the procedure of a preparation process according to the first embodiment. FIG. 16 is a diagram showing an example of point cloud data of a target object according to the first embodiment. FIG. 17 is a flowchart showing the procedure of a deep learning process according to the first embodiment. FIG. 18 is an image diagram showing multiple examples of bounding boxes assigned to an image of a target object according to the first embodiment. FIG. 19 is a diagram for explaining an operation of acquiring a three-dimensional posture of a target object relative to a 3D camera according to the first embodiment. FIG. 20 is a flowchart showing the procedure of a preparation process according to the seventh embodiment. FIG. 21 is a diagram for explaining an inference operation of a specific position of a target object according to the ninth embodiment.

[0021] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Each of the embodiments described below will be described as an application of the present invention to constructing a learning database (teacher database) used when a robot (movable body) performs a predetermined process on a target object (for example, grasping the target object or processing a specific position on the target object). In the following description, identical parts and components are denoted by the same reference numerals. Since their names and functions are also the same, detailed description thereof will not be repeated.

[0022] First Embodiment In this embodiment, a case will be described in which the present invention is applied to construct a learning database used for estimating the three-dimensional posture of a target object relative to a 3D camera (three-dimensional camera).

[0023] -Overall configuration and operation overview of the learning database construction system- First, the overall configuration of a learning database construction system 1 according to this embodiment will be described with reference to Figure 1. The learning database construction system 1 includes, as its main components, a control device 10, a photography robot 20, and a mounting device 30.

[0024] The control device 10 is realized by a server, a computer, or the like, and performs various processes such as acquiring information on images captured by the 3D camera 24 mounted on the photo-taking robot 20 by transmitting and receiving various data to and from the photo-taking robot 20 and the mounting device 30 via a wired LAN or a wireless LAN. The processes performed by the control device 10 will be described in detail later.

[0025] The photography robot 20 is composed of an articulated robot equipped with an arm section 27 and a working section 28 attached to the tip of the arm section 27, and performs various tasks such as moving the arm section 27, working section 28, etc. to various positions or tilting them into various postures based on command information sent from the control device 10 or according to its own judgment processing (processing by the CPU 21 described below).

[0026] The mounting device 30 has a mounting table 31 on which a target object 40 to be subjected to deep learning or annotation is placed. The mounting table 31 can be rotated and tilted.

[0027] The control device 10 operates the arm unit 27 of the imaging robot 20 to change the position and posture of the 3D camera 24, thereby photographing the target object 40 placed on the mounting table 31 from various angles, automatically annotating the target object 40, automatically assigning a bounding box to the photographed image of the target object 40, or performing segmentation of the target object 40 from the photographed image, and even obtaining the three-dimensional posture of the target object 40 relative to the 3D camera 24 as learning data (details will be described later).

[0028] In this way, the learning database construction system 1 according to this embodiment enables deep learning with reduced operator effort. The configuration and operation of each part of the learning database construction system 1 will be described in detail below.

[0029] -Configuration of the control device- One aspect of the configuration of the control device 10, which is a component of the learning database construction system 1 according to this embodiment, will be described with reference to Figure 2. The control device 10 is configured to include, as its main components, a CPU (Central Processing Unit) 11, a memory 12, an operation unit 13, and a communication interface 14.

[0030] The CPU 11 controls each part of the control device 10 by executing a program stored in the memory 12. For example, the CPU 11 executes a program stored in the memory 12 and performs various processes described below by referring to various data.

[0031] The CPU 11 has a three-dimensional information generating unit 11a, an annotation information generating unit 11b, and a learning database constructing unit 11c as functional units realized by the programs. The functions of these units will be outlined below.

[0032] The three-dimensional information generator 11a has a function of generating three-dimensional information of the target object 40 by controlling the imaging robot 20 (particularly the arm unit 27 of the imaging robot 20) to perform three-dimensional imaging in which the target object 40 is imaged from multiple directions using the 3D camera 24. Specifically, the three-dimensional information generator 11a has a function of acquiring three-dimensional information (three-dimensional image information) of only the target object 40, and has a function of performing three-dimensional imaging of the entire 360-degree circumference of the target object 40 by moving and rotating the arm unit 27 of the imaging robot 20, and creating three-dimensional point cloud data of the target object 40 from the acquired RGB+depth map for 360 degrees.

[0033] The annotation information generation unit 11b has a function of automatically generating annotation information based on the three-dimensional information for an image of the target object 40 captured by the 3D camera 24. Specifically, the annotation information generation unit 11b has a function of automatically generating a bounding box that approximately circumscribes the target object 40 (or a contour line of the target object 40 obtained by segmentation) from three-dimensional point cloud data of the target object 40 and information on the imaging direction when the target object 40 is imaged by the 3D camera 24 (the imaging direction is determined by the attitude of the 3D camera 24 according to the attitude of the arm unit 27).

[0034] The learning database construction unit 11c constructs a learning database using, as learning data, a plurality of pieces of data each of which is the annotation information and information on the three-dimensional posture of the target object 40 relative to the 3D camera 24 (referred to as three-dimensional physical information in the present invention). Specifically, the learning database construction unit 11c has a function of photographing the target object 40 from a plurality of directions, thereby learning (acquiring learning data) information on the bounding box (or the contour of the target object 40 obtained by segmentation) assigned to the target object 40 corresponding to each direction, and information on the three-dimensional posture of the target object 40, and constructing a learning database using this learning data.

[0035] The memory 12 is realized by various types of RAM, various types of ROM, etc., and may be included in the control device 10, may be detachable from various interfaces of the control device 10, or may be a recording medium of another device accessible from the control device 10. The memory 12 stores programs executed by the CPU 11, data generated by execution of programs by the CPU 11, input data, other databases used in this embodiment, etc.

[0036] The operation unit 13 receives commands from a user, an administrator, or the like, and inputs the commands to the CPU 11 .

[0037] The communication interface 14 transmits data from the CPU 11 to the photographing robot 20 via a wired LAN or a wireless LAN, and conversely, receives data (including three-dimensional image information from the 3D camera 24) from the photographing robot 20 and passes it on to the CPU 11.

[0038] 3, one embodiment of the configuration of the photography robot 20, which is a component of the learning database construction system 1. The photography robot 20 is configured to include, as its main components, a CPU 21, a memory 22, an operation unit 23, a 3D camera 24, a light 25, a communication interface 26, an arm unit 27, and a working unit 28.

[0039] The CPU 21 controls each part of the photography robot 20 by executing a program stored in the memory 22. For example, the CPU 21 controls the attitude of the arm 27 by adjusting the rotation angle of a motor provided in each joint of the photography robot 20.

[0040] The memory 22 is realized by various RAMs, various ROMs, etc., and stores various programs, data generated by the execution of programs by the CPU 21, operation commands given from the control device 10, data input via the operation unit 23, etc.

[0041] The operation unit 23 is composed of buttons, switches, etc., and receives various commands from the user and inputs the commands to the CPU 21 .

[0042] The 3D camera 24 is realized by an RGB-D camera or the like. This 3D camera 24 can acquire the distance to each part of the captured image by using, for example, two cameras. The 3D camera 24 performs three-dimensional imaging or normal two-dimensional imaging based on instructions from the CPU 21.

[0043] The light 25 irradiates light in front of the 3D camera 24 in accordance with instructions from the CPU 21 .

[0044] The communication interface 26 transmits and receives data to and from other devices such as the control device 10 via the Internet, a carrier network, a router, etc. For example, the communication interface 26 receives an operation command from the control device 10 and passes it to the CPU 21.

[0045] The arm unit 27 controls the position and attitude of the 3D camera 24 attached to the arm unit 27 and the position and attitude of the working unit 28 in accordance with instructions from the CPU 21. The 3D camera 24 is fixed to the arm unit 27. The attitude of the arm unit 27 can be determined by monitoring the rotation angle positions of each joint of the imaging robot 20. Therefore, by recognizing the attitude of the arm unit 27, the position and attitude of the 3D camera 24 can be grasped. For example, by recognizing the attitude of the arm unit 27, it is possible to grasp the coordinate position in a direction along the optical axis of the lens of the 3D camera 24 (hereinafter referred to as the X-axis direction: see FIG. 8 ) (hereinafter referred to as the X-axis coordinate position), the coordinate position in a direction perpendicular to the X-axis direction and extending horizontally (hereinafter referred to as the Y-axis direction) (hereinafter referred to as the Y-axis coordinate position), and the coordinate position in a direction extending perpendicular to each of the X-axis direction and the Y-axis direction (hereinafter referred to as the Z-axis coordinate position).

[0046] The working unit 28 corresponds to a human hand attached to the tip of the arm unit 27 and performs grasping operations, etc., and performs various operations to grasp the target object 40 or change the position or orientation of the target object 40 according to instructions from the CPU 21.

[0047] -Information processing of control device- Next, information processing of the control device 10 will be described in detail with reference to Fig. 4. As a preparatory process for deep learning, the CPU 11 of the control device 10 executes the process shown in Fig. 4 in accordance with the program in the memory 12. The process shown in Fig. 4 is performed by the three-dimensional information generator 11a.

[0048] First, the CPU 11 receives 3D CAD data such as 3D shape and position information of the imaging environment (table, stage, robot itself, etc.) and registers it in the memory 12 (environment information registration) (step ST1). The imaging environment here refers to objects other than the target object 40 in the image captured by the 3D camera 24 (objects to be subtracted from the captured image in future image processing), such as the mounting device 30, the background wall surface, and, when the working unit 28 is within the field of view of the 3D camera 24, the working unit 28 itself.

[0049] The CPU 11 causes the 3D camera 24 attached to the arm 27 of the image capturing robot 20 to capture an image of the target object 40 and acquires an RGB+depth map (step ST2). As described above, since the 3D camera 24 is fixed to the arm 27 of the image capturing robot 20, the CPU 11 can calculate the orientation information of the 3D camera 24 (the aforementioned X-axis coordinate position, Y-axis coordinate position, Z-axis coordinate position, etc.) from the orientation information of the image capturing robot 20 and the arm 27.

[0050] The CPU 11 obtains three-dimensional information (three-dimensional image information) of only the target object 40 by subtracting the data of the surrounding objects (data of the shooting environment) registered in step ST1 from the RGB+depth map captured in step ST2 (object data extraction in step ST3).

[0051] The CPU 11 moves or rotates the arm 27 of the imaging robot 20, or rotates or tilts the mounting base 31 (changing the imaging direction in step ST4), to capture an image from a different angle (3D imaging in step ST2). That is, the CPU 11 repeats the processes in steps ST2 to ST5 until three-dimensional imaging of the entire 360-degree circumference of the target object 40 is completed (until a YES determination is made in step ST5).

[0052] The CPU 11 creates three-dimensional point cloud data of the target object 40 from the 360-degree RGB+depth map of the target object 40 (3D point cloud data creation in step ST6). Specifically, as shown in FIG. 5 , three-dimensional point cloud data is created from the three-dimensional captured image of the target object 40.

[0053] Furthermore, based on the point cloud created in step ST6, it is preferable that the CPU 11 moves the arm unit 27 to take additional photographs with the 3D camera 24 so that areas where there are insufficient point clouds or areas where there is noise can be photographed in more detail, and then re-synthesizes the three-dimensional point clouds (data complementation processing in step ST7).

[0054] Since three-dimensional point cloud data is created in this manner, this created point cloud data corresponds to the three-dimensional information referred to in this invention, and the process of creating the point cloud data corresponds to the three-dimensional information generation process referred to in this invention.

[0055] After the preparatory processing shown in Fig. 4 described above, the CPU 11 of the control device 10 subsequently executes the processing shown in Fig. 6 as deep learning processing in accordance with the program stored in the memory 12. The processing shown in Fig. 6 is performed by the annotation information generation unit 11b and the learning database construction unit 11c.

[0056] The CPU 11 causes the 3D camera 24 attached to the arm 27 of the photographing robot 20 to take a two-dimensional photograph of the target object 40 (step ST11).

[0057] The CPU 11 calculates how the target object 40 appears and automatically creates annotation information based on the position information and posture information of the shooting robot 20 and the arm unit 27, the position information and posture information of the target object 40, and the three-dimensional point cloud data of the target object 40 (step ST12).

[0058] Specifically, in this embodiment, as shown in, for example, Figures 7(a) to 7(c), the CPU 11 automatically generates a bounding box that approximately circumscribes the target object 40 as annotation information from the three-dimensional point cloud data of the target object 40 and the shooting direction. Figure 7(a) shows a bounding box that is added when the target object 40 is photographed from the front, and Figure 7(b) shows a bounding box that is added when the target object 40 is photographed from diagonally in front. Figures 7(a) and 7(b) show cases where a bounding box surrounded by horizontal and vertical lines is added. This bounding box defines the coordinate positions of four points, which are the intersections of the horizontal and vertical lines, and in this embodiment, the coordinate positions of these four points become learning data.

[0059] 7(c) shows a rotated bounding box assigned when the target object 40 is photographed from diagonally behind. A rotated bounding box is obtained by removing the restriction that the lines constituting the bounding box must be horizontal and vertical lines when setting a minimum circumscribing rectangle for the target object 40, and it is possible to set a minimum circumscribing rectangle for the target object 40 that is an area smaller than a bounding box defined by horizontal and vertical lines. When this rotated bounding box is assigned, the coordinate positions of the four points that are the intersections of the lines constituting the rotated bounding box become training data.

[0060] Furthermore, instead of a bounding box, the annotation information may be an automatically generated contour line of the target object 40 (a boundary line separating the target object 40 from the background) obtained by performing segmentation. In this case, the boundary line separating the target object 40 from the background serves as learning data.

[0061] The above-described processing in steps ST11 and ST12 corresponds to the annotation information generating step performed by the annotation information generating unit 11b.

[0062] In step ST13, the three-dimensional orientation of the target object 40 as seen from the 3D camera 24 (relative to the 3D camera 24) at the orientation of the 3D camera 24 at the time of capturing the image is acquired from the orientation of the arm unit 27 of the imaging robot 20 (corresponding to the orientation of the 3D camera 24) at the time of capturing the image, in addition to the information acquired in step ST12, and the three-dimensional orientation information of the target object 40 acquired in the preparation process (acquired in step ST2). This three-dimensional orientation also serves as learning data in the present invention. Generally, as shown in FIG. 8 , the three-dimensional orientation of the target object 40 relative to the 3D camera 24 can be acquired from the amount of deviation in the X-axis direction (horizontal direction), Y-axis direction (horizontal direction perpendicular to the X-axis direction), and Z-axis direction (vertical direction) of the lens of the 3D camera 24 at the installation position of the target object 40.

[0063] Thereafter, in step ST14, the CPU 11 moves or rotates the arm unit 27 of the imaging robot 20 (changing the imaging direction in step ST14) to capture an image from a different angle (imaging in step ST11). That is, the CPU 11 repeats the processes of steps ST11 to ST15 until two-dimensional imaging of the entire 360-degree circumference of the target object 40 is completed (until a YES determination is made in step ST15). As a result, information on the bounding box (or the contour of the target object 40 obtained by segmentation) assigned to the target object 40 corresponding to each of the multiple directions and information on the three-dimensional posture of the target object 40 (the three-dimensional posture of the target object 40 relative to the 3D camera 24) are learned (acquired as learning data), and a learning database is constructed from this learning data.

[0064] By constructing such a learning database, it becomes possible to estimate the three-dimensional posture of the target object 40 (to generate an inference model) simply by acquiring one two-dimensional image with the 3D camera 24. For example, when grasping the target object 40 with the working unit 28 of the imaging robot 20, it becomes possible to estimate the three-dimensional posture of the target object 40 by simply acquiring one two-dimensional image with the 3D camera 24 while referring to the learning database (for example, by extracting learning data that matches the acquired two-dimensional image), and to move the working unit 28 toward an optimal grasping position.

[0065] The above-described processing of steps ST13 to ST15 corresponds to the learning database construction step carried out by the learning database construction unit 11c.

[0066] Effect of the Embodiment As described above, in the present embodiment, the constructed learning database is constructed using multiple pieces of learning data, each of which is annotation information and three-dimensional posture information of the target object 40 relative to the 3D camera 24. This makes it possible to construct a learning database that can improve the recognition accuracy of the target object 40 (the recognition accuracy of the three-dimensional posture of the target object 40). For example, when the target object 40 is grasped by the working unit 28 of the imaging robot 20, the relative three-dimensional posture information of the target object 40 can be obtained simply by acquiring a two-dimensional image of the target object 40. This three-dimensional posture information can be used to improve the recognition accuracy of the target object 40, and the target object 40 can be grasped at an optimal position (for example, near the center of gravity of the target object 40).

[0067] Other embodiments of the present invention will be described below.

[0068] Second Embodiment In addition to the above-described embodiment, the CPU 11 uses the recognition result of the target object 40 using deep learning based on the annotation information that was previously automatically created, and thereafter calculates the similarity between the annotation information of the target object 40 calculated in step ST11 from the captured image including the target object 40 and the information recognized by deep learning, and if the similarity is large, it is preferable to perform annotation processing with a greater emphasis on angles that are close to a similar angle.

[0069] Third Embodiment In addition to the above-described embodiments, lights may be provided on the image capturing robot 20, the mounting device 30, the ceiling, the wall, etc. Then, in step ST14, the CPU 11 moves or rotates the arm unit 27 of the image capturing robot 20, turns the light on or off, changes the light intensity, or changes the color of the light to capture images from different angles. That is, the CPU 11 repeats the processes of steps ST11 to ST15 until two-dimensional image capturing under various lighting conditions for 360 degrees around the target object 40 is completed (until a YES determination is made in step ST15).

[0070] Fourth Embodiment In addition to the above-described embodiments, the orientation and posture of the target object 40 may be changed by the working unit 28 mounted on the image capturing robot 20. In this case, the three-dimensional shape of the target object 40 changes, and therefore, information relating to the changed orientation and posture of the target object 40 is linked to the three-dimensional shape of the target object 40 at that time, and is stored separately in the memory 12 for each posture of the target object 40.

[0071] When performing the deep learning process (the process shown in the flowchart of Figure 6), the CPU 11 reads out the orientation and posture of the target object 40 stored in the memory 12, and then has the working unit 28 of the shooting robot 20 set the orientation and posture of the target object 40 to the registered state so that the target object 40 has the correct posture, and then performs the deep learning process.

[0072] Fifth Embodiment In addition to the above-described embodiments, in step ST11, the CPU 11 causes the image capturing robot 20 to perform two-dimensional image capturing. However, the image capturing robot 20 may also perform three-dimensional image capturing. Then, annotation information may be added to each piece of three-dimensional image capturing data based on 3D point cloud data.

[0073] Sixth Embodiment In the above-described embodiment, the 3D camera 24 used in the preparation process shown in Fig. 4 is also used to capture images for the deep learning process shown in Fig. 6. However, the present invention is not limited to this, and a camera other than the 3D camera 24 used in the preparation process shown in Fig. 4 may be used to capture images for the deep learning process shown in Fig. 6.

[0074] <Seventh embodiment> In the above-described embodiment, three-dimensional CAD data such as three-dimensional shape and position information of the imaging environment (table, stage, robot itself, etc.) is received and registered in memory 12, but the three-dimensional information of the imaging environment may also be acquired in the same manner as the target object 40.

[0075] Referring to Figure 9, in this embodiment, before placing the target object 40, three-dimensional information of the shooting environment is acquired using a method similar to the target object acquisition method shown in Figure 4, and if the environmental information has not been registered (if a NO judgment is made in step ST8), the acquired three-dimensional information is registered in memory 12 as environmental data (step ST9).

[0076] Thereafter, in the same manner as in the first embodiment, the CPU 11 repeats the processing from step ST2 to step ST5 until 3D imaging of the entire 360-degree circumference of the target object 40 is completed (until a YES judgment is made in step ST5).

[0077] Eighth Embodiment Some or all of the roles of each device, such as the control device 10 and the photographing robot 20, of the learning database construction system 1 of the above-described embodiments may be performed by other devices. For example, some of the roles of the control device 10 may be performed by the photographing robot 20, multiple personal computers, or multiple servers on the cloud.

[0078] Ninth Embodiment The first embodiment described above is intended to construct a learning database used to estimate the three-dimensional orientation of the target object 40 relative to the 3D camera 24. In this embodiment, instead, the present invention is applied to construct a learning database used to grasp a specific position on the target object 40 or to perform processing on the specific position, etc.

[0079] FIG. 10 is a diagram illustrating an inference operation for a specific position of the target object 40 according to this embodiment. Position P1 of the target object 40 in FIG. 10 is a position to be grasped by the working unit 28. For example, this is a position near the center of gravity of the target object 40. In this embodiment, the learning data acquired by the learning database construction unit 11c described above is learned as the position (position P1) on the target object 40 to be grasped by the working unit 28. That is, in step ST13 of the deep learning process described with reference to FIG. 6 , a specific position (e.g., position P1) of the target object 40 as seen from the 3D camera 24 (relative to the 3D camera 24) in the orientation of the 3D camera 24 at the time of imaging is acquired from the orientation of the arm unit 27 of the imaging robot 20 at the time of imaging (corresponding to the orientation of the 3D camera 24) and the three-dimensional orientation information of the target object 40 acquired in the preparation process described above. In this way, by photographing the target object 40 from multiple directions, information on the bounding box (or the contour of the target object 40 obtained by segmentation) assigned to the target object 40 corresponding to each direction and information on specific positions of the target object 40 are learned (learning data is acquired), and a learning database is constructed using this learning data. In this way, the position on the target object 40 that should be grasped by the working unit 28 can be estimated by simply acquiring one two-dimensional image.

[0080] In this case, the camera may also be made to learn the TCP (Tool Center Point) of the image capturing robot 20. That is, the operation of the arm unit 27 of the image capturing robot 20 is controlled based on this TCP (e.g., position P2 in FIG. 10), and when grasping the target object 40, the working unit 28 is controlled based on the position to be grasped (position P1).

[0081] Furthermore, the configuration of this embodiment is not limited to gripping the target object 40, but can also be used to process a specific position on the target object 40. For example, when processing (e.g., welding) position P3 on the target object 40 in Fig. 10, the learning data acquired by the learning database construction unit 11c is learned as the processing position on the target object 40, and the processing position on the target object 40 can be estimated by simply acquiring one two-dimensional image.

[0082] Other Embodiments The present invention is not limited to the above-described embodiments, and all modifications and applications that fall within the scope of the claims and equivalents thereto are possible.

[0083] For example, in the above-described embodiments, the present invention has been described as being applied to constructing a learning database used when the imaging robot 20 grasps the target object 40 or processes a specific position on the target object 40. However, the present invention is not limited to this, and a learning database used for other purposes may be constructed. For example, a learning database may be constructed to be used in a system that recognizes the orientation of the target object 40 transported on a conveyor when it reaches a predetermined position, notifies an operator of the target object 40 that is not in the predetermined orientation, or corrects the orientation using the imaging robot 20.

[0084] In addition, in each of the above embodiments, the mounting device 30 is included as a component of the learning database construction system 1, but the mounting device 30 is not an essential component of the present invention. For example, the learning database may be constructed by taking images of a target object 40 placed on the floor.

[0085] This application claims priority from Japanese Patent Application No. 2023-169882, filed on September 29, 2023, the entire contents of which are incorporated herein by reference.

[0086] The present invention is applicable to a learning database construction system for improving the recognition accuracy of target objects.

[0087] REFERENCE SIGNS LIST 1 Learning database construction system 10 Control device 11a Three-dimensional information generation unit 11b Annotation information generation unit 11c Learning database construction unit 20 Photography robot (movable body) 24 3D camera (three-dimensional camera) 40 Target object

Claims

1. A learning database construction system comprising: a three-dimensional camera capable of photographing a target object; a movable body capable of changing the position and attitude of the three-dimensional camera; and a control device capable of acquiring information on images of the target object photographed by the three-dimensional camera, wherein the control device comprises: a three-dimensional information generation unit that generates three-dimensional information of the target object by controlling the movable body to perform three-dimensional photography in which the target object is photographed from multiple directions with the three-dimensional camera; an annotation information generation unit that automatically generates annotation information based on the three-dimensional information for images of the target object photographed by the three-dimensional camera or a camera other than the three-dimensional camera; and a learning database construction unit that constructs a learning database using as learning data multiple data sets of the annotation information and relative three-dimensional physical information of the target object with respect to the camera used when automatically generating the annotation information.

2. A learning database construction system as described in claim 1, characterized in that the three-dimensional physical information is relative three-dimensional posture information of the target object with respect to the camera used when automatically generating the annotation information.

3. A learning database construction system as described in claim 1, characterized in that the three-dimensional physical information is information on a specific three-dimensional position of the target object relative to the camera used when automatically generating the annotation information.

4. A control device capable of acquiring information on images of a target object captured by a three-dimensional camera supported on a movable body and capable of changing the position and posture by operating the movable body, comprising: a three-dimensional information generation unit that generates three-dimensional information of the target object by controlling the movable body to perform three-dimensional photography of the target object from multiple directions using the three-dimensional camera; an annotation information generation unit that automatically generates annotation information based on the three-dimensional information for an image of the target object captured by the three-dimensional camera or a camera other than the three-dimensional camera; and a learning database construction unit that constructs a learning database using as learning data multiple data on the annotation information and the relative three-dimensional physical information of the target object with respect to the camera used when automatically generating the annotation information.

5. A learning database construction method comprising: a three-dimensional information generation step of generating three-dimensional information of a target object by controlling a movable body supporting a three-dimensional camera to perform three-dimensional photography in which the target object is photographed from multiple directions by the three-dimensional camera; an annotation information generation step of automatically generating annotation information based on the three-dimensional information for an image of the target object photographed by the three-dimensional camera or a camera other than the three-dimensional camera; and a learning database construction step of constructing a learning database using as learning data multiple data of the annotation information and relative three-dimensional physical information of the target object with respect to the camera used when automatically generating the annotation information.

Images