Teacher data generation device, teacher data generation method, and process determination device and process determination method

Abstract

To provide a device capable of determining a work process with high accuracy, even in a work for an object having no attitude detection sensor or the like.SOLUTION: A teacher data creation device for generating teacher data for each work process of assembling a plurality of components into a product, on the basis of 3DCAD models of the plurality of components has: a model processing part for reading the 3DCAD models of the plurality of components and photography conditions of the work processes; a capture diagram generation part for generating the 3DCAD model for each of the work processes, and generating a capture diagram obtained by virtually photographing the generated 3DCAD models for each work process under different photographic conditions; and a teacher data generation part for tagging a work process name to the generated capture diagram, and generating teacher data.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a teacher data generation device and a teacher data generation method that create teacher data based on a 3D CAD model of a part, and a process determination device and a process determination method that determine the progress of a work process based on the generated teacher data. [Background technology]

[0002] When recognizing the progress of work processes in a manufacturing site, it is difficult to accurately identify the current work process from the worker's position alone. Therefore, a technology has been proposed that prepares a large amount of training data in which tags indicating the work process are added to work videos, and uses the results of deep learning on this training data to identify the work process that corresponds to the filmed work video.

[0003] However, tagging videos is a heavy burden on workers, and because there is little experience with each task in assembly work involving high-mix, low-volume production, it is difficult to prepare a large amount of training data for each task. Therefore, there has been a demand for technology that can easily prepare large amounts of training data for deep learning, even for tasks with little experience, such as assembly work involving high-mix, low-volume production.

[0004] As a technology that meets this need, for example, the abstract of Patent Document 1 discloses that "the manufacturing method of a trained task classification estimation model 180 comprises obtaining training data 188A including a specific viewpoint image 206 in which a three-dimensional model showing the three-dimensional shape of the work machine while working is viewed from a specific viewpoint position, and a task classification result 204 showing the content of the operation of the work machine, which is labeled with the specific viewpoint image 206, and training the task classification estimation model 180 using the training data 188A."

[0005] Furthermore, paragraph 0123 of the same document explains, "As shown in Figures 8 and 9, by using action state image 205 of the 3D model as training data and freely changing the viewing direction of the 3D model by changing the viewpoint position, it is possible to easily increase the amount of training data for learning task classification estimation model 180. By training task classification estimation model 180 with a large amount of training data, it is possible to improve the accuracy of task classification estimation using the trained task classification estimation model 180, making it possible to perform task classification with high accuracy." [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Patent Publication No. 2021-8747 Summary of the Invention [Problem to be solved by the invention]

[0007] However, in order to realize "improved accuracy of task classification estimation" by "simply increasing the number of teacher data" as described in paragraph 0123 of Patent Document 1, as a prerequisite, an angle detector that detects the posture of the work machine (wheel loader) is essential, as described in paragraph 0031 of the same document, which states that "the wheel loader 1 is equipped with..., a first angle detector 29, a second angle detector 48,..." and as described in paragraph 0103, which states that "the posture data generation unit 163 detects the boom angle θ1 and bucket angle θ2 at the relevant time based on the detection results of the first angle detector 29 and the second angle detector 48, and generates posture data 203 (FIG. 8).", there is a problem in that teacher data cannot be generated for targets that do not have a sensor similar to an angle detector.

[0008] Therefore, an object of the present invention is to provide an apparatus and method that can easily create training data and determine the work process with high accuracy, even when working on an object that does not have an orientation detection sensor, etc. [Means for solving the problem]

[0009] In order to solve the above problems, the teacher data creation device of the present invention is a teacher data creation device that generates teacher data for each work process of assembling a plurality of parts into a product based on a 3D CAD model of the plurality of parts, and has a model processing unit that reads the 3D CAD model of the plurality of parts and the shooting conditions of the work process, a capture image generation unit that generates a 3D CAD model for each work process and generates a capture image obtained by virtually photographing the generated 3D CAD model for each work process under different shooting conditions, and a teacher data generation unit that tags the generated capture image with the name of the work process to generate teacher data.

[0010] In addition, the process judgment device of the present invention is a process judgment device that uses teacher data generated by the above-mentioned teacher data creation device, and has a deep learning unit that deep learns an object detection model that detects objects from within a work image captured by a camera by transfer learning the teacher data, and a process analysis unit that judges a work process by comparing an object detected from the work image using the object detection model with an object in the captured image. Effect of the Invention

[0011] According to the present invention, even when working on an object that does not have an orientation detection sensor, training data can be easily created and the work process can be analyzed with high accuracy. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a configuration diagram of a process analysis system according to an embodiment. [Diagram 2] A flowchart showing a method for creating training data in one embodiment. [Diagram 3] An example of a photographed state of an assembly work in one embodiment [Figure 4] An example of assembly process information in one embodiment [Diagram 5]An example of an assembly process and its 3D CAD model in one embodiment [Figure 6] 1 is an example of training data for a deep learning object detection model in one embodiment. [Figure 7] A flowchart showing a method for determining a work process in one embodiment. [Figure 8] 13 is an example of a method for calculating a work time of a work process in an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, the embodiments of the present invention will be described with reference to the drawings. Note that the embodiments described below do not limit the scope of the invention, and not all of the elements and combinations thereof described in the embodiments are necessarily essential to the solution of the invention.

[0014] 1 is a configuration diagram of a process judgment system 100 in one embodiment of the present invention. As shown in the figure, the process judgment system 100 of this embodiment is a system in which a process judgment device 1, a client computer 2, and a camera 3 are connected via a wired or wireless network. In the following, the client computer 2 and the camera 3 will be briefly described, and then the process judgment device 1 will be described in detail.

[0015] <Client computer 2> The client computer 2 is a terminal used by a user when using the process judgment system 100, and is a general personal computer, tablet, smartphone, or the like, on which software for linking with the process judgment device 1 is installed. This client computer 2 is connected to or has built-in an input device 21, such as a keyboard, mouse, or touch panel, which the user uses when operating the system, and a display device 22, such as a liquid crystal display, which the system uses when presenting information to the user.

[0016] <Camera 3> The camera 3 is an imaging device placed in a position where it can capture images of assembly work at the manufacturing site, and satisfies the camera specifications, imaging distance, and imaging height conditions that allow it to clearly capture the contours of the parts that are the subject of the work. Note that, although Fig. 1 illustrates an example of an assembly work in which a relatively small part P2 is attached to the top surface of a relatively large part P1, the work targeted by this system is not limited to the assembly work shown in Fig. 1, and may be other types of work as long as the appearance of the target changes before and after the work.

[0017] <Process judgment device 1> The process judgment device 1 is a device that integrates a teacher data generation function that automatically generates a large amount of teacher data based on a 3D CAD model and a process judgment function that judges the work process being photographed by the camera 3, and includes an input / output unit 11, a storage unit 12, and a calculation unit 13. Specifically, the process judgment device 1 is a server that includes hardware such as a calculation device such as a CPU, a main storage device such as a semiconductor memory, an auxiliary storage device such as a hard disk or SSD, and a communication device. The calculation device executes a predetermined program to realize each function described below, but the following description will be given while omitting such well-known techniques as appropriate. In FIG. 1, the process judgment device 1 and the client computer 2 are illustrated as independent systems, but they may be integrated into one system.

[0018] <<I / O section 11>> The input / output unit 11 is a communication interface for communicating with the client computer 2 and the camera 3 via a network.

[0019] <<Storage section 12>> 1, the storage unit 12 holds a 3D CAD model 12a, assembly process information 12b, photography conditions 12c, an object detection model 12d, and presentation information 12e. These will be described in detail later, but the outline is as follows.

[0020] The 3D CAD model 12a is a 3D CAD model obtained by assembling, as an assembly model, 3D CAD models of the three-dimensional shapes of parts P (for example, parts P1 and P2) that are to be the objects of assembly work at a manufacturing site.

[0021] The assembly process information 12b includes a 3D CAD model ID linked to the 3D CAD model 12a, an assembly sequence indicating the order in which the assembly work of each part is performed, work posture information indicating the work posture of the part, and information identifying a background image for each process.

[0022] The photographing conditions 12c are photographing conditions for the camera 3 that photographs the assembly work at the manufacturing site, and include at least the photographing distance L and the photographing height H, which are information for calculating the photographing angle and size of the assembly parts.

[0023] The object detection model 12d is an algorithm that detects where a learned object appears in a captured image.

[0024] The presentation information 12e is a display format and alert information used when displaying the results of the process judgment.

[0025] <<Arithmetic unit 13>> 1, the calculation unit 13 includes a model processing unit 13a, a capture diagram generating unit 13b, a teacher data generating unit 13c, a deep learning unit 13d, a process analysis unit 13e, and a result display execution unit 13f. The details of these will be described later, but the overview is as follows.

[0026] The model processing unit 13a processes the 3D CAD model 12a stored in the storage unit 12 in accordance with the assembly process information 12b.

[0027] The capture drawing generating unit 13b generates a capture drawing by virtually photographing a 3D CAD model of each assembly process.

[0028] The teacher data generating unit 13c pastes the captured image generated by the captured image generating unit 13b onto a specified background image, and generates teacher data in which the paste position and the process name are entered.

[0029] The deep learning unit 13d learns the object detection model 12d by using the teacher data generated by the teacher data generating unit 13c. Note that, for example, transfer learning is used in the learning here to improve the previous object detection model 12d.

[0030] The process analysis unit 13e uses the object detection model 12d, which is the learning result of the deep learning unit 13d, to recognize processes and calculate work times for images captured at the manufacturing site.

[0031] The result display execution unit 13f displays on the display device 22 the process information or the operation time and the alert corresponding to the result of the analysis by the process analysis unit 13e in accordance with the display format defined by the presentation information 12e.

[0032] <How to create training data> Next, a method for generating teacher data in this embodiment will be described with reference to the flowchart of FIG.

[0033] Step S1: First, the model processing unit 13a reads the 3D CAD model 12a and the photographing conditions 12c from the storage unit 12. Fig. 3 shows an example of the read 3D CAD model 12a and the photographing conditions 12c. The 3D CAD model 12a is, for example, a 3D CAD model of each of parts P1 and P2. The photographing conditions 12c are, for example, a condition that an assembly operation of the 3D CAD model 12a is photographed from a position at a horizontal distance L and a vertical distance H from the product.

[0034] Step S2: Next, the model processing unit 13a adjusts the 3D CAD model 12a according to the photographing conditions 12c, and sets the size and angle to be captured.

[0035] Step S3: The capture diagram generating unit 13b reads the assembly process information 12b from the storage unit 12. An example of the assembly process information is shown in Fig. 4. As shown in Fig. 4, the assembly process information 12b includes an assembly sequence, a process name, a part name, information on the working posture, and information for specifying a background image. Note that the background image is an image taken of the actual work site in a situation where no parts to be worked on are present, and if the working locations for each process are different, a background image is prepared for each process.

[0036] Step S4: The capture diagram generator 13b uses the loaded assembly process information 12b to generate 3D CAD models of assembled parts for each process. Fig. 5 shows the 3D CAD model at the time of the first process and the 3D CAD model at the time of the second process. Since the first process of assembly sequence "1" is the first process, only the 3D CAD model of part P1 is generated. On the other hand, in the second process of assembly sequence "2", the 3D CAD model of the product in which the 3D CAD model of part P2 is added to the 3D CAD model of part P1 in the first process is generated.

[0037] Step S5: The capture diagram generating unit 13b adjusts the working posture of the assembled 3D CAD model generated in step S4 using the read assembly process information 12b, and saves it as the basic posture for each process. In the example of Fig. 4, the working posture adjustment angle for the first and second processes is 0°, so the rotation angle is also 0°. On the other hand, when there is a specified angle (e.g., 45°) as the working posture adjustment angle, as in the assembly sequence "3" in Fig. 4, the 3D CAD model is rotated to the specified angle and displayed in the correct working posture, and then the basic posture is saved.

[0038] Step S6: The capture image generating unit 13b automatically generates capture images of the 3D CAD model of each process, virtually photographed, based on the 3D CAD model and its basic posture stored in step S5. For example, the 3D CAD model in the basic posture is rotated in 5-degree increments within a specified angle range of 30 degrees in each of the X, Y, and Z axes directions to generate multiple capture images of the 3D CAD model virtually photographed from various viewpoints. The range of the rotation angle and the rotation unit angle may be appropriately adjusted according to the size of the target product, the photographing conditions, and the like.

[0039] Step S7: The teacher data generating unit 13c makes the background of the captured image of the 3D CAD model corresponding to each process generated in step S6 transparent and performs binarization processing. This makes it possible to generate an image of the 3D CAD model expressed in black and white.

[0040] Step S8: The teacher data generating unit 13c reads the background image of the assembly process information 12b stored in the storage unit 12.

[0041] Step S9: The teacher data generating unit 13c randomly pastes the captured images of each process binarized in step S7 onto the background image read in step S8. Then, the name of the work process of the captured image for each process, and the pasted position and size are tagged to the annotation data to automatically generate teacher data. Figure 6 shows an example of teacher data corresponding to the second process illustrated in Figure 5.

[0042] <How to update the object detection model> After generating a large amount of training data for each process in the procedure of Figure 2, the deep learning unit 13d obtains a new trained object detection model 12d by re-learning (transfer learning) the training data for each process as training data for the deep learning object detection model.

[0043] <How to determine the work process> Next, a procedure for determining the current work process from a video image taken at a manufacturing site based on the object detection model 12d re-learned by the deep learning unit 13d will be described with reference to the flowchart of FIG.

[0044] Step S11: First, the process analysis unit 13e reads the video captured by the camera 3 at the manufacturing site.

[0045] Step S12: Next, the process analysis unit 13e extracts work images at predetermined time intervals from the read video.

[0046] Step S13: The process analysis unit 13e processes the size of the operation image so as to match the size of the background image read in step S8.

[0047] Step S14: The process analysis unit 13e performs binarization processing on the size-adjusted working image.

[0048] Step S15: The process analysis unit 13e calculates the height of the detection target based on the 3D CAD model 12a and the assembly process information 12b read in step S1, and selects the image range of the detection target.

[0049] Step S16: The process analysis unit 13e performs process detection for the image range of the selected work image using the trained object detection model, and detects the process name and process recognition accuracy in the work image. The process recognition accuracy detected in this step is calculated by comparing the part shape in the work image with the 3D CAD model of each process. For example, if the part shape in the work image and the 3D CAD model of a certain process are very similar to each other, the possibility that the current work process is the process is calculated to be 95%.

[0050] Step S17: The process analysis unit 13e judges whether the detected process name is the currently targeted process. If the detected process is a process before the target process, it is ignored and the process returns to step S12. On the other hand, if the detected process is the target process or the next process, the process proceeds to step S18.

[0051] Step S18: The process analysis unit 13e compares the process recognition accuracy with the stored maximum recognition accuracy, and if the process recognition accuracy detected in step S16 is greater than the stored process recognition accuracy, the process proceeds to step S19.

[0052] Step S19: The process analysis unit 13e updates the process name detected in step S16, the recognition accuracy of the process, and the photographing time.

[0053] Step S1a: The process analysis unit 13e calculates the operation time using the process name and the photographing time updated in step S19. The operation time of the second process can be calculated by subtracting the photographing time of the first process, which is the previous process, from the photographing time of the second process. FIG. 8 shows an image of the calculation of the operation time. As shown here, the time when the 3D CAD model of the first process matches the image photographed at the manufacturing site can be recognized as the end time T1 of the first process, and the time when the 3D CAD model of the second process matches the image photographed at the manufacturing site can be recognized as the end time T2 of the second process, so that T2-T1 can be calculated as the time required for the second process. Then, if the operation time calculated in this step exceeds the standard operation time set for each process, the process analysis unit 13e commands the result display execution unit 13f to issue an alert indicating that the time has exceeded.

[0054] Step S1b: The result display execution unit 13f displays the process name and operation time obtained by the above procedure on the display device 22 according to the format defined by the presentation information 12e. This allows the user to know which operation process the current process is, whether the operation time required for each process has exceeded the standard operation time, etc.

[0055] Step S1c: The process analysis unit 13e checks whether the work image extracted in step S12 is an image of the final process. If it is not an image of the final process, steps S12 to S1b are repeated. On the other hand, if it is the final work image, the process analysis unit 13e stops process detection. In addition, the result display execution unit 13f notifies the user via the display device 22 that all work processes have been completed. <Effects of this embodiment> As described above, according to this embodiment, even in the case of an assembly operation for a part that does not have an orientation detection sensor, teacher data can be easily created and the work process can be analyzed with high accuracy. [Explanation of symbols]

[0056] 100 Process Judgment System 1 Process judgment device 11 Input / output section 12 Storage section 12a 3D CAD model 12b Assembly process information 12c Shooting conditions 12d object detection model 12e Presentation information 13 Arithmetic section 13a Model processing section 13b Capture diagram generation section 13c Teacher data generation unit 13d Deep Learning Department 13e Process Analysis Department 13f Result display execution section 2. Client computer 21 Input Devices 22 Display device 3. Camera P parts

Claims

1. A training data generation device that generates training data for each work process of assembling a plurality of parts into a product based on a 3D CAD model of the plurality of parts, a model processing unit that reads 3D CAD models of the plurality of parts and imaging conditions of the work process; A capture image generating unit that generates a 3D CAD model for each work process and generates a capture image by virtually photographing the generated 3D CAD model for each work process under different photographing conditions; a training data generation unit that generates training data by tagging the generated capture diagram with a work process name; A teacher data creation device comprising:

2. A process judgment device that uses the teacher data generated by the teacher data creation device of claim 1, A deep learning unit that performs deep learning of an object detection model that detects an object from within a work image captured by a camera by transfer learning the teacher data; a process analysis unit that determines a work process by comparing an object detected from the work image using the object detection model with an object in the capture drawing; A process judgment device comprising:

3. 3. The process judgment device according to claim 2, The process analysis unit calculates a process recognition accuracy by comparing objects detected from the work image with objects in a capture drawing of each work process, and determines the work process corresponding to the capture drawing with the highest calculated process recognition accuracy as the work process corresponding to the work image, characterized by the process determination device.

4. 4. The process judgment device according to claim 3, The process analysis unit includes: Calculating the operation time of the subsequent operation process from the time difference between the completion time of the previous operation process and the completion time of the subsequent operation process; A process judgment device characterized in that an alert is issued when the calculated work time exceeds the standard work time of the subsequent work process.

5. A method for generating training data for each work process for assembling a plurality of parts into a product based on a 3D CAD model of the parts, comprising: Reading 3D CAD models of the plurality of parts and photographing conditions of the work process; A step of generating a 3D CAD model for each work process and generating a capture image by virtually photographing the generated 3D CAD model for each work process under different photographing conditions; A step of generating training data by tagging the generated capture diagram with a work process name; A teacher data creation method comprising the steps of:

6. A process judgment method using teacher data generated by the teacher data creation method of claim 5, A step of deep learning an object detection model that detects an object from within a work image captured by a camera by transfer learning the teacher data; A step of determining a work process by comparing an object detected from the work image using the object detection model with an object in the capture drawing; A process judgment method comprising the steps of:

Images