Work process management system

Abstract

This technology enables accurate detection of parent and child objects from images captured by cameras, such as wearable cameras, whose camera angle and position change. [Solution] The parent component detection unit 20 detects the parent component image region from the image captured by the wearable camera 40. The child component detection unit 30 detects the child component image region within the parent component image. The parent component detection unit 20 includes a parent component neural network 22 that has been trained on images including the parent component image region. The parent component neural network 22 sets up multiple candidate regions (bounding boxes) in the captured image and identifies the candidate region with the highest confidence score for the parent component image region. When the obtained confidence score is equal to or greater than the parent component threshold over a predetermined number of frames of the video, the parent component detection unit 20 confirms the candidate region that is equal to or greater than the parent component threshold, detected in the captured image after the predetermined number of frames, as the parent component image region.

Description

Technical Field

[0001] This specification discloses work process management.

[0002] For example, Patent Document 1 discloses an object detection system. This system includes a parent object detection neural network and a child object detection neural network. The parent object detection neural network detects a predetermined building (parent object) from, for example, an aerial photograph. And the child object detection neural network detects facilities (child objects) in the image of the detected building. That is, the object detection system first cuts out an image of the parent object, and then detects child objects in the image of the parent object.

[0003] The parent object detection neural network and the child object detection neural network each output the position information of the parent object and the position information of the child object. For example, the positions where the parent object and the child object are included are indicated by rectangles. These neural networks are trained using learning data in which an image is associated with the position information of the parent object and the child object in the image.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] By the way, in order to manage the product assembly process, there are cases where a wearable camera is attached to an operator and an image of the assembly process is captured from the operator's line of sight. Since the wearable camera moves according to the movement of the operator's line of sight, the camera angle and the camera position change dynamically. That is, throughout the working time, the parent parts and the child parts do not fit into fixed positions in the captured image. As a result, it becomes difficult to detect the positions of the parent object and the child object.

[0006] Therefore, this specification discloses a work process management system that can accurately detect parent and child objects from images captured by a camera, such as a wearable camera, whose camera angle and position change. [Means for solving the problem]

[0007] This specification discloses a work process management system. This system comprises a camera, a parent component detection unit, and a child component detection unit. The camera captures a video with a dynamically changing imaging range. The parent component detection unit detects the parent component image region from the image captured by the camera. The child component detection unit detects the child component image region within the parent component image region. The parent component detection unit includes a parent component neural network that has been trained on images containing the parent component image region. The parent component neural network sets multiple candidate regions within the captured image and identifies the candidate region with the highest confidence score for the parent component image region. When the obtained confidence score is equal to or greater than the parent component threshold over a predetermined number of frames of the video, the parent component detection unit confirms the candidate region found in the image captured after the predetermined number of frames that is equal to or greater than the parent component threshold as the parent component image region.

[0008] According to the above configuration, even if the confidence score of a predetermined candidate region (e.g., bounding box) in the captured image exceeds the parent component threshold, the parent component image region is not determined unless that state persists for several frames. Because the camera angle and camera position change, the position of the parent component in the captured image is undefined. However, by determining the parent component image region based on the continuous maintenance of a high confidence score for the parent component, it is possible to extract a parent component with high confidence.

[0009] Furthermore, in the above configuration, the child component detection unit may include a child component neural network that has been trained using the parent component image region. In this case, the child component neural network outputs a confidence score indicating that the child component is not installed in the parent component image region, and a confidence score indicating that the child component is installed.

[0010] With the above configuration, both the state where the sub-components are not set and the state where they are set can be indicated by a bounding box or the like.

[0011] Furthermore, in the above configuration, the child component detection unit may put the child component neural network into a paused state until the parent component detection unit determines the parent component image region.

[0012] With the above configuration, the computational load on the system can be reduced by temporarily pausing the child component neural network.

[0013] Furthermore, in the above configuration, multiple parent components may be used in the assembly process. In this case, the parent component neural network can output confidence scores for different parent components. The parent component detection unit keeps the parent component neural network running continuously while the worker is performing their tasks.

[0014] With the above configuration, by constantly monitoring multiple parent components, it is possible to detect work errors such as a new parent component being assembled while a child component is still being assembled.

[0015] Furthermore, in the above configuration, the child component detection unit may specify a child component neural network that cooperates with each parent component image region. In the child component neural network that cooperates with the parent component image region in which the candidate region has been determined, if the confidence score indicating that a child component has already been placed is less than the child component threshold, a new candidate region may be determined for a parent component different from the parent component corresponding to the determined parent component image region. In such cases, the parent component detection unit outputs a warning signal.

[0016] With the above configuration, by constantly monitoring multiple parent components, it is possible to detect work errors such as assembling the next parent component while a child component is missing.

[0017] This specification also discloses a work process management system. This system comprises a camera, a parent region detection unit, and a child region detection unit. The camera captures a video with a dynamically changing imaging range. The parent region detection unit detects the parent image region from the image captured by the camera. The child region detection unit detects the child image region within the parent image region. The parent region detection unit includes a parent region neural network that has been trained on images containing the parent image region. The parent region neural network sets up multiple candidate regions within the captured image, and the parent region neural network outputs a confidence score for each candidate region. When the confidence score of any candidate region is equal to or greater than the parent region threshold over a predetermined number of frames of the video, the parent region detection unit determines the candidate region detected in the image captured after the predetermined number of frames that is equal to or greater than the parent region threshold as the parent image region.

[0018] According to the above configuration, for example, in the visual inspection of parts using a camera whose imaging range changes dynamically, such as a wearable camera, the parent region can be determined by maintaining a consistently high confidence score in that parent region, thereby enabling the extraction of a highly reliable parent image.

[0019] In the above configuration, the camera may image the object to be inspected visually. In this case, the object is a polyhedron. The parent region detection unit divides one face of the object into multiple parent image regions.

[0020] When an item being inspected is, for example, long in length, the way light hits it may differ on a single surface. With the above configuration, by dividing one surface of the item into multiple parent images, it becomes possible to extract parent image regions with less variation in brightness. [Effects of the Invention]

[0021] According to the work process management system disclosed herein, it is possible to accurately detect parent and child objects from images captured by cameras such as wearable cameras, whose camera angle and position change. [Brief explanation of the drawing]

[0022] [Figure 1] It is a diagram illustrating the hardware configuration of the work process management system according to this embodiment. [Figure 2] It is a diagram illustrating an assembly work process. [Figure 3] It is a diagram illustrating a captured image (1 / 3) by a wearable camera. [Figure 4] It is a diagram illustrating a captured image (2 / 3) by a wearable camera. [Figure 5] It is a diagram illustrating a captured image (3 / 3) by a wearable camera. [Figure 6] It is a diagram illustrating the functional blocks of the work process management system according to this embodiment. [Figure 7] It is a diagram illustrating a sub-component neural network. [Figure 8] It is a diagram illustrating the work process management flow (1 / 2) according to this embodiment. [Figure 9] It is a diagram illustrating the work process management flow (2 / 2) according to this embodiment. [Figure 10] It is a diagram illustrating the management screen displayed on the display in the assembly work process. [[ID=3X]] [Figure 11] It is a diagram illustrating a captured image of the maintenance work process. [Figure 12] It is a diagram illustrating the management screen displayed on the display in the maintenance work process. [Figure 13] It is a diagram illustrating a captured image of the appearance inspection work. [Figure 14] It is a diagram illustrating the functional blocks of the work process management system in the appearance inspection work. [Figure 15] It is a diagram illustrating the work process management flow in the appearance inspection work. [Figure 16] It is a diagram illustrating the management screen displayed on the display in the appearance inspection work.

Mode for Carrying Out the Invention

[0023] Note: There seems to be a typo in the original text where "[[ID=3X]]" should probably be something else. I've left it as is in the translation. 1. Overall Structure Figure 1 illustrates a work process management system according to this embodiment. As will be described later, this system manages the product assembly process. The system also determines whether or not there are any missing parts during product assembly. This work process management system consists of a process control device 10, a wearable camera 40, an input device 42, and a display device 44.

[0024] A wearable camera 40, an input device 42, and a display device 44 are connected to the process control device 10. The wearable camera 40 is a camera that captures video with a dynamically changing imaging range. The wearable camera 40 captures video along the worker's line of sight. For example, the wearable camera 40 may be a head-mounted camera. The display device 44 displays the image captured by the wearable camera 40. The input device 42 transmits an abnormality resolution signal to the assembly device control unit 28 (see Figure 6), which will be described later. The input device 42 may be, for example, a release button.

[0025] The process control device 10 is composed of, for example, a computer. The process control device 10 includes a CPU 11 as an arithmetic unit, and system memory 12 and storage 13 as storage means. The storage 13 may be a non-transient storage device such as a hard disk drive (HDD) or solid state drive (SSD). The process control device 10 also includes an input / output controller 14 that manages the input and output of information with external devices such as a wearable camera 40.

[0026] Furthermore, the process control device 10 includes a GPU 15 (Graphics Processing Unit), a frame memory 16, a RAMDAC 17 (Random Access Memory Digital-to-Analog Converter), and a display control unit 18 as means for processing the images captured by the wearable camera 40.

[0027] The GPU 15 is an image processing unit and is primarily used to detect parent and child components, which will be described later. The frame memory 16 is a memory device that stores images captured by the wearable camera 40 and processed by the GPU 15. The RAMDAC 17 converts the image data stored in the frame memory 16 into analog signals for the display unit 44, which is an analog display. For example, the display unit 44 is placed near the worker.

[0028] The display control unit 18 displays the image processed through the GPU 15, frame memory 16, and RAMDAC 17 on the display unit 44. For example, the display control unit 18 superimposes the image captured by the wearable camera 40 and candidate regions surrounding parent and child components, such as bounding boxes 60, 61A-61D (see Figure 10), on the display unit 44.

[0029] As will be described later, if the process control unit 26 (see Figure 6) determines that a part is missing, the display control unit 18 outputs a warning signal. Upon receiving the warning signal, a warning message is displayed on the display unit 44.

[0030] Figure 6 illustrates the functional blocks of the process control device 10 in a mixed form with the hardware blocks shown in Figure 1. This functional block diagram is composed of the CPU 11 executing a program that is stored, for example, in the storage device 13 or on a computer-readable, non-transient storage medium such as a DVD.

[0031] The process control device 10 comprises a parent component detection unit 20, a process control unit 26, an assembly device control unit 28, and a child component detection unit 30 as functional blocks. The parent component detection unit 20 comprises a parent component neural network 22 and a memory 24. The child component detection unit 30 comprises multiple child component neural networks 32-1 to 32-k and 34-1 to 34-m. Details of the parent component detection unit 20 and the child component detection unit 30 will be described later.

[0032] The process control unit 26 controls the product assembly process. Specifically, the process control unit 26 receives the detection results from the parent component detection unit 20 and the child component detection unit 30. Based on these detection results, the process control unit 26 determines whether or not there are any missing parts in the assembly process. For example, the process control unit 26 executes the work process control flow exemplified in Figures 8 and 9. Details of the work process control flow will be described later.

[0033] The assembly device control unit 28 controls the product assembly device. For example, as illustrated in Figure 3, the product is assembled on the workbench 100. The operation of the workbench and other functions are controlled by the assembly device control unit 28. For example, when the worker determines that the assembly is complete, the workbench 100 rotates, but when the process control unit 26 determines that a part is missing, the rotation of the workbench 100 stops.

[0034] 2. Parent component and child component In this specification, the terms "parent component" and "child component" are used to refer to the parts that make up a product. For example, a product may consist of multiple parent components and multiple child components. A parent component refers to a component to which other components are attached. A child component refers to the aforementioned "other components." The attachment points for other components on a parent component are formed on the exposed surface of the parent component. In other words, the attachment points for other components are formed on the surface of the parent component that can be imaged by the wearable camera 40. Furthermore, parent components are selected from the product's assembly parts that do not have any similarly shaped components. The absence of similar shapes means that it is difficult to classify them into separate classes using a neural network.

[0035] A component that covers the entire exposed surface of a parent component becomes the next parent component. For example, multiple parent components are used in the assembly of a product. These parent components may all be approximately the same size. For the next parent component as well, mounting points for child components are formed on the exposed surface.

[0036] For example, parent components are larger than child components. For instance, base components such as product housings are included in parent components. Child components include relatively small parts such as steel balls, screws, O-rings, and washers.

[0037] In the product assembly process, as illustrated in Figures 2 and 3, the first parent component 50 is placed on the workbench 100 (S10). Then, after multiple child components are assembled onto the first parent component (S12), a new second parent component is placed on top of the first parent component (S14). Then, multiple child components are assembled onto the second parent component (S16). If the product consists of three or more layers of parent components, steps S14 and S16 are repeated for each layer.

[0038] Since the child component is assembled on the parent component, the image area of ​​the child component is provided within the image area of ​​the parent component in the image captured by the wearable camera 40. As will be described later, in the work process management flow according to this embodiment, once the image area of ​​the parent component is determined, the presence or absence of the child component is determined only within that parent component image area. This avoids detecting child components that are outside the parent component image area in the captured image, for example, those that appear in a parts stocker.

[0039] 3. Images captured by a wearable camera Figures 3-5 illustrate images captured by the wearable camera 40. In the video captured by the wearable camera 40, the camera angle and position change according to the worker's line of sight. For example, Figure 3 illustrates the process of fixing the main component 50 to the workbench 100.

[0040] Referring to Figure 4, after fixing the main component 50 to the workbench 100, the worker shifts their gaze to retrieve the sub-component 56 (large O-ring) from the stocker 76. After assembling the sub-component 56 to the main component 50, the worker retrieves the sub-components 53 (steel ball), 54 (nut), 55 (small O-ring), and 57 (bolt) stored in stockers 73, 74, 75, and 77, and assembles them to the main component 50. Once all sub-components 53-57 are assembled to the main component 50, referring to Figure 5, a new main component 52 is placed on top of the main component 50. Then, the sub-components are assembled onto the main component 52.

[0041] 4. Parent component: Neural Network Referring to Figure 6, the parent component detection unit 20 detects the parent component image region from the image captured by the wearable camera 40. The parent component detection unit 20 includes a trained parent component neural network 22. From the perspective of image recognition, the parent component neural network 22 is composed of a convolutional neural network (CNN). For example, an object detection algorithm is implemented in the parent component neural network 22.

[0042] In object detection algorithms, multiple candidate regions are defined within an image. These candidate regions may be rectangular bounding boxes. The positional accuracy and class probability for each bounding box are then calculated. The product of the positional accuracy and class probability is then calculated as the confidence score. Finally, the candidate region (bounding box) with the highest confidence score is identified.

[0043] In other words, the parent component neural network 22 sets multiple candidate regions within the captured image and identifies the candidate region with the highest confidence score among them. The identified parent component image region will be referred to as the parent component image region below.

[0044] The parent component neural network 22 is pre-trained using training data. The training data includes images captured by the wearable camera 40 as input data. The output data includes the position and image region of the parent component in the captured image. If the product assembly involves multiple parent components, the position and image region of each parent component are provided as output data. In the example in Figure 6, there are n parent components required for the product assembly, and each node in the output layer outputs confidence scores for parent components 1 through n.

[0045] Furthermore, referring to Figures 3 and 5, the training data is processed so that the parent component neural network 22 does not misdetect the next parent component 52, which is prepared on the workbench 100, before it is assembled onto the parent component 50. For example, as illustrated in Figure 5, the next parent component 52 is stacked on top of the parent component 50, and the image region in which the child components 58, 58 protrude from the through holes 52E, 52F of the parent component 52 is provided as the correct image for the parent component 52 in the training data.

[0046] Furthermore, images captured by the wearable camera 40 are used as training data. In the images captured by the wearable camera 40, the camera angle and camera position change as the worker's gaze moves. In other words, the parent and child parts do not remain in fixed positions in the image. To put it another way, the parent and child parts can be ubiquitous in the captured image (they can exist at any position). For this reason, the positional accuracy of the object detection algorithm will be a value that does not differ significantly regardless of the position in the captured image. In other words, the class probability becomes the main element of the confidence score.

[0047] 5. Sub-component neural network Referring to Figure 6, the child component detection unit 30 detects child component image regions from the parent component image region. The child component detection unit 30 includes multiple child component neural networks 32-1 to 32-k, 34-1 to 34-m that have been trained using the parent component image region. Similar to the parent component neural network 22, the child component neural networks 32-1 to 32-k, 34-1 to 34-m consist of convolutional neural networks (CNNs) on which object detection algorithms are implemented.

[0048] Note that the sub-component neural networks 32-1~32-k and 34-1~34-m do not need to be set up for all sub-components required for product assembly. For example, sub-components whose absence can be detected during post-assembly functional testing may be excluded from detection by the neural networks.

[0049] The child component neural networks 32-1~32-k and 34-1~34-m are grouped by parent component (more specifically, by the parent component image region). That is, the k child components assembled to the first parent component 50 (see Figure 3) are linked to the first parent component 50 as a group of child component neural networks 32-1~32-k. More specifically, the image region of the first parent component 50 is linked to the child component neural networks 32-1~32-k. Similarly, the m child components assembled to the second parent component are linked to the second parent component 52 as a group of child component neural networks 34-1~34-m. More specifically, the image region of the second parent component 52 is linked to the child component neural networks 34-1~34-m.

[0050] Each of the child component neural networks 32-1~32-k and 34-1~34-m detects whether a single child component is installed in the parent component or not. In other words, each of the child component neural networks 32-1~32-k and 34-1~34-m outputs confidence scores for installed and uninstalled components in its output layer.

[0051] For example, referring to Figure 4, the parent component 50 has recesses 50A, 50B, 50C and a ring groove 50D formed therein as mounting locations for the child components. A child component 53 is placed in recess 50A. Another child component 53 is placed in recess 50B. A child component 55 (small O-ring) is placed in recess 50C, and a child component 56 (large O-ring) is placed in the ring groove 50D.

[0052] For example, the child component neural network 32-1 that monitors the installation process of recess 50A learns from images of both the state in which the child component 53 is placed in recess 50A and the state in which it is not placed, as training data. For example, as input data, an image region of the parent component 50 (see Figure 7) is cropped. The output data then includes the position and image region of recess 50A within that image region. The input data includes images of when the child component 53 is placed in recess 50A and images of when it is not placed.

[0053] When the image region of the parent component 50 is input to the child component neural network 32-1, which has been trained with such training data, the output layer outputs both the confidence score for the unset component and the confidence score for the set component, as illustrated in Figure 7.

[0054] 7.Work process management screen Figure 10 shows an example of a work process management screen. The work process management screen is displayed on the display unit 44 (see Figure 6). On the work process management screen, annotation results of parent and child components are superimposed on the image captured by the wearable camera 40.

[0055] For example, once the parent component neural network 22 identifies (detects) the parent component 50, after a confirmation period described later, the area surrounding the parent component 50, i.e., the bounding box 60, is displayed. Following the display of the bounding box 60, bounding boxes 61A-61D indicating the child components (more precisely, indicating the placement locations of the child components) are displayed.

[0056] Furthermore, the work process management screen displays, for example, a work record field 62 in the upper left. The work record field 62 displays the date and time of the work, the number of units manufactured, the number of layers of the parent part, the assembly time, etc. Also, the parts field 64 is displayed in the upper right. The parts field lists the parent part 50, whose bounding box has been determined, and its child parts. For example, the text corresponding to an installed child part, such as child part 53, is displayed in green in the parts field 64. In Figure 10, the green display format is shown alternatively with an underline. Also, child parts that have not been installed are displayed in red. In Figure 10, the display format without an underline is shown alternatively.

[0057] As will be described later, if a new parent component 52 is stacked on the parent component 50 before all the child components that are to be assembled to the parent component 50 have been assembled to the parent component 50, the display unit 44 will display a warning. For example, the work process management screen will flash red. This allows the worker to recognize that a component has been missing.

[0058] 6. Work Process Management Flow Figures 8 and 9 illustrate the work process management flow executed by the process control device 10 (Figure 6). In this example, the number of parent parts required for product assembly is assumed to be two.

[0059] Images (videos) captured by the wearable camera 40 are transmitted to the input layer of the parent component detection unit 20 of the process control device 10. The parent component detection unit 20 refers to the output layer of the parent component neural network 22 and determines whether the confidence score of the first parent component 50 (see Figure 3) is equal to or greater than a predetermined threshold (parent component threshold) (S20). The parent component threshold is set to a value of, for example, 0.80 or higher.

[0060] If the reliability score of the first parent component 50 is less than the parent component threshold, the flow returns to step S20. If the reliability score of the first parent component 50 is equal to or greater than the parent component threshold, the parent component detection unit 20 determines whether the reliability score of the first parent component 50 remains equal to or greater than the parent component threshold for a predetermined number of frames of the video (S22). For example, the parent component detection unit 20 determines whether the reliability score of the first parent component 50 remains equal to or greater than the parent component threshold for a number of frames corresponding to one second. For example, the parent component detection unit 20 stores the reliability score of the parent component 50 for several frames in memory 24.

[0061] If the confidence score of the first parent component 50 falls below the parent component threshold before reaching a predetermined number of frames, the flow returns to step S20. In other words, the frame count is reset to 0.

[0062] If the confidence score of the first parent component 50 remains above the parent component threshold for a predetermined number of frames, the process of determining the parent component image region begins. That is, in the frame following the predetermined number of frames, the parent component detection unit 20 determines the bounding box 60 (see Figure 1) where the confidence score of the first parent component 50 is above the parent component threshold as the parent component image region. Furthermore, the display of the bounding box 60 of the first parent component 50 continues from this determination process until the parent component 50 leaves the field of view, or until the determination process for the next parent component 52.

[0063] The above detection results are transmitted from the parent component detection unit 20 to the display control unit 18. As illustrated in Figure 10, the display control unit 18 superimposes the bounding box 60 (candidate region) of the parent component 50 onto the captured image (S24). The display control unit 18 may also display the class (e.g., parent component 1) and confidence score (e.g., 0.93) near the bounding box 60.

[0064] Next, the data of the confirmed parent component image region and the class "Parent Component 1" are transmitted to the child component detection unit 30. That is, until the parent component image region is confirmed, the child component detection unit 30 puts the child component neural networks 32-1~32-k and 34-1~34-m into a dormant state. By intermittently activating the child component neural networks 32-1~32-k and 34-1~34-m in this manner, the computational load on the process control device 10 is reduced.

[0065] The child component detection unit 30 detects child components within the bounding box of the first parent component 50 (S26). That is, the child component detection unit 30 inputs data of the parent component image region to each input layer of the child component neural networks 32-1 to 32-k that cooperate with the first parent component 50, which corresponds to class "parent component 1". As a result, the position and confidence score of each child component are output to each output layer of the child component neural networks 32-1 to 32-k. For example, immediately after the image region of parent component 1 is determined, none of the child components are assembled to the parent component 50, so a high confidence score is output for class "not installed" in all of the child component neural networks 32-1 to 32-k.

[0066] Referring to Figure 10, the display control unit 18 superimposes the bounding boxes 61A to 61D of each child component onto the captured image based on the bounding box position information output from the child component neural networks 32-1 to 32-k. The display control unit 18 may also display the respective class and confidence score in the vicinity of the bounding boxes 61A to 61D.

[0067] For example, the output layers of the child component neural networks 32-1 to 32-k consist of classes "installed" and "not installed". If child components 53-57 are not installed in the first parent component 50, a high confidence score is output from the "not installed" class of the child component neural networks 32-1 to 32-k. If child components 53-57 are installed in the first parent component 50, a high confidence score is output from the "installed" class of the child component neural networks 32-1 to 32-k. In either case, a high confidence score is output from the child component neural networks 32-1 to 32-k, so the bounding boxes 61A to 61D of each child component are always superimposed within the bounding box 60 of the first parent component 50.

[0068] The display control unit 18 reflects the output results of the parent component neural network 22 and the child component neural networks 32-1 to 32-k in the component column 64 of the captured image. For example, as described above, the output layer of the parent component neural network 22 outputs a high confidence score for the parent component 50. In this case, the text indicating the parent component 50 is displayed in green (represented by an underline in Figure 10) in the component column 64. On the other hand, corresponding to the child component neural networks 32-1 to 32-k, which output a high confidence score for the class "Not Installed," the text indicating each of the child components is displayed in red.

[0069] While the worker's assembly work progresses, the child component detection unit 30 repeatedly performs calculations using the child component neural networks 32-1 to 32-k, for example, in accordance with the clock of the CPU 11 (see Figure 1). The parent component detection unit 20 also continuously repeats calculations using the parent component neural network 22, in accordance with the clock of the CPU 11.

[0070] The child component detection unit 30 determines whether the confidence scores of the child component neural networks 32-1 to 32-k for the class "installed" are all equal to or above a threshold (child component threshold) (S28). The child component threshold is set to a value of, for example, 0.80 or higher.

[0071] If the confidence scores for the child component neural networks 32-1 to 32-k of the "Installed" class are all equal to or greater than the child component threshold, the child component detection unit 30 transmits a detection result to the process control unit 26 and the display control unit 18 indicating that the assembly to the first parent component 50 has been successfully completed.

[0072] As described above, the parent component neural network 22 is constantly running while the operator is working, and the confidence score for each class is constantly output to the output layer. The parent component detection unit 20 determines whether the confidence score of class "parent component 2" is equal to or greater than the parent component threshold (S30). If the confidence score of class "parent component 2" is less than the parent component threshold, the flow returns to step S30.

[0073] As shown in Figure 5, when the operator places the second parent component 52 on the first parent component 50, the confidence score of class "parent component 2" corresponding to the second parent component 52 becomes equal to or greater than the parent component threshold in the output layer of the parent component neural network 22.

[0074] In response, the parent component detection unit 20 determines whether the reliability score of the second parent component 52 remains above the parent component threshold for a predetermined number of frames in the video (S32). For example, the parent component detection unit 20 determines whether the reliability score of the second parent component 52 remains above the parent component threshold for a number of frames corresponding to one second. For example, the parent component detection unit 20 stores the reliability score of the parent component 52 for several frames in the memory 24.

[0075] If the confidence score of the second parent component 52 falls below the parent component threshold before reaching a predetermined number of frames, the flow returns to step S30. If the confidence score of the second parent component 52 remains above the parent component threshold for a predetermined number of frames, the process moves to determining the parent component image region. That is, in the frame following the predetermined number of frames, the parent component detection unit 20 determines the bounding box (candidate region) where the confidence score of the second parent component 52 is above the parent component threshold as the new parent component image region.

[0076] This detection result is transmitted from the parent component detection unit 20 to the display control unit 18. The display control unit 18 superimposes the bounding box (candidate region) of the parent component 52 onto the captured image (S34). The display control unit 18 may also display a confidence score near the bounding box.

[0077] Next, the data of the confirmed parent component image region and the class "Parent Component 2" are transmitted to the child component detection unit 30. The child component detection unit 30 detects child components within the bounding box of the second parent component 52 (S36). The child component detection unit 30 inputs the image region data of the parent component 52 to each input layer of the child component neural network 34-1 to 34-m, which is linked to the class "Parent Component 2," i.e., the second parent component 52. As a result, the position and confidence score of each child component are output to each output layer of the child component neural network 34-1 to 34-m.

[0078] Furthermore, based on the bounding box position information output from the child component neural networks 34-1 to 34-m, the display control unit 18 superimposes the bounding boxes of each child component onto the captured image. The display control unit 18 may also display the respective class and confidence score near the bounding boxes of the child components. In addition, the display control unit 18 reflects the output results of the parent component neural network 22 and the child component neural networks 34-1 to 34-m in the component column 64 of the captured image.

[0079] The sub-component detection unit 30 then determines whether the confidence scores for the class "installed" for the sub-component neural networks 34-1 to 34-m are all equal to or greater than the sub-component threshold (S38).

[0080] If any of the confidence scores for the "Installed" class are below the child component threshold, the flow returns to step S38. For child component neural networks 34-1 to 34-m, if the confidence scores for the "Installed" class are all equal to or greater than the child component threshold, the child component detection unit 30 transmits a detection result indicating that assembly to the second parent component 52 has been successfully completed to the process control unit 26 and the display control unit 18. The process control unit 26 receives the detection result and determines that assembly is complete.

[0081] Returning to step S28, the parent component detection unit 20 determines whether the next parent component 52 is mistakenly placed on the first parent component 50 during the assembly of the child component to the first parent component 50. In step S28, if any of the confidence scores for the class "Installed" of the child component neural networks 32-1 to 32-k are below the child component threshold, the parent component detection unit 20 determines whether there is an assembly defect. That is, the parent component detection unit 20 determines whether the confidence score of the class "Parent Component 2" of the parent component neural network 22 is equal to or greater than the parent component threshold (S40). If the confidence score of the class "Parent Component 2" is below the parent component threshold, the flow returns to step S26.

[0082] When the confidence score of class "parent component 2" becomes equal to or greater than the parent component threshold, the parent component detection unit 20 determines whether the confidence score of the second parent component 52 remains equal to or greater than the parent component threshold for a predetermined number of frames of the video (S42). For example, the parent component detection unit 20 determines whether the confidence score of the second parent component 52 remains equal to or greater than the parent component threshold for a number of frames corresponding to 1 second.

[0083] If the reliability score of the second parent component 52 falls below the parent component threshold before reaching a predetermined number of frames, the flow returns to step S26. If the reliability score of the second parent component 52 remains above the parent component threshold for a predetermined number of frames, that is, if a candidate area (partition) for a new parent component 52 is determined, the parent component detection unit 20 transmits a warning signal to the process control unit 26 and the display control unit 18. The process control unit 26 transmits an abnormality command to the assembly device control unit 28. The assembly device control unit 28 interrupts the assembly work and outputs an alarm (S44). The display control unit 18 also causes the display unit 44 to display a warning.

[0084] Thus, in the work process management system according to this embodiment, the parent component is determined when a high confidence score is maintained across multiple frames of the video. This makes it possible to identify components with high accuracy even in images captured by a wearable camera 40, where the camera angle and camera position are undefined.

[0085] 7. Management Process for Maintenance Work Operations In the above-described embodiment, the product assembly process was the subject of management, but the work process management system according to this embodiment is not limited to this embodiment. For example, the work process management system according to this embodiment can also be applied to maintenance work, as illustrated in Figure 11.

[0086] For example, in product maintenance, it may be necessary to disassemble a part of the product and then reassemble it. Figure 11 shows an example of an image captured by a wearable camera 40. In the maintenance work illustrated in Figure 11, the parent component 150 is removed from the parent component 152.

[0087] The main component 150 is fastened to the main component 152 by the sub-components 153A-153M (bolts). Therefore, during maintenance work, after the sub-components 153A-153M are removed from the main component 152, they must be reattached to the main component 152.

[0088] In such cases, the work management system according to this embodiment is used to detect detachment or insufficient fastening of sub-parts 153A-153M using images captured by the wearable camera 40.

[0089] For example, referring to Figure 6, the parent component detection unit 20 is configured with parent components that are removed from the product during maintenance work as a class. The child component detection unit 30 implements a neural network for each child component attached to the parent component. The output layer of this neural network is configured with classes such as "installed" and "not installed".

[0090] By using such a work management system for maintenance work, an image like that shown in Figure 12 is displayed on the display unit 44 (see Figure 6). In other words, the work process management screen exemplified in Figure 12 overlays the annotation results of parent and child parts onto the image captured by the wearable camera 40.

[0091] For example, once the parent component neural network 22 identifies (detects) the parent component 150, a candidate region surrounding the parent component 150, i.e., a bounding box 160, is displayed after a confirmation period described later. Following the display of the bounding box 160, bounding boxes 161A-161M indicating the child components (more precisely, indicating the placement locations of the child components) are displayed.

[0092] Furthermore, the work record field 162 is displayed, for example, in the upper left of the work process management screen. The parts field 164 is displayed, for example, in the upper right of the work process management screen. The parts field 164 lists the parent part 150, whose candidate area (bounding box) has been determined, and its child parts 153A-153M (see Figure 11). For example, the text corresponding to an installed child part, such as child part 153A-153C, is displayed in green in the parts field 164. Figure 12 shows the green display format with an underline as an alternative. Also, child parts that have not been installed are displayed in red. Figure 12 shows the display format without an underline as an alternative.

[0093] Thus, if any of the child parts that are linked to the parent part 150, or in other words, that should be assembled to the parent part 150, are not installed, a red bounding box will be displayed on the work process management screen. This allows the worker to recognize that a part is missing.

[0094] In this maintenance work process management, the same flow as in Figures 8 and 9 is executed. That is, if the reliability score of the first parent component 150 is equal to or greater than a predetermined threshold (parent component threshold), it is determined whether the reliability score of the first parent component 150 remains equal to or greater than the parent component threshold for a predetermined number of frames of the video (S22).

[0095] If the confidence score of the first parent component 150 falls below the parent component threshold before reaching a predetermined number of frames, the flow returns to step S20. In other words, the frame count is reset to 0. On the other hand, if the confidence score of the first parent component 150 remains above the parent component threshold for a predetermined number of frames, the process moves to the confirmation process of the parent component image region. That is, in the frame following the predetermined number of frames, the parent component detection unit 20 confirms the bounding box 160 (see Figure 1) where the confidence score of the first parent component 150 is above the parent component threshold as the parent component image region. Furthermore, the display of the bounding box 160 of the first parent component 150 continues from this confirmation process until the parent component 150 leaves the field of view, or until the confirmation process of the next parent component. Child component detection is then performed within the confirmed parent component image region.

[0096] 8. Visual inspection process for items Figures 11-16 show an example of applying the work management system according to this embodiment to a visual inspection operation. Figure 11 shows an example of an image captured by a wearable camera 40. The wearable camera 40 is a camera that captures video with a dynamically changing imaging range. The wearable camera 40 is, for example, a head-mounted camera. In a visual inspection operation, for example, the presence or absence of defects on the surface of an item 250 that is the subject of the visual inspection is determined by visual inspection.

[0097] In the example shown in Figure 11, defective portions 253A and 253B are formed on article 250. For example, defective portion 253A is formed on the front surface 251A of article 250. Defective portion 253B is formed on the upper surface 251B of article. Defective portions 253A and 253B may be, for example, discoloration abnormalities such as burning that may occur during resin molding.

[0098] The article 250 subject to visual inspection is, for example, a single component. Furthermore, the article 250 is a polyhedron, for example, a rectangular parallelepiped. In addition, the article 250 has multiple three-dimensional feature parts. For example, multiple insertion holes 250A-250E are drilled in the upper surface 251B of the article 250.

[0099] In the visual inspection process, the entire surface of item 250 is inspected for defects. This means that not only the top surface 251B and front surface 251A of item 250 are visually inspected, but also the bottom surface, rear surface, and both sides, which are hidden surfaces in Figure 11, to determine whether or not there are any defects.

[0100] In this type of visual inspection work, the work management system has the configuration shown in Figure 14. The process control device 10 comprises a display control unit 18, a process control unit 26, a parent area detection unit 120, and a child area detection unit 130. Image data captured from the wearable camera 40 is input to the process control device 10. The display control unit 18 is connected to the display unit 44.

[0101] The parent region detection unit 120 includes a parent region neural network 122. The parent region neural network 122 is trained using images that include the parent image region. The parent region neural network 122 has nodes in its output layer that output confidence scores for multiple parent regions.

[0102] The parent region neural network 122 sets multiple candidate regions (sections) within the captured image and outputs a confidence score for each candidate region. For example, as illustrated in Figure 16, one face of the item 250 is divided into multiple sections. For example, in the example in Figure 16, the top surface 251B of the item 250 is divided into multiple bounding boxes 260A-260E (candidate regions). Such division is performed based on characteristic shapes, such as insertion holes 250A-250E.

[0103] For example, if the item 250 is a long object, the brightness may differ between the front and back sides of the top surface 251B depending on the imaging environment. In such cases, if the top surface 251B is partitioned as a single parent image region, there is a risk that the relatively darker areas may be mistakenly identified as defects 261. Therefore, in the visual inspection process according to this embodiment, one surface of the item 250 is divided into multiple parent image regions.

[0104] The parent image region is assigned to all faces of the object 250. In other words, the parent region neural network is given images of all faces of the object 250 as training data. Furthermore, for each face, images under various imaging conditions such as different angles, magnifications, and partial loss due to being covered by fingers, etc., are provided as training data. Training data images can also be obtained, for example, by applying geometric transformations, image quality changes, cropping, etc., to existing captured images.

[0105] The child region detection unit 130 detects child image regions within the parent image region. The child region detection unit 130 comprises multiple child region neural networks 232-1 to 232-k. Each child region neural network 232-1 to 232-k includes, for example, one node that outputs a confidence score as an output layer.

[0106] For example, the child-region neural networks 232-1 to 232-k are each trained using images (child image regions) of defect areas 261 with different shapes and colors as training data. In other words, when a defect area 261 of a specific shape and color is included in the parent image region, a high confidence score is output from the output layer of the child-region neural network 232, which has been trained based on that defect area 261 of that shape and color.

[0107] 9. Process management flow for visual inspection work Figure 15 illustrates the workflow management flow for visual inspection performed by the process control device 10. In this example, the item to be inspected is assumed to be a single component.

[0108] First, the parent region counter i is set to an initial value of 1 (S100). Then, the image (video) captured by the wearable camera 40 is transmitted to the input layer of the parent region detection unit 120 of the process control device 10 (see Figure 14).

[0109] The parent region detection unit 120 refers to the output layer of the parent region neural network 122 and determines whether the confidence score of class "parent region i" is equal to or greater than a predetermined threshold (parent region threshold). The parent region threshold is set to a value of, for example, 0.80 or higher.

[0110] If the confidence score of class "parent region i" is equal to or greater than the parent region threshold, the parent region detection unit 120 determines whether the confidence score of class "parent region i" remains equal to or greater than the parent region threshold for a predetermined number of frames of the video (S102). For example, the parent region detection unit 120 determines whether the confidence score of class "parent region i" remains equal to or greater than the parent region threshold for a number of frames corresponding to one second. For example, the parent region detection unit 120 stores the confidence score of class "parent region i" for several frames in memory 24.

[0111] If the confidence score of class "parent region i" falls below the parent component threshold before reaching a predetermined number of frames, the parent region detection unit 120 determines whether the parent region counter i is at its maximum value n (S116). If the parent region counter i is at its maximum value n, the flow is returned to the starting point.

[0112] If the parent region counter i is not at its maximum value n, the parent region detection unit 120 increments the parent region counter i (S120). The flow then returns to step S102. In other words, it is determined whether the confidence score of the next parent region is equal to or greater than the parent region threshold for several frames.

[0113] In step S102, if the confidence score of class "parent region i" is equal to or greater than the parent region threshold over a predetermined number of frames, the process of confirming the parent region image area proceeds. That is, in the frame following a predetermined number of frames, the parent region detection unit 120 confirms the bounding box, for example, bounding box 260A (see Figure 16), whose confidence score is equal to or greater than the parent region threshold, as the parent region image area. Furthermore, the display of the bounding box 260A of the parent image area continues from this confirmation process until the parent component 50 leaves the field of view.

[0114] The above detection results are transmitted from the parent region detection unit 120 to the display control unit 18. As illustrated in Figure 16, the display control unit 18 superimposes the bounding box 260A (candidate region) of the parent region i onto the captured image (S104). The display control unit 18 may also display a class (e.g., parent region 1-1) and a confidence score (e.g., 0.96) near the bounding box 260A.

[0115] Upon confirmation of the parent image region, the child region detection unit 130, which was in a dormant state, is activated. The child region detection unit 130 sets the child region counter a to an initial value of 1 (S106). Next, the child region detection unit 130 detects child region a within the bounding box 260A of the parent region i. In other words, the child region detection unit 130 inputs the parent region image data into the child region neural network 232-1 and outputs a confidence score. In the example in Figure 16, since the bounding box 260A does not contain any defective areas 261, the child region neural network 232-1 outputs a relatively low confidence score.

[0116] The child region detection unit 130 determines whether the confidence score of the child region neural network 232-1 is equal to or greater than the child region threshold (S110). The child region threshold is set to a value of, for example, 0.80 or higher.

[0117] If the confidence score of the child region neural network 232-1 is less than the child region threshold, the child region detection unit 130 determines whether the child region counter a is at its maximum value k (S114). If the child region counter a is at its maximum value k, the process proceeds to step S116, which checks the parent region counter.

[0118] On the other hand, if the child region counter a is different from the maximum value k, the child region detection unit 130 increments the child region counter a (S118). The flow then returns to step S108, where it is determined whether or not there are defects of other shapes or colors within the parent region.

[0119] In step S110, if the confidence score of the child region neural network 232-a (a=1~k) is equal to or greater than the child region threshold, the child region detection unit 130 highlights child component a (a=1~k) within the bounding box 260i (i=1~n) of the parent region i (S112). For example, as illustrated in Figure 16, the bounding box 261A of child component 1 is displayed in the parent region 260D. Alternatively, the image region (feature area) that is the basis for the relatively high confidence score output for child component 1 is displayed by a heatmap.

[0120] The process control unit 26 receives the detection result for sub-component a. The process control unit 26 sends an abnormality notification to an external party if the confidence score of the sub-region neural network 232-a (a=1~k) is equal to or greater than the sub-region threshold. For example, the process control unit 26 emits a warning sound.

[0121] After the highlighting in step S112, the child region detection unit 130 determines whether the child region counter a is at its maximum value in order to detect the next shape or color defect (S114).

[0122] By performing the visual inspection process described above, the presence or absence of all child regions (defects) is determined for all parent regions. Furthermore, when the article 250 is rotated in the captured image, a visual inspection is performed on the main surface of the article 250 in the captured image. [Explanation of Symbols]

[0123] 10 Process control device, 20 Parent component detection unit, 22 Parent component neural network, 26 Process control unit, 28 Assembly device control unit, 30 Child component detection unit, 32-1~32-k, 34-1~34-m Child component neural network, 40 Wearable camera, 50, 52 Parent component, 53-57 Child component, 60, 61A-61D Bounding box (candidate region).

Claims

1. A camera that captures video with a dynamically changing imaging range, A parent component detection unit detects the parent component image region from the image captured by the aforementioned camera, A child component detection unit that detects the child component image region in the parent component image region, Equipped with, The parent component detection unit includes a parent component neural network that has been trained using an image that includes the parent component image region. The parent component neural network sets multiple candidate regions within the captured image and identifies the candidate region with the highest confidence score among the parent component image regions. When the calculated confidence score is equal to or greater than the parent component threshold over a predetermined number of frames of the video, the parent component detection unit determines the candidate region detected in the captured image after the predetermined number of frames that is equal to or greater than the parent component threshold as the parent component image region. Work process management system.

2. A work process management system according to claim 1, The child component detection unit includes a child component neural network that has been trained using the parent component image region. The child component neural network outputs the confidence score for when the child component is not installed in the parent component image region, and the confidence score for when the child component is installed. Work process management system.

3. A work process management system according to claim 2, Until the parent component detection unit determines the parent component image region, the child component detection unit puts the child component neural network into a paused state. Work process management system.

4. A work process management system according to claim 3, Multiple parent components are used in the assembly process. The parent component neural network is capable of outputting the respective confidence scores for different parent components. The parent component detection unit keeps the parent component neural network running continuously while the worker is working. Work process management system.

5. A work process management system according to claim 4, The child component detection unit specifies the child component neural network that cooperates with each of the parent component image regions, In the child component neural network that works in conjunction with the parent component image region in which the candidate region has been determined, when the confidence score of the child component already installed is less than the child component threshold, the parent component detection unit outputs a warning signal when a new candidate region is determined for a parent component different from the parent component corresponding to the determined parent component image region. Work process management system.

6. A camera that captures video with a dynamically changing imaging range, A parent region detection unit that detects a parent image region from an image captured by the aforementioned camera, A child region detection unit for detecting child image regions in the parent image region, Equipped with, The parent region detection unit includes a parent region neural network that has been trained using an image that includes the parent image region. The parent region neural network sets multiple candidate regions within the captured image and outputs a confidence score for each of the candidate regions. When the confidence score of any of the candidate regions is equal to or greater than the parent region threshold over a predetermined number of frames of the video, the parent region detection unit determines the candidate region detected in the captured image after the predetermined number of frames that is equal to or greater than the parent region threshold to be the parent image region. Work process management system.

7. A work process management system according to claim 6, The aforementioned camera captures images of the item to be inspected visually, The aforementioned article is a polyhedron, The parent region detection unit divides one surface of the article into a plurality of parent image regions. Work process management system.

Images