Work support system, work support method, and program
The work support system uses cameras to capture and analyze worker movements, automatically advancing work model videos when correlations are detected, addressing inefficiencies in conventional methods by enhancing worker training efficiency and accuracy.
Patent Information
- Application Number
- JP2023084364
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-23
- Publication Date
- 2024-12-05
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Conventional methods for supporting new workers in performing multiple work steps through video display are inefficient, requiring frequent frame-by-frame advancement to understand the next step, which is cumbersome and time-consuming.
A work support system that uses a ceiling camera and a rear camera to capture images of a skilled worker's process, extracts feature amounts, and automatically advances a work model video frame-by-frame when a correlation is detected, reducing the need for manual advancement.
Provides highly convenient work support by allowing new workers to follow the work process seamlessly, improving efficiency and accuracy by minimizing manual frame-by-frame operations and enabling real-time feedback on work progress.
Smart Images

Figure 2024167994000001_ABST
Abstract
Description
[Technical field]
[0001] An embodiment of the present invention relates to a work assistance system, a work assistance method, and a program. [Background technology]
[0002] Conventionally, in a factory or the like, a worker may perform a series of work or tasks consisting of multiple work steps (hereinafter, work of multiple work steps). In such cases, in order to support the work of a new worker, for example, a method is used in which a video of the work performed by a skilled worker (veteran worker) is displayed and the new worker works while watching the video. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] JP 2020-17283 A [Patent Document 2] International Publication No. 2017 / 150293 [Patent Document 3] JP 2018-92478 A [Patent Document 4] JP 2012-216221 A [Patent Document 5] JP 2001-331513 A Summary of the Invention [Problem to be solved by the invention]
[0004] However, in the above-mentioned conventional technology, when a video is normally displayed, a series of work steps are displayed continuously. Therefore, for example, when a new worker wants to know the content of the next work step, the display of the video of that work step may have already ended.
[0005] As a countermeasure, there is a method in which the new worker stops the video at the current work process and then operates the frame-by-frame operation when it is time to understand the next work process. However, with this method, the new worker needs to operate the frame-by-frame operation every time a work process is completed while working, which is inconvenient.
[0006] Therefore, an embodiment of the present invention has been made in consideration of the above circumstances, and has an objective of providing a work assistance system, a work assistance method, and a program that can provide highly convenient work assistance to a worker performing work involving multiple work processes. [Means for solving the problem]
[0007] The work support system of the embodiment includes an acquisition unit that acquires captured images from a camera that captures images of workers performing a plurality of work processes; a judgment unit that judges whether or not there is a predetermined correlation or more between features of the captured images for the most recent past work process and features of a sample captured image obtained by previously photographing a sample maker performing the work; and a control unit that displays the sample captured image on a display unit viewable by the worker while working, and if the judgment unit determines that there is the predetermined correlation or more, displays the sample captured image of the next work process on the display unit, and if the judgment unit determines that there is not the predetermined correlation or more, displays the sample captured image of the next work process on the display unit when a frame-by-frame operation is performed. [Brief description of the drawings]
[0008] [Figure 1] FIG. 1 is a diagram showing an outline of the overall configuration of a work support system according to an embodiment. [Diagram 2] FIG. 2 is a diagram showing a state of a work place according to the embodiment. [Diagram 3] FIG. 3 is a diagram illustrating an example of a display screen on the smartphone according to the embodiment. [Figure 4] FIG. 4 is an explanatory diagram relating to the determination of the frame advance timing of the display screen in the embodiment. [Diagram 5] FIG. 5 is a diagram showing an example of transition of a display image in a comparative example (when no frame-feed timing control is performed). [Figure 6] FIG. 6 is a diagram showing a first transition example of a display image in the embodiment (when timing control of frame-by-frame advance is performed). [Figure 7] FIG. 7 is a diagram showing a second transition example of a display image in the embodiment. [Figure 8] FIG. 8 is a flowchart showing the process of shooting and editing a working model moving image in the embodiment. [Figure 9] FIG. 9 is a flowchart showing the process of monitoring the work of a worker and when an automatic frame advance error occurs in this embodiment. [Figure 10] FIG. 10 is a diagram showing an example of a display screen in a case where both a working model moving image and a worker are displayed in the embodiment. [Figure 11] FIG. 11 is a diagram showing an example of a display screen in a case where both a working model moving image and an image of a skilled worker captured by a ceiling camera are displayed in the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0009] Hereinafter, embodiments of a work assistance system, a work assistance method, and a program according to the present invention will be described with reference to the accompanying drawings.
[0010] In order to facilitate understanding of the embodiment, the prior art will be described in detail once again. Due to the need to avoid country risk and the ongoing depreciation of the yen, there is a trend of returning production to Japan and back to factories. On the other hand, multi-skilling has been a long-standing issue at factory sites, but for many years workers have relied on oral Technology Transfer (TT) from the work supervisor and paper manuals to learn new skills.
[0011] For example, in the past, the way to teach new workers at factories was for the work supervisor to teach them each step of the work process step by step so that they would remember the work. The new worker would be taught everything according to the paper manual. After that, they would try to do the work themselves, but at first they would make many mistakes, and the work supervisor would have to teach them again each time.
[0012] In tasks with many steps, mistakes are likely to occur even after learning. For new workers, self-study using paper manuals is difficult when the tasks are slightly different even if they look similar. It has been said for a long time that people are moving away from printed matter, and this is thought to be due to the fact that paper manuals are difficult to understand and to get started with.
[0013] New workers could check the paper manual every time the work process progresses, but this takes time and is tedious, making it difficult for them to continue doing so. Rather, they tend to be pressed for time and proceed with the work based on their own assumptions without checking the paper manual. As a result, problems arise in maintaining and improving product quality due to work mistakes caused by missing tasks or failing to check.
[0014] Furthermore, there is the task of filling out paper checklists. After completing a work process, the worker must switch from a tool to a writing implement and fill out the form, which is time-consuming and laborious. In this case, too, the worker may be reluctant to take the time to fill out the checklist later, instead of filling out the checklist on paper one by one after each process.
[0015] Another existing technology is AR (Augmented Reality). One example of a consumer product that applied AR and became a hit is the smartphone game "Pokemon (registered trademark) GO." The characters from the game are displayed on the smartphone screen superimposed on the scenery in front of the user, creating a sense of reality as if Pokemon had appeared in the real world. In addition, by using location information from the Global Positioning System (GPS), the game creates a level of expansion and immersion that is far greater than games that are completed solely with on-screen CG (Computer Graphics), and has become a hit around the world.
[0016] By superimposing the real scenery seen through special glasses such as AR glasses that use this type of AR, explanations and instructions can be given, allowing users to understand the situation and select and execute the next action without having to look at tourist booklets or paper manuals. For example, see JP 2021-51620 A as a reference.
[0017] xReality is a collective term that includes virtual reality (VR) and mixed reality (MR), which analyzes the user's field of vision by integrating a PC (Personal Computer) and a display, such as Microsoft's HoloLens. Both technologies fuse the world the user sees with digital images, and are fields in which technology is emerging and advancing rapidly. The introduction of these new technologies requires expensive HW (hardware) and SW (software) resources for real-time response. In addition, some workers may become sick from viewing the real world through special goggles. In addition, the viewing angle may be insufficient. These hurdles make it difficult to introduce these technologies in many workplaces.
[0018] Another method is to replace paper manuals with videos of experienced workers (example: a sample manufacturer) performing their work, and have new workers watch and learn from them. If such work model videos are played normally, a series of work steps will be played at once. This makes it difficult for new workers to understand the next work step, which is what they most want to know, while they are working. In the end, even with a video, they end up having to advance frame by frame to check the instructions for the next work step. During work, it is necessary to perform the frame-by-frame advance operation for each work step, which is cumbersome and takes extra time.
[0019] For example, suppose the incoming inspection and adjustment of a K1 size circuit board is carried out over 100 work steps, taking just under two hours. In one day, four circuit boards are adjusted over eight hours. If one work instruction screen is displayed per step, one frame advance is required for each work step, so in one day, the video frame advance operation must be performed 4 x 100 times = 400 times.
[0020] It is necessary to move away from paper manuals and improve the decline in worker operability and accuracy that occurs incidentally due to paper manuals. To achieve this, it is not enough to simply shoot a video and show it to the worker; it is necessary to introduce a mechanism that allows the worker to view the work model video at an appropriate time. In the following embodiment, a shooting environment with two shooting devices is considered: an image sensor (ceiling camera) installed on the ceiling and a camera installed close to the worker.
[0021] There are several considerations to be made when implementing this system. The first is the installation location of the image sensor. At a work site, the easiest place to install it is on the ceiling, from the viewpoint of work and safety. In some cases, it is possible to create a task model video using an image sensor installed on the ceiling, but in many cases, such as when performing operations on a control panel, the ceiling creates a shadow in the field of view, making it impossible to create a task model video.
[0022] To film operations on the operation panel, it is necessary to use a camera installed close to the workbench. In most cases, the operation status needs to be filmed using a camera installed behind or diagonally behind the worker. Since obstacles cannot be installed permanently in the work space, the camera used to film the working model video can be installed temporarily at the time of filming, but must be removed when a new worker starts working afterwards. In that case, if you want to film the new worker, you will need to use an image sensor installed on the ceiling.
[0023] The second method is to film the work of an experienced worker as a work model video, and then display the work model video appropriately. There are two types of images taken at different times and by different workers, as well as different image sizes, directions, and fps (frames per second). Each worker in the video follows the same work procedure and makes similar movements, but they do not match completely. We want to display these two types of images on the screen in a synchronized manner.
[0024] For example, a video of an experienced worker is taken at 24 fps to 60 fps using a camera or smartphone mounted on a tripod, and used as a work model video for learning. However, in most cases, the camera installed behind the worker or on the work desk cannot be left installed all the time due to workability and safety considerations. When a new worker starts working afterwards, the camera or smartphone mounted on the tripod that was only temporarily installed is removed.
[0025] On the other hand, it is possible to record the work status of new workers using a ceiling camera for monitoring. The work speed of new workers depends on their skills, and the fps of the ceiling camera may differ from that of a smartphone. There is a demand for recording the movements of new workers with a ceiling camera and monitoring their movements in real time, while also displaying videos of experienced workers recorded in advance as work model videos at appropriate times. There are many advantages to be found by switching from paper manuals to work model videos.
[0026] Under these circumstances, a technology that can provide highly convenient work assistance to a worker who performs work in a plurality of work processes will be described below.
[0027] (overview) First, an overview of the embodiment will be described. A work performed by an experienced worker is captured as a work model video for learning by a rear camera (such as a smartphone) mounted on a tripod, while simultaneously capturing images with a ceiling camera. Then, characteristic information (hereinafter also referred to as feature amounts) such as hand movements, body orientation and position, and the state of tools and measuring instruments are extracted from the images captured by the ceiling camera of the work performed by a novice worker.
[0028] In addition, by comparing the feature values of a new worker with those of an experienced worker performing the same task, which has already been photographed by a ceiling camera, it is possible to ascertain which task step the new worker is currently performing. By knowing which task step the new worker is performing, it is possible to determine the next task step to be shown in the task model video, and therefore to determine the video to be shown to the new worker. This allows the new worker to check the next task step without having to advance the video frame by frame.
[0029] If a new worker performs the same actions as in the working model video, the frame advance will be automatic, which is less work than manually advancing the frames. Also, by repeating this process, the new worker will naturally perform the same actions as in the working model video, and his / her proficiency will improve. And since manual frame advance will be required if there is an oversight in the work, an oversight in checking, or a mistake in the work, the new worker will try to avoid performing actions different from the working model video as much as possible, which will lead to stable product quality.
[0030] In addition, by counting the number of manual frame advances, it is possible to measure the degree of proficiency and to understand the work processes in which new workers are likely to make mistakes, and the work model video can be improved based on this. In addition, the target users are not limited to new workers. For example, even experienced workers may have their own style of movements, and this can be used to check whether the work they have done matches the movements in the work model video.
[0031] Furthermore, when a process is completed, such as when a product is placed in a designated location or a designated operation button is pressed on a measuring device, the worker can confirm the completion of the process by pressing a check button on a video viewing device or by image processing of images captured by a ceiling camera. At that time, the system determines the content of the completed work without human intervention and performs check processing on an electronic check sheet. This eliminates the need to use paper check sheets, and allows checks to be performed on an electronic check sheet when a work process is completed, improving the worker's work efficiency and accuracy.
[0032] (System Configuration) 1 is a diagram showing an outline of the overall configuration of a work support system 1 according to an embodiment. The work support system 1 includes a ceiling camera 2, a rear camera 3, a smartphone 5, and a server 10.
[0033] The smartphone 5 is an example of a display unit that can be viewed by the worker 4 (e.g., a new worker) while working. The smartphone 5 displays the working model video (including still images) using a browser or a dedicated application. The working model video can be edited to add text, audio, markers, and other processing, as described below, to improve the workability of the worker 4 viewing it. Note that the display unit is not limited to a smartphone, and may be a tablet terminal, PC, etc.
[0034] Ceiling camera 2 is installed on the ceiling above where worker 4 works, and has a wide shooting area. The shooting range of ceiling camera 2 is set as shooting area A. Rear camera 3 is installed behind worker 4 when worker 4 works, and has a narrower shooting range than ceiling camera 2, so it captures close-up images of worker 4 working. Rear camera 3 is a device that captures the work of a skilled worker as a work model video (sample shot image).
[0035] The reason for providing a nearby rear camera 3 is that many tasks are done by hand, and the operation procedures of the measuring instrument 6 (Fig. 3) that is the target of the work are not captured in the field of view of the ceiling camera 2. Shooting area B, which is the shooting area of the rear camera 3, partially overlaps with shooting area A, which is the shooting area of the ceiling camera 2. Since the rear camera 3 cannot be permanently installed in many cases, it is assumed that it will be installed only when shooting the working model video. Images captured by the ceiling camera 2 and the rear camera 3 are transmitted to the server 10.
[0036] The server 10 is a computer device, and may be a cloud server or may be a site-specific server installed on-premise. The server 10 includes a storage unit 11, an input unit 12, a display unit 13, a communication unit 14, and a processing unit 15.
[0037] The storage unit 11 is a means for storing information, and is composed of, for example, a RAM (Random Access Memory), a ROM (Read Only Memory), a HDD (Hard Disk Drive), an SSD (Solid State Drive), etc. The storage unit 11 stores various information such as images captured by the ceiling camera 2, images captured by the rear camera 3, and edited working model videos.
[0038] The input unit 12 is a means for inputting information, and is, for example, a keyboard, a mouse, a touch panel, or the like.
[0039] The display unit 13 is a means for displaying information, and is, for example, an LCD (Liquid Crystal Display).
[0040] The communication unit 14 is a communication interface for communicating with an external device.
[0041] The processing unit 15 is a means for performing various arithmetic processing, and is, for example, a CPU (Central Processing Unit). The processing unit 15 includes, as functional components, an acquisition unit 151, a determination unit 152, an analysis unit 153, and a control unit 154.
[0042] The acquisition unit 151 acquires various pieces of information from various components. The acquisition unit 151 acquires images captured by the ceiling camera 2 and the rear camera 3, for example.
[0043] The determination unit 152 executes various determinations. For example, the determination unit 152 determines whether or not there is a predetermined correlation or more between the feature amount of the captured image of the work performed by the worker 4 and the feature amount of the work model video obtained by capturing the work performed by a skilled worker in advance, for the most recent work process in the past.
[0044] For example, the determination unit 152 determines whether or not there is a predetermined correlation or more between the feature amount of at least one captured image and the feature amount of at least one working model video for the most recent past work process.
[0045] In addition, the determination unit 152 may determine whether or not there is a predetermined correlation or more, including the shooting order, using the feature amounts of at least two or more captured images and the feature amounts of at least two or more working model videos for the most recent work process in the past.
[0046] In addition, when determining whether or not there is a correlation of a predetermined level or more between the feature amounts of the captured images and the feature amounts of the working model video for the most recent work process in the past, the determination unit 152 may be configured to make the determination using a predetermined portion of both images (for example, the area around the worker's hands).
[0047] The analysis unit 153 executes various analyses. For example, the analysis unit 153 performs analysis based on the manual frame-forward operation (including the frame-back operation) performed by the worker 4. For example, for each work process, the proficiency level is evaluated based on the number of manual frame-forward operations. For example, the fewer the number of manual frame-forward operations, the higher the proficiency level is evaluated, and the more the number of manual frame-forward operations, the lower the proficiency level is evaluated.
[0048] The control unit 154 causes the smartphone 5 to display the working model moving image. When the determination unit 152 determines that there is a correlation between the two images that is equal to or greater than a predetermined level, the control unit 154 causes the smartphone 5 to display the working model moving image of the next work process by automatically frame-by-frame advancing.
[0049] In addition, when the determination unit 152 determines that there is no correlation between the two images that is equal to or greater than a predetermined level, the control unit 154 causes the smartphone 5 to display the working model video of the next work process when the worker 4 manually performs a frame-by-frame advance operation.
[0050] In addition, when the determination unit 152 determines that there is no correlation between the two images at a predetermined level, the control unit 154 may cause the smartphone 5 to display the reason for the low correlation by using an image, text display, flashing light, or the like.
[0051] Furthermore, when the determining unit 152 determines that there is a correlation between the two images that is greater than or equal to a predetermined level, the control unit 154 may electronically mark the corresponding task item in the checklist data. Alternatively, a voice recognition function may be realized in the processing unit 15. This may enable, for example, recognition of the voice uttered by the worker 4 with both hands full, and electronic marking processing or changing of the screen displayed on the smartphone 5.
[0052] 2 is a diagram showing the state of a workplace of the embodiment. In the workplace, a ceiling camera 2 is installed on the ceiling, and a rear camera 3 is installed behind a worker 4 by a tripod 31. Images captured by the ceiling camera 2 and the rear camera 3 are transmitted to a server 10.
[0053] When collecting working model videos of a skilled worker working, the rear camera 3 may be placed on the work desk, but is usually placed behind or diagonally behind the worker 4, or in a close position above and behind. This is because it is necessary to collect images of detailed parts such as the operation panel (measuring instrument 6, etc.) of the object being worked on by the worker 4.
[0054] Furthermore, the rear camera 3 cannot be permanently installed when the worker 4 is working for operational and safety reasons, and is expected to be removed after the working model video is shot. When shooting the working model video, multi-camera shooting is performed using the ceiling camera 2 and the rear camera 3. Even if the fps of both cameras is different, synchronous display can be achieved by using the shooting time information of both cameras. Note that during the worker's practical work, such as training new employees or retraining, only the ceiling camera 2 is used to shoot.
[0055] 3 is a diagram showing an example of a display screen on the smartphone 5 of the embodiment. Area R1 shows an image of a work desk viewed from the ceiling by the ceiling camera 2. A worker 40 (worker 4) is in front of a measuring instrument 6, which is the object of the work. The smartphone 5 displaying the working model video is placed to the left of the measuring instrument 6. A tool stand 7, which holds a flathead screwdriver 71 and a Phillips screwdriver 72 in their designated positions, is placed in the upper right corner of the screen of the worker 40. The hand 41 of the worker 40 is also shown on the screen.
[0056] Additionally, area R2 shows a video (working model video) previously captured by rear camera 3. Measuring instruments 6 are shown large, making it easy to confirm the operation status. Hands 91 of the skilled worker when the working model video was captured are also shown, making it possible to confirm which buttons the skilled worker pressed to perform the operation.
[0057] Additionally, buttons 51 are displayed at the bottom of the screen. The buttons 51 are a touch panel and can be operated. The buttons 51 include buttons for frame-by-frame forward / frame-by-frame rewinding of a moving image. By operating the buttons 51, the worker 40 can perform operations such as frame-by-frame forwarding and frame-by-frame rewinding of the moving image.
[0058] The smartphone 5 is placed in a position where the touch panel can be easily operated and the screen can be easily viewed, such as on a work desk, so that the worker 40 can easily view the working model video and operate the smartphone 5. In this example, the image in area R1 is an image captured by the ceiling camera 2 at the current time, and the image in area R2 is a working model video captured in the past, which shows the operation procedure, etc. The worker 40 can confirm and recognize the work content and precautions by looking at the images and comments in area R2. The button 51 is not limited to the one shown in the figure, and may include a button that allows appropriate operation of which image to display.
[0059] 4 is an explanatory diagram regarding the determination of the timing of frame-by-frame advancement of the display screen in the embodiment. In order for the worker 40 to view, a screen showing the current situation of the worker 40 is displayed in an area R1 of the smartphone 5, and an operation image (work model video) of a skilled worker captured in the past is displayed in an area R2.
[0060] Meanwhile, as background processing, an image of the work performed by the skilled worker 9 shown in area R3 captured by the ceiling camera 2 as a work model video is compared with a screen showing the current situation of the worker 40 shown in area R1. As a premise, the measuring instrument 6 and the tool stand 7 are arranged in the same position in both areas R1 and R3. This makes it possible to compare the images in areas R1 and R3. Meanwhile, the smartphone 5 placed in the left front of the worker 40 and the smartphone 5 placed in the left front of the skilled worker 9 do not necessarily need to be in the same position. This is because the smartphone 5 is not an object directly related to the work content.
[0061] FIG. 5 is a diagram showing an example of transition of a display image in a comparative example (when no frame-by-frame timing control is performed). In FIG. 5, the screen as time passes is shown from left to right. The time pass of each image and the procedure for editing the working model will be explained using FIG. 5 and the following figures. In FIG. 5, the image group (M1-1 to M1-5) in (a) and the image group (M2-1 to M2-5) in (b) were photographed in advance of a skilled worker 9, with (a) being photographed by the ceiling camera 2 and (b) being photographed by the rear camera 3. For example, M1-1 and M2-1 are at the same position on the horizontal axis, indicating that they were photographed at the same time.
[0062] (c) is an image manual for work (work model video). This is a group of images (MM2-1 to MM2-5) in which comments such as work instructions and points to note have been added to the images (M2-1 to M2-5) in (b). This group of images has buttons 51 on the touch panel so that worker 4 can advance or rewind the images frame by frame. In other words, worker 4 can advance or rewind the images frame by frame as necessary to check the next work content, etc. In this example, work numbers A01 to A05 corresponding to the work process are attached to the top right corner of the images.
[0063] And while worker 4 was working, the group of images in (c) was automatically played frame by frame on smartphone 5 placed on the work desk, and the images captured by ceiling camera 2 were F1-1 to F1-4 in (d). Compared to M1-1 to M1-5 of skilled worker 9 in (a), worker 4's movements are slower and his work is delayed. In other words, it can be seen that worker 4 is not able to complete the same number of work steps in the same amount of time.
[0064] Specifically, it is as follows. Worker 4 sits in front of the measuring instrument 6 at the time of F1-1, and turns on the power button of the measuring instrument 6 at the time of F1-2. After that, it takes time for Worker 4 to find the screwdriver, causing a delay in the work. In other words, for F1-3, it takes time for the work process of taking out the Phillips screwdriver 72 from the tool stand 7. During this time, in the video of (c), at the time of MM2-3, it is displayed "Bring the Phillips screwdriver closer to you to open the lid," and at the time of MM2-4, it is displayed "Remove the screws on the lid of the box to be measured," but the work by Worker 4 does not keep up with this.
[0065] In other words, when worker 4 brings the Phillips screwdriver 72 to hand at the time of F1-4, the screen of MM2-5 in the video of (c) displays "Insert the cable into the box to be measured." Therefore, if the screen of MM2-4 in (c) does not display "Remove the screws from the lid of the box to be measured" at this point, worker 4 will not be able to recognize the correct work process.
[0066] In other words, simply playing the video means that the worker 4 cannot check the video in accordance with his / her own work progress unless he / she can proceed through all the work steps at the same speed as the skilled worker 9. As a result, the worker 4 has no choice but to proceed with the work by watching the video frame by frame for each work step. Alternatively, there is a possibility that the worker 4 will proceed with the work based on his / her own assumptions without checking the video for each next work step.
[0067] The series of images may be a group of so-called still images, or may be images with a frame rate of about 30 fps to 60 fps, so-called a moving image. When the shooting rate of the ceiling camera 2 is different from that of the rear camera 3 mounted on the tripod 31, images generated by thinning out images from the moving image in accordance with the shooting timing of the group of still images may be used. When generating images by thinning out images, the group of thinned out images may be re-edited for each work process using a notification means such as a comment with work instructions or points to note written on the image, or a voice.
[0068] 6 is a diagram showing a first transition example of a display image in an embodiment (when controlling the timing of frame-by-frame advance). (a) and (b) are the same as FIG. 5. When editing the working model video of (c), a "pause marker" (a mark for pausing) is set. The pause marker is set so that frame-by-frame advance can be performed by releasing it between one work process and the next, or when a peculiar movement indicating the progress of the work is confirmed by the monitored ceiling camera 2, such as when a hand reaches for the tool stand 7 on which a screwdriver is placed.
[0069] For example, in (c), frames were automatically sent from MM2-1 to MM2-2 and MM2-3, but the frame feed is temporarily stopped by the "pause marker 1". The worker 4 visually confirms the work instruction "Move the rightmost Phillips screwdriver closer to you to open the lid", and performs the work action of reaching out his right hand (the right side of the two hands 41) to pick up the Phillips screwdriver 72 (F1-3). When the work action (F1-3) is confirmed by the image captured by the ceiling camera 2, the control unit 154 cancels the "pause marker 1" as it is determined that the worker 4 has performed the movement in accordance with the work instruction. Then, the smartphone 5 viewed by the worker 4 displays the frame feed from MM2-3 in (c), and the next MM2-4 is displayed. The worker 4 reads the work instruction written in MM2-4, "Remove the screw from the lid of the box to be measured", and understands the content of the next work process. Operator 4 holds a Phillips head screwdriver 72 in his hand 41 and removes the screws (not shown) from the box to be measured (= the product to be observed and adjusted). When the screw removal task is completed, (F1-4) the pause marker 2 is released and frame advance is made from MM2-4 to MM2-5. Operator 4 confirms the next task instruction, "Insert the cable into the box to be measured."
[0070] In this way, when worker 4 performs the same actions as the work actions (M1-1 to M1-5) of the skilled worker, the working model video of (c) is automatically advanced frame by frame. As worker 4 repeats a series of tasks, he or she learns that performing the same movements as the working model video of (c) avoids the need to perform the frame-by-frame operation himself or herself. This naturally makes worker 4 realize that he or she can eliminate the time required to perform the frame-by-frame operation manually. Therefore, worker 4 gradually becomes accustomed to the automatic frame-by-frame advancement and begins to perform the same movements as the work actions of the skilled worker.
[0071] Furthermore, if the worker 4 does not perform the work according to the work procedure and makes a mistake, work instructions on what to do can be given. For example, an image can be displayed to give the instructions again. For example, "most" can be added before "on the right" in MM2-3 in (c) to more clearly show the location of the Phillips screwdriver 72. Another possible method is to display a screen showing the work mistake captured by the ceiling camera 2 and show the correct work content.
[0072] For example, in the example of a mistake, it may be found that worker 4 is prone to mistaking a Phillips screwdriver 72 for a flathead screwdriver 71. On the other hand, the resolution of the image captured by ceiling camera 2 may not be sufficient to identify the type of screwdriver by its shape. However, there is no need to increase the resolution of ceiling camera 2. A straightforward method would be to add text to the work instructions to prevent work mistakes from occurring. Other preventive measures that use images (such as using the color of the tool or an empty stand) may also be used. This flow is shown in the flowchart of Figure 9, which will be described later.
[0073] Fig. 7 is a diagram showing a second transition example of a display image in an embodiment. The flow of frames of an image showing a process when frame advance is stopped will be described with reference to Fig. 7. Specifically, a method for releasing the pause marker and an example of a method for presenting a countermeasure when the worker 4 cannot release the pause marker will be described.
[0074] (a) is the working model video watched by the worker 4 after editing. (b) is the movement of the worker 4 captured by the ceiling camera 2. First, (a) and (b) will be described.
[0075] Following the work instruction in MM2-2 to "Turn on the measuring device (measuring device 6)," worker 4 performs the work. F1-2 shows worker 4 turning on the measuring device 6. In this case, since hand 41 of worker 40 is near the power supply of measuring device 6, judgment unit 152 sets this area as a designated area (predetermined area) and can judge that the power has been turned on when hand 41 comes into this designated area.
[0076] Next, the worker 4 confirms the work instruction in MM2-3 that "to open the lid, move the Phillips screwdriver on the far right closer to you." It takes the worker 4 time to find the Phillips screwdriver 72, causing a work delay. It can be seen in F1-3-1 that the worker 40 is bringing his hand 41 closer to the tool stand 7.
[0077] The determination unit 152 determines that the worker 40 has reached out to grab the Phillips head screwdriver 72, and the control unit 154 releases the pause marker 1 and automatically advances the screen to MM2-4 frame by frame. However, as can be seen from F1-3-2, the worker 4 does not follow the work instruction in MM2-3, which is to "move the Phillips head screwdriver on the right end closer to you to open the lid." In other words, the Phillips head screwdriver 72 is not taken out from the tool stand 7. Because the worker 4 does not take the prescribed action, the "pause marker 2" is not released, and the next screen, MM2-4, remains displayed. (a) and (b) are examples in which the worker 4 made a mistake.
[0078] Next, (c) and (d) show examples of how to deal with this mistake. They are similar to (a) and (b) until halfway through. In F1-3, the hand 41 of the worker 4 approaches the tool stand 7, so it is assumed that the worker 4 has followed the work instruction in MM2-3, "Move the Phillips screwdriver on the right end closer to you to open the lid." The pause marker 1 is released and automatic frame advance is performed to MM2-4.
[0079] However, the next image F1-4 captured by the ceiling camera 2 cannot be confirmed, so the pause marker 2 cannot be released. Because this cannot be released for a predetermined period of time, the determination unit 152 determines that the worker 4 has stopped working, and the control unit 154 displays M1-4 (similar to M1-4 in Figs. 5 and 6), which is one of the images captured in advance by the ceiling camera 2. This reminds the worker 4 that he needs to take the Phillips screwdriver 72 in his hand 41, and he brings the Phillips screwdriver 72 to his hand. This is confirmed in the image (F1-4) captured by the ceiling camera, and the screen automatically advances frame by frame to the next MM2-5 screen.
[0080] In this way, by implementing measures to deal with mistakes made by worker 4, worker 4 can easily proceed with the work without becoming confused.
[0081] 8 is a flowchart showing a process of shooting and editing a working model video in the embodiment. Here, it is assumed that a working model video is shot of a skilled worker working in the environment shown in FIG.
[0082] In step S1, the determination unit 152 determines whether the shooting environment for the working model video is ready or not, and if Yes, proceeds to step S2, and if No, returns to step S1. In step S1, for example, if the ceiling camera 2 and the rear camera 3 installed on the tripod 31 are ready, the determination is Yes.
[0083] In step S2, the acquisition unit 151 acquires images of the worker 4 captured by the ceiling camera 2 and the rear camera 3.
[0084] Next, in step S5, the control unit 154 sets "start / stop marking (pause marking)" for realizing frame-by-frame advance processing of the video when checking the work action of the worker 4 using the image captured by the ceiling camera 2 between one work instruction and the next work instruction. Note that the "start / stop marking" may be "pause marking" that stops frame-by-frame advance for a predetermined period of time.
[0085] In addition, the comparison area (designated area) is set in order to check using images from the ceiling camera 2 whether a specified action has been performed in the set area (for example, a hand entering the area, or a specified tool being removed from the tool stand 7, etc.).
[0086] Next, in step S6, the determination unit 152 determines whether or not "start / stop marking" and "comparison area designation" have been set. If Yes, the process proceeds to step S7, and if No, the process proceeds to step S4.
[0087] In step S4, the control unit 154 determines that there is a portion in the collected image group that cannot be set, and confirms that correction work has been performed, mainly on the shooting environment, and returns to step S1.
[0088] In step S7, the control unit 154 sets a link operation between the group of images captured by the ceiling camera 2 and the working model moving image created by editing the group of images captured by the rear camera 3 installed on the tripod 31.
[0089] Next, in step S8, the determination unit 152 determines whether the link operation is appropriate or can be handled with fine adjustment. If Yes, the process proceeds to step S9, and if No, re-imaging is required, so the process proceeds to step S4.
[0090] In step S9, the control unit 154 confirms that the settings related to time adjustment, such as the setting for changing the maximum stop time from the default value when a pause marker is set, the specification of the error processing screen to be displayed frame by frame thereafter, the entry of work instructions, and the settings for using sound and light, have been implemented in "Perform fine adjustment". In addition, other settings such as numbering of each frame and setting of frame forward / rewind functions are also implemented. In this way, the settings and fine adjustments for creating a series of working model videos (image group) are completed.
[0091] Next, in step S10, the acquisition unit 151 starts collecting examples of working model videos. For example, images captured by the ceiling camera 2 are collected.
[0092] Next, in step S11, the judgment unit 152 judges whether or not an example of the performed work has been collected, that is, whether or not an image of the worker 4 (new worker) working has been collected by the ceiling camera 2. If the answer is Yes, the process proceeds to step S12, and if the answer is No, the process proceeds to step S4 since there was a problem with the captured video or the shooting environment.
[0093] Next, in step S12, the analysis unit 153 executes a statistical analysis. For example, a statistical analysis is performed on whether the worker 4 (new worker) whose image was collected performed the work based on the work model video. Specifically, for example, by counting the number of manual frame advances for each work process, it is possible to measure the proficiency level and to grasp the work process in which the worker 4 is likely to make a mistake.
[0094] Next, in step S13, the control unit 154 confirms that the corrections have been addressed based on the analysis results. For example, improvements such as adding comments to the working model video are made based on the analysis results.
[0095] Next, in step S14, the control unit 154 determines whether or not to complete collection of performed work examples. If Yes, the process ends as creation of the work model video is complete, and if No, the process proceeds to step S4 to continue improvement.
[0096] Note that steps S11 and after relate to actual data collection and correction based on the analysis results, and may be repeated as necessary. In addition, in the flowchart, description of writing / reading data to / from the memory unit 11 in FIG. 1 is omitted. In addition, the target worker 4 is not limited to a new worker, but is a general worker who is inexperienced in the corresponding work. In other words, the technology of this embodiment can be used not only for training new workers, but also for training when multi-skilling, re-education of veteran workers to maintain / improve the work quality, and other general promotion of a work environment according to a manual.
[0097] 9 is a flowchart showing the process of monitoring the work of the worker 4 and the process when an error occurs in the automatic frame advance in the embodiment. When the automatic frame advance described above does not operate normally, the process shows the procedure for checking the current state of the worker 4 using the image from the ceiling camera 2 and for processing when an error occurs.
[0098] First, in step S20, the determination unit 152 determines whether or not a setting is made to issue a teaching / instruction when an automatic frame advance error occurs. If Yes, the process proceeds to step S21, and if No, the process ends.
[0099] In step S21, the determination unit 152 determines whether or not the working model video for the target work process has start / stop markings and comparison area designation. If Yes, the process proceeds to step S22; if No, the process proceeds to step S34.
[0100] In step S34, the control unit 154 moves to monitoring the next work process (trigger), and returns to step S21.
[0101] In step S22, the control unit 154 monitors the work of the worker 4 using the ceiling camera 2.
[0102] Next, in step S23, the determination unit 152 determines whether or not a "frame forward / backward" operation by the worker 4 has been detected, and if Yes, the process proceeds to step S24, and if No, the process proceeds to step S26. Note that this detection may be an estimation based on an image captured by the ceiling camera 2, or may be a confirmation of the presence or absence of a "frame forward / backward" operation command.
[0103] In step S24, the determination unit 152 determines whether or not the worker 4 has not left the work area. If Yes, the process proceeds to step S25, and if No, the process proceeds to step S26.
[0104] In step S25, the judgment unit 152 judges whether or not the set work time for each work process has been exceeded. If Yes, it means that the predetermined time has been exceeded and proceeds to step S26, and if No, it means that the predetermined time is not exceeded and returns to step S22 to continue work monitoring.
[0105] In step S26, the control unit 154 checks the current state of the worker 4 based on the image captured by the ceiling camera 2.
[0106] Next, in step S27, the control unit 154 checks the display animation model number.
[0107] Next, in step S28, the determination unit 152 checks whether there is a mismatch between the display animation model number and the current state, and if Yes, proceeds to step S29, and if No, proceeds to step S34. In step S28, specifically, for example, based on the image by the ceiling camera 2, it is confirmed whether the worker 4 has not performed the expected work and there is a mismatch, that is, whether the server 10 cannot perform automatic frame advance. If Yes in step S28, it is determined that there is a mismatch, and the next step S29 is performed. If No in step S28, it is determined that there is a state in which the automatic frame advance is not triggered for some reason. In order to continue the work, the process proceeds to step S34 once. This is processing assuming that the automatic frame advance does not operate, but the worker 4 will perform manual frame advance. There is a problem with the image that determines whether to perform automatic frame advance, and this is a part that requires subsequent improvement.
[0108] In step S29, the control unit 154 gives instructions and guidance in the event of an error. Specifically, for example, when monitoring with the ceiling camera 2, processing is performed in the case where automatic frame advance is not operating because the worker 4 is not performing an appropriate work behavior. That is, the above-mentioned working status of the skilled worker previously photographed by the ceiling camera 2 is presented (M1-4 in (c) of FIG. 7). Also, a message on how to deal with an error in work may be displayed.
[0109] Next, in step S30, the control unit 154 monitors the work of the worker 4 and checks how the worker 4 responds to the teaching / instruction given at the time of the error.
[0110] Next, in step S31, the determination unit 152 determines whether the state of the operator 4 is consistent and whether the automatic frame-by-frame advance operation is in progress. If Yes, the process proceeds to step S33, and if No, the process proceeds to step S32.
[0111] In step S32, the judgment unit 152 judges whether the number of automatic frame-forward operations N is equal to or less than a predetermined number N0. If Yes, the judgment unit 152 returns to step S26 to check the current situation of the worker 4, and if No, the judgment unit 152 proceeds to step S34 to monitor the next work process.
[0112] If step S31 is Yes (progress consistency), this means that the worker 4 has performed a frame-by-frame advance operation by performing a predetermined work action, and the process proceeds to step S33. Then, in step S33, the determination unit 152 determines whether to end the video, and if Yes, the end of the working model video has been reached and the process is terminated, whereas if No, the end of the working model video has not been reached and the process proceeds to step S34.
[0113] Next, Fig. 10 is a diagram showing an example of a display screen in a case where both a working model video and a worker 4 are displayed in an embodiment. It has already been explained that the display example shown in Fig. 3 can be used so that a worker 4 (a new worker) can check the work content. In Fig. 10, another display example will be explained.
[0114] 10, the aforementioned close-up video (video captured by rear camera 3) captured in advance by a skilled worker is displayed in area R13. Also, an image captured in advance by skilled worker 9 captured by ceiling camera 2 is displayed in area R12. And an image captured by ceiling camera 2 of where worker 4 is currently working is displayed in area R11.
[0115] Thus, Fig. 10 is an example of displaying a comparison of the working states of a skilled worker and worker 4. Note that image processing may be used to superimpose the working states of both workers to make the differences in hand movements more clear.
[0116] Next, Fig. 11 is a diagram showing an example of a display screen in an embodiment in which both a working model video and an image of a skilled worker captured by ceiling camera 2 are displayed. Unlike the example of Fig. 3 which shows the working state of worker 4 captured by ceiling camera 2, this is an example in which both the working state of a skilled worker captured in the past by ceiling camera 2 and a working model video are displayed. Area R22 is similar to area R13 in Fig. 10. Area R21 is similar to area R12 in Fig. 10.
[0117] This can prevent the worker 4 from feeling the mental burden of being filmed. In order to display the working model video in a large size, it is also possible to display only the working model video without displaying past and present images taken by the ceiling camera 2.
[0118] In this case, as background processing, the validity of the work by the skilled worker 9 shown in area R21 can be determined by comparing the image of the work currently being performed by the worker 40 shown in area R23.
[0119] In this way, according to the work support system 1 of this embodiment, when a new worker (worker 4) performs a certain work process, if the work is highly correlated with the work of an experienced worker, the work model video is automatically advanced frame by frame, and if not, the new worker is made to operate the work model video frame by frame. This makes it possible to provide work support that is highly convenient for the new worker. In other words, the new worker will naturally perform the same movements as the experienced worker to reduce the effort of frame-by-frame advancement, and as the proficiency level increases, the effort of frame-by-frame advancement decreases.
[0120] In addition, by identifying (statistical analysis) which work processes new workers are likely to make mistakes in after the fact, it is possible to make improvements such as adding comments to the work model video or reviewing the process.
[0121] In addition, by counting the number of times the frame advance is manually performed, the proficiency of the new worker can be quantitatively measured.
[0122] In addition, even for tasks that involve many steps and that would be overlooked in a conventional paper manual, it is possible to confirm whether or not checks have been carried out.
[0123] In addition, for example, by determining whether or not there is a predetermined correlation between the features of at least one or more captured images and the features of at least one or more working model videos for the most recent work process in the past, it is possible to determine whether or not to automatically frame-by-frame advance the working model video.
[0124] In addition, by using the features of at least two or more captured images and the features of at least two or more working model videos for the most recent past work process to determine whether there is a predetermined level of correlation, including the shooting order, it is possible to suppress erroneous judgments even in cases where erroneous judgments are likely to occur based on only one image.
[0125] In addition, when determining whether there is a correlation greater than a predetermined level between the features of the captured image and the features of the working model video for the most recent work process in the past, by using specified parts of both images, it is possible to determine with high accuracy, for example, whether a specified part of the device was operated with the hand.
[0126] In addition, if it is determined that there is not a predetermined level of correlation, the reason for the low correlation can be displayed using an image, text display, flashing lights, etc., to encourage new workers to take measures to increase the correlation.
[0127] Furthermore, if it is determined that there is a correlation greater than a predetermined level, the corresponding work item in the checklist data can be electronically marked, thereby eliminating the need for new workers to mark the work manually.
[0128] In the above embodiment, the ceiling camera 2 and the rear camera 3 are used as the image capturing means, but the present invention is not limited to this and only one of them may be used. For example, if the details of the work content can be captured by only the ceiling camera 2, the rear camera 3 may not be used.
[0129] Also, for example, if the details of the work can be captured by only the rear camera 3, the ceiling camera 2 may not be used, and the work of the worker 4 (new worker) may also be captured by the rear camera 3.
[0130] Furthermore, the program executed by the task support system 1 of this embodiment can be provided by recording it in an installable or executable file format on a recording medium readable by a computer device, such as a CD (Compact Disc)-ROM (Read Only Memory), a flexible disk (FD), a CD-R (Recordable), or a DVD (Digital Versatile Disk). The program may also be provided or distributed via a network such as the Internet.
[0131] Although an embodiment of the present invention has been described, this embodiment is presented as an example and is not intended to limit the scope of the invention. This novel embodiment can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. This embodiment and its modifications are included in the scope and spirit of the invention, and are included in the invention and its equivalents described in the claims.
[0132] For example, when changing the layout of the work area, an image linked to the working model video may be acquired by re-shooting using the ceiling camera 2. In that case, depending on the situation, re-shooting of the working model video itself may not be necessary. [Explanation of symbols]
[0133] 1...work support system, 2...ceiling camera, 3...rear camera, 4...worker, 5...smartphone, 10...server, 11...storage unit, 12...input unit, 13...display unit, 14...communication unit, 15...processing unit, 151...acquisition unit, 152...determination unit, 153...analysis unit, 154...control unit
Claims
1. an acquisition unit that acquires captured images from a camera that captures images of workers performing a plurality of work processes; a determination unit that determines whether or not there is a predetermined correlation between the feature amount of the photographed image and the feature amount of a sample photographed image obtained by photographing a sample maker performing the work, for the most recent past work process; displaying the sample photographed image on a display unit that is visible to the worker during the work; When the determining unit determines that there is a correlation equal to or greater than the predetermined correlation, the display unit displays the sample photographed image of the next work process. A work support system comprising: a control unit that, when the judgment unit determines that there is no correlation equal to or greater than the specified level, displays the sample photographed image of the next work process on the display unit when a frame-by-frame advance operation is performed.
2. The work support system according to claim 1 , wherein the determination unit determines whether or not there is a predetermined correlation or more between features of at least one of the captured images and features of at least one of the sample captured images for the most recent work process in the past.
3. 2. The work support system according to claim 1, wherein the determination unit determines whether or not there is a predetermined correlation or more, including an order of shooting, for the most recent work process in the past, using feature amounts of at least two of the captured images and feature amounts of at least two of the sample captured images.
4. 2. The work support system according to claim 1, wherein the determination unit, when determining whether or not there is a predetermined correlation or more between the feature amounts of the captured image and the feature amounts of the sample captured image for the most recent past work process, makes the determination using predetermined portions of both images.
5. The work support system according to claim 1 , wherein the control unit, when the determination unit determines that the correlation is not equal to or greater than the predetermined level, causes the display unit to display a reason why the correlation is low.
6. 2. The work support system according to claim 1, wherein the control unit, when the determining unit determines that there is a correlation equal to or greater than the predetermined level, electronically marks the corresponding work item in the check sheet data.
7. An acquisition step in which an acquisition unit acquires captured images from a camera that captures images of workers performing a plurality of work processes; a determining step in which a determining unit determines whether or not there is a predetermined correlation or more between the feature amount of the photographed image and the feature amount of a sample photographed image obtained by photographing a sample maker performing the work, for the most recent past work process; a control unit that displays the sample photographed image on a display unit that is visible to the worker during work; When it is determined that there is a correlation equal to or greater than the predetermined correlation in the determining step, the sample photographed image of the next work process is displayed on the display unit; and if the judgment step determines that there is no correlation equal to or greater than the specified level, a control step of displaying the sample photographed image of the next work process on the display unit when a frame-by-frame advance operation is performed.
8. Computer, an acquisition unit that acquires captured images from a camera that captures images of workers performing a plurality of work processes; a determination unit that determines whether or not there is a predetermined correlation between the feature amount of the photographed image and the feature amount of a sample photographed image obtained by photographing a sample maker performing the work, for the most recent past work process; displaying the sample photographed image on a display unit that is visible to the worker during the work; When the determining unit determines that there is a correlation equal to or greater than the predetermined correlation, the display unit displays the sample photographed image of the next work process. A program for functioning as a control unit that, when the judgment unit determines that there is no correlation above the specified level, displays the sample photographed image of the next work process on the display unit when a frame-by-frame advance operation is performed.
Citation Information
Patent Citations
Work data managing method
JP2001331513A
Operator safety testing apparatus
JP2012216221A
Work instruction system
JP2018092478A
Program, information processing device, and information processing method
JP2020017283A
Information processing system, information processing device, information processing method, and program
WO2017150293A1
