Work analysis device, program, and work analysis method

Abstract

A work analysis device (120) comprises: an image feature calculation unit (124) that calculates a plurality of image feature amounts by inputting, to an image encoder, a plurality of pieces of target image data indicating, in time series, a plurality of target images included in a video obtained by imaging work; a text feature calculation unit (126) that calculates a plurality of text feature amounts by inputting, to a text encoder, a plurality of pieces of text data respectively indicating contents of a plurality of steps constituting the work; a comparison unit (127) that calculates a degree of similarity between each of the plurality of image feature amounts and each of the plurality of text feature amounts; and a step identification unit (128) that identifies one or more steps in time series by identifying, in each of the plurality of image feature amounts, a step of which the contents are indicated by text corresponding to the text feature amount having the highest degree of similarity.

Description

Work analysis device, program, and work analysis method

[0001] The present disclosure relates to an activity analysis device, a program, and an activity analysis method.

[0002] In manufacturing and other industries, improving work processes to increase productivity is important. To improve work processes, the first step is to analyze the current situation and perform a work analysis that measures the time required for each process that makes up the work. Work analysis is often carried out visually based on images captured by a camera, but this takes a huge amount of time. In response to this, a work analysis device has been proposed that shortens the time required for work analysis.

[0003] For example, Patent Document 1 describes a work data management system in which criteria are set in advance, such as the positions through which a worker's body parts pass for each process, and the process performed at each time is identified when the criteria are met.

[0004] Japanese Patent Application Laid-Open No. 2019-016226

[0005] However, in the conventional technology, it is necessary to predetermine the criteria for each process, which is time-consuming.

[0006] Therefore, one or more aspects of the present disclosure aim to make it possible to easily identify work processes with high accuracy.

[0007] A work analysis device according to one aspect of the present disclosure includes an image feature calculation unit that calculates a plurality of image features, which are a plurality of features corresponding to each of a plurality of target images included in a video of a captured task, by inputting a plurality of pieces of target image data that show, in time series, a plurality of target images included in the video of the task into an image encoder; a text feature calculation unit that calculates a plurality of text features, which are a plurality of features corresponding to each of a plurality of steps that make up the task, by inputting a plurality of pieces of text data that show the contents of each of the plurality of steps; a comparison unit that calculates a similarity between each of the plurality of image features and each of the plurality of text features; and a process identification unit that identifies, for each of the plurality of image features, a process whose content is indicated by text corresponding to the text feature with the highest similarity, thereby identifying one or more steps in the time series; wherein the image encoder and the text encoder are pre-trained models so that, for each step included in the plurality of steps, a high similarity is achieved between an image feature calculated by inputting image data that shows an image of the step into the image encoder and a text feature calculated by inputting text data that shows the content of the step into the text encoder.

[0008] A program according to one aspect of the present disclosure causes a computer to function as an image feature calculation unit that calculates a plurality of image features, each of which corresponds to a plurality of target images, by inputting a plurality of pieces of target image data that show, in time series, a plurality of target images included in a video of a task into an image encoder; a text feature calculation unit that calculates a plurality of text features, each of which corresponds to a plurality of steps, by inputting a plurality of pieces of text data that show the contents of a plurality of steps that make up the task into a text encoder; a comparison unit that calculates a similarity between each of the plurality of image features and each of the plurality of text features; and a process identification unit that identifies, for each of the plurality of image features, a process whose content is indicated by text corresponding to the text feature with the highest similarity, thereby identifying one or more steps in the time series. The image encoder and the text encoder are pre-trained models so that, for each step included in the plurality of steps, a high similarity is achieved between an image feature calculated by inputting, into the image encoder, image data that show an image of the step, and a text feature calculated by inputting, into the text encoder, text data that show the content of the step.

[0009] A task analysis method according to one aspect of the present disclosure is a task analysis method that inputs a plurality of pieces of target image data that show a plurality of target images included in a video of a task in chronological order into an image encoder, thereby calculating a plurality of image features that are a plurality of features corresponding respectively to the plurality of target images, inputs a plurality of pieces of text data that show the contents of a plurality of steps that make up the task into a text encoder, thereby calculating a plurality of text features that are a plurality of features corresponding respectively to the plurality of steps, calculates a similarity between each of the plurality of image features and each of the plurality of text features, and identifies, for each of the plurality of image features, a step whose content is indicated by text corresponding to the text feature with the highest similarity, thereby identifying one or more steps in the chronological order, wherein the image encoder and the text encoder are pre-trained models so that, for each step included in the plurality of steps, the image feature calculated by inputting image data that shows an image of the step into the image encoder is highly similar to the text feature calculated by inputting text data that shows the content of the step into the text encoder.

[0010] According to one or more aspects of the present disclosure, it is possible to easily identify steps in a job with high accuracy.

[0011] Fig. 1 is a block diagram showing an outline of the configuration of a work analysis system according to embodiments 1 and 2. Fig. 2 is a block diagram showing an outline of the configuration of a work analysis device in embodiment 1. Fig. 3 is a block diagram showing an outline of the configuration of a PC. Fig. 4 is a flowchart showing the operation of the work analysis device in embodiment 1. Fig. 5 is a block diagram showing an outline of the configuration of a work analysis device in embodiment 2. Fig. 6 is a flowchart showing the operation of the work analysis device in embodiment 2.

[0012] 1 is a block diagram showing a schematic configuration of an activity analysis system 100 according to embodiment 1. The activity analysis system 100 includes a camera 110 as an imaging device, and an activity analysis device 120.

[0013] Although the camera 110 and the work analysis device 120 are connected to the network 101, the first embodiment is not limited to this example. For example, the camera 110 may be connected to the work analysis device 120 via a connection interface compatible with a USB (Universal Serial Bus) or the like.

[0014] The camera 110 is installed where the work is performed, captures video of the work, and generates video data representing the captured video. The camera 110 then transmits the video data to the work analysis device 120 via the network 101.

[0015] 2 is a block diagram showing a schematic configuration of the work analysis device 120 according to embodiment 1. The work analysis device 120 includes a communication unit 121, a video acquisition unit 122, a region-of-interest extraction unit 123, an image feature calculation unit 124, a text acquisition unit 125, a text feature calculation unit 126, a comparison unit 127, a process identification unit 128, and an output unit 129.

[0016] The communication unit 121 performs communication via the network 101. In this embodiment, the communication unit 121 receives video data from the camera 110 via the network 101. The received video data is provided to the video acquisition unit 122.

[0017] The video acquisition unit 122 acquires video data. The acquired video data is provided to the attention area extraction unit 123. Here, the video acquisition unit 122 acquires the video data from the camera 110 via the communication unit 121, but the embodiment is not limited to this example. For example, if the video data is stored in a storage unit (not shown), the video acquisition unit 122 may acquire the video data by reading the video data from the storage unit. Furthermore, if the video data is stored in a server (not shown), the video acquisition unit 122 may acquire the video data from the server via the communication unit 121.

[0018] The attention area extraction unit 123 extracts one or more attention areas, which are one or more predetermined parts, from each of the multiple frames that make up the video, and constructs multiple target image data using one or more attention area data, which are data that indicate the one or more attention areas.

[0019] Here, the attention area extraction unit 123 extracts one or more portions from each of the multiple frames constituting the video data as one or more attention areas. For example, the attention area extraction unit 123 extracts at least one of the peripheral area of ​​a worker who is performing work and is being imaged by the camera 110, the peripheral area of ​​the worker's hands, and a changed area from the previous frame as one or more attention areas.

[0020] Specifically, when a worker is working using his or her entire body, or when the worker's posture affects the work content, it is desirable to set the area around the worker as the region of interest. Furthermore, when a worker is working with his or her hands while maintaining a constant posture, such as while sitting, or when a worker is working with different tools in hand depending on the process, it is desirable to set the area around the hands as the region of interest. Furthermore, when work is performed using an inspection machine and the inspection results are displayed on the inspection machine monitor, it is desirable to set the changed area as the region of interest. Here, an example is described in which the area around the worker, the area around the hands, and the changed area are all extracted as multiple regions of interest, but it is sufficient if at least one of these is extracted. One or more pieces of region of interest data indicating the extracted one or more regions of interest (here, multiple regions of interest) are provided to the image feature calculation unit 124.

[0021] The image feature calculation unit 124 inputs a plurality of pieces of target image data representing a plurality of target images included in a video of a captured work operation in time series into the image encoder, and calculates a plurality of image feature amounts, which are a plurality of feature amounts corresponding to the plurality of target images. Here, the target images are the region of interest, but the frames constituting the video may also be the target images. In this case, the region of interest extraction unit 123 is not necessary.

[0022] For example, the image feature calculation unit 124 calculates one or more image feature amounts, which are one or more feature amounts, by inputting each of one or more pieces of attention region data from the attention region extraction unit 123 to a known image encoder. The calculated one or more image feature amounts are provided to the comparison unit 127. Here, since multiple attention regions are provided from the attention region extraction unit 123, the image feature calculation unit 124 calculates the multiple image feature amounts by inputting data indicating each of the multiple attention regions to a known image encoder.

[0023] The text acquisition unit 125 acquires text data indicating text describing each of a plurality of steps included in the work imaged by the camera 110. The acquired text data is provided to the text feature calculation unit 126. For example, if the text data is stored in a storage unit (not shown), the text acquisition unit 125 may acquire the text data by reading the text data from the storage unit. Alternatively, if the text data is stored in a server (not shown), the text acquisition unit 125 may acquire the text data from the server via the communication unit 121.

[0024] The text feature calculation unit 126 inputs a plurality of pieces of text data indicating the contents of a plurality of steps that make up the work into a text encoder, and calculates a plurality of text features that are a plurality of features that respectively correspond to the plurality of steps. Here, the text feature calculation unit 126 inputs the text data provided by the text acquisition unit 125 into a known text encoder, and calculates the text features that are the features for each step.

[0025] Here, we will explain the image encoder used by the image feature calculation unit 124 and the text encoder used by the text feature calculation unit 126. The image encoder and text encoder used here are assumed to have been trained in advance to convert image data of a process included in a job and text data indicating the content of that process into similar feature quantities in the same space. The content of the process may be any content related to that process.

[0026] In other words, the image encoder and the text encoder are pre-trained models that, for each process included in the plurality of processes, increase the similarity between the image feature calculated by inputting image data representing an image of that process into the image encoder and the text feature calculated by inputting text data representing the content of that process into the text encoder. Such image encoders and text encoders are described, for example, in the following literature: Literature: Alec Radford et al., "Learning Transferable Visual Models From Natural Language Supervision," arXiv:2103.00020 [cs. CV], 26 February 2021

[0027] The comparison unit 127 calculates the similarity between each of the multiple image features and each of the multiple text features. For example, the comparison unit 127 calculates a similarity vector indicating the similarity between one or more image features and the text features for each process, for each frame, from one or more image features for each frame and the text features for each process. The similarity calculated here may be, for example, cosine similarity, but is not limited to this. The calculated similarity vector is provided to the process identification unit 128. Here, one similarity vector indicates the similarity between one image feature in a certain frame and a text feature for each process.

[0028] Here, the comparison unit 127 acquires a plurality of image feature amounts from the image feature calculation unit 124, and therefore calculates a similarity vector for each frame for each of the plurality of image feature amounts. Specifically, the comparison unit 127 calculates a similarity vector for each frame in which the peripheral portion of the worker is the attention region, a similarity vector for each frame in which the peripheral portion of the hand is the attention region, and a similarity vector for each frame in which the changed portion is the attention region.

[0029] The process identification unit 128 identifies one or more processes in chronological order by identifying a process whose content is indicated by text corresponding to the text feature with the highest calculated similarity for each of the multiple image features. Here, the process identification unit 128 identifies the process indicated by the frame from the similarity vector for each frame. For example, if only one region of interest is extracted, the process identification unit 128 may determine the process described in the text data with the highest similarity indicated by the similarity vector for each frame as the process indicated by that frame.

[0030] On the other hand, when multiple regions of interest are extracted, the process identification unit 128 determines the process indicated in that frame from the multiple similarity vectors for each frame. For example, the process identification unit 128 may determine the process described in the text data with the highest similarity among the similarities indicated by the multiple similarity vectors for each frame as the process indicated in that frame. Alternatively, the process identification unit 128 may summarize the similarities indicated by the multiple similarity vectors for each frame for each text data, and may determine, for example, the process described in the text data with the highest added or multiplied value as the process indicated in that frame.

[0031] The output unit 129 outputs data indicating the process determined for each frame. For example, the output unit 129 may display the data on a display (not shown) in a predetermined display format. The output unit 129 may also send such data to a predetermined destination via the communication unit 121.

[0032] The work analysis device 120 described above can be realized by, for example, a computer such as the PC 10 shown in Fig. 3. The PC 10 includes storage 11 such as a hard disk drive (HDD) and a solid state drive (SSD), memory 12, a processor 13 such as a central processing unit (CPU), a communication interface (I / F) 14 such as a network interface card (NIC), an input interface 15 such as a keyboard and a mouse, and a display 16.

[0033] For example, the communication unit 121 can be realized by the communication I / F 14. The video acquisition unit 122, the attention area extraction unit 123, the image feature calculation unit 124, the text acquisition unit 125, the text feature calculation unit 126, the comparison unit 127, the process identification unit 128, and the output unit 129 can be realized by loading a program stored in the storage 11 into the memory 12 and having the processor 13 execute the program.

[0034] The program may be downloaded to the storage 11 from a recording medium (not shown) via a reader / writer (not shown) or from the network 101 via the communication I / F 14, and then loaded onto the memory 12 and executed by the processor 13. Alternatively, the program may be directly loaded onto the memory 12 from a recording medium via the reader / writer or from the network 101 via the communication I / F 14, and then executed by the processor 13. In other words, the program may be provided by a computer program product such as a recording medium.

[0035] 4 is a flowchart showing the operation of the work analysis apparatus 120 in embodiment 1. First, the video acquisition unit 122 acquires video data (S10). The acquired video data is provided to the attention area extraction unit 123.

[0036] The region-of-interest extraction unit 123 sequentially selects one unprocessed frame from the multiple frames constituting the video data (S11). Then, the region-of-interest extraction unit 123 extracts one or more regions of interest from the selected frame (S12). Here, it is assumed that multiple regions of interest are extracted. A plurality of pieces of region-of-interest data indicating the extracted regions of interest are provided to the image feature calculation unit 124.

[0037] The image feature calculation unit 124 calculates a plurality of image feature amounts corresponding to the plurality of regions of interest by inputting each of the plurality of region of interest data from the region of interest extraction unit 123 to a known image encoder (S13). The calculated plurality of image feature amounts are provided to the comparison unit 127.

[0038] The text acquisition unit 125 acquires text data indicating text describing each of a plurality of steps included in the work imaged by the camera 110, and the text feature calculation unit 126 inputs the text data into a known text encoder to calculate text features, which are features for each step (S14). The calculated text features are provided to the comparison unit 127.

[0039] The comparison unit 127 calculates the similarity between each of the plurality of image features and the text feature for each process, thereby calculating a plurality of similarity vectors corresponding to the plurality of image features (S15). The plurality of similarity vectors are provided to the process identification unit 128.

[0040] The process identification unit 128 identifies the process indicated by the frame from the plurality of similarity vectors (S16). Here, the process identification unit 128 uses the similarity for each process indicated by each of the plurality of similarity vectors to identify the process that can be determined to have the highest similarity among the processes depicted in the frame, based on a predetermined determination criterion.

[0041] Then, the attention area extraction unit 123 determines whether or not all frames of the video data have been selected in step S11 (S17). If all frames have been selected (Yes in S17), the process proceeds to step S18, and if there are frames that have not yet been selected (No in S17), the process returns to step S11.

[0042] In step S18, the output unit 129 outputs data indicating the process determined for each frame.

[0043] As described above, according to the first embodiment, the steps in a job can be easily identified with high accuracy.

[0044] 1, an activity analysis system 200 according to the second embodiment includes a camera 110 and an activity analysis device 220. The camera 110 of the activity analysis system 200 according to the second embodiment is similar to the camera 110 of the activity analysis system 100 according to the first embodiment. However, in the second embodiment, the camera 110 transmits video data to the activity analysis device 220 via the network 101.

[0045] 5 is a block diagram showing a schematic configuration of a work analysis device 220 according to embodiment 2. The work analysis device 220 includes a communication unit 121, a video acquisition unit 122, a region-of-interest extraction unit 123, an image feature calculation unit 124, a text acquisition unit 125, a text feature calculation unit 126, a comparison unit 127, a process identification unit 128, an output unit 129, and a process correction unit 230.

[0046] The communication unit 121, video acquisition unit 122, area of ​​interest extraction unit 123, image feature calculation unit 124, text acquisition unit 125, text feature calculation unit 126, comparison unit 127, process identification unit 128, and output unit 129 of the work analysis apparatus 220 in embodiment 2 are similar to the communication unit 121, video acquisition unit 122, area of ​​interest extraction unit 123, image feature calculation unit 124, text acquisition unit 125, text feature calculation unit 126, comparison unit 127, process identification unit 128, and output unit 129 of the work analysis apparatus 120 in embodiment 1. However, in embodiment 2, the process identification unit 128 notifies the process correction unit 230 of the process identified for each frame, and the output unit 129 outputs data indicating the process corrected by the process correction unit 230.

[0047] The process correction unit 230 corrects errors contained in the processes for each frame identified by the process identification unit 128. For example, the process correction unit 230 corrects a predetermined number of processes by using the most frequently occurring process among a predetermined number of processes consecutively identified in time series. Specifically, the process correction unit 230 may determine the most frequently identified process for each predetermined number of frames as the process for the predetermined number of frames.

[0048] Furthermore, the process correction unit 230 may statistically correct the identified process error. Specifically, the process correction unit 230 may statistically correct the identified process error using a known algorithm such as a hidden Markov model.

[0049] The work analysis device 220 described above can also be realized by a computer such as the PC 10 shown in Fig. 2. In the second embodiment, the process correction unit 230 can also be realized by loading a program stored in the storage 11 into the memory 12 and having the processor 13 execute the program.

[0050] Fig. 6 is a flowchart showing the operation of the work analysis device 220 in embodiment 2. Note that, among the processes included in the flowchart shown in Fig. 6, the processes that are the same as the processes included in the flowchart shown in Fig. 4 are assigned the same reference numerals as those used in the flowchart shown in Fig. 4.

[0051] The processing of steps S10 to S17 in Fig. 6 is the same as the processing of steps S10 to S17 in Fig. 4. However, in Fig. 6, if it is determined in step S17 that all frames have been selected (Yes in S17), the processing proceeds to step S20.

[0052] In step S20, the process correction unit 230 corrects the error contained in the process for each frame identified by the process identification unit 128. The corrected process for each frame is provided to the output unit 129. Then, the process proceeds to step S18.

[0053] In step S18, the output unit 129 outputs data indicating the process for each frame.

[0054] As described above, according to the second embodiment, the process is corrected for each frame, so that the process can be specified with higher accuracy.

[0055] In the above-described first and second embodiments, a process is identified for each frame, but the first and second embodiments are not limited to such examples. For example, one sample frame serving as one sample may be extracted for each of a predetermined number of frames included in a video, and a process may be identified for each sample frame by the above-described operation.

[0056] 100, 200 Work analysis system, 110 Camera, 120, 220 Work analysis device, 121 Communication unit, 122 Video acquisition unit, 123 Attention area extraction unit, 124 Image feature calculation unit, 125 Text acquisition unit, 126 Text feature calculation unit, 127 Comparison unit, 128 Process identification unit, 129 Output unit, 230 Process correction unit.

Claims

1. A work analysis device comprising: an image feature calculation unit that calculates a plurality of image features corresponding to a plurality of target images included in a video of a task by inputting a plurality of pieces of target image data showing, in time series, a plurality of target images included in the video of the task into an image encoder; a text feature calculation unit that calculates a plurality of text features corresponding to a plurality of steps by inputting a plurality of text data showing the contents of a plurality of steps that make up the task into a text encoder; a comparison unit that calculates the similarity between each of the plurality of image features and each of the plurality of text features; and a process identification unit that identifies one or more steps in the time series by identifying, for each of the plurality of image features, a step whose content is indicated by text corresponding to the text feature with the highest similarity; wherein the image encoder and the text encoder are pre-trained models so that, for each step included in the plurality of steps, the image feature calculated by inputting image data showing an image of the step into the image encoder is highly similar to the text feature calculated by inputting text data showing the content of the step into the text encoder.

2. The work analysis device described in claim 1, further comprising an attention area extraction unit that extracts one or more attention areas, which are one or more predetermined parts, from each of the multiple frames that make up the video, and constructs the multiple target image data using data indicating the one or more attention areas.

3. The work analysis device according to claim 2, wherein the attention area extraction unit selects the area surrounding the worker performing the work as one of the one or more attention areas.

4. The work analysis device according to claim 2, characterized in that the attention area extraction unit selects the area around the hand of the worker performing the work as one of the one or more attention areas.

5. The work analysis device according to claim 2, characterized in that the attention area extraction unit selects a changed portion of the video image that has changed from the previous frame as one of the one or more attention areas.

6. The work analysis device according to any one of claims 1 to 5, further comprising a process correction unit that corrects the identified process error.

7. The work analysis device according to claim 6, characterized in that the process correction unit corrects the predetermined number of processes using the most frequently occurring process among the predetermined number of processes identified consecutively in the time series.

8. The work analysis device according to claim 6, wherein the process correction unit statistically corrects the identified process errors.

9. A program that causes a computer to function as: an image feature calculation unit that calculates a plurality of image features, which are a plurality of features corresponding respectively to a plurality of target images included in a video of a task, by inputting a plurality of target image data pieces that show, in chronological order, a plurality of target images included in the video of the task into an image encoder; a text feature calculation unit that calculates a plurality of text features, which are a plurality of features corresponding respectively to a plurality of steps that make up the task, by inputting a plurality of text data pieces that show the contents of a plurality of steps that make up the task into a text encoder; a comparison unit that calculates the similarity between each of the plurality of image features and each of the plurality of text features; and a process identification unit that identifies one or more steps in the chronological order by identifying, for each of the plurality of image features, a step whose content is indicated by text corresponding to the text feature with the highest similarity, wherein the image encoder and the text encoder are pre-trained models so that, for each step included in the plurality of steps, the similarity between the image feature calculated by inputting, into the image encoder, image data showing an image of the step and the text feature calculated by inputting, into the text encoder, text data showing the contents of the step 10. A work analysis method that calculates a plurality of image features corresponding to each of the plurality of target images by inputting a plurality of pieces of target image data that show, in time series, a plurality of target images included in video footage of a task into an image encoder; calculates a plurality of text features corresponding to each of the plurality of steps by inputting a plurality of text data that show the contents of a plurality of steps that make up the task into a text encoder; calculates the similarity between each of the plurality of image features and each of the plurality of text features; and identifies, for each of the plurality of image features, the step whose content is indicated by text corresponding to the text feature with the highest similarity, thereby identifying one or more steps in the time series, wherein the image encoder and the text encoder are pre-trained models so that, for each step included in the plurality of steps, the image feature calculated by inputting image data that show an image of the step into the image encoder is highly similar to the text feature calculated by inputting text data that show the content of the step into the text encoder.

Images