Work time measuring device

Abstract

The present invention provides a work time measuring device that, despite its simple configuration, can determine the transition between work processes and accurately measure the work time for each work process. [Solution] The device has a main body comprising an imaging means, a sound collection means, a calculation unit that performs various processing on image data and audio data, and a storage unit that stores various data, and is characterized by: image AI processing that detects characteristic movements of the target work from image data and narrows down the time before and after the timing when the characteristic movements are detected as the operation time; audio AI processing that detects characteristic sounds of the target work from audio data; and logic processing that determines whether there is a time period in the time period when the characteristic sound occurs after the audio AI processing that matches the time period narrowed down by the image AI processing, and if there is a match, determines that the time when the corresponding characteristic sound occurred is the time when the work is completed.

Description

Technical Field

[0004] , , , , , [[ID=​​​​​​​​​​​​​​​​​​​​​​​​​​​​​ Japanese Patent Publication No. 2023-167448 [Overview of the project] [Problems that the invention aims to solve]

[0005] Therefore, the present invention aims to provide a work time measuring device that has a simple configuration, can determine the transition between work processes, and can accurately measure the work time for each work process. [Means for solving the problem]

[0006] To achieve the above objective, the present invention provides a work time measuring device comprising: an imaging means for acquiring image data; a sound collection means for acquiring sound data; a calculation unit for performing various processing on the image data and the sound data; a storage unit for storing data related to an AI that performs judgments based on pre-learned training data; an image AI processing that detects characteristic movements of the target work from the image data and narrows down the time period before and after the timing at which the characteristic movements are detected as the operation time; a sound AI processing that detects characteristic sounds of the target work from the sound data; and a logic processing that determines whether there is a time period at which the characteristic sound occurs after the sound AI processing matches the time period narrowed down by the image AI processing, and if there is a match, determines that the time at which the corresponding characteristic sound occurred is the time of work completion.

[0007] Furthermore, in a work time measuring device having the above-described features, the imaging means and the sound collection means may be configured in a single device, and after acquiring the combined image data and sound data as video data, it may be imported into the storage unit and then split into the image data and the sound data. Having such features allows for the use of general-purpose equipment and reduces the amount of base material.

[0008] Furthermore, in a work time measurement device having the above-described features, it is desirable that the image AI processing divides the image data into unit image data of predetermined time units and processes each of the multiple unit image data in parallel. Having such features makes it possible to shorten the processing time of image data with a large data volume. [Effects of the Invention]

[0009] A work time measuring device with these characteristics can determine the transition between work processes and accurately measure the work time for each work process. [Brief explanation of the drawing]

[0010] [Figure 1] This is a block diagram showing the schematic configuration of a work time measuring device according to an embodiment. [Figure 2] This diagram illustrates the process of splitting video data acquired by the work time measurement device according to the embodiment into image data and audio data. [Figure 3] This diagram illustrates the process of AI processing image data. [Figure 4] This diagram illustrates the process of AI processing audio data. [Figure 5] This diagram illustrates the logic for determining the completion of a task using AI-processed image and audio data. [Figure 6] This table summarizes the work procedures, including characteristic movements, and characteristic sounds associated with the process of creating a cable equipped with couplers at both ends. [Figure 7] This diagram shows the workflow for attaching a coupler to a cable, along with the actions and characteristic sounds associated with each step. [Figure 8] This diagram illustrates the process for handling cases where duplicate feature behaviors, feature sounds that do not match image data, or similar behaviors that are difficult to distinguish are detected during the process of attaching a coupler to a cable. [Modes for carrying out the invention]

[0011] Hereinafter, embodiments of the work time measuring device of the present invention will be described in detail with reference to the drawings. The embodiments shown below are some of the preferred forms for carrying out the present invention, and even if some of their configurations are modified, they can still be considered part of the present invention, as long as they achieve their intended effects.

[0012] [composition] First, with reference to Figure 1, the schematic configuration of the work time measurement device according to this embodiment will be described. The work time measurement device 10 according to this embodiment is basically composed of an imaging means 12, a sound collection means 14, and a device body 16. The imaging means 12 is a means for photographing a worker performing the target work and acquiring image data, and can be a camera or the like. The sound collection means 14 is a means for acquiring sounds generated when performing the target work, and can be a microphone or the like.

[0013] Here, the imaging means 12 and the sound collection means 14 do not necessarily have to be separate components. For example, they may be devices capable of acquiring video data (data that combines image data and audio data), such as a video camera. In this case, the video data will be taken into the main body of the device 16, which will be described in detail later, and then separated into image data and audio data.

[0014] The main body of the device 16 has at least a processing unit 18 and a storage unit 24, and is responsible for acquiring data directly or indirectly from the imaging means 12 and the sound collection means 14 and executing various processes. The processing unit 18 can be, for example, one that has a CPU (Central Processing Unit) 20a, a GPU (Graphics Processing Unit) 22a, a CPU memory 20b, and a GPU memory 22b. In this embodiment, the CPU 20a is an element for executing commands when performing processes such as splitting video data into image data and audio data, splitting image data into multiple data, and processing judgment logic. On the other hand, the GPU 22a is an element for performing minor calculations simultaneously (in parallel) among the calculation processes mainly handled by the CPU 20a, as well as performing AI prediction execution processes. The CPU memory 20b is a temporary storage unit that temporarily stores information recorded in the storage unit 24, which will be described in detail later, and information during calculation processing when the CPU 20a performs calculation processing. Furthermore, the GPU memory 22b is a temporary storage unit that temporarily stores information recorded in the storage unit 24 (details to be described later) and information during computation processing when the GPU 22a performs computational processing.

[0015] The memory unit 24 is an element for storing image data acquired by the imaging means 12 and audio data acquired by the sound collection means 14 (including video data combining both), programs for executing processes according to various instructions, pre-trained training data, and data related to the AI ​​that makes a determination of task completion based on the training data. Specifically, the memory unit 24 can be an HDD (Hard Disk Drive) or an SSD (Solid State Drive). When the main unit 16 indirectly acquires image data, audio data, etc., the various data (e.g., video data) can be transferred from the video camera (combining the imaging means 12 and the sound collection means 14) to the memory unit 24 via a storage medium 26 that provides temporary storage such as memory.

[0016] [Effects / Effects] Next, the operation time measurement and determination operation by the work time measurement device 10 configured as described above will be described. The work time measurement and determination according to this embodiment is roughly carried out in three stages: the first process, the second process, and the third process. The first process is the process of capturing and recording image data and audio data. The second process is AI processing related to characteristic operations in the image data and audio data, and the presence or absence of characteristic operations and characteristic sounds. And the third process is logic processing such as identification at the end of work based on the data of characteristic operations and characteristic sounds narrowed down by AI, and measurement of the time until the end of work.

[0017] First, the first process will be described with reference to FIG. 2. In the form shown in FIG. 2, it is assumed that the image data and audio data are acquired as one video data by a video camera and indirectly taken into the device main body 16 via a storage medium 26 such as an SD card.

[0018] The first process performs a process of dividing the video data into image data and audio data. Specifically, the video data acquired via the camera 12 and the microphone 14 is taken into the storage unit 24 via the storage medium 26 that records the video data (S10). The video data taken into the storage unit 24 is read into the CPU memory 20b of the arithmetic unit 18 (S20), and the CPU 20a executes arithmetic processing along with the processing program recorded in the storage unit 24 (a program for dividing the video data into image data and audio data) to divide the video data into image data and audio data. Further, at this time, the image data is divided into a plurality of data at predetermined intervals (for each frame corresponding to a predetermined time: for example, every 5000 frames). This is because parallel processing of a plurality of short-time image data can shorten the processing time required for analysis compared to analyzing long-time image data (S30). The image data and audio data obtained by the division process are stored in the storage unit 24 in a mutually associated state (S40).

[0019] Next, the second process will be explained with reference to Figures 3 and 4. In the second process, the analysis of image data and the analysis of audio data are performed in parallel, but for convenience, they will be described in the following order.

[0020] First, the image data analysis process will be explained with reference to Figure 3. Initially, processing by CPU 20a is performed. The processing by CPU 20a is an element that prepares the system for AI processing by the GPU, which will be described in detail later. Specifically, it is sufficient to perform processing such as loading the image data into CPU memory 20b and expanding it (S110).

[0021] Image data processed by CPU 20a (CPU memory 20b) is read into GPU memory 22b (S120) by instructions from the AI ​​running via GPU 22a, and AI analysis processing is performed. The AI ​​analysis processing here involves detecting parts of the divided image data that are predicted to contain characteristic actions (feature actions) during the operation (S130). Image data for which the AI ​​has made a prediction judgment regarding the presence or absence of feature actions is temporarily stored in CPU memory 20b via CPU 20a (S140). Image data with the AI ​​prediction judgment results temporarily stored in CPU memory 20b is converted into a CSV file via CPU 20a and saved to storage unit 24. Converting the storage format reduces the data size and improves the processing speed required for reading, etc. (S150). This analysis is processed in parallel for each divided image file.

[0022] Next, the audio data analysis process will be explained with reference to Figure 4. First, processing by CPU 20a is performed. The processing by CPU 20a is an element that prepares the system for AI processing by the GPU, which will be described in detail later. Specifically, it is sufficient to perform processing such as reading the audio data into CPU memory 20b and expanding it (S210).

[0023] The audio data processed by CPU 20a (CPU memory 20b) is read into GPU memory 22b (S220) at the instruction of the AI ​​running via GPU 22a, and the AI ​​performs analysis processing. The AI ​​analysis processing here involves narrowing down the portion where characteristic operational sounds (characteristic sounds) are predicted to exist during the work (narrowing down the time period in which characteristic sounds occur) (S230). The audio data after the AI ​​prediction and narrowing processing is temporarily stored in CPU memory 20b via CPU 20a (S240). The AI ​​prediction and narrowing results temporarily stored in CPU memory 20b are converted into a CSV file via CPU 20a and saved to storage unit 24 (S250).

[0024] Next, the third process will be explained with reference to Figure 5. The third process is logic processing based on the AI ​​processing results of the image data and audio data in the second process. In the third process, first, data with the AI ​​processing results of the image data (image data in which a characteristic action is predicted to exist and whose frame range has been identified) is read from the storage unit 24 to the CPU memory 20b (S310), and the frame range in which the characteristic action is performed is identified via the CPU 20a. When identifying the range of action, if a characteristic action is detected multiple times during a single operation, the action that occurred in the latter chronological order is identified as the characteristic action (S320).

[0025] Next, data containing the AI ​​processing results of the audio data (audio data in which the time period in which the characteristic sound occurred has been narrowed down) is read from the storage unit 24 to the CPU memory 20b (S330). The occurrence time of the characteristic sound in the time period that overlaps with the narrowed-down range of the characteristic action in the image data is detected, and the target time in which the specific task is expected to be completed is determined. Here, when determining the target time, if the characteristic sound is detected multiple times, the characteristic sound that can be determined to be a characteristic sound associated with the characteristic action in chronological order is identified as the characteristic sound for that task (S340).

[0026] After determining the target time, the time required from the start to the end of the work is calculated (S350) and saved as the final result in the storage unit 24 (S360).

[0027] With a work time measurement device 10 capable of performing such processing, it is possible to determine the transition between work processes and accurately measure the work time for each work process, despite having a simple configuration consisting of an imaging means 12, a sound collection means 14, and the device body 16. Furthermore, by using both image data and sound data in the final determination, it is possible to accurately measure the work time of a specific worker even if multiple workers are performing the same task in adjacent spaces.

[0028] [Application to actual work] The work time measuring device 10, having the features described above, can also measure the work time (lap time) for each task when multiple tasks are performed in succession. For example, in the case of creating a cable with couplers at both ends, three steps are performed in succession: stripping the insulation from the insulated cable (task A), crimping terminals onto the stripped core wires (task B), and inserting the terminals into the couplers (task C). In such a series of tasks, each work operation can be broadly divided into five operations ae shown in Figure 6. These are: holding the cable in one hand and a tool in the other (operation a), bringing both hands close together to work with the tool (operation b), separating both hands to strip the insulation (operation c), holding the cable in one hand and a component in the other (operation d), and bringing both hands close together to assemble (operation e).

[0029] Then, when these actions are applied to the time sequence of each task (task A, task B, task C), it can be shown as in Figure 7. First, task A proceeds as follows: holding the cable in one hand and the stripper in the other (action a1), bringing both hands closer together to cut the cable's insulation (action b1), and separating both hands to strip the insulation (action c). A characteristic sound α, "squeak," is emitted when action c is performed.

[0030] Next, task B proceeds as follows: holding the cable in one hand and the terminal in the other (action d1), bringing both hands closer together to align the terminal with the core wire (action e1), holding the cable (with terminal attached) in one hand and the crimping tool in the other (action a2), and bringing both hands closer together to crimp the terminal (action b2). A characteristic sound β, "kashu," is emitted when action b2 is performed.

[0031] Furthermore, task C involves holding a cable (with terminals) in one hand and a coupler in the other (action d2), and then bringing both hands closer together to insert the terminals into the coupler (action e2). A characteristic sound γ, "snap," is emitted when action e2 is performed. In other words, each action accompanied by a characteristic sound is considered a characteristic action.

[0032] When measuring the working time of such actual work using the work time measuring device 10, it is possible to accurately determine the completion of each task and measure the time required for the task by pre-learning the nine actions and three characteristic sounds (α, β, γ) for the three tasks mentioned above, and by teaching the device that the three actions (c, b2, e2) accompanied by these characteristic sounds are characteristic actions.

[0033] Then, in the image data filtering process, multiple image frames (times) containing characteristic actions are selected from among these multiple actions. Furthermore, a characteristic sound is audio data that includes the frequency band of a previously learned characteristic sound, and the process of filtering the audio data involves selecting a predetermined time period that includes the detected frequency band.

[0034] Furthermore, in the logic processing, it is determined whether a characteristic sound is emitted during a time period that overlaps with the time period of the image frame containing the characteristic action. If there is a time period in which the characteristic action and the characteristic sound overlap, this is taken as the time of completion of the task. Based on the time difference between the completion of the previous task (e.g., task A) and the time when the characteristic action b2 and characteristic sound β overlap, the work time (wrap type) for task B (crimping terminals onto stripped core wires) is calculated. By making such a determination regarding the presence or absence of a characteristic sound, even when there are actions similar to a characteristic action (for example, in the case of action b2, action b1 is a similar action, and in the case of action e2, action e1 is a similar action), and it is difficult to determine whether or not it is a characteristic action based on the image alone, adding the presence or absence of a characteristic sound as a determination criterion makes it easier to determine that it is a characteristic action.

[0035] When performing the actual work, as shown in Figure 8, duplicate feature actions (1), feature sounds that do not match the image data (2), and similar actions that cannot be distinguished (3) may be mixed in. Such feature actions, feature sounds, and similar actions are removed at the logic processing stage as false detections for various reasons (they are not recognized as feature actions or feature sounds).

[0036] For example, the overlapping feature action shown in (1) does not coincide with the occurrence of feature sound α. Therefore, there was a mistake in the stripping process, and the identical action that occurs later in the timeline and coincides with feature sound α is determined to be feature action c, and is considered to be the end of task A.

[0037] Furthermore, the detection of characteristic sounds not caused by the actions shown in (2) is determined to be sounds emitted by surrounding workers performing the same task, and is not considered a characteristic sound of task C in the task being measured.

[0038] Furthermore, a similar action b (an action that appears to be the worker performing some kind of task with a tool, with both hands brought close together) detected after characteristic action b2 as shown in (3) is not considered a characteristic action because it is not accompanied by any characteristic sound and is therefore judged to be simply an action mistake by the worker. Thus, the time when characteristic action b2 accompanied by characteristic sound β is detected is determined to be the completion of task B. [Industrial applicability]

[0039] In the above embodiment, the process of attaching a coupler to a cable was used as an example of actual work. However, if the work as a whole, or each step of the work, produces characteristic movements and characteristic sounds upon completion, the work time can be measured using the work time measuring device 10 according to the present invention. [Explanation of symbols]

[0040] 10...Work time measurement device, 12...Imaging means, 14...Sound collection means, 16...Device body, 18...Calculation unit, 20a...CPU, 20b...CPU memory, 22a...GPU, 22b...GPU memory, 24...Storage unit, 26...Storage medium.

Claims

1. An imaging means for acquiring image data, A sound collection method for acquiring audio data, The device comprises a main body comprising: a calculation unit that performs various processing on the image data and the audio data; a storage unit that stores data related to AI that performs judgments based on pre-learned training data, a program that executes instructions for various processing via the calculation unit, and the image data and the audio data, A work time measuring device characterized by: detecting a single characteristic action defined for each target work from the image data according to the instructions of the AI, and if the characteristic action is detected multiple times in succession, narrowing down the operation time to a predetermined time before and after the timing when the last detected action was detected in chronological order; detecting a single characteristic sound defined for each characteristic action of the target work from the audio data; and, if the characteristic sound after the audio AI processing is detected multiple times, determining whether a characteristic sound corresponding to the characteristic action exists in a time period that matches the time period narrowed down by the image AI processing in chronological order, and if a match exists, executing a logic process that determines the time when the corresponding characteristic sound occurred as the completion of the work.

2. The imaging means and the sound collection means are configured in a single device, and after acquiring the image data and the sound data as video data, The work time measuring device according to claim 1, characterized in that the data is taken into the storage unit and divided into the image data and the audio data.

3. The work time measuring device according to claim 1 or 2, characterized in that the image AI processing divides the image data into unit image data of a predetermined time unit and processes each of the multiple unit image data in parallel.

Images