Information processing device and information processing method

Abstract

To provide an information processor that can determine the type of a work being done in a work field on the basis of video image data obtained by imaging with a visible light camera.SOLUTION: The information processor acquires plural pieces of consecutive frame image data obtained by imaging with a visible light camera. The visible light camera takes an image of a work field as an object where a worker is doing a work which involves blinking of light. The information processor detects the region of the worker in first frame image data of the plural pieces of frame image data. The information processor generates image data showing the change of state of the object on the basis of the first frame image data and second frame image data ahead of the first frame image data by a predetermined number of pieces of data. The information processor extracts the image data of the region of the worker from the generated image data, and determines the type of the work being done in the work field on the basis of the extracted image data.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present disclosure relates to an information processing device and an information processing method. [Background technology]

[0002] 2. Description of the Related Art Conventionally, as disclosed in Japanese Patent Laid-Open Publication No. 7-146897 (Patent Document 1), for example, a system is known that classifies the work content in a workplace while playing back an image of the workplace captured by an infrared video camera.

[0003] More specifically, in this system, the contents of a video tape recorded by an infrared video camera are played back on a video deck. The human body recognition unit in the system's controller tracks the red part displayed on the infrared video monitor that corresponds to the worker, and inputs the movement of the red part to the condition comparison unit via the mode switching unit. The condition comparison unit in the controller reads the conditions stored in the condition storage unit, compares them with the data sent from the human body recognition unit, and classifies the work being performed by the worker. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 7-146897 Summary of the Invention [Problem to be solved by the invention]

[0005] The system of Patent Document 1 requires that the subject be imaged by an infrared video camera, and therefore lacks versatility.

[0006] The present disclosure provides an information processing device and an information processing method capable of determining the type of work being performed in a workplace based on moving image data (a plurality of frame image data) captured by a visible light camera. [Means for solving the problem]

[0007] According to an aspect of the present disclosure, an information processing device includes an acquisition means for acquiring a plurality of consecutive frame image data obtained by imaging with a visible light camera. The visible light camera is fixed in its installation position and posture, and captures an image of a workplace where a worker is performing work involving blinking light as a subject. The information processing device further includes a detection means for detecting an area of ​​the worker in a first frame image data of the plurality of frame image data, a generation means for generating image data showing a change in state of the subject based on the first frame image data and a second frame image data of the plurality of frame image data that is a predetermined number of frames before the first frame image data, an extraction means for extracting image data of an area corresponding to the detected area of ​​the worker from the generated image data, and a determination means for determining a type of work being performed in the workplace based on the extracted image data.

[0008] According to another aspect of the present disclosure, an information processing method includes a step of capturing an image of a workplace where a worker is performing work involving blinking light by a visible light camera as a subject. The visible light camera is installed at a fixed position and attitude. The information processing method further includes a step of acquiring a plurality of consecutive frame image data captured by the visible light camera, a step of detecting an area of ​​the worker in a first frame image data of the plurality of frame image data, a step of generating image data showing a change in a state of the subject based on the first frame image data and a second frame image data of the plurality of frame image data that is a predetermined number of frames before the first frame image data, a step of extracting image data of an area corresponding to the area of ​​the detected worker from the generated image data, and a step of determining a type of work being performed in the workplace based on the extracted image data. Effect of the Invention

[0009] According to the present disclosure, it is possible to determine the type of work being performed in a workplace based on a plurality of frame image data obtained by capturing images using a visible light camera. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram for explaining a schematic configuration of a determination system. [Diagram 2] FIG. 2 is a diagram illustrating a hardware configuration of an information processing device. [Diagram 3] FIG. 2 is a diagram for explaining an overview of a process executed by an information processing device. [Figure 4] FIG. 11 is a flowchart showing the flow of a determination process. [Diagram 5] FIG. 5 is a flow chart showing details of the process in step S1 of FIG. 4. [Figure 6] FIG. 5 is a flow chart showing details of the process in step S2 of FIG. [Figure 7] FIG. 7 is a flow diagram for explaining details of the process of step S202 in FIG. 6. [Figure 8] FIG. 8 is a diagram for explaining the processing of FIG. 7 using image data. [Figure 9] FIG. 7 is a flowchart for explaining details of the process of step S206 in FIG. 6. [Figure 10] 13 is a diagram showing data including the final judgment result stored in memory in a highly compatible file format. FIG. [Figure 11] FIG. 8 is a flowchart showing a modified example of the series of processes in step S202 shown in FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment will be described with reference to the drawings. In the following description, the same components are denoted by the same reference numerals. The names and functions of the components are also the same. Therefore, detailed description thereof will not be repeated.

[0012] First, some of the terms used in the present embodiment will be explained.

[0013] The "trained model" refers to an "inference program" incorporated with "trained parameters". The "trained parameters" refer to the obtained parameters (coefficients) as a result of learning using a learning dataset. The trained parameters are generated by mechanically adjusting them for a certain purpose by inputting the learning dataset to the learning program. The "inference program" refers to a program that enables a certain result to be output for an input by applying the incorporated trained parameters.

[0014] The "learning program" refers to a program that executes an algorithm for finding certain rules from a learning dataset and generating a model that represents those rules. Specifically, a program that defines the procedures to be executed by a computer to realize learning by the adopted learning method corresponds to this.

[0015] The "learning dataset" refers to secondary processed data generated by performing preprocessing such as removing missing values and outliers, adding separate data such as label information (correct data), or combining these on raw data, and performing transformation and / or processing to facilitate analysis by the target learning method. The learning dataset is a collection of data (hereinafter also referred to as "learning data"). In the present embodiment, the learning data includes one piece of image data and label information.

[0016] <A. System Configuration> FIG. 1 is a diagram for explaining the schematic configuration of the determination system according to the present embodiment.

[0017] As shown in FIG. 1, the determination system 1000 includes a camera 1 and an information processing device 2.

[0018] Camera 1 is a visible light camera. The installation position and posture of camera 1 are fixed. Camera 1 captures an image of a workplace where a worker 900 is performing work accompanied by blinking light as a subject. Examples of work accompanied by blinking light include welding work, grinding work, and gouging work. Note that welding work includes spot welding.

[0019] Moving image data Da captured by the camera 1 is sent to an information processing device 2. The moving image data Da includes a plurality of consecutive frame image data #1, #2, #3, . . .

[0020] The information processing device 2 is used by a user 950. The information processing device 2 is typically a personal computer. The information processing device 2 acquires moving image data Da captured by the camera 1 from the camera 1. The information processing device 2 may acquire the moving image data Da via another device such as a server device. The information processing device 2 may also acquire the moving image data Da via a storage medium such as an IC card or a USB memory.

[0021] FIG. 2 is a diagram showing a hardware configuration of the information processing device 2. As shown in FIG.

[0022] 2, the information processing device 2 includes a processor 201, a memory 202, a display 203, an input device 204, a communication interface 205, a card reader 206, and a USB port 207. The memory 202 includes a read only memory (ROM) 221, a random access memory (RAM) 222, a solid state drive (SSD) 223, and a hard disk drive (HDD) 224.

[0023] An operating system and various programs including a trained model are stored in the memory 202. Programs for executing various processes described below are stored in the memory 202. The processor 201 executes the operating system and the above programs.

[0024] The input device 204 receives operation inputs from the user 950. The input device 204 is typically a keyboard or a mouse.

[0025] Based on the operations received by the input device 204, the processor 201 executes various processes. The processor 201 displays various information on the display 203. The processor 201 displays the execution results of the program on the display.

[0026] The communication interface 205 is an interface for communicating with external devices. The processor 201 acquires moving image data Da from the camera 1 via the communication interface 205.

[0027] The card reader 206 reads data stored in the IC card. A USB memory is connected to the USB port 207. The processor 201 can also acquire the moving image data Da via the card reader 206 or the USB port 207.

[0028] <B. Determination of Work Type> FIG. 3 is a diagram for explaining the outline of the processes executed by the information processing apparatus 2.

[0029] As shown in FIG. 3, the information processing apparatus 2 includes a moving image data acquisition unit 10, a person region detection unit 20, a generation unit 30, an extraction unit 40, a work type determination unit 50, a storage unit 60, a display control unit 70, and a display unit 80.

[0030] Note that the moving image data acquisition unit 10 corresponds to the communication interface 205, the card reader 206, or the USB port 207. The person region detection unit 20, the generation unit 30, the extraction unit 40, the work type determination unit 50, and the display control unit 70 are functional blocks realized by the processor 201 executing programs and the like stored in the memory 202. The storage unit 60 corresponds to the memory 202. The display unit 80 corresponds to the display 203.

[0031] The moving image data acquisition unit 10 acquires moving image data Da obtained by imaging using the camera 1 from the camera 1. In detail, the moving image data acquisition unit 10 acquires a plurality of consecutive frame image data #1, #2, #3, . . . obtained by the camera 1.

[0032] In information processing device 2, detection processing by person area detection section 20 and image generation processing by generation section 30 are performed separately for a plurality of frame image data #1, #2, #3, .... Note that the timing at which the detection processing by person area detection section 20 and the image generation processing by generation section 30 are performed may be either first or simultaneously.

[0033] In detail, the moving image data Da acquired by the moving image data acquisition unit 10 is temporarily stored in a memory 202 such as the storage unit 60, and is then read out by the person area detection unit 20 and the generation unit 30.

[0034] The following mainly describes the person area detection unit 20, the generation unit 30, the extraction unit 40, the task type determination unit 50, and the display control unit 70.

[0035] (b1. Person area detection unit 20) The person area detection unit 20 detects the area of ​​the worker (hereinafter also referred to as "person area") in each of the multiple frame image data #1, #2, #3, ... (hereinafter referred to as "frame image data #N" where N is a natural number). In this example, the person area detection unit 20 is realized by the trained model M20. Note that in this example, the person area in frame image data #1 is not used, so there is no need to detect the person area in frame image data #1.

[0036] The trained model M20 receives frame image data #N as input, and outputs information indicating a person area. Specifically, the trained model M20 outputs the coordinates of the person area. More specifically, the trained model M20 extracts a rectangular area as the person area. The area can typically be represented as the coordinate values ​​of the four corners of the rectangle. In this way, the trained model M20 outputs four coordinate values ​​as the coordinates of the person area.

[0037] Person area detection unit 20 sends the coordinates of the person area to extraction unit 40 together with an identifier (e.g., frame number, timestamp) indicating frame image data #N. Specifically, person area detection unit 20 sends the coordinates of the person area in frame image data #N associated with the identifier of frame image data #N to extraction unit 40. For example, when video data Da includes K (1≦N≦K) frame image data, person area detection unit 20 sends a set of coordinates of K person areas (coordinate values ​​are 4K=4×K in total) to extraction unit 40. More specifically, in this example, the coordinates of the K person areas are temporarily stored in memory 202 such as storage unit 60, and then read out by extraction unit 40.

[0038] (b2. Generation unit 30) The generating unit 30 generates image data showing a change in the subject's state based on frame image data #N and frame image data #N-1, which is the frame immediately preceding frame image data #1. More specifically, the generating unit 30 generates image data showing a change in the subject's state by a frame difference method (also called an "inter-frame difference method"). Note that since frame image data #0 does not exist, image data showing a change in the subject's state based on frame image data #0 and frame image data #1 is not generated.

[0039] More specifically, the generation unit 30 generates masked image data #N-1 (2≦N≦K) by a frame difference method from consecutive frame image data #N-1 and frame image data #N. The frame difference method and mask processing will be described later.

[0040] A plurality of post-mask processing image data #N-1 are generated by the generating unit 30. In this manner, the generating unit 30 generates a plurality of image data each of which indicates a change in the state of the subject.

[0041] The generating unit 30 sends the masked image data #N-1 to the extracting unit 40 together with an identifier indicating the frame image data #N. Specifically, the generating unit 30 sends the masked image data #N-1 associated with the identifier of the frame image data #N to the extracting unit 40. For example, when the moving image data Da includes K frame image data, the generating unit 30 sends the K-1 masked image data to the extracting unit 40. More specifically, in this example, the K-1 masked image data are temporarily stored in a memory 202 such as the storage unit 60, and then read out by the extracting unit 40.

[0042] (b3. Extraction section 40) The extraction unit 40 extracts image data of a region corresponding to the person region detected by the person region detection unit 20 from the image data (masked image data #N-1) generated by the generation unit 30. In other words, the extraction unit 40 cuts out the image.

[0043] In detail, the extraction unit 40 uses the coordinates of the person area and the masked image data, which are assigned the same identifier, to extract image data of a rectangular area corresponding to the person area from the masked image data.

[0044] Specifically, the extraction unit 40 extracts image data of an area (more specifically, a rectangular area specified by the coordinates of four vertices of the person area) corresponding to the person area detected in the frame image data #2 from the masked image data #1 generated by applying the frame difference method to the frame image data #1 and the frame image data #2. Similarly, the extraction unit 40 extracts image data of an area corresponding to the person area detected in the frame image data #3 from the masked image data #2 generated by applying the frame difference method to the frame image data #2 and the frame image data #3. When the moving image data Da includes K frame image data, the extraction unit 40 performs such extraction processing K-1 times in total.

[0045] The extraction unit 40 sends the extracted image data #N-1 to the work type determination unit 50. In detail, in this example, the K-1 extracted image data are temporarily stored in a memory 202 such as the storage unit 60, and then read out by the work type determination unit 50.

[0046] (b4. Work type determination unit 50) The work type determination unit 50 determines the type of work being performed in the workplace based on the image data extracted by the extraction unit 40. In other words, the work type determination unit 50 classifies the work being performed in the workplace. The work type determination unit 50 includes a welding work determination unit 51, a grinding work determination unit 52, a gouging work determination unit 53, and a final determination unit 54.

[0047] Welding is a joining process for joining metals together. When welding is performed, light emission called arc light occurs. Grinding is a grinding process for cutting metal with a grindstone or the like. When grinding is performed, the ground metal powder emits light. Gouging is a process for cutting, melting, and removing metal. When gouging is performed, light emission occurs due to arc discharge. In this way, welding, grinding, and gouging are all metal processing, and light is emitted from the part to be processed during the work. However, welding, grinding, and gouging each have different light emission states. In this embodiment, the information processing device 2 focuses on the difference in the light emission states and determines the work type.

[0048] The welding operation determination unit 51 determines whether or not the operation being performed at the workplace is welding operation, based on the image data #N-1 extracted by the extraction unit 40. More specifically, in this example, the welding operation determination unit 51 is realized by a trained model M51.

[0049] The trained model M51 receives as input image data #N-1 (one piece of image data) extracted by the extraction unit 40, and outputs a degree of accuracy indicating that the work being performed in the workplace is welding work. In this example, the degree of accuracy is a value between 0 and 1 inclusive. In this way, the trained model M51 normalizes the degree of accuracy (in this example, the minimum value is 0 and the maximum value is 1) and outputs it. The higher the degree of accuracy, the more likely it is that the work being performed in the workplace is welding work.

[0050] The trained model M51 associates the accuracy calculated for each extracted image data #N-1 with the identifier of the extracted image data #N-1 and outputs the accuracy to the final decision unit 54. For example, when the video image data Da includes K frame image data, the trained model M51 outputs K-1 accuracy values ​​to the final decision unit 54.

[0051] The grinding operation determination unit 52 determines whether or not the operation being performed in the workplace is grinding operation, based on the image data #N-1 extracted by the extraction unit 40. In detail, in this example, the grinding operation determination unit is realized by the trained model M52.

[0052] The trained model M52 receives as input image data #N-1 (one piece of image data) extracted by the extraction unit 40, and outputs a degree of accuracy indicating that the work being performed in the workshop is grinding work. In this example, the degree of accuracy is a value between 0 and 1 inclusive. In this way, like the trained model M51, the trained model M52 normalizes the degree of accuracy (in this example, the minimum value is 0 and the maximum value is 1) and outputs it. The higher the degree of accuracy, the more likely it is that the work being performed in the workshop is grinding work.

[0053] Similar to the trained model M51, the trained model M52 associates the accuracy calculated for each extracted image data #N-1 with the identifier of the extracted image data #N-1 and outputs the accuracy to the final decision unit 54. For example, when the video image data Da includes K frame image data, the trained model M52 outputs K-1 accuracy to the final decision unit 54.

[0054] The gouging operation determination unit 53 determines whether or not the operation being performed in the workplace is a gouging operation, based on the image data #N-1 extracted by the extraction unit 40. In more detail, in this example, the gouging operation determination unit is realized by a trained model M53.

[0055] The trained model M53 receives as input the image data #N-1 (one piece of image data) extracted by the extraction unit 40, and outputs a degree of accuracy indicating that the work being performed in the workplace is gouging work. In this example, the degree of accuracy is a value between 0 and 1 inclusive. In this way, like the trained models M51 and M52, the trained model M53 normalizes the degree of accuracy (in this example, the minimum value is 0 and the maximum value is 1) and outputs it. The higher the degree of accuracy, the more likely it is that the work being performed in the workplace is gouging work.

[0056] Similar to the trained models M51 and M52, the trained model M53 associates the accuracy calculated for each extracted image data #N-1 with the identifier of the extracted image data #N-1 and outputs the accuracy to the final decision unit 54. For example, when the video data Da includes K frame image data, the trained model M53 outputs the K-1 accuracy to the final decision unit 54.

[0057] The final determination unit 54 determines whether the work being performed in the workplace is welding work, grinding work, gouging work, or work that cannot be classified, based on the results of the determination by the trained model M51, the results of the determination by the trained model M52, and the results of the determination by the trained model M53. Examples of other work include the movement of a worker.

[0058] Specifically, the final determination unit 54 determines whether or not there is any work whose accuracy is equal to or greater than a threshold value (e.g., 0.6) for each of the extracted image data #N-1 (each piece of image data). If there is any work whose accuracy is equal to or greater than the threshold value, the final determination unit 54 determines that the work is work being performed in a workplace.

[0059] For example, for one extracted image data #1, if the accuracy output from the trained model M51 is 0.7, the accuracy output from the trained model M52 is 0.1, and the accuracy output from the trained model M53 is 0.05, the final determination unit 54 determines that the work being performed in the workplace is welding work. Also, for one extracted image data #2, if the accuracy output from the trained model M51 is 0.5, the accuracy output from the trained model M52 is 0.2, and the accuracy output from the trained model M53 is 0.1, there is no accuracy that is equal to or greater than the threshold value (0.6 in this example), so the final determination unit 54 determines that the work being performed in the workplace cannot be classified. Such a determination is made for each extracted image data #N-1.

[0060] Furthermore, the final determination unit 54 performs the final determination at a predetermined cycle (for example, every second). The cycle can be appropriately set based on the frame rate of the video data Da. For example, the frame rate of the video data Da is assumed to be 60 fps (frames per second). In this case, the video data Da includes 60 frame image data per second.

[0061] Therefore, the final determination unit 54 obtains 60 determination results during one second of the video data Da. The final determination unit 54 determines (hereinafter also referred to as the "final determination") that the work type with the largest number of results among the 60 determination results is the work being performed in the workplace during that period (one second).

[0062] For example, if, out of 60 judgment results in a one-second period, there were 40 judgments of welding work, 4 judgments of grinding work, 0 judgments of gouging work, and 16 judgments that classification was not possible, the final judgment unit 54 will judge the work type for that period to be welding (final judgment).

[0063] The final judgment unit 54 stores the result of the final judgment in the storage unit 60. Specifically, the final judgment unit 54 stores the result of the final judgment in the storage unit 60 in association with the moving image data Da. More specifically, the final judgment unit 54 synchronizes the result of the final judgment with the moving image data Da. The final judgment unit 54 associates the result of the final judgment with each frame image data on which the final judgment was made.

[0064] The association may be based on the identifier of the frame image data, or on the elapsed time from the start of playback of the video data Da.

[0065] (b5. Display control unit 70) The display control unit 70 controls the display on the display unit 80. Based on user operations, the display control unit 70 displays the result of the final determination together with the moving image data Da. According to the above-described association, the result of the final determination changes sequentially as the reproduction of the moving image data Da progresses. In this example, when the moving image data Da is reproduced, the result of the final determination is updated every second.

[0066] <C trained model> The trained models M20, M51, M52, and M53 will be described.

[0067] The trained model M20 is generated by a pre-prepared training dataset and a training program. The training dataset includes a plurality of training data. Each piece of training data is obtained by adding label information (correct data) indicating the person area in the image data to the image data (still image data, frame image data) of a worker working at the workplace. In this example, the person area of the label information is specified by a rectangular area.

[0068] Similar to the trained model M20, the trained model M51 is generated by a pre-prepared training dataset and a training program. The training dataset includes a plurality of training data. Each piece of training data is obtained by adding label information (correct data) indicating that the work type is welding to the image data of a worker performing welding work at the workplace.

[0069] Similar to the trained models M20 and M51, the trained model M52 is generated by a pre-prepared training dataset and a training program. The training dataset includes a plurality of training data. Each piece of training data is obtained by adding label information indicating that the work type is a grinder to the image data of a worker performing grinder work at the workplace.

[0070] The trained model M53, like the trained models M20, M51, and M52, is generated by a training data set and a training program that are prepared in advance. The training data set includes a plurality of training data. Each training data is image data of a worker performing gouging work in a workplace, to which label information indicating that the work type is gouging has been added.

[0071] The trained models M20, M51, M52, and M53 are networks classified as DNNs (Deep Neural Networks). The trained models M20, M51, M52, and M53 include a preprocessing network classified as a CNN (Convolutional Neural Network), an intermediate layer, an activation function corresponding to an output layer, and a Softmax function.

[0072] The preprocessing network is expected to function as a kind of filter for extracting effective features for calculating estimation results from features with relatively large degrees. The preprocessing network has a configuration in which convolutional layers (CONV) and pooling layers (Pooling) are arranged alternately. Note that the number of convolutional layers and pooling layers does not have to be the same, and an activation function such as ReLU (rectified linear unit) is arranged on the output side of the convolutional layer.

[0073] More specifically, the preprocessing network is constructed to receive input of features and output internal features that indicate predetermined attribute information. The intermediate layer is made up of a fully connected network having a predetermined number of layers, and sequentially connects the outputs from the preprocessing network for each node using weights and biases determined for each node.

[0074] An activation function such as ReLU is placed on the output side of the intermediate layer, and finally, the estimation result is output after being normalized to a probability distribution using the Softmax function.

[0075] When the learning program optimizes the values of the parameters, any optimization algorithm can be used. More specifically, as the optimization algorithm, for example, gradient methods such as SGD (Stochastic Gradient Descent), Momentum SGD (SGD with added inertia term), AdaGrad, RMSprop, AdaDelta, Adam (Adaptive moment estimation) can be used.

[0076] <D. Flow of processing> The flow of the above-described processing in the information processing apparatus 2 will be further described with reference to a flowchart or the like.

[0077] FIG. 4 is a flowchart showing the flow of the determination process.

[0078] As shown in FIG. 4, in step S1, the information processing apparatus 2 detects a person area in each frame image data #N constituting the moving image data Da. In step S2, the information processing apparatus 2 determines the work type for each frame image data #N using the detection result of the person area, and further executes a final determination every second based on the determination result for each frame image data #N.

[0079] The processes of step S1 and step S2 are executed by the processor 201. The process of step S1 is executed by the person area detection unit 20 (FIG. 3). Specifically, the process of step S1 is realized by the learned model M20. The process of step S2 is executed by the work type determination unit 50 (FIG. 3). Specifically, the process of step S2 is realized by the learned models M51, M52, M53.

[0080] FIG. 5 is a flowchart showing the details of the process of step S1 in FIG. 4.

[0081] 5, in step S101, the processor 201 accepts a setting input of the number of people to be determined via the input device 204. Typically, the processor 201 accepts an input indicating one person (single person work) or two people (two people work). The reason for accepting a setting input of the number of people is to improve the accuracy of detecting a person area.

[0082] In step S102, the processor 201 reads the video data Da from the memory 202. In step S103, the processor 201 reads the trained model M20 for human area detection from the memory 202. In step S104, the processor 201 executes human area detection processing using the trained model M20 for each frame image data #N.

[0083] In step S105, processor 201 adjusts the determination in accordance with the number of people set in step S101. Specifically, processor 201 selects person regions for the set number of people in the same frame image data in descending order of accuracy of the determination of the person region. For example, when the set number of people is one, processor 201 selects the person region with the highest accuracy from multiple person regions (candidate regions). When the set number of people is two, processor 201 selects the person region with the highest accuracy and the person region with the second highest accuracy from multiple person regions (candidate regions). In step S106, processor 201 executes post-processing such as front and back complementation.

[0084] In step S107, processor 201 saves the person area detection result (coordinates) in a predetermined format in memory 202 for each frame image data #N. Processor 201 typically saves the person area detection result as data in one of the highly compatible file formats (for example, csv (Comma Separated Value) format). In detail, processor 201 determines the person area detection (coordinates) in association with information such as a timestamp, the frame number of the frame image data, the object number, and the accuracy of the determination of the person area detection.

[0085] In step S108, the processor 201 creates video data Dc of the person region detection result, and stores it in the memory 202. Note that the video data Dc is video data Da in which a figure (rectangle) indicating a person region is superimposed.

[0086] FIG. 6 is a flow diagram showing details of the process of step S2 in FIG.

[0087] As shown in Fig. 6, in step S201, the processor 201 reads video data Da from the memory 202. In step S202, the processor 201 creates frame difference video data Db using a frame difference method. This process is executed by the generator 30 (Fig. 3). Details of the process of step S202 will be described later (Fig. 7).

[0088] In step S203, the processor 201 reads the person area detection result from the memory 202. Note that the person area detection result includes information such as the coordinates of the person area and the identifier (timestamp or frame number) of the frame image data described above.

[0089] In step S204, the processor 201 extracts an area corresponding to the person area from each frame image data P constituting the frame difference moving image data Db based on the person area detection result. In other words, the processor 201 cuts out the image. More specifically, the processor 201 uses the person area detection result associated with each frame image data P to extract an area corresponding to the person area from each frame image data P. More specifically, the processor 201 cuts out the image using a different person area detection result for each frame image data P.

[0090] In step S205, the processor 201 reads the trained models M51, M52, and M53 for determining the type of work from the memory 202. In step S206, the processor 201 executes the trained models M51, M52, and M53 to perform a process of determining the type of work from each image data Q (image data of the cut-out portion) extracted in step S204. Details of the process in step S206 will be described later (FIG. 9).

[0091] In step S207, processor 201 outputs the task type determination result in step S206 as the final determination result for each second. Specifically, as described above, processor 201 determines (final determination) the task type that occurs most frequently in each second as the task being performed in the workplace during that second. The final determination results for each second are sequentially temporarily stored in a work area of ​​memory 202 (typically, RAM 222).

[0092] In step S208, when the processor 201 determines that the work in one second cannot be classified, the processor 201 executes a movement determination process to determine whether the work is a movement of a worker. The movement determination process will be described later in detail.

[0093] In step S209, the processor 201 executes post-processing such as front and rear complementation. In step S210, the processor 201 stores the final determination result for each second in a predetermined format in the memory 202 in a non-volatile manner. Typically, the processor 201 stores the final determination result in the SSD 223 or the HDD 224 in a CSV format. In detail, the processor 201 determines the final determination result in association with information such as a timestamp and a frame number.

[0094] In step S211, the processor 201 creates video data Dd including the final determination result, and stores it in the memory 202. Note that the video data Dd is obtained by superimposing an image indicating the final determination result using identification information such as characters on the video data Da.

[0095] When the user 950 plays back the video data Dd on the information processing device 2, the work type is displayed superimposed on the image of the work in the workplace on the display 203. The display of the work type is updated every second.

[0096] Next, the movement determination process in step S208 will be described. In the movement determination process, first, the processor 201 calculates the average values ​​of the width and height of each detected person area. That is, the processor 201 calculates the position of the center of gravity of each person area within frame image data #N.

[0097] Next, the processor 201 calculates the average value of the center of gravity position for each second. After that, the processor 201 associates each average value of the center of gravity position with the final determination result at the same time (timing). Furthermore, the processor 201 calculates the amount of movement of the person area for each second based on each average value of the center of gravity position.

[0098] Processor 201 extracts the final determination result that is other work (unclassifiable) from the work area of ​​memory 202. Processor 201 then determines whether or not the movement amount associated with the extracted final determination result is within a predetermined range (between a lower threshold and an upper threshold). If the movement amount is within the predetermined range, processor 201 replaces the information of other work (unclassifiable) with "movement".

[0099] Fig. 7 is a flow diagram for explaining details of the process of step S202 in Fig. 6. Fig. 8 is a diagram for explaining the process of Fig. 7 using image data.

[0100] As shown in Fig. 7, in step S2201, the processor 201 sets the value of the above-mentioned variable N (#N) to 2. In step S2202, the processor 201 acquires two consecutive frame image data #N-1, #N from the loaded video data Da. Fig. 8 shows examples of frame image data #N-1 and frame image data #N.

[0101] In step S2203, the processor 201 generates differential image data R (see FIG. 8) representing the difference between frame image data #N-1 and frame image data #N. In step S2204, the processor 201 binarizes the differential image data R to generate binarized image data T (see FIG. 8).

[0102] In step S2205, the processor 201 performs a closing process on the binarized image data T to generate closing image data U (see FIG. 8). The closing image data U is used as a mask image. In step S2206, the processor 201 performs a masking process on the frame image data #N with the closing image data U. As a result, image data V after the mask process (see FIG. 8) is generated.

[0103] In step S2207, the processor 201 saves the masked image data V as new image data #N-1 in the memory 202. In step S2208, the processor 201 determines whether the video data Da is complete. Specifically, the processor 201 determines whether the above-mentioned process has been performed on all two consecutive frame image data of the video data Da.

[0104] If it is determined that the video data Da is not complete (NO in step S2208), the processor 201 increments the value of N by 1 in step S2210. If it is determined that the video data Da is complete (YES in step S2208), the processor 201 generates the above-mentioned frame difference video data Db by connecting all of the new image data #N in chronological order in step S2209, and stores it in the memory 202.

[0105] After step S2209, the processor 201 advances the process to step S203 in FIG.

[0106] FIG. 9 is a flowchart for explaining details of the process in step S206 in FIG.

[0107] As shown in FIG. 9, in step S2601, the processor 201 reads each piece of extracted image data Q from the memory 202 (image data of the cut-out portion).

[0108] In step S2602, the processor 201 performs a judgment using each of the trained models M51, M52, and M53 for one image data Q. Specifically, the processor 201 executes each of the trained models M51, M52, and M53 to calculate the accuracy of each operation (welding operation, grinding operation, and gouging operation) as described above. In step S2603, the processor 201 determines whether the maximum accuracy among the calculated accuracy of the three operations exceeds a threshold value (for example, 0.6).

[0109] If it is determined that the maximum accuracy exceeds the threshold (YES in step S2603), the processor 201 adopts the determination of the maximum accuracy in step S2604. As a specific example, if the accuracy output from the trained model M51 is 0.7, the accuracy output from the trained model M52 is 0.1, and the accuracy output from the trained model M53 is 0.05, the processor 201 adopts the determination by the trained model M51. The processor 201 determines that the work type is welding.

[0110] If it is determined that the maximum accuracy does not exceed the threshold value (NO in step S2603), the processor 201 determines in step S2607 that classification is impossible. In step S2605, the processor 201 stores the determination result in the memory 202.

[0111] In step S2606, the processor 201 determines whether the image data Q is the last one. If it is determined that the image data Q is the last one (YES in step S2606), the processor 201 ends the series of processes in step S206 and advances the process to step S207 in FIG. 6. If it is determined that the image data Q is not the last one (NO in step S2606), the processor 201 switches the processing target to the next image data Q in step S2608. Then, the processor 201 advances the process to step S2602.

[0112] <E. Example of final determination result> FIG. 10 is a diagram showing data including the final determination result stored in the memory 202 in a highly compatible file format (in this example, the csv format).

[0113] As shown in FIG. 10, the final determination results are recorded every second in the "Predict" column of the data. "Indistinguishable" represents "unclassifiable", "Moving" represents "moving", and "Welding" represents "welding".

[0114] Note that "G_X" and "G_Y" represent the X coordinate and Y coordinate of the center of gravity of the person area, respectively. Specifically, "G_X" and "G_Y" are the average values of the center of gravity positions within one second. "Width" and "Height" represent the width and height of the person area, respectively.

[0115] In this way, according to the information processing apparatus 2, the work type is specified every second. Therefore, if the time is accumulated for each work type, the time required for the work of each work type can be calculated. The information processing apparatus 2 calculates such time, for example, in response to a user operation, and causes the calculated result (work time for each work type) to be displayed on the display 203.

[0116] <F. Parentheses> Part of the processing executed by the information processing apparatus 2 is enclosed in parentheses as follows.

[0117] (1) Information processing device 2 includes a moving image data acquisition unit 10 that acquires a plurality of consecutive frame image data (moving image data, video data) captured by camera 1 (visible light camera). Camera 1 is fixed in its installation position and orientation, and captures an image of a workplace where workers are performing work involving blinking light as a subject.

[0118] The information processing device 2 further includes a person area detection unit 20 that detects the worker's area (person area) in frame image data #N of the above-mentioned multiple frame image data, a generation unit 30 that generates image data indicating a change in the state of the subject based on the frame image data #N and frame image data #N-1 that precedes frame image data #N of the above-mentioned multiple frame image data, an extraction unit 40 that extracts image data of an area corresponding to the detected worker's area from the generated image data, and a work type determination unit 50 that determines the type of work being performed in the workplace based on the extracted image data.

[0119] According to the information processing device 2 configured as above, it is possible to extract image data showing a state change in the area of ​​the worker from a plurality of frame image data obtained by imaging with a visible light camera. Furthermore, according to the information processing device 2, the type of work being performed in the work field is determined based on the image data showing the state change in the area of ​​the worker.

[0120] Therefore, the information processing device 2 can determine the type of work being performed in the workplace based on the video data captured by the visible light camera. In particular, the type of work that involves blinking of light can be determined with high accuracy for each frame of image data.

[0121] (2) The information processing device 2 further includes a storage unit 60 that stores information indicating the type of work that has been determined. With this configuration, the information processing device 2 stores the results of the determination, and can therefore perform various post-processing operations (for example, the above-mentioned final determination process and display process) using the results.

[0122] (3) The generator 30 generates image data showing a change in the state of the subject by a frame difference method using frame image data #N and frame image data #N-1. With this configuration, image data showing a change in the state of the subject can be generated by using the frame difference method, which is one of the methods for detecting moving objects.

[0123] (4) The work type determination unit 50 determines whether or not the work being performed in the workplace is welding. With this configuration, it is possible to determine whether or not the work accompanied by blinking of light is welding.

[0124] (5) The task type determination unit 50 determines which of a plurality of pre-specified tasks is being performed in the workplace. With this configuration, it is possible to determine which of the plurality of tasks is being performed that involves blinking of light.

[0125] (6) The plurality of operations include a welding operation, a grinding operation, and a gouging operation. With this configuration, it is possible to determine whether the operation accompanied by the blinking of the light is a welding operation, a grinding operation, or a gouging operation.

[0126] (7) The work type determination unit 50 includes a trained model M51 that receives the extracted image data as input and determines whether the work being performed in the workplace is welding work, a trained model M52 that receives the extracted image data as input and determines whether the work being performed in the workplace is grinding work, and a trained model M53 that receives the extracted image data as input and determines whether the work being performed in the workplace is gouging work.

[0127] The work type determination unit 50 determines whether the work being performed in the workplace is welding work, grinding work, or gouging work based on the results of determination by the trained model M51, the results of determination by the trained model M52, and the results of determination by the trained model M53.

[0128] According to such a configuration, the information processing apparatus 2 determines the type of work using the learned models M51, M52, and M53. Therefore, according to the information processing apparatus 2, a determination with higher accuracy can be made compared to the rule-based determination process that does not use the learned model.

[0129] (8) The person area detection unit 20 is a learned model M20 that takes the frame image data #N as input and outputs information indicating the area of the worker. According to such a configuration, the information processing apparatus 2 detects the area of the worker (person area) using the learned model M20. Therefore, according to the information processing apparatus 2, a detection with higher accuracy can be made compared to the rule-based detection process that does not use the learned model.

[0130] (9) For each of the plurality of frame image data, the information processing apparatus 2 performs detection by the person area detection unit 20, generation by the generation unit 30, extraction by the extraction unit 40, and determination by the work type determination unit 50. The information processing apparatus 2 calculates the working time for each work type based on the number of determinations for each work type determined by the work type determination unit 50.

[0131] According to such a configuration, the information processing apparatus 2 calculates the total working time for each work based on the plurality of frame image data obtained by imaging with the camera 1. Therefore, the user 950 of the information processing apparatus 2 can grasp how much time is required for each work.

[0132] <G. Modification Example> (1) FIG. 11 is a flowchart showing a modification example of the series of processes in step S202 shown in FIG. 7. Hereinafter, a configuration using the frame difference method using three consecutive frame image data will be described.

[0133] 11, the series of processes shown in Fig. 11 differs from the series of processes shown in Fig. 7 in the following points. The series of processes shown in Fig. 11 includes steps S2202A, S2203A, S2204A, and S2205A instead of steps S2202, S2203, S2204, and S2205 (see Fig. 7). Furthermore, the series of processes shown in Fig. 11 differs from Fig. 7 in that it includes step S2211, which does not include this step.

[0134] The processing of other steps in Fig. 11 is the same as the processing described in Fig. 7. Therefore, the following will describe these steps S2202A, S2203A, S2204A, S2205A, and S2211.

[0135] In step S2202A after step S2201, the processor 201 obtains three consecutive frame image data #N-1, #N, and #N+1 from the loaded video image data Da. In step S2203A, the processor 201 generates differential image data R representing the difference between frame image data #N-1 and frame image data #N, and differential image data R representing the difference between frame image data #N and frame image data #N+1.

[0136] In step S2204A, the processor 201 binarizes each differential image data R to generate two binary image data T. In step S2211, the processor 201 extracts the shared portion of the two binary image data T to generate image data W. Specifically, the processor 201 sets the portions of the two binary image data T that are both white (i.e., portions with a value of 1) to white (value 1), and sets the rest to black (value 0).

[0137] In step S2205A, the processor 201 performs a closing process on the extracted image data W to generate closing image data U. After that, the processor 201 advances the process to step S2206.

[0138] More specifically, in this modified example, the generator 30 generates image data indicating a change in the state of the subject by a frame difference method using frame image data #N, frame image data #N-1, and frame image data #N+1.

[0139] According to such a process, a more accurate determination process than that in the configuration of FIG. 7 is possible.

[0140] (2) In the above, an example has been described in which the generating unit 30 generates image data showing a change in the subject's state using consecutive frame image data #N and #N-1, but the present invention is not necessarily limited to this. The generating unit 30 may be configured to generate image data showing a change in the subject's state based on frame image data #N and a frame image data that is a predetermined number of frames before (one or more frames before) frame image data #N. For example, the generating unit 30 may generate image data showing a change in the subject's state based on frame image data #N and frame image data #N-2 that is two frames before frame image data #N.

[0141] (3) If image data showing changes in the state of a subject can be generated, moving object detection methods other than the frame difference method can also be applied.

[0142] (4) It is not necessary to use a trained model to detect the worker's area (person area). The worker's area may be detected by a rule-based method.

[0143] The embodiments disclosed herein are merely examples and are not limited to the above. The scope of the present invention is defined by the claims, and it is intended to include all modifications within the scope and meaning equivalent to the claims. [Explanation of symbols]

[0144] 1 camera, 2 information processing device, 10 video data acquisition unit, 20 person area detection unit, 30 generation unit, 40 extraction unit, 50 work type determination unit, 51 welding work determination unit, 52 grinding work determination unit, 53 gouging work determination unit, 54 final determination unit, 60 storage unit, 70 display control unit, 80 display unit, 201 processor, 202 memory, 203 display, 204 input device, 205 communication interface, 206 card reader, 207 port, 900 worker, 950 user, 1000 determination system, M20, M51, M52, M53 trained models.

Claims

1. an acquisition means for acquiring a plurality of consecutive frame image data obtained by imaging with a visible light camera, the visible light camera being installed at a fixed position and attitude, and capturing an image of a workplace where a worker is performing work involving blinking light as a subject; a detection means for detecting an area of ​​the worker in a first frame image data of the plurality of frame image data; a generating means for generating difference data between the first frame image data and a second frame image data that is a predetermined number of frames before the first frame image data among the plurality of frame image data by a frame difference method, and for generating image data that indicates a change in state of the subject based on the generated difference data and the first frame image data; an extraction means for extracting image data of an area corresponding to an area of ​​the detected worker from the generated image data; The information processing device further comprises a determination means for determining whether a type of work being performed at the workplace is welding work, grinding work, or gouging work, based on the extracted image data.

2. The information processing apparatus according to claim 1 , wherein the second frame image data is a frame image data immediately preceding the first frame image data.

3. The determination means is A first trained model that receives the extracted image data as an input and determines whether the work being performed at the workplace is the welding work; and A second trained model that receives the extracted image data as an input and determines whether the work being performed in the workplace is the grinding work; and a third trained model that receives the extracted image data as an input and determines whether the work being performed at the work site is the gouging work; 3. The information processing device according to claim 1 or 2, which determines whether the work being performed in the workplace is the welding work, the grinding work, or the gouging work based on a result of judgment using the first trained model, a result of judgment using the second trained model, and a result of judgment using the third trained model.

4. The information processing device according to claim 1 , wherein the detection means is a fourth trained model that receives the first frame image data as an input and outputs information indicating an area of ​​the worker.

5. For each of the plurality of frame image data, detection by the detection means, generation by the generation means, extraction by the extraction means, and determination by the determination means are performed; The information processing apparatus according to claim 1 , further comprising: a determining unit that determines a task time for each of the task types based on a determination number for each of the task types determined by the determining unit.

6. a step of capturing an image of a workplace where a worker is performing work involving blinking light by a visible light camera as a subject, the visible light camera having a fixed installation position and attitude; acquiring a plurality of consecutive frame image data captured by the visible light camera; detecting an area of ​​the worker in a first frame image data of the plurality of frame image data; generating difference data between the first frame image data and a second frame image data that is a predetermined number of frames before the first frame image data among the plurality of frame image data by a frame difference method; generating image data representing a change in a state of the subject based on the generated difference data and the first frame image data; extracting image data of an area corresponding to an area of ​​the detected worker from the generated image data; The information processing method further comprises a step of determining whether a type of work being performed at the workplace is welding work, grinding work, or gouging work based on the extracted image data.

Images