Grip period determination program, grip period determination method and information processing device

Abstract

To determine with high accuracy a period during which an operator is gripping an object.SOLUTION: A grip period determination program makes a computer execute: object recognition processing for determining whether or not an object exists for each frame of input moving image data; grip determination processing for determining whether or not the object is gripped for a frame in which the object is determined to exist by the object recognition processing; and processing for determining first frames C2, C4 in which the object is determined to be gripped by the grip determination processing and a second frame C3 in which the object is determined not to exist by the object recognition processing as a period 11d in which the object is gripped.SELECTED DRAWING: Figure 10

Description

[Technical field]

[0001] The present invention relates to a holding period determination program, a holding period determination method, and an information processing device. [Background technology]

[0002] A methodology known as "industrial engineering" (IE) is gaining attention as it accurately grasps the actual work conditions of workers at production sites and utilizes this information to implement measures to improve productivity and safety.

[0003] "Time study" is known as one of the IE methods. Time study is a method of dividing up the work content of workers into detailed processes, measuring the work time spent on each detailed process, and using the results to devise measures to improve work efficiency, such as reducing work time.

[0004] Methods for measuring the task time of a detailed process include, for example, the "stopwatch method" and the "VTR (Video Tape Recorder) method," which measure the task time of each detailed process performed by a worker. The stopwatch method is a method in which a stopwatch is used to measure task time at the work site, while the VTR method is a method in which task time is measured by analyzing a VTR that records the tasks performed by the worker. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2010-244413 A [Patent Document 2] International Publication No. 2009 / 096208 Brochure [Patent Document 3] Patent Publication No. 2021-018465 Summary of the Invention [Problem to be solved by the invention]

[0006] The above-mentioned methods of measuring work time all require manual measurement, which requires high human labor costs. In order to measure the work time of at least one detailed process among multiple detailed processes without human intervention, it is possible to use a trained machine learning model that performs AI (Artificial Intelligence) tasks such as object recognition tasks.

[0007] For example, the information processing device performs object recognition of the tool, recognition of the worker's posture, and judgment of the worker's grip of the tool for each image obtained by dividing video data of the work of the worker at the production site into frames.Then, the information processing device estimates a period during which the worker is gripping the tool based on a number of images in which it is judged that the tool is being gripped.

[0008] For example, in object recognition of a tool, an information processing device inputs an image into a machine learning model to obtain an object area indicating the position of the object and a recognition score, and if the recognition score is greater than or equal to a recognition threshold, determines that an object (tool) is present in the object area.

[0009] In a state where the entire tool is displayed in the image, such as when the tool is placed on a workbench, the recognition score for object recognition is high, and the presence of the tool is easily recognized. On the other hand, in a state where a part of the tool is not displayed (hidden or not shown) in the image due to the worker's body or the object being worked on, such as while the worker is working, the recognition score for object recognition is low, and the presence of the tool is difficult to recognize.

[0010] In this way, if part of the tool is not displayed in the image, the information processing device may determine that the worker is not holding the tool even though the worker is actually holding the tool.

[0011] In one aspect, an object of the present invention is to enable highly accurate determination of a period during which a worker is gripping an object. [Means for solving the problem]

[0012] In one aspect, the gripping period determination program may cause a computer to execute the following processes. The process may execute an object recognition process for determining the presence or absence of an object for each frame of input video data. The process may also execute a gripping determination process for determining whether or not the object is being gripped for a frame in which it is determined by the object recognition process that the object is present. Furthermore, the process may determine, as a period in which the object is being gripped, a first frame in which it is determined by the gripping determination process that the object is being gripped and a second frame in which it is determined by the object recognition process that the object is not present. Effect of the Invention

[0013] In one aspect, the present invention can determine with high accuracy the period during which an operator is gripping an object. [Brief description of the drawings]

[0014] [Figure 1] FIG. 2 is a block diagram showing an example of the hardware configuration of a computer that realizes the functions of a server according to an embodiment. [Diagram 2] FIG. 2 is a diagram for explaining an example of an imaging range of a camera. [Diagram 3] FIG. 2 is a block diagram showing an example of a software configuration of a server according to an embodiment. [Figure 4] 11A to 11C are diagrams for explaining an example of a screw fastening operation process. [Diagram 5] 11A and 11B are diagrams illustrating an example of a recognition result of a posture recognition process. [Figure 6] FIG. 11 is a diagram showing an example of a recognition result of a tool recognition process. [Figure 7] 13A and 13B are diagrams illustrating an example of a determination result of a grip determination process. [Figure 8] 13A to 13C are diagrams illustrating other examples of the determination results of the grip determination process. [Figure 9] FIG. 13 is a diagram showing an example in which the grip determination result is False. [Figure 10]FIG. 13 is a diagram for explaining an example of a holding period estimation method. [Figure 11] FIG. 13 is a diagram for explaining an example of a holding period estimation method. [Figure 12] 11 is a flowchart illustrating an example of an operation of a server according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. However, the embodiment described below is merely an example, and is not intended to exclude various modifications or application of techniques not specified below. For example, this embodiment can be modified in various ways without departing from the spirit of the invention. In the drawings used in the following description, parts with the same reference numerals represent the same or similar parts unless otherwise specified.

[0016] [A] Configuration example of one embodiment The server 10 according to an embodiment (see FIG. 3) is an example of an information processing device or a computer that determines a period during which a worker holds an object such as a tool.

[0017] [A-1] Hardware configuration example The server 10 may be a virtual server (VM: Virtual Machine) or a physical server. The functions of the server 10 may be realized by one computer or by two or more computers. Furthermore, at least a part of the functions of the server 10 may be realized using hardware (HW: Hardware) resources and network (NW: Network) resources provided by a cloud environment.

[0018] 1 is a block diagram showing an example of a HW configuration of a computer 1 that realizes the functions of a server 10 according to an embodiment. When multiple computers are used as HW resources that realize the functions of the server 10, each computer may have the HW configuration shown in FIG.

[0019] As shown in FIG. 1, the computer 1 may, as a HW configuration, illustratively include a processor 1a, a graphics processing unit 1b, a memory 1c, a storage device 1d, an IF (Interface) device 1e, an IO (Input / Output) device 1f, and a reading device 1g.

[0020] The processor 1a is an example of a processing unit that performs various controls and calculations. The processor 1a may be connected to each block in the computer 1 via a bus 1j so that they can communicate with each other. The processor 1a may be a multiprocessor including a plurality of processors, a multicore processor having a plurality of processor cores, or a configuration having a plurality of multicore processors.

[0021] Examples of the processor 1a include integrated circuits (ICs) such as a CPU, MPU, APU, DSP, ASIC, and FPGA. Note that a combination of two or more of these integrated circuits may be used as the processor 1a. CPU is an abbreviation for Central Processing Unit, and MPU is an abbreviation for Micro Processing Unit. APU is an abbreviation for Accelerated Processing Unit. DSP is an abbreviation for Digital Signal Processor, ASIC is an abbreviation for Application Specific IC, and FPGA is an abbreviation for Field-Programmable Gate Array.

[0022] The graphics processing device 1b performs screen display control for an output device such as a monitor of the IO device 1f. The graphics processing device 1b may have a configuration as an accelerator that executes machine learning processing and inference processing using a machine learning model, for example, a human recognition model 11b and a tool recognition model 11c shown in Fig. 3. The graphics processing device 1b may be various arithmetic processing devices, for example, integrated circuits (ICs) such as a GPU (Graphics Processing Unit), an APU, a DSP, an ASIC, or an FPGA.

[0023] The memory 1c is an example of HW that stores various data, programs, and other information. Examples of the memory 1c include one or both of a volatile memory such as a dynamic random access memory (DRAM) and a non-volatile memory such as a persistent memory (PM).

[0024] The storage device 1d is an example of HW that stores various data, programs, and other information. Examples of the storage device 1d include various storage devices such as a magnetic disk device such as a hard disk drive (HDD), a semiconductor drive device such as a solid state drive (SSD), and a non-volatile memory. Examples of the non-volatile memory include a flash memory, a storage class memory (SCM), and a read only memory (ROM).

[0025] The storage device 1d may store a program 1h that realizes all or part of various functions of the computer 1. The program 1h may include, for example, a holding period determination program that determines a holding period. The holding period determination program may include an inference program for executing an inference process using a trained machine learning model, or may include a machine learning program for training the machine learning model.

[0026] The processor 1a of the server 10 can realize the functions of the server 10 (for example, the control unit 19 shown in FIG. 3) by, for example, loading a program 1h stored in the storage device 1d into the memory 1c and executing it.

[0027] The IF device 1e is an example of a communication IF that controls the connection and communication between the server 10 and other computers. For example, the IF device 1e may include an adapter that complies with electrical communications such as Ethernet (registered trademark) or optical communications such as FC (Fibre Channel). The adapter may support one or both of wireless and wired communication methods.

[0028] For example, the server 10 may be connected to the camera 2 via the IF device 1e and the network 2a so that they can communicate with each other. The network 2a may include, for example, an intranet such as a LAN (Local Area Network) and / or the Internet. The server 10 may be connected to an administrator terminal via the IF device 1e and the network 2a or a network not shown so that they can communicate with each other. The administrator terminal may be used by an administrator who causes the server 10 to estimate the holding period. The program 1h may be downloaded from the network to the computer 1 via the IF device 1e and stored in the storage device 1d.

[0029] The IO device 1f may include one or both of an input device and an output device. Examples of the input device include a keyboard, a mouse, a touch panel, etc. Examples of the output device include a display device such as a monitor, a projector, a printer, etc. The IO device 1f may also include a touch panel or the like in which the input device and the output device are integrated. The output device may be connected to the graphic processing device 1b.

[0030] The reading device 1g is an example of a reader that reads out data or program information recorded on the recording medium 1i. The reading device 1g may include a connection terminal or device to which the recording medium 1i can be connected or inserted. Examples of the reading device 1g include an adapter that complies with USB (Universal Serial Bus) or the like, a drive device that accesses a recording disk, and a card reader that accesses a flash memory such as an SD card. Note that the recording medium 1i may store a program 1h, and the reading device 1g may read the program 1h from the recording medium 1i and store it in the storage device 1d.

[0031] Examples of the recording medium 1i include non-transitory computer-readable recording media such as magnetic / optical disks and flash memories. Examples of the magnetic / optical disks include flexible disks, CDs (Compact Discs), DVDs (Digital Versatile Discs), Blu-ray Discs, and HVDs (Holographic Versatile Discs). Examples of the flash memory include semiconductor memories such as USB memories and SD cards.

[0032] The above-described HW configuration of the computer 1 is an example. Therefore, the HW in the computer 1 may be increased or decreased (for example, adding or deleting any block), divided, or integrated in any combination, or buses may be added or deleted, etc. as appropriate.

[0033] Camera 2 is an example of an imaging device that outputs video data obtained by capturing (taking a picture) of an imaging range. In one embodiment, camera 2 may be a surveillance camera that is installed, for example, at a production site, for example, at a production line, and captures images of one or more workers so that they are included in the imaging range.

[0034] 2 is a diagram for explaining an example of an imaging range 20 of the camera 2. As shown in FIG. 2, the imaging range 20 may include a worker 21, a tool 22, a work target 23, a workbench 24, and a line 25.

[0035] The worker 21 is a person who performs work on an object 23 at a production site, and is the observed person who is the subject of time research in IE.

[0036] The work target 23 is an object to be produced at a production site.

[0037] The workbench 24 is a work place such as a table, for example, a desk, on which the tool 22 and the work target 23 are placed and on which the worker 21 works on the work target 23. The worker 21 moves around the workbench 24, for example.

[0038] The line 25 is an example of a conveyor belt that conveys the work piece 23. Examples of the line 25 include various conveyors such as a belt conveyor, a chain conveyor, and a roller conveyor. Note that the line 25 does not have to be included in the imaging range 20. In the following description, the display of the line 25 in the imaging range 20 may be omitted.

[0039] The tool 22 is an implement used in work by the worker 21, and is an example of an object for which a holding period is to be determined. Examples of the tool 22 include various hand-held tools such as an impact driver, a drill driver, a grinder, and a heat tool. The tool 22 is not limited to these, and may be various tools that can be held at least in part to process the work target 23. The tool 22 may be placed on the workbench 24, attached or suspended at the tool holding position, etc., while not being used by the worker 21, and may be held by the worker 21 from the workbench 24 or the tool holding position when used by the worker 21.

[0040] In one embodiment, in the imaging range 20, the position of the workbench 24 is constant (the position does not change) between frames of the video data, in other words, the background is constant in the imaging range 20. Hereinafter, a "frame" may be written as a "frame image" or an "image".

[0041] The tool 22 is assumed to be in either a first state in which it is placed on the workbench 24 or the like, or a second state in which it is held by the worker 21. In the first and second states, the tool 22 is assumed to be in either an exposed state in which the entire tool 22 is displayed (not blocked) in the imaging range 20, or a blocked state in which at least a portion of the tool 22 is not displayed (blocked) in the imaging range 20. In the blocked state, at least a portion of the tool 22 is blocked by an "object" such as the worker 21 or the work target 23, and thus can be said to be located in a blind spot from the camera 2.

[0042] In the following description, the image capturing range 20 includes one worker 21, one tool 22, one or more work objects 23, and one workbench 24, but is not limited thereto. In the posture recognition and object recognition described below, if each of the workers 21 and the tools 22 can be individually identified by, for example, an identification label, the image capturing range 20 may include multiple workers 21 and multiple tools 22. Alternatively, in the image capturing range 20, if the movement ranges of the multiple workers 21 are each within a certain range, a part of the image area of ​​the frame may be used as the image area to be processed in the posture recognition and object recognition described below. The image area to be processed may be, for example, an image area that is within a range including the movement range or the vicinity of one worker 21 to be estimated for the gripping period and includes one tool 22.

[0043] [A-2] Software configuration example Fig. 3 is a block diagram showing an example of a software configuration of the server 10 according to an embodiment. As shown in Fig. 3, the server 10 may exemplarily include a memory unit 11, an acquisition unit 12, a posture recognition unit 13, a tool recognition unit 14, a grip determination unit 15, a recognition threshold calculation unit 16, a grip period estimation unit 17, and an output unit 18. The acquisition unit 12, the posture recognition unit 13, the tool recognition unit 14, the grip determination unit 15, the recognition threshold calculation unit 16, the grip period estimation unit 17, and the output unit 18 are examples of a control unit 19. The function of the control unit 19 may be realized, for example, by the processor 1a (see Fig. 1) of the server 10 executing a program 1h expanded in the memory 1c.

[0044] The memory unit 11 is an example of a storage area, and stores various data used by the server 10. The memory unit 11 may be realized, for example, by a storage area included in one or both of the memory 1c and the storage device 1d (see FIG. 1) of the server 10.

[0045] As shown in FIG. 3, the memory unit 11 may be capable of storing, for example, a plurality of frame images 11a, a person recognition model 11b, a tool recognition model 11c, and a holding period 11d.

[0046] The acquisition unit 12 acquires various information used by the server 10. For example, the acquisition unit 12 may acquire video data output from the camera 2 capturing an image of the imaging range 20, and store a plurality of frame images 11a included in the video data in the memory unit 11. The frame images 11a may be image data obtained by capturing an image of the imaging range 20 shown in FIG.

[0047] As described above, the camera 2 captures an image of a series of work tasks at a production site. The server 10 according to an embodiment focuses on at least a part of the series of work tasks and estimates a period during which the worker 21 holds the tool 22. Hereinafter, an example will be described in which the work task is "screw tightening work" and the work task of screw tightening is included in multiple frame images 11a.

[0048] 4 is a diagram for explaining an example of the steps of the screw fastening operation. As shown in FIG. 4, the screw fastening operation may include the following steps A1 to A4.

[0049] (Process A1) The worker 21 places the work target 23 that has been conveyed from the line 25 on the work table 24. The worker 21 takes out screws to be attached to the work target 23 from the tray.

[0050] (Process A2) The worker 21 holds a tool 22 (for example, an electric screwdriver) in his / her hand and uses the tool 22 to screw a screw into the work target 23. The worker 21 visually checks the state of the screw being screwed into the work target 23.

[0051] (Process A3) A worker 21 places a tool 22 on a workbench 24 .

[0052] (Process A4) A worker 21 places an object 23 to be worked on onto a line 25 .

[0053] Among the series of processes described above, for example, if an attempt is made to shorten the working time of process A2 (attaching screws to the work object 23 and visually checking the attached state), there is a possibility that the quality of the product, i.e., the work object 23, will decrease. For this reason, in IE time research, it is conceivable to consider shortening the time in processes other than process A2. As a temporary analysis for this purpose, it is important to calculate the ratio of the work time other than process A2 to the total work time of the series of processes and estimate the room for time reduction.

[0054] In one embodiment, the server 10 focuses on the period of process A2, i.e., the period during which the worker 21 holds the tool 22, and estimates the period with high accuracy, thereby making it possible to estimate the room for time reduction with high accuracy.

[0055] The posture recognition unit 13 executes a posture recognition process for recognizing the posture of the worker 21 in each of the plurality of frame images 11a.

[0056] The posture recognition unit 13, for example, recognizes the worker 21 as a person included in the frame image 11a by inputting the frame image 11a into the person recognition model 11b, and obtains the recognition result of the position coordinates in the frame image 11a of the joint positions of the recognized worker 21.

[0057] Fig. 5 is a diagram showing an example of the recognition result of the posture recognition process. Fig. 5 shows an example of the recognition result (posture recognition result) when the posture recognition process is performed on the frame image 11a in which the worker 21 is performing the screw fastening work (step A2).

[0058] As shown in FIG. 5, the posture recognition result may include a person area 210 indicating the position coordinates where the worker 21 is present, a plurality of joint positions (which may be referred to as "key points") 211 of the worker 21, and line segments (e.g., vectors) 212 connecting the joints. The person area 210 is a frame indicating the range that the worker 21 occupies in the frame image 11a, and may be, for example, rectangular. The person area 210 may include a label indicating the posture of the worker 21 (indicated as "Bend" in FIG. 5). The joint positions 211 indicate the position coordinates of the main joints of the worker 21, such as the hands (wrists, palms, etc.), elbows, shoulders, waists, knees, ankles, etc., with circles. In the example of FIG. 5, the joint positions 211 and line segments 212 displayed in the frame image 11a are indicated by solid lines, and the joint positions 211 and line segments 212 not displayed in the frame image 11a are indicated by dotted lines.

[0059] The posture recognition unit 13 may acquire a person area 210 of the worker 21 obtained by inputting the frame image 11a into the person recognition model 11b, and identify each of the multiple joint positions 211 in the person area 210 by comparing the person area 210 with a model of the human body (human model).

[0060] The tool recognition unit 14 executes a tool recognition process for recognizing the tool 22 in each of the plurality of frame images 11a. The tool recognition process is an example of an object recognition process for determining the presence or absence of the tool 22 for each frame image 11a of the input video data.

[0061] The tool recognition unit 14, for example, inputs the frame image 11a to a tool recognition model 11c to obtain a recognition result of the tool 22 included in the frame image 11a.

[0062] Fig. 6 is a diagram showing an example of a recognition result of the tool recognition process. Fig. 6 shows an example of a recognition result (tool recognition result) when the tool recognition process is performed on the same frame image 11a as the frame image 11a used in Fig. 5.

[0063] As shown in Fig. 6, the tool recognition result may include a tool area 220 indicating the position coordinates where the tool 22 exists, and a recognition score 221 (indicated as "0.7" in Fig. 6) indicating the likelihood that the recognized object is the tool 22. The tool area 220 is a frame indicating the range that the tool 22 occupies in the frame image 11a, and may be, for example, rectangular. The tool area 220 may include a label indicating the type of the tool 22 (indicated as "Electric screwdriver" in Fig. 6).

[0064] An example of each of the person recognition model 11b and the tool recognition model 11c is an object recognition model using a convolutional neural network (CNN), such as YOLO (You Only Look Once). Note that the object recognition model is not limited to YOLO, and various machine learning models capable of executing a person or tool recognition process for each frame image 11a and outputting a recognition result, such as the person region 210 or the tool region 220, and a recognition score, may be used.

[0065] One or both of the person recognition model 11b and the tool recognition model 11c may be, for example, a dedicated trained machine learning model trained to recognize the worker 21 or the tool 22. Alternatively, one or both of the person recognition model 11b and the tool recognition model 11c may be a general-purpose trained machine learning model trained to recognize people and tools. Furthermore, a common object recognition model trained to recognize people and tools may be used as the person recognition model 11b and the tool recognition model 11c.

[0066] The grip determination unit 15 executes a grip determination process for determining whether or not the worker 21 is gripping the tool 22 in the frame image 11a used in the recognition processes, based on the posture recognition result by the posture recognition unit 13 and the tool recognition result by the tool recognition unit 14. The grip determination process is an example of a process for determining whether or not the tool 22 is gripped for the frame image 11a in which it has been determined by the tool recognition process that the tool 22 is present.

[0067] The grip determination unit 15 may determine whether or not the tool 22 is being gripped, depending on, for example, an overlap state between the position coordinates of the hand (e.g., palm) of the worker 21 included in the posture recognition result and the tool region 220 included in the tool recognition result. As an example, the grip determination unit 15 may determine that the worker 21 is gripping the tool 22 when the distance between the position coordinates of the joint position 211 of the palm of the worker 21 and the position coordinates of the tool 22 (tool region 220) is equal to or less than a threshold. Furthermore, the grip determination unit 15 may determine that the worker 21 is not gripping the tool 22 when the distance is greater than the threshold or when the worker 21 or the tool 22 does not exist (is not recognized).

[0068] The position coordinates of the tool 22 (tool area 220) may be, for example, the coordinates in the tool area 220 that are closest to the palm joint position 211, or may be representative coordinates of the tool area 220, such as the center, center of gravity, or the coordinates of the handle if recognizable. The distance may be, for example, the number of pixels in the frame image 11a. The threshold may be set in advance according to conditions such as the resolution of the frame image 11a and the distance between the installation position of the camera 2 and the worker 21.

[0069] Fig. 7 is a diagram showing an example of a determination result of the grip determination process, which illustrates the posture recognition result (see Fig. 5), the tool recognition result (see Fig. 6), and the determination result of the grip determination process (grip determination result).

[0070] As shown in FIG. 7, the grasping determination result may include grasping presence / absence 150 indicating whether or not the worker 21 is grasping the tool 22 (in FIG. 7, this is represented as “Grabbing=True”, indicating that the tool is being grasped).

[0071] As described above, the posture recognition unit 13, the tool recognition unit 14, and the grip determination unit 15 determine whether or not the worker 21 is gripping the tool 22 for each frame image 11a.

[0072] At least one type of information among the posture recognition result, the tool recognition result, and the grip determination result may be stored in, for example, the memory unit 11, and may be referenced and used from the memory unit 11 in subsequent processing.

[0073] However, when at least a portion of the tool 22 in the frame image 11a is obscured by an object, such as a worker 21 or a workpiece 23, the accuracy of the tool recognition result based on the tool recognition model 11c decreases, and as a result, the accuracy of the grip determination result may decrease.

[0074] Fig. 8 is a diagram showing another example of the determination result of the grip determination process. Fig. 8 shows an example in which the tool 22 held by the worker 21 is hidden by the body of the worker 21, and therefore the tool 22 is not recognized as being present in the tool recognition process, and as a result, it is determined in the grip determination process that the worker 21 is not holding the tool 22. For such a frame image 11a, even though the worker 21 is actually holding the tool 22, it is not included in the grip period, so the period of the above-mentioned process A2 cannot be accurately estimated, and there is a risk that the estimation accuracy of the room for time reduction in the time study will decrease.

[0075] In the object recognition process, an object recognition model, for example, a CNN such as YOLO, resizes the frame image 11a into a square and divides the resized image into tiny squares. The CNN estimates a square in which an object is captured from among the divided squares, and estimates the object region by connecting the estimated squares. The CNN then assigns a recognition score to the estimated object region, and if the recognition score is equal to or greater than a predetermined recognition threshold, outputs a result indicating that an object exists in the object region.

[0076] In this way, since the object recognition model performs object recognition based on the results of comparing the recognition score with the recognition threshold, it is important to set an appropriate recognition threshold in order to achieve accurate object recognition.

[0077] For example, the higher the recognition threshold is set, the more likely it is that frame images 11a will not be recognized as objects even in a state where the tool 22 is easily recognized, such as an exposed state where the tool 22 is placed on the workbench 24. On the other hand, if the recognition threshold is lowered too much to enable recognition even when the tool 22 is hidden (see FIG. 8, for example), there is a risk that frame images 11a will be generated in which the tool 22 is determined to be present even when the tool 22 is not actually present. Therefore, when the display content changes from frame to frame due to the actions of the worker 21, it is difficult to set an appropriate recognition threshold.

[0078] Therefore, in one embodiment, accurate object recognition is achieved by utilizing the properties of the recognition score in two states, that is, a state in which the tool 22 is placed on the workbench 24 and a state in which the worker 21 is holding the tool 22 and working, and setting an appropriate recognition threshold.

[0079] The recognition threshold calculation unit 16 calculates the first and second recognition thresholds by the following processes (a) to (d).

[0080] (a) The recognition threshold calculation unit 16 applies the following processes (a-1) to (a-3) to each of the plurality of frame images 11a.

[0081] (a-1) The recognition threshold calculation unit 16 causes the posture recognition unit 13 to execute posture recognition processing and output position coordinates of key points of the hand of the worker 21 in each of the plurality of frame images 11a. In the posture recognition processing, a default threshold (for example, a default value in the conventional example) may be used as the person recognition threshold.

[0082] (a-2) The recognition threshold calculation unit 16 causes the tool recognition unit 14 to execute the tool recognition process with the recognition threshold set to a very small value, and causes the tool recognition unit 14 to output a tool recognition result including a recognition score.

[0083] Setting the recognition threshold to a very small value may mean, for example, setting a predetermined recognition threshold that can reliably recognize the tool 22. Such a recognition threshold may be, for example, a sufficiently small value that enables complete object recognition in the tool recognition process, and may be, for example, a minimum value that can be set in the tool recognition process. Note that even if the tool 22 is not recognized in the process (a-2), this can be followed up in the process described below, so there is no need to use a strict recognition threshold that can recognize all the tools 22.

[0084] Although this increases the possibility that an object other than the tool 22 will be erroneously detected as a "tool," it is possible to detect all of the tools 22 of which at least a portion is displayed in all of the frame images 11a.

[0085] Note that the process (a-1) and the process (a-2) may be executed in parallel (independently) with each other.

[0086] (a-3) The recognition threshold calculation unit 16 causes the grip determination unit 15 to execute a grip determination process using the posture recognition result of the process (a-1) and the tool recognition result of the process (a-2).

[0087] By the above-mentioned process (a), for example, for each frame image 11a in the video data of all 1000 frames, a process result having the following tendency is obtained.

[0088] In the process (a-1), the entire body of the worker 21 is not occluded, so the key points of the hands are determined relatively accurately (see FIG. 5).

[0089] In the process (a-2), it is determined that the tool 22 is present in the frame image 11a (see FIG. 6). Note that since the recognition threshold is smaller than the default value, it may be determined that "the tool is present" in multiple areas other than the area where the tool 22 is present.

[0090] In process (a-3), if the worker 21 is actually holding the tool 22, it is determined that "the worker is holding the tool" (holding determination result: True) in one of the areas determined as "a tool is present" (see Figure 7).

[0091] Fig. 9 is a diagram showing an example in which the grip determination result is False. In the process (a-3), when the operator 21 is not actually gripping the tool 22, it is determined that "the operator is not gripping the tool" (grippling determination result: False) in all areas where it is determined that "the tool is present" (see Fig. 9).

[0092] If a false positive detection of the tool 22 occurs in an area near the hand key point, it may be erroneously determined that "the worker is holding a tool." However, since the hand key point always moves during work, while the false positive detection area of ​​the tool 22 does not move, it is not determined that "the worker is holding a tool" over multiple frames, and such a false positive determination may be tolerated.

[0093] (b) The recognition threshold calculation unit 16 calculates a first recognition threshold for recognizing the tool 22 whose grip determination result is True, based on the processing result of process (a). The first recognition threshold is a threshold for recognizing the gripped tool 22 in all frame images 11a whose grip determination result is True.

[0094] The recognition threshold calculation unit 16 specifies, for example, the minimum value of the recognition scores of the tool 22 determined to be held among all frame images 11a in which the grip determination result is True, as the first recognition threshold. The tool 22 determined to be held is, for example, a tool 22 in a state in which the position of the hand of the worker 21 and the position of the tool 22 overlap. FIG. 7 shows a case in which the recognition score 221 is "0.2" in a frame image 11a in which the grip presence / absence 150 is True. When the recognition score 221 shown in FIG. 7 is the minimum value among all frame images 11a in which the grip determination result is True, the first recognition threshold becomes "0.2".

[0095] In addition, even if the worker 21 is not actually holding the tool 22, the recognition threshold calculation unit 16 may set a value larger than the minimum value as the first recognition threshold, instead of the minimum value of the recognition score, in consideration of the possibility that there may be a frame image 11a that is determined by chance to be "held", even if the worker 21 is not actually holding the tool 22. An example of a value larger than the minimum value is the 95th percentile point of the recognition score of the tool 22 that is determined to be held among all frame images 11a in which the holding determination result is True. In this case, there is a possibility that a frame image 11a that is determined to be "not held" even though the tool is actually being held may appear, but this can be handled in the process described below.

[0096] By the above-mentioned process (b), for example, for each frame image 11a of 300 frames in which the grasping judgment result is True out of a total of 1000 frames, the minimum recognition score of the tool recognition result judged to be grasped, “0.2”, is set as the first recognition threshold.

[0097] The first recognition threshold ensures that the tool 22 is recognized reliably in the frame image 11a in which the worker 21 is actually holding the tool 22, so that the frame image 11a in which the tool 22 is being held can be reliably determined.

[0098] (c) The recognition threshold calculation unit 16 selects frame images 11a for which the grip judgment result is False based on the grip judgment using the first recognition threshold (the grip judgment based on the tool recognition result using the first recognition threshold). Tool recognition results that are not selected by the process (c) are rejected as recognition errors.

[0099] In addition, for the frame image 11a in which the grip determination result in process (a) is True, the grip determination result based on the first recognition threshold will always be True. This is because the first recognition threshold calculated in process (b) is the minimum value of the recognition score of the tool 22 among all the frame images 11a in which the grip determination result in process (a) is True. Therefore, in process (c), it is not necessary to redo the tool recognition process and the grip determination process based on the first recognition threshold, and the tool recognition result in process (a-2) and the grip determination result in process (a-3) may be used. In other words, the tool recognition process in process (a-2) is an example of the tool recognition process based on the first recognition threshold (first object recognition process), and the grip determination process in process (a-3) is an example of the grip determination process based on the first recognition threshold (first grip determination process).

[0100] In the process (c), the grip determination result is False when it is determined that the tool 22 is present at a point other than the key point of the hand. For example, the following two patterns can be mentioned. When the tool 22 is displayed away from the hand key points (when not truly grasped: see Figure 9). The tool 22 is not recognized because the tool 22 is completely hidden behind the worker 21 or the work object 23 even though the tool 22 is being held (see FIG. 8). Note that the tool may be erroneously detected in a different location.

[0101] As described above, the recognition score of the tool recognition result tends to be low when the tool is being held (obstructed state) and high when the tool is not being held (exposed state). Nevertheless, the frame image 11a in which the grip determination result becomes False after the tool recognition process and grip determination process using the first recognition threshold value corresponds to the second pattern above. Note that, on the premise that there is only one tool 22 to be recognized (or a tool 22 with a specific label), all areas other than the vicinity of the hand key point where it is determined that "a tool is present" are erroneous detections. Therefore, in process (c), such frame images 11a are selected.

[0102] By the above-mentioned process (c), for example, 700 frame images 11a are selected from the total 1000 frames, excluding 300 frames in which the grip determination result is True.

[0103] (d) The recognition threshold calculation unit 16 calculates a second recognition threshold based on the tool recognition result of the frame image 11a selected in process (c) through the following processes (d-1) to (d-3).

[0104] (d-1) The recognition threshold calculation unit 16 selects frame images 11a that include tool recognition results that are equal to or greater than the first recognition threshold (e.g., 0.2) from the frame images 11a selected in process (c). In other words, the recognition threshold calculation unit 16 eliminates frame images 11a that do not include tool recognition results that are equal to or greater than the first recognition threshold from the frame images 11a selected in process (c). This is because tools with recognition scores below the first recognition threshold can be considered to be erroneous detections.

[0105] (d-2) The recognition threshold calculation unit 16 extracts the maximum value of the tool recognition result (recognition score) included in each frame image 11a selected in the process (d-1) because there is a possibility that the frame image 11a includes multiple tool recognition results.

[0106] (d-3) The recognition threshold calculation unit 16 specifies the minimum value of the tool recognition results (recognition scores) extracted from the multiple frame images 11a selected in the process (d-1) as the second recognition threshold.

[0107] In the frame image 11a selected in process (d-1), the gripping determination result is False, and therefore the tool 22 is placed somewhere in the frame image 11a. Therefore, by setting a second recognition threshold value that allows the tool 22 to be recognized in each of the multiple frame images 11a selected in process (d-1), it becomes possible to reliably recognize one tool 22 (a tool 22 that is not gripped).

[0108] By the above-mentioned process (d-1), for example, from each frame image 11a of 700 frames in which the grip judgment result selected in process (c) is False, 500 frames including tool recognition results that have been assigned a recognition score equal to or greater than the first recognition threshold value of “0.2” are selected.

[0109] In the above-mentioned process (d-2), the maximum recognition score given to the tool recognition result included in each frame image 11a selected in the process (d-1) is extracted. For example, 500 pieces of numerical information are obtained from 500 frames.

[0110] By the above-mentioned process (d-3), the minimum value of the 500 pieces of numerical information (for example, "0.7" shown in FIG. 9) is specified as the second recognition threshold.

[0111] The process (d) obtains a second recognition threshold for reliably recognizing the tool 22 when the grip determination result is False and the tool 22 appears in the frame image 11a. The process (b) described above obtains a first recognition threshold for reliably recognizing the tool 22 when the grip determination result is True and the tool 22 appears in the frame image 11a.

[0112] When the tool 22 is not being held, the tool 22 is not hidden and is therefore easier to recognize than when the tool 22 is being held. Therefore, the first and second recognition thresholds have a relationship of first recognition threshold (e.g., 0.2)<second recognition threshold (e.g., 0.7).

[0113] After the recognition threshold calculation unit 16 completes the above-described processes (a) to (d), the holding duration estimation unit 17 executes the following process (e).

[0114] (e) The holding period estimation unit 17 estimates the holding period 11d during which the worker 21 holds the tool 22 based on the multiple frame images 11a and the first recognition threshold and the second recognition threshold calculated by the recognition threshold calculation unit 16, by the following processes (e-1) to (e-3).

[0115] Fig. 10 is a diagram for explaining an example of a gripping period estimation method. In Fig. 10, the symbol B indicates a determination result, and the symbol C indicates a frame image 11a. For example, the horizontal axis of the symbol C indicates a plurality of frame images 11a arranged in chronological order, that is, time.

[0116] (e-1) The holding period estimation unit 17 causes the tool recognition unit 14 to execute tool recognition processing to which a first recognition threshold is applied for the multiple frame images 11a. The holding period estimation unit 17 also causes the holding determination unit 15 to execute a holding determination processing using the posture recognition result of the process (a-1) and the tool recognition result of the process (e-1), and acquires a holding determination result. Note that the holding determination result acquired in the process (e-1) may be acquired by utilizing the holding determination result acquired in the process (a-3).

[0117] As shown by reference symbol B1, the holding period estimation unit 17 identifies a frame image 11a (see reference symbols C2 and C4) in which the tool 22 is recognized at the first recognition threshold and the holding determination result is True, as a "holding frame image" in which the worker 21 holds the tool 22. This makes it possible to reliably identify a frame image 11a such as that shown in Fig. 7 as a holding frame image. The frame images 11a shown by reference symbols C2 and C4 are an example of a first frame in which it is determined by the holding determination process that the tool 22 is being held.

[0118] (e-2) The holding period estimation unit 17 causes the tool recognition unit 14 to execute tool recognition processing to which the second recognition threshold is applied for the multiple frame images 11a. Also, the holding period estimation unit 17 causes the holding determination unit 15 to execute a holding determination processing using the posture recognition result of the process (a-1) and the tool recognition result of the process (e-2), and acquires a holding determination result.

[0119] As shown by the reference symbol B2, the gripping period estimation unit 17 identifies the frame image 11a (see the reference symbols C1 and C5) in which the tool 22 is recognized at the second recognition threshold and the gripping determination result is False as a "non-grip frame image" in which the worker 21 is not gripping the tool 22. This makes it possible to reliably identify, for example, the frame image 11a shown in Fig. 9 as a "non-grip frame image."

[0120] In this way, the holding period estimation unit 17 may cause the tool recognition unit 14 to execute a first object recognition process for determining the presence or absence of the tool 22 based on the first recognition threshold, and a second object recognition process for determining the presence or absence of the tool 22 based on the second recognition threshold. The holding period estimation unit 17 may cause the holding determination unit 15 to execute a first holding determination process for determining whether or not the tool 22 is held for the frame image 11a in which it is determined that the tool 22 is present by the first object recognition process, and a second holding determination process for determining whether or not the tool 22 is held for the frame image 11a in which it is determined that the tool 22 is present by the second object recognition process. As described above, the first object recognition process and the first holding determination process may be realized by the process (a-2) and the process (a-3), or the process (c).

[0121] (e-3) The holding period estimation unit 17 treats a frame image 11a in which the tool 22 is not recognized at all even by the first and second recognition thresholds as a frame image 11a in which the holding determination result is True. This is because, in such a frame image 11a, as shown in Fig. 8, the tool 22 is completely occluded by the worker 21 or the work target 23, and it is estimated to be a frame image 11a in which no object recognition was performed. The frame image 11a in which the tool 22 is not recognized at all even by the first and second recognition thresholds is an example of a second frame in which it is determined by the object recognition process that the tool 22 does not exist.

[0122] As shown by the reference symbol B3, the holding period estimation unit 17 identifies the frame image 11a (see the reference symbol C3) in which the tool 22 is not recognized even by the tool recognition processing in the process (e-1) and the process (e-2) as a holding frame image in which the worker 21 holds the tool 22. This makes it possible to reliably identify, for example, the frame image 11a shown in FIG. 8 as a holding frame image.

[0123] The holding period estimation unit 17 may, for example, calculate the holding period 11d from the determined holding frame image and store it in the memory unit 11. The holding period 11d may include at least one type of information, for example, the frame number of the holding frame image in the video data, the total number of frames of the holding frame image, the duration of the holding frame image based on the number of frames and the frame rate of the video data, etc. In the following description, the identified frame image 11a and the holding period 11d may be expressed as being the same.

[0124] In this way, the holding period estimation unit 17 determines the first frame in which it is determined that the tool 22 is being held and the second frame in which it is determined that the tool 22 is not present as the holding period 11d. The second frame may be a frame that is continuous with the first frame. By determining the first frame and the second frame as the holding period 11d when they are continuous, it is possible to reliably include the frame image 11a in a state in which the tool 22 is truly being held, such as when the tool 22 is hidden while being held, in the holding period 11d. In addition, it is possible to prevent the frame image 11a in a state in which the tool 22 is truly not being held, such as when the tool 22 is hidden while the tool 22 is not truly being held (see symbol B2), from being erroneously detected as the holding period 11d.

[0125] For example, by including frame images 11a in which the recognition score of the recognized tool 22 is less than the first recognition threshold as the frame image 11a in which the "tool is not recognized" shown in reference symbol B3, it is possible to specify a frame image 11a in which the tool 22 is "held" and "completely hidden" (see reference symbol C3). This makes it possible to estimate with high accuracy the period of reference symbols C2 to C4, including reference symbol C3, as the holding period 11d in which the worker 21 holds the tool 22.

[0126] The series of frame images 11a indicated by symbols C1 to C5 shown in Fig. 10 may be included in multiple sets in the video data. For example, in the case of a screw tightening operation, the steps indicated by symbols A1 to A4 in Fig. 4 are repeatedly recorded in the video data. Therefore, the series of frame images 11a indicated by symbols C1 to C5 will be present in the video data every time the process indicated by symbol A2 including the gripping period 11d is executed.

[0127] 10 may include the frame image 11a of code C3 that is discontinuous in time depending on the movement of the worker 21. For example, when the frame image 11a in which the determination of code B1 has been made and the frame image 11a in which the determination of code B3 has been made are continuous in time (in other words, in terms of frame number), the holding period estimation unit 17 may include these frame images 11a in the holding period 11d.

[0128] Fig. 11 is a diagram for explaining an example of a gripping period estimation method. Fig. 11 shows an example in which frame images 11a (shaded: codes C3 and C7) in which a code B3 has been determined exist in two separate periods during a gripping period 11d with frame images 11a (shaded: codes C2 and C4) in which a code B1 has been determined at both ends. In the example of Fig. 11, a frame image 11a (shaded: code C6) in which a code B1 has been determined exists between codes C3 and C7.

[0129] As described above, when frames C3 and C7 exist between frames C2, C4, and C6 which are sandwiched between frames C1 and C5, the holding period estimation unit 17 estimates the frames C2 to C4, C6, and C7 as the holding period 11d.

[0130] The output unit 18 outputs output data. The output data may include at least one of the following: the gripping period 11d, the ratio of the work time other than the gripping period 11d to the total work time, etc. The output data may also include intermediate data such as the first recognition threshold, the second recognition threshold, the posture recognition result, the tool recognition result, and the grip determination result.

[0131] In "outputting" the output data, the output unit 18 may, for example, transmit (provide) the output data to another computer (not shown), or may store the output data in the memory unit 11 and manage it so that it can be acquired from the server 10 or another computer. Alternatively, in "outputting" the output data, the output unit 18 may output information indicating the output data to a screen on an output device such as the server 10 or an administrator terminal, or may output the output data in various other modes.

[0132] [B] Example of operation Next, an example of the operation of the server 10 according to an embodiment will be described below. Fig. 12 is a flowchart illustrating an example of the operation of the server 10 according to an embodiment.

[0133] As illustrated in FIG. 12, the acquisition unit 12 acquires video data from the camera 2 (step S1).

[0134] The acquisition unit 12 divides the video data into a plurality of frame images 11a (step S2) and stores them in the memory unit 11.

[0135] The server 10 selects one of the unselected frame images 11a (step S3).

[0136] The recognition threshold calculation unit 16 causes the posture recognition unit 13 to execute posture recognition processing (step S4: processing (a-1)). The posture recognition unit 13, for example, inputs the selected frame image 11a to the person recognition model 11b to obtain a person recognition result, and obtains the posture (and key points) of the worker 21 by comparing the person recognition result with the human model. The recognition threshold in the person recognition processing may be the same as a default (e.g., conventional) recognition threshold.

[0137] The recognition threshold calculation unit 16 causes the tool recognition unit 14 to execute tool recognition processing (step S5: processing (a-2)). The tool recognition unit 14, for example, inputs the selected frame image 11a to the tool recognition model 11c to obtain a tool recognition result including a recognition score. The recognition threshold in the tool recognition processing may be a settable minimum value.

[0138] The recognition threshold calculation unit 16 causes the grip determination unit 15 to execute a grip determination process (step S6: process (a-3)). The grip determination unit 15 determines whether the worker 21 is gripping the tool 22 or not based on, for example, the distance between the position coordinates of the hand key point in the posture recognition result and the position coordinates of the tool 22 in the tool recognition result, and outputs a grip determination result.

[0139] The recognition threshold calculation unit 16 stores the grip determination result in the memory unit 11 (step S7). Note that the recognition threshold calculation unit 16 may store in the memory unit 11 either or both of the posture recognition result and the tool recognition result.

[0140] The server 10 judges whether or not there is an unselected frame image 11a (step S8). If there is an unselected frame image 11a (YES in step S8), the process proceeds to step S3.

[0141] If there is no unselected frame image 11a (NO in step S8), the recognition threshold calculation unit 16 calculates a first recognition threshold based on the processing result in which the grip determination result stored in the memory unit 11 is True (step S9: processing (b)). The recognition threshold calculation unit 16 determines, for example, the minimum value of the recognition scores of the tool recognition results determined to be gripped as the first recognition threshold.

[0142] The recognition threshold calculation unit 16 calculates a second recognition threshold based on the processing result in which the grip determination result is False (step S10: processing (c) and processing (d)). For example, the recognition threshold calculation unit 16 selects a frame image 11a in which the grip determination result is False (processing (c)). The recognition threshold calculation unit 16 also selects a frame image 11a that includes a tool recognition result equal to or greater than the first recognition threshold from the frame images 11a selected in processing (c) (processing (d-1)). Furthermore, the recognition threshold calculation unit 16 extracts the maximum recognition score included in each of the frame images 11a selected in processing (d-1) (processing (d-2)). Then, the recognition threshold calculation unit 16 determines the minimum value of the extracted recognition score among the frame images 11a selected in processing (d-1) as the second recognition threshold.

[0143] The holding period estimation unit 17 causes the tool recognition unit 14 to execute a tool recognition process using a second recognition threshold. For example, the tool recognition unit 14 inputs each of the multiple frame images 11a into the tool recognition model 11c to obtain a tool recognition result including a recognition score. At this time, the tool recognition unit 14 recognizes that the tool 22 is present when the recognition score is equal to or greater than the second recognition threshold.

[0144] Further, the holding period estimation unit 17 causes the holding determination unit 15 to execute a holding determination process. The holding determination unit 15 determines whether or not the worker 21 is holding the tool 22 based on, for example, the distance between the position coordinates of the hand key point in the posture recognition result in step S4 and the position coordinates of the tool 22 in the tool recognition result using the second recognition threshold, and outputs a holding determination result.

[0145] Then, the holding period estimation unit 17 includes in the holding frame image the frame image 11a in which the holding judgment result obtained in step S9 is True and the frame image 11a in which the tool 22 is not recognized even using the first and second recognition thresholds (step S11: processing (e)).

[0146] The holding period estimation unit 17 outputs the holding period 11d calculated based on the holding frame image (step S12), and the process ends.

[0147] [C] Other The technology according to the above-described embodiment can be modified and changed as follows.

[0148] For example, the acquisition unit 12, posture recognition unit 13, tool recognition unit 14, grip determination unit 15, recognition threshold calculation unit 16, grip period estimation unit 17, and output unit 18 provided in the server 10 shown in Figure 3 may be merged in any combination or may be divided separately.

[0149] 3 may be configured such that a plurality of devices cooperate with each other via a network to realize each processing function. As an example, the acquisition unit 12 and the output unit 18 may be a Web server and an application server, the posture recognition unit 13, the tool recognition unit 14, the grip determination unit 15, the recognition threshold calculation unit 16, and the grip period estimation unit 17 may be an application server, and the memory unit 11 may be a DB (Database) server, etc. In this case, the Web server, the application server, and the DB server may cooperate with each other via a network to realize the processing function of the server 10.

[0150] [D] Notes The following supplementary notes are further disclosed regarding the above embodiment.

[0151] (Appendix 1) Executes object recognition processing for each frame of the input video data to determine whether or not an object is present; executing a gripping determination process for determining whether or not the object is being gripped for a frame in which it is determined by the object recognition process that the object is present; A first frame in which it is determined by the gripping determination process that the object is being gripped, and a second frame in which it is determined by the object recognition process that the object is not present, are determined as a period in which the object is being gripped. A gripping period determination program that causes a computer to execute processing.

[0152] (Appendix 2) the process of determining includes a process of determining, when the first frame and the second frame are temporally continuous, that the first frame and the second frame are the period. The holding period determination program according to appendix 1.

[0153] (Appendix 3) The object recognition process includes: a first object recognition process for determining whether or not the object exists in each frame of the video data based on a first recognition threshold; a second object recognition process for determining whether or not the object exists for each frame of the video data based on a second recognition threshold that is greater than the first recognition threshold; The grip determination process includes: a first gripping determination process for determining whether or not the object is being gripped for a frame in which it has been determined by the first object recognition process that the object is present; The process of determining includes: determining, as the period, the first frame in which it is determined by the first grip determination process that the object is being gripped and the second frame in which it is determined by the second object recognition process that the object is not present; The holding period determination program according to claim 1 or 2.

[0154] (Appendix 4) determining, as the first recognition threshold, a minimum value of the recognition scores of the object among all frames in which it is determined that the object is being grasped by the grasping determination process based on a processing result of the object recognition process using a predetermined recognition threshold capable of recognizing the object; The holding period determination program according to claim 3, which causes the computer to execute processing.

[0155] (Appendix 5) determining, as the second recognition threshold, a minimum value among maximum values ​​of the recognition scores of the object in each frame in which it is determined that the object is not being grasped by the first grasp determination process, in all frames in which it is determined that the object is not being grasped; The holding period determination program according to claim 3 or 4, which causes the computer to execute processing.

[0156] (Appendix 6) Executes object recognition processing for each frame of the input video data to determine whether or not an object is present; executing a gripping determination process for determining whether or not the object is being gripped for a frame in which it is determined by the object recognition process that the object is present; A first frame in which it is determined by the gripping determination process that the object is being gripped, and a second frame in which it is determined by the object recognition process that the object is not present, are determined as a period in which the object is being gripped. A method for determining a holding period, the processing of which is executed by a computer.

[0157] (Appendix 7) the process of determining includes a process of determining, when the first frame and the second frame are temporally continuous, that the first frame and the second frame are the period. A method for determining holding period as described in Appendix 6.

[0158] (Appendix 8) The object recognition process includes: a first object recognition process for determining whether or not the object exists in each frame of the video data based on a first recognition threshold; a second object recognition process for determining whether or not the object exists for each frame of the video data based on a second recognition threshold that is greater than the first recognition threshold; The grip determination process includes: a first gripping determination process for determining whether or not the object is being gripped for a frame in which it has been determined by the first object recognition process that the object is present; The process of determining includes: determining, as the period, the first frame in which it is determined by the first grip determination process that the object is being gripped and the second frame in which it is determined by the second object recognition process that the object is not present; A method for determining a holding period according to claim 6 or 7.

[0159] (Appendix 9) determining, as the first recognition threshold, a minimum value of the recognition scores of the object among all frames in which it is determined that the object is being grasped by the grasping determination process based on a processing result of the object recognition process using a predetermined recognition threshold capable of recognizing the object; The holding period determination method according to claim 8, wherein the processing is executed by the computer.

[0160] (Appendix 10) determining, as the second recognition threshold, a minimum value among maximum values ​​of the recognition scores of the object in each frame in which it is determined that the object is not being grasped by the first grasp determination process, in all frames in which it is determined that the object is not being grasped; 10. The method for determining a holding period according to claim 8 or 9, wherein the processing is executed by the computer.

[0161] (Appendix 11) Executes object recognition processing for each frame of the input video data to determine whether or not an object is present; executing a gripping determination process for determining whether or not the object is being gripped for a frame in which it is determined by the object recognition process that the object is present; A first frame in which it is determined by the gripping determination process that the object is being gripped, and a second frame in which it is determined by the object recognition process that the object is not present, are determined as a period in which the object is being gripped. An information processing device comprising a control unit.

[0162] (Appendix 12) the control unit, in the process of determining, determines that the first frame and the second frame are the period when the first frame and the second frame are temporally continuous. 12. The information processing device according to claim 11.

[0163] (Appendix 13) The control unit is In the object recognition process, a first object recognition process for determining whether or not the object exists in each frame of the video data based on a first recognition threshold; a second object recognition process for determining whether or not the object exists for each frame of the video data based on a second recognition threshold that is greater than the first recognition threshold; In the grip determination process, executing a first gripping determination process for determining whether or not the object is being gripped for a frame in which it has been determined by the first object recognition process that the object is present; In the determination process, determining, as the period, the first frame in which it is determined by the first grip determination process that the object is being gripped and the second frame in which it is determined by the second object recognition process that the object is not present; 13. The information processing device according to claim 11 or 12.

[0164] (Appendix 14) The control unit is determining, as the first recognition threshold, a minimum value of the recognition scores of the object among all frames in which it is determined that the object is being grasped by the grasping determination process based on a processing result of the object recognition process using a predetermined recognition threshold capable of recognizing the object; 14. The information processing device according to claim 13.

[0165] (Appendix 15) The control unit is determining, as the second recognition threshold, a minimum value among maximum values ​​of the recognition scores of the object in each frame in which it is determined that the object is not being grasped by the first grasp determination process, in all frames in which it is determined that the object is not being grasped; 15. The information processing device according to claim 13 or 14. [Explanation of symbols]

[0166] 1. Computer 10 Server 11 Memory section 11a Frame image 11b Person Recognition Model 11c Tool Recognition Model 11d Grasping period 12 Acquisition Department 13 Posture recognition section 14 Tool recognition section 15 Grip determination section 150 Gripping presence / absence 16 Recognition threshold calculation unit 17 Grasping period estimator 18 Output section 19 Control section 2 Camera 2a Network 20 Image Range 21 Worker 22 Tools 23 Worked object 24 Workbench 25 Line 210 People area 211 Joint Position 212 Line Segments 220 Tool area 221 Recognition Score

Claims

1. Executes object recognition processing for each frame of the input video data to determine whether or not an object is present; executing a gripping determination process for determining whether or not the object is being gripped for a frame in which it is determined by the object recognition process that the object is present; A first frame in which it is determined by the gripping determination process that the object is being gripped, and a second frame in which it is determined by the object recognition process that the object is not present, are determined as a period in which the object is being gripped. A gripping period determination program that causes a computer to execute processing.

2. the determining process includes a process of determining, when the first frame and the second frame are temporally continuous, that the first frame and the second frame are the period. The holding period determination program according to claim 1 .

3. The object recognition process includes: a first object recognition process for determining whether or not the object exists in each frame of the video data based on a first recognition threshold; a second object recognition process for determining whether or not the object exists for each frame of the video data based on a second recognition threshold that is greater than the first recognition threshold; The grip determination process includes: a first gripping determination process for determining whether or not the object is being gripped for a frame in which it has been determined by the first object recognition process that the object is present; The process of determining includes: determining, as the period, the first frame in which it is determined by the first grip determination process that the object is being gripped and the second frame in which it is determined by the second object recognition process that the object is not present; The holding period determination program according to claim 1 or 2.

4. determining, as the first recognition threshold, a minimum value of the recognition scores of the object among all frames in which it is determined that the object is being grasped by the grasping determination process based on a processing result of the object recognition process using a predetermined recognition threshold capable of recognizing the object; The holding period determination program according to claim 3 , which causes the computer to execute processing.

5. determining, as the second recognition threshold, a minimum value among maximum values ​​of the recognition scores of the object in each frame in which it is determined that the object is not being grasped by the first grasp determination process, in all frames in which it is determined that the object is not being grasped; The holding period determination program according to claim 3 , which causes the computer to execute processing.

6. Executes object recognition processing for each frame of the input video data to determine whether or not an object is present; executing a gripping determination process for determining whether or not the object is being gripped for a frame in which it is determined by the object recognition process that the object is present; A first frame in which it is determined by the gripping determination process that the object is being gripped, and a second frame in which it is determined by the object recognition process that the object is not present, are determined as a period in which the object is being gripped. A method for determining a holding period, the processing of which is executed by a computer.

7. Executes object recognition processing for each frame of the input video data to determine whether or not an object is present; executing a gripping determination process for determining whether or not the object is being gripped for a frame in which it is determined by the object recognition process that the object is present; A first frame in which it is determined by the gripping determination process that the object is being gripped, and a second frame in which it is determined by the object recognition process that the object is not present, are determined as a period in which the object is being gripped. An information processing device comprising a control unit.

Images