Information processing device, information processing method, and information processing program

JPWO2024142193A5Active Publication Date: 2024-11-26MITSUBISHI ELECTRIC CORP
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
JP2024552757
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2024-09-04
Publication Date
2024-11-26
Estimated Expiration
2042-12-27

AI Technical Summary

Technical Problem

Current technologies fail to automatically select an appropriate estimation model for work processes on assembly lines based on the combination of worker and workpiece characteristics, leading to inefficient manual mode switching.

Method used

An information processing device that acquires worker movement information and work object images to select an appropriate estimation model from a plurality of models based on the characteristics of the worker and workpiece, using skeleton detection and product detection units to calculate similarity and select the corresponding model for accurate work process estimation.

Benefits of technology

Enables automatic and efficient selection of the appropriate estimation model, improving the accuracy of work process estimation by adapting to the specific characteristics of workers and workpieces, thereby reducing the time-consuming manual switching of operation modes.

✦ Generated by Eureka AI based on patent content.
Patent Text Reader

Abstract

Acquisition units (22, 23) acquire worker motion information, which is information on motions of a worker working on a workpiece, and a workpiece image, which is an image of the workpiece. A model selection unit (26) uses the worker motion information and the workpiece image to select, as an estimation model to be used, an estimation model to be used for estimating a work process being performed on the workpiece by the worker from among a plurality of estimation models for estimating a work process being performed among a plurality of work processes.
Need to check novelty before this filing date? Find Prior Art

Description

Information processing device, information processing method, and information processing program

[0001] The present disclosure relates to a technique for estimating a work process being performed by a worker.

[0002] In the industrial field, there are assembly lines where the characteristics of workers (e.g., the dominant hand of the worker) and / or the characteristics of the work object (e.g., the model of the product that is the work object) change depending on the time of day. In such assembly lines, there is a need for technology that can accurately estimate the work process being performed from video. A work process is an element that makes up a work procedure. A single action procedure is made up of a combination of multiple work processes. A work process is also referred to as a work element hereinafter.

[0003] In order to accurately estimate the work process being performed from video, it is necessary to switch the operation mode (hereinafter sometimes simply referred to as "mode") depending on the characteristics of the worker and the characteristics of the work object. In other words, it is necessary to select an appropriate estimation model from multiple estimation models for estimating the work process being performed depending on the characteristics of the worker and / or the characteristics of the work object. Then, it is necessary to switch to the operation mode corresponding to the selected estimation model and execute the selected estimation model. Currently, such switching of operation modes is performed manually. However, switching operation modes manually is time-consuming, and automation of the switching process is desired.

[0004] Technologies related to the present disclosure include those disclosed in Patent Document 1 and Patent Document 2. Patent Document 1 discloses a method for automatically switching operation modes based on the amount of movement, speed, and trajectory of a person's fingers acquired from a video. Patent Document 2 discloses a method for automatically switching operation modes by determining the presence or absence of an object from an image.

[0005] JP 2004-246814 A JP 2019-215711 A

[0006] However, the techniques of Patent Document 1 and Patent Document 2 have a problem in that they are unable to select an appropriate estimation model depending on the combination of the characteristics of the worker (e.g., the worker's dominant hand) and the characteristics of the work object (e.g., the model of the product that is the work object).

[0007] The main objective of the present disclosure is to solve these problems, specifically to select an appropriate estimation model depending on the combination of the worker's characteristics and the characteristics of the work object.

[0008] The information processing device according to the present disclosure has an acquisition unit that acquires worker movement information, which is movement information of a worker when working on a work object, and a work object image, which is an image of the work object; and a model selection unit that uses the worker movement information and the work object image to select, as an estimation model to be used, from a plurality of estimation models that are models for estimating a work process being performed among a plurality of work processes, an estimation model to be used to estimate the work process being performed by the worker on the work object.

[0009] According to the present disclosure, an appropriate estimation model can be selected depending on the combination of the characteristics of the worker and the characteristics of the work object.

[0010] 1 is a diagram showing an example of the configuration of a task element identification device according to the first embodiment. FIG. 2 is a flowchart showing pre-operation processing according to the first embodiment. FIG. 3 is a diagram showing an example of skeleton detection processing in pre-operation processing according to the first embodiment. FIG. 4 is a diagram showing an example of product detection processing in pre-operation processing according to the first embodiment. FIG. 5 is a flowchart showing operation processing according to the first embodiment. FIG. 6 is a diagram showing an example of skeleton detection processing in operation processing according to the first embodiment. FIG. 7 is a diagram showing an example of product detection processing in operation processing according to the first embodiment. FIG. 8 is a diagram showing an example of movement similarity calculation processing in operation processing according to the first embodiment. FIG. 9 is a diagram showing an example of movement similarity calculation processing in operation processing according to the first embodiment. FIG. 10 is a diagram showing an example of product similarity calculation processing in operation processing according to the first embodiment. FIG. 11 is a diagram showing an example of a correspondence table between index information and task element identification models according to the first embodiment. FIG. 12 is a diagram showing examples of selection criteria for task element identification models according to the first embodiment. FIG. 13 is a diagram showing an example of the configuration of a task element identification device according to a second modification of the first embodiment. FIG. 14 is a diagram showing an example of the configuration of a task element identification device according to a third modification of the first embodiment. FIG. 15 is a diagram showing an example of the configuration of a task element identification device according to the second embodiment. A diagram showing an example of a work element table according to embodiment 2. A diagram showing an example of a search map according to embodiment 2. A diagram showing an example of a representative image according to embodiment 2. A diagram showing an example of an updated search map according to embodiment 2. A flowchart showing an operation process according to embodiment 2. A flowchart showing an operation process according to embodiment 2. A diagram showing an example of an updated search map according to embodiment 2.

[0011] Hereinafter, embodiments will be described with reference to the drawings. In the following description of the embodiments and the drawings, the same reference numerals denote the same or corresponding parts.

[0012] Embodiment 1. ***Description of Configuration*** An example configuration of a work element identification device 10 according to embodiment 1 will be described with reference to FIG. 1. The work element identification device 10 corresponds to an information processing device. The operation procedure of the work element identification device 10 corresponds to an information processing method. The program that realizes the operation of the work element identification device 10 corresponds to an information processing program. As described above, the "work elements" identified by the work element identification device 10 are elements (work processes) that make up a work procedure (assembly work in this embodiment). Two or more work elements are combined to make up one work procedure.

[0013] The work element identification device 10 is a computer. The work element identification device 10 includes, as hardware, a processor 11, a memory 12, a storage 13, and a communication interface 14. The processor 11 is connected to other hardware via signal lines and controls this other hardware. The processor 11 is an IC (Integrated Circuit) that performs processing. The processor 11 is, for example, a CPU (Central Processing Unit), a DSP (Digital Signal Processor), or a GPU (Graphics Processing Unit). The memory 12 is a storage device that temporarily stores data. The memory 12 is, for example, a static random access memory (SRAM) or a dynamic random access memory (DRAM). The storage 13 is a storage device that stores data. The storage 13 is, for example, a hard disk drive (HDD). The storage 13 may also be a portable recording medium such as an SD (Secure Digital, registered trademark) memory card, a CF (CompactFlash, registered trademark), a NAND flash, a flexible disk, an optical disk, a compact disk, a Blu-ray (registered trademark) disk, or a DVD (Digital Versatile Disk). The communication interface 14 is an interface for communicating with external devices. The communication interface 14 is, for example, an Ethernet (registered trademark), a USB (Universal Serial Bus), or an HDMI (registered trademark, High-Definition Multimedia Interface) port.

[0014] The task element identification device 10 includes, as functional components, a video acquisition unit 21, a skeleton detection unit 22, a product detection unit 23, a movement similarity calculation unit 24, a product similarity calculation unit 25, a mode selection unit 26, and a task element identification unit 27. The functions of each functional component of the task element identification device 10 are realized by software. The storage 13 stores a program that realizes each functional component of the task element identification device 10. This program is loaded into the memory 12 by the processor 11 and executed by the processor 11. In this way, each functional component of the task element identification device 10 is realized. The storage 13 also stores a training video 31, a true value 32 of the training video, and a task element identification model 33.

[0015] 1, there is one processor 11. However, there may be a plurality of processors 11, and the plurality of processors 11 may cooperate to execute programs that realize the respective functional components.

[0016] The learning video 31 is a video showing one or more workers performing assembly work and a product as the work target. In this embodiment, there are multiple learning videos 31. Each learning video 31 corresponds to one of multiple work procedures.

[0017] The true value 32 of the learning video is label information of the task element being performed in each frame of the learning video 31 .

[0018] The work element identification model 33 is a model generated for each training video 31. In this embodiment, there are multiple work element identification models 33. Each work element identification model 33 corresponds to one of the training videos 31. In other words, each work element identification model 33 corresponds to one of the multiple work procedures. The work element identification model 33 is an example of an estimation model.

[0019] The task element identification device 10 can estimate the task elements being performed by the worker shown in the video by applying the worker's skeletal information acquired from the video to the task element identification model 33. The skeletal information is movement information that represents the movement of the worker's body while working. The skeletal information includes time-series information on the coordinates of multiple joints, such as the worker's neck and wrists, or time-series information on the relative positional relationships of multiple joints.

[0020] In this embodiment, as will be described later, the worker's handedness (right-handedness, left-handedness) is assumed as a worker characteristic. Furthermore, the product model (model i, model ii) is assumed as a work object characteristic. In this embodiment, there are four learning videos 31 that cover all patterns of worker handedness and product models. That is, in this embodiment, there is a learning video 31 in which a right-handed worker works on a product model i (hereinafter referred to as a "right-handed & model i learning video"). There is also a learning video 31 in which a right-handed worker works on a product model ii (hereinafter referred to as a "right-handed & model ii learning video"). There is also a learning video 31 in which a left-handed worker works on a product model i (hereinafter referred to as a "left-handed & model i learning video"). There is also a learning video 31 in which a left-handed worker works on a product model ii (hereinafter referred to as a "left-handed & model ii learning video").

[0021] The task element identification model 33 is also similar to the learning video 31. That is, there are four task element identification models 33 that cover all patterns of worker handedness and product models. Specifically, in this embodiment, there is a task element identification model 33 (hereinafter referred to as the "right-handed & model i model") that is applied when a right-handed worker is working on a product of model i. There is also a task element identification model 33 (hereinafter referred to as the "right-handed & model ii model") that is applied when a right-handed worker is working on a product of model ii. There is also a task element identification model 33 (hereinafter referred to as the "left-handed & model i model") that is applied when a left-handed worker is working on a product of model i. There is also a task element identification model 33 (hereinafter referred to as the "left-handed & model ii model") that is applied when a left-handed worker is working on a product of model ii.

[0022] Furthermore, the learning video true value 32 indicates the name of the work element corresponding to each frame of the learning video 31. For example, assume that the work procedure corresponding to the learning video 31 includes work element a, work element b, and work element c. In this case, the learning video true value 32 indicates, for example, that frames 1 to 10 are frames of work element a, frames 11 to 20 are frames of work element b, and frames 21 to 30 are frames of work element c.

[0023] The functional components of the task element identification device 10 will be described in detail later, but an overview of each functional component will be given here.

[0024] The video acquisition unit 21 acquires video via the communication interface 14. Hereinafter, the video acquired by the video acquisition unit 21 via the communication interface 14 is referred to as input video 46. The input video 46 is a video of a worker performing work on a product, which is the work target.

[0025] The skeleton detection unit 22 acquires skeleton information, which is motion information of the worker, from the learning video 31 and the input video 46. The skeleton detection unit 22 corresponds to an acquisition unit. The processing performed by the skeleton detection unit 22 corresponds to an acquisition process. The skeleton information acquired by the skeleton detection unit 22 from the input video 46 corresponds to worker motion information.

[0026] Furthermore, the product detection unit 23 acquires product images from the learning video 31 and the input video 46. The product detection unit 23 also corresponds to an acquisition unit. Furthermore, the processing performed by the product detection unit 23 also corresponds to acquisition processing. Furthermore, the product images acquired by the product detection unit 23 from the input video 46 correspond to work object images.

[0027] The movement similarity calculation unit 24 calculates the similarity between the skeleton information acquired from the training video 31 and the skeleton video acquired from the input video 46. The movement similarity calculation unit 24 corresponds to a similarity calculation unit.

[0028] The product similarity calculation unit 25 calculates the similarity between the product image acquired from the training video 31 and the product image acquired from the input video 46. The product similarity calculation unit 25 also corresponds to a similarity calculation unit.

[0029] The mode selection unit 26 uses the movement similarity calculated by the movement similarity calculation unit 24 and the product similarity calculated by the product similarity calculation unit 25 to select a work element identification model 33 to be used to identify a work element from a plurality of work element identification models 33. The mode selection unit 26 corresponds to a model selection unit. The processing performed by the mode selection unit 26 corresponds to a model selection process. The work element identification model 33 selected by the mode selection unit 26 corresponds to a usage estimation model.

[0030] The task element identification unit 27 uses the task element identification model 33 selected by the mode selection unit 26 to identify the task element being performed by the worker.

[0031] ***Explanation of Operation*** Next, an example of the operation of the task element identifying device 10 according to the first embodiment will be described.

[0032] The task element identification device 10 according to the first embodiment performs pre-operation processing and operation processing. In the pre-operation processing, the task element identification device 10 uses a learning video 31 to prepare for the operation processing. In the operation processing, the task element identification device 10 analyzes an input video 46 that has been input, and selects a task element identification model 33 to be used to identify a task element. First, the pre-operation processing according to the first embodiment will be described with reference to FIG. 2 .

[0033] (Step S11: Skeleton detection process) The skeleton detection unit 22 reads all learning videos 31 from the storage 13. As shown in FIG. 3 , the skeleton detection unit 22 detects skeleton information 42 of one worker 41 from each learning video 31. FIG. 3 shows skeletal information 42 for one joint and one cycle of a work procedure. However, in reality, the skeleton detection unit 22 detects skeletal information 42 for multiple joints and multiple cycles. One cycle of a work procedure includes all work elements.

[0034] The skeleton detection unit 22 determines one task identification target from one or more workers 41 shown in the learning video 31. Then, the skeleton detection unit 22 detects skeletal information 42 of the task identification target. Furthermore, the skeleton detection unit 22 assigns index information of the task element identification model 33 to the skeletal information 42 of the task identification target, corresponding to the learning video 31 from which the skeletal information 42 was detected. Then, the skeletal information 42 is written to the memory 12 together with the index information. In this embodiment, the skeleton detection unit 22 detects skeletal information 42 of the task identification target from each of the "right-handed & model i learning video," "right-handed & model ii learning video," "left-handed & model i learning video," and "left-handed & model ii learning video." Then, the skeleton detection unit 22 assigns index information indicating "right-handed & model i model" to the skeletal information 42 detected from the "right-handed & model i learning video." The skeleton detection unit 22 similarly detects skeleton information 42 for the "right-handed & model II learning video," "left-handed & model I learning video," and "left-handed & model II learning video," and assigns index information. One method for detecting the skeleton information 42 is to use deep learning. Another method for determining the specific task target is to select the person whose Euclidean distance between the detected right and left shoulder coordinates is the largest.

[0035] (Step S12: Product Detection Processing) The product detection unit 23 reads all training videos 31 from the storage 13. As shown in FIG. 4 , the product detection unit 23 detects a product 44 to be assembled from a representative frame 43 of each training video 31, and cuts out a product image 45 showing the product 44 to be assembled. The product detection unit 23 assigns index information of the work element identification model 33 to the cut-out product image 45, corresponding to the training video 31 from which the product image 45 was cut out. Then, the product detection unit 23 writes the product image 45 together with the index information to the memory 12.

[0036] Methods for selecting the representative frame 43 include a method in which the product detection unit 23 selects the representative frame 43 from the training video 31 at regular intervals, and a method using the true value 32 of the training video. In the method using the true value 32 of the training video, the product detection unit 23 reads the true value 32 of the training video from the memory 12 and identifies the frame where each task element starts and ends from the true value 32 of the training video. The product detection unit 23 then selects a central frame or the like between the frame where each task element starts and the frame where it ends as the representative frame 43. There may be multiple frames where a single task element starts and ends. That is, there may be multiple instances where a single task element is performed multiple times in the training video 31. In this case, the product detection unit 23 may prepare a representative frame 43 for each time. Alternatively, the product detection unit 23 may select multiple representative frames 43 between the frame where each task element starts and the frame where it ends. Methods for cutting out the product image 45 include a method using deep learning, a method of cutting out the area around the coordinates of the wrist of the specific work target detected in step S11, and a method of cutting out a predetermined range.

[0037] The operation process according to the first embodiment will be described with reference to FIG.

[0038] (Step S21: Video Acquisition Process) The video acquisition unit 21 acquires the input video 46 via the communication interface 14. The video acquisition unit 21 writes the input video 46 to the memory 12. The input video 46 is a video of an arbitrary time that shows one or more workers 41 performing assembly work and a product 44 to be assembled.

[0039] (Step S22: End Determination Process) The video acquisition unit 21 determines whether or not it has been able to acquire the input video 46. If it has not been able to acquire the input video 46, it ends the operation process.

[0040] (Step S23: Skeleton Detection Processing) The skeleton detection unit 22 reads the input video 46 from the memory 12. As shown in FIG. 6 , the skeleton detection unit 22 detects skeletal information 42 of one worker 41 from the input video 46. FIG. 6 shows skeletal information 42 for one joint. However, in reality, the skeleton detection unit 22 detects skeletal information 42 for multiple joints. The skeleton detection unit 22 determines one specified task target from one or more workers 41 shown in the input video 46. Then, the skeleton detection unit 22 detects skeletal information 42 of the specified task target, and further, the skeleton detection unit 22 writes the skeletal information 42 of the specified task target to the memory 12. Methods for detecting the skeletal information 42 include a method using deep learning. Methods for determining the specified task target include a method of selecting a person whose Euclidean distance between the detected coordinates of the right shoulder and left shoulder is the largest.

[0041] (Step S24: Product Detection Processing) The product detection unit 23 reads the input video 46 from the memory 12. As shown in FIG. 7 , the product detection unit 23 detects the product 44 to be assembled from the representative frame 43 of the input video 46, and cuts out a product image 45 showing the product 44 to be assembled. The product detection unit 23 writes the cut-out product image 45 to the memory 12. The product detection unit 23 selects one frame, such as the center frame of the input video 46, as the representative frame 43. Methods for cutting out the product image 45 include a method using deep learning, a method of cutting out the area around the coordinates of the wrist of the task identification target detected in step S23, and a method of cutting out a predetermined range.

[0042] (Step S25: Movement similarity calculation process) The movement similarity calculation unit 24 reads the skeletal information 42 of all learning videos 31 detected in step S11 and the skeletal information 42 of the input video 46 detected in step S23 from the memory 12. As shown in Figure 8, the movement similarity calculation unit 24 calculates the similarity between each piece of skeletal information 42 of the learning videos 31 detected in step S11 and the skeletal information 42 of the input video 46 detected in step S23 in a brute force manner. The movement similarity calculation unit 24 writes the calculated similarity to the memory 12.

[0043] As shown in FIG. 9 , when calculating the similarity of the skeletal information 42, the movement similarity calculation unit 24 divides each piece of skeletal information 42 of the training video 31 detected in step S11 into work elements. Specifically, the movement similarity calculation unit 24 reads the true values ​​32 of the training video from the memory 12 and identifies the frames at which each work element starts and ends from the true values ​​32 of the training video. The movement similarity calculation unit 24 then divides the skeletal information 42 of the training video 31 into work elements based on the identified frames. Furthermore, the movement similarity calculation unit 24 calculates the similarity between the divided skeletal information 42 of the training video 31 and the skeletal information 42 of the input video 46 detected in step S23 in a brute-force manner. In the examples of FIGS. 8 and 9 , for each of the two pieces of skeletal information 42 shown on the left side of FIG. 8 , the similarity between the two pieces of skeletal information 42 of the training video 31 and the skeletal information 42 of the input video 46 detected in step S23 is calculated for each work element shown in FIG. 9 in a brute-force manner. As a method for determining the similarity of the skeleton information 42, there is a method in which the Euclidean distance between two pieces of skeleton information 42 is calculated for each joint, and the skeleton information 42 with a smaller total Euclidean distance is deemed to have a higher similarity.

[0044] (Step S26: Product Similarity Calculation Process) The product similarity calculation unit 25 reads all product images 45 of the learning video 31 cut out in step S12 and the product images 45 of the input video 46 cut out in step S24 from the memory 12. As shown in FIG. 10 , the product similarity calculation unit 25 calculates the similarity between each product image 45 of the learning video 31 cut out in step S12 and the product image 45 of the input video 46 cut out in step S24 in a brute-force manner. The method for calculating the similarity of the product images may be a well-known method such as Bag of Features. The product similarity calculation process 24 writes the calculated similarity to the memory 12.

[0045] (Step S27: Mode Selection Process) The mode selection unit 26 reads from the memory 12 the skeletal information 42 of the learning video 31 detected in step S11, the product image 45 of the learning video 31 extracted in step S12, and the similarities calculated in steps S25 and S26. The mode selection unit 26 selects a work element identification model 33 based on the magnitude of the similarity of the skeletal information 42 calculated in step S25, the magnitude of the similarity of the product image 45 calculated in step S26, and the index information assigned to the skeletal information 42 of the learning video 31 used in the similarity calculation and the index information assigned to the product image 45 of the learning video 31. The mode selection unit 26 selects a mode corresponding to the selected work element identification model 33 as the operating mode (work element identification mode) of the work element identification unit 27.

[0046] Here, consider a case in which the mode selection unit 26 selects one of the four types of task element identification models 33, namely, the "right-handed & model i model," "right-handed & model ii model," "left-handed & model i model," and "left-handed & model ii model." The task element identification device 10 also has a separate correspondence table, such as that shown in FIG. 11 , that indicates the correspondence between the index information of each task element identification model 33 and the four types of task element identification models. In this case, the mode selection unit 26 selects the task element identification model 33 based on a combination of "dominant hand," which is a characteristic of the task identification target person shown in the input video 46, and "model," which is a characteristic of the product 44 to be assembled.

[0047] First, the mode selection unit 26 selects, on a work element basis, the skeletal information 42 of the learning video 31 detected in step S11 that has the greatest similarity to the skeletal information 42 of the input video 46 detected in step S23. That is, the mode selection unit 26 selects the work element with the greatest similarity from the work elements of the multiple pieces of skeletal information 42 of the learning video 31. Then, the mode selection unit 26 identifies the work element identification model 33 indicated by the index information assigned to the selected work element as a first candidate model.

[0048] Next, the mode selection unit 26 selects, from the product images 45 of the learning video 31 extracted in step S12, the product image 45 that has the greatest similarity to the product image 45 of the input video 46 extracted in step S24. Then, the mode selection unit 26 identifies the work element identification model 33 indicated by the index information assigned to the selected product image 45 as the second candidate model.

[0049] The mode selection unit 26 compares a first worker characteristic (handedness), which is a characteristic of a worker corresponding to the first candidate model, with a second worker characteristic (handedness), which is a characteristic of a worker corresponding to the second candidate model. The mode selection unit 26 also compares a first work object characteristic (model), which is a characteristic of a work object corresponding to the first candidate model, with a second work object characteristic (model), which is a characteristic of a work object corresponding to the second candidate model. If the first worker characteristic matches the second worker characteristic but the first work object characteristic does not match the second work object characteristic, the mode selection unit 26 prioritizes the calculation result of the similarity of the skeletal information 42 and selects one of the candidate models. That is, the mode selection unit 26 selects the candidate model with the greater similarity in terms of the skeletal information 42 from the first candidate model and the second candidate model. Furthermore, when the first worker characteristics and the second worker characteristics do not match, but the first work object characteristics and the second work object characteristics match, the mode selection unit 26 prioritizes the calculation result of the similarity of the product image 45 and selects one of the candidate models. That is, the mode selection unit 26 selects the candidate model with the greater similarity for the product image 45 from the first candidate model and the second candidate model. When the first worker characteristics and the second worker characteristics do not match, and the first work object characteristics and the second work object characteristics do not match, the mode selection unit 26 normalizes the similarity of the skeleton information 42 and the similarity of the product image 45 to scores between 0 and 1, and selects one of the candidate models with the greater total score. That is, the mode selection unit 26 selects the candidate model with the greater total score obtained by normalizing the similarity for the skeleton information 42 and the similarity for the product image 45 from the first candidate model and the second candidate model.

[0050] 12 shows a specific example of the selection criteria for the task element identification model 33. When the first and second worker characteristics match and the first and second work object characteristics match, i.e., when the first and second candidate models match, the mode selection unit 26 selects the first candidate model (= the second candidate model). When the first and second candidate models do not match, but only the first and second work object characteristics do not match (the product models do not match), such as when "right-handed & model i model" and "right-handed & model ii model," the mode selection unit 26 selects a candidate model with a high degree of similarity in terms of the product image 45. When only the first and second worker characteristics do not match, such as when "right-handed & model i model" and "left-handed & model i model," the mode selection unit 26 selects a candidate model with a high degree of similarity in terms of the skeleton information 42. Furthermore, when the first worker characteristic and the second worker characteristic do not match, and further when the first work object characteristic and the second work object characteristic do not match, such as in the case of a "right-handed & machine type i model" and a "left-handed & machine type ii model," the mode selection unit 26 normalizes the similarity for the skeleton information 42 and the similarity for the product image 45 to scores between 0 and 1. Then, the mode selection unit 26 selects the candidate model with the largest total score for the skeleton information 42 and the score for the product image 45.

[0051] (Step S28: Work element identification process) In the work element identification mode selected in step S27, the work element identification unit 27 uses the selected work element identification model 33 to identify the work element being performed by the worker 41 and output the work element name.

[0052] ***Effects of First Embodiment*** As described above, according to this embodiment, it is possible to select an appropriate task element identification model 33 according to the combination of the characteristics of the worker and the characteristics of the task object.

[0053] More specifically, the task element identification device 10 according to this embodiment calculates the similarity of the skeletal information 42 between the input video 46 and the learning video 31, and selects a task element identification mode based on the calculated similarity. The skeletal information 42 is human motion information, such as time-series information on the coordinates of the human joints and time-series information on the relative positional relationships of multiple joints. Methods that use the Euclidean distance of the skeletal information 42 can also be used to calculate the similarity. In methods that use the Euclidean distance, even if the same task element portion of two pieces of skeletal information 42 contains different movements, it is possible to determine that the similarity is high based on the common movement portion. Therefore, according to this embodiment, an appropriate task element identification model 33 can be selected using human motion information.

[0054] Furthermore, the task element identification device 10 according to this embodiment calculates the similarity of the product images 45 between the input video 46 and the learning video 31, and determines the task element identification mode based on the calculated similarity. The similarity of the product images 45 is high if the products 44 are of the same model, and low if the products 44 are of different models. Therefore, according to this embodiment, it is possible to select an appropriate task element identification model 33 depending on the model of the product 44.

[0055] ***Other Configurations*** <Variation 1> In the first embodiment, as shown in Figures 2 to 4, the task element identification device 10 acquires skeleton information 42 and product images 45 from the learning video 31 and input video 46 as pre-operation processing, and writes them to the memory 12. However, the task element identification device 10 may write skeleton information 42 and product images 45 generated externally in advance to the memory 12. Furthermore, the entire pre-operation processing (from assigning index information to the skeleton information 42 to assigning index information to the product images 45) may be performed externally. In this case, the task element identification device 10 performs operation processing using the skeleton information 42 and index information, and the product images 45 and index information generated externally.

[0056] <Modification 2> In the first embodiment, as shown in FIG. 1 , the task element identification device 10 is a single device. However, the task element identification device 10 may be composed of multiple devices. As a specific example, as shown in FIG. 13 , the task element identification device 10 may be composed of two devices: a detection device 110 and an identification device 120. The detection device 110 acquires skeleton information 42 and product images 45 from a training video 31. The identification device 120 identifies task elements from an input video 46. In this case, the training video 31, the true value 32 of the training video, and the task element identification model 33 may be stored in storage provided external to the detection device 110 and the identification device 120. Furthermore, the training video 31, the true value 32 of the training video, and the task element identification model 33 may be stored in storage provided external to either the detection device 110 or the identification device 120. Note that the hardware of the detection device 110 and the identification device 120 is not shown in FIG. 13 . Similar to the work element identifying device 10, the detecting device 110 and the identifying device 120 each include a processor, a memory, a storage device, and a communication interface as hardware.

[0057] <Modification 3> In the first embodiment, the functional components of the work element identification device 10 are realized by software. However, in a third modification, the functional components of the work element identification device 10 may be realized by hardware. Differences between this third modification and the first embodiment will be described below. An example configuration of the work element identification device 10 according to the third modification will be described with reference to FIG. 14 . When the functional components are realized by hardware, the work element identification device 10 includes an electronic circuit 15 instead of the processor 11, memory 12, and storage 13. The electronic circuit 15 is a dedicated circuit that realizes the functions of the functional components, memory 12, and storage 13. Possible examples of the electronic circuit 15 include a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, a logic IC, a GA (Gate Array), an ASIC (Application Specific Integrated Circuit), and an FPGA (Field-Programmable Gate Array). All functional components may be implemented in a single electronic circuit 15, or the functional components may be distributed across multiple electronic circuits 15.

[0058] <Modification 4> As a modification 4, some of the functional components of the task element identification device 10 may be realized by hardware, and other functional components may be realized by software.

[0059] <Variation 5> In the first embodiment, the mode selection unit 26 determines whether to prioritize the calculation result of the similarity between the skeleton information 42 or the product image 45, based on the relationship between the first and second candidate models narrowed down based on the similarity between the skeleton information 42 and the product image 45. However, the mode selection unit 26 may prioritize the calculation result of the similarity of the product image 45 if the maximum value of the similarity of the product image 45 calculated in step S26 is equal to or greater than a threshold, and may prioritize the calculation result of the similarity of the skeleton information 42 if the maximum value is less than the threshold.

[0060] Embodiment 2. Embodiment 2 differs from embodiment 1 in that a search map is generated in pre-operation processing to narrow down the selection candidates for work element identification models. Also, embodiment 2 differs from embodiment 1 in that a work element identification model is selected from the selection candidates narrowed down using the search map and / or information from the previous cycle in operation processing. In embodiment 2, these differences will be mainly described, and a description of the same points will be omitted.

[0061] In the second embodiment, multiple work procedures are assumed. In the second embodiment, the work elements included in each work procedure are specified, and the order in which the work elements are performed is also specified. In the second embodiment, it is assumed that one model of product 44 to be assembled is determined for each work procedure. For example, if work procedures A and B exist, model A of product 44 is to be assembled for work procedure A, and model B of product 44 is to be assembled for work procedure B. Furthermore, if the workers 41 are differently handed, the work procedures and the work elements that make up the work procedures are treated as separate even if the model of the product 44 to be assembled is the same. For example, as work procedures for assembling products A and B, there are work procedures A and B for right-handed workers, and work procedures A' and B' for left-handed workers. Furthermore, if work procedure A is composed of work elements a, b, ..., work procedure A' is composed of work elements a', b', .... The same applies to the work elements that make up work procedures B and B'. A learning video 31 exists for each work procedure, and a work element identification model 33 exists for each learning video 31. For example, work procedures A, B, A', and B' each have work videos A, B, A', and B', and work videos A, B, A', and B' each have work element identification models A, B, A', and B'. In the first embodiment, the worker's dominant hand (right-handed or left-handed) is assumed as the worker's characteristic, and the product model (model i, model ii) is assumed as the characteristic of the work object. The "right-handed & model i learning video," "right-handed & model ii learning video," "left-handed & model i learning video," and "left-handed & model ii learning video" assumed in the first embodiment correspond to the work videos A, B, A', and B' in the second embodiment. Furthermore, the "right-handed & model i model," "right-handed & model ii model," "left-handed & model i model," and "left-handed & model ii model" correspond to task element identification models A, B, A', and B' in embodiment 2. Meanwhile, in this embodiment, the worker's dominant hand is assumed to be either right-handed or left-handed, and the task object's characteristic is assumed to be the product model (models A to F). Furthermore, embodiment 2 is premised on the fact that multiple learning videos 31 contain at least one task element in common.Furthermore, in the second embodiment, it is assumed that the input video starts from the beginning of the work, not from the middle of the work.

[0062] ***Description of Configuration*** The configuration of an task element identification device 10 according to the second embodiment will be described with reference to FIG. 15 . The task element identification device 10 differs from the task element identification device 10 shown in FIG. 1 in that it includes a search map generation unit 28 and a model search unit 29. Furthermore, the second embodiment differs from the first embodiment in that the task element identification model 33 outputs not only the name of the identified task element but also the reliability of the identification. Furthermore, in the second embodiment, the movement similarity calculation unit 24 prioritizes the task element identification model 33 previously selected by the mode selection unit 26 when calculating the similarity with the skeletal information 42 of the input video 46. Similarly, the product similarity calculation unit 25 prioritizes the task element identification model 33 previously selected by the mode selection unit 26 when calculating the similarity with the product image 45 of the input video 46. Furthermore, in the second embodiment, the movement similarity calculation unit 24 sets priorities among the work element identification models 33 by analyzing the input video 46, and calculates the similarity between the input video 46 and the skeletal information 42 of the work element identification model 33 in descending order of priority. Similarly, the product similarity calculation unit 25 calculates the similarity between the input video 46 and the product image 45 of the work element identification model 33 in descending order of priority.

[0063] ***Description of Operation*** The operation of the work element identification device 10 according to embodiment 2 will be described with reference to Fig. 16 to Fig. 24. The operation of the work element identification device 10 according to embodiment 2 includes pre-operation processing and operation processing.

[0064] Pre-operation processing according to the second embodiment will be described with reference to Fig. 16. The processing from step S31 to step S32 is the same as the processing from step S11 to step S12 in Fig. 2. Therefore, the description of these processing will be omitted.

[0065] (Step S33: Search map generation process) The search map generation unit 28 reads the true values ​​32 of the learning video from the storage 13, and reads the skeleton information 42 detected in step S31 and the product image 45 cut out in step S32 from the memory 12. The search map generation unit 28 generates a work element table 47 using the skeleton information 42 and the true values ​​32 of the learning video. The search map generation unit 28 also generates a search map 48 from the work element table 47, the skipped portions obtained from the work element table 47, and the product image 45. The search map generation unit 28 then writes the search map 48 to the memory 12. The work element table 47 is a table that chronologically lists the work elements being performed in each learning video 31. The work element table 47 only includes work elements for one cycle of the work procedure. The search map 48 is a map for narrowing down the work element identification models 33 to be selected when the work element identification device 10 selects the work element identification model 33 from the input video 46 .

[0066] The contents and generation method of the work element table 47 will now be described in detail. The search map generation unit 28 identifies the start and end frames of each work element from the true value 32 of the learning video. The search map generation unit 28 then divides the skeletal information 42 into work elements based on the identified frames. Furthermore, as shown in FIG. 17 , the search map generation unit 28 calculates the similarity for each work element in a round-robin manner for each piece of skeletal information 42 from different learning videos 31 that detected the work element. One method for calculating the similarity of the skeletal information 42 is to calculate the Euclidean distance between two pieces of skeletal information 42 for each joint, and determine that the skeletal information 42 with a smaller sum of Euclidean distances has a higher similarity. The search map generation unit 28 treats work elements whose similarity in the skeletal information 42 is equal to or greater than a threshold as the same work element and generates a work element table 47 as shown in FIG. 18 . In FIG. 18 , the work element numbers indicate the order of the work elements. For example, learning video A corresponds to work procedure A. In work procedure A, work elements are performed in the order of work elements a, c, d, f, and h. Note that work element numbers 1 to 5 in Figure 18 constitute one cycle of the work procedure.

[0067] Next, the contents of the search map 48 and the method for generating it will be described in detail. The search map generation unit 28 generates a search map 48 as shown in FIG. 19. FIG. 19 shows the search map 48 generated from the work element table 47 of FIG. 18. The search map 48 is a tree-structured map in which the order of performing two or more work processes for each training video (work procedure) is indicated by node connections. In FIG. 19, the work element identification models 33 generated from training videos A, B, ... are respectively represented as work element identification models A, B, ....

[0068] 19, for example, when the work element number is 1, if the previously identified work element is a, work element identification models A, B, and C, which are leaf nodes of the subtree rooted at node a, become selection candidates. (skip) in FIG. 19 indicates that there is no need to change the work element identification model 33 from the time the previous work element was performed. In other words, (skip) indicates that the similarity calculation for selecting the work element identification model 33 can be skipped.

[0069] The search map generation unit 28 determines up to one representative image 49 from the product images 45 for each learning video 31 from which the images were extracted and for each work element, and generates an updated search map 50 using the representative image 49 .

[0070] An example of the representative image 49 is shown in FIG. 20 . FIG. 20 corresponds to the work element table 47 in FIG. 18 , and is a list in which the product image 45 of the start frame of each work element is used as the representative image 49. Note that the start frame of work element number 1 is a frame before the product is set, so no representative image 49 exists. When there are multiple representative images 49 in FIG. 20 corresponding to each node in the search map 48, the search map generator 28 calculates the similarity of the representative images 49 of that node in a brute-force manner. Well-known methods such as bag of features may be used to calculate the similarity of the representative images 49. If the calculated similarity of the representative images 49 is less than a threshold, the search map generator 28 generates an updated search map 50. For example, node a in FIG. 19 corresponds to A1, B1, and C1 in FIG. 20 , but because no representative image exists, the search map generator 28 does not perform a similarity calculation. Node d of the subtree rooted at node a in Figure 19 corresponds to A3 and B3 in Figure 20. Therefore, the search map generation unit 28 calculates the similarity between representative image A3 and representative image B3. If the similarity is less than a threshold, the search map generation unit 28 splits node d into nodes d1 and d2. Figure 21 shows an updated search map 50, which is the search map 48 after all node splits have been performed.

[0071] The operation process according to the second embodiment will be described with reference to Figures 22 and 23. The processes from step S41 to step S44 are the same as the processes from step S21 to step S24 in Figure 5. The process from step S52 is the same as the process from step S28 in Figure 5. Therefore, the description of these processes will be omitted.

[0072] (Step S45: Search Implementation Determination Processing) If step S45 is being performed for the second or subsequent time, the model search unit 29 proceeds to step S46. If step S45 is being performed for the first time, the model search unit 29 sets all work element identification models 33 as selection candidates and writes index information of the selection candidates to the memory 12. Then, the process proceeds to step S48.

[0073] (Step S46: Model search processing) The model search unit 29 reads the updated search map 50 from the memory 12. The model search unit 29 determines selection candidates for the work element identification model 33 using the updated search map 50 from the work element number of the input video 46 at that time and the past work elements identified by the work element identification unit 27. The model search unit 29 also determines the search results using the updated search map 50 for the work element number of the input video 46 at that time from the work element number of the input video 46 at that time and the past work elements identified by the work element identification unit 27. The model search unit 29 writes the index information of the selection candidates for the work element identification model 33 and the search results using the updated search map 50 to the memory 12.

[0074] (Step S47: Skip Determination Processing) The model search unit 29 reads the search result by the updated search map 50 from the memory 12. If the search result is (skip), the model search unit 29 skips the similarity calculation processing and proceeds to step S50. On the other hand, if the search result is not (skip), the model search unit 29 proceeds to step S48.

[0075] (Step S48: Movement Similarity Calculation Process) The second embodiment differs from the process of step S25 in that the movement similarity calculation unit 24 reads index information of selection candidates for the work element identification model 33 and the updated search map 50 from the memory 12. Furthermore, the second embodiment differs from the process of step S25 in that when reading out the skeletal information 42 of the learning video 31 detected in step S31 from the memory 12, not all of the skeletal information 42 is read out. Furthermore, the second embodiment differs from the process of step S25 in that the order in which similarity calculations are performed using the read skeletal information 42 is determined, and if the similarity is equal to or greater than a threshold, the similarity calculations are terminated and the selection candidates for the work element identification model 33 are narrowed down.

[0076] Specifically, the movement similarity calculation unit 24 reads from the memory 12, among the skeletal information 42 of the learning video 31 detected in step S31, the skeletal information 42 whose assigned index information matches the selection candidate for the work element identification model 33. When the input video 46 shows a work procedure from the second cycle onward, the movement similarity calculation unit 24 performs a first similarity calculation using, among the read skeletal information 42, the skeletal information 42 whose assigned index information matches the work element identification model 33 selected in the previous cycle. For example, assume that the input video 46 shows the work procedure for the third cycle and that work element identification model A is selected for the work procedure for the second cycle. In this case, the movement similarity calculation unit 24 performs a similarity calculation using the skeletal information 42 having the index information of work element identification model A. Work procedures on a product assembly line are often performed not just once but multiple times. For this reason, it is considered that the probability of using the same work element identification model 33 as in the previous cycle is higher than the probability of using a work element identification model 33 different from the previous cycle. Therefore, the movement similarity calculation unit 24 prioritizes the task element identification model 33 selected in the previous cycle in calculating the similarity between the input video 46 and the skeleton information 42 .

[0077] Furthermore, the movement similarity calculation unit 24 references the update search map 50 and sets the priority order between the work element identification models 33, i.e., the order of similarity calculation, based on the number of leaf nodes of the work element identification models 33 remaining as selection candidates at that time. For example, assume a situation in which work process b is extracted by analyzing the input video 46. That is, assume a situation in which one of nodes d and e branching from node b corresponding to work process b in the update search map 50 of FIG. 24 is to be selected. In this case, work element identification models D, E, and F remain as selection candidates. At this time, the subtree rooted at node d has one leaf node D, and the subtree rooted at node e has two leaf nodes E and F. Therefore, the movement similarity calculation unit 24 first performs similarity calculation using skeleton information 42 having index information for work element identification model E or work element identification model F. Then, the movement similarity calculation unit 24 subsequently performs similarity calculation using skeleton information 42 having index information for work element identification model D. Since it is considered that there is a high probability that the work element corresponding to the node that is the root of a subtree with a large number of leaf nodes is being performed in the input video 46, the movement similarity calculation unit 24 sets priorities between the work element identification models 33 in this manner.

[0078] If the similarity obtained by the calculation is equal to or greater than a threshold, the movement similarity calculation unit 24 determines, as a selection candidate, the work element specifying model 33 indicated by the index information assigned to the skeleton information 42 used in the calculation. Then, the movement similarity calculation unit 24 writes the index information of the work element specifying model 33 determined as a selection candidate and the similarity to the memory 12, and ends the processing of step S48.

[0079] If a previous cycle exists, the movement similarity calculation unit 24 performs a similarity calculation based on the selection result of the previous cycle. On the other hand, if a previous cycle does not exist, the movement similarity calculation unit 24 performs a similarity calculation based on the number of leaf nodes. Specifically, in the example of FIG. 24 , in a situation where one of nodes d and e branching from node b is to be selected, if work element identification model D was selected in the previous cycle, the movement similarity calculation unit 24 performs a similarity calculation based on the number of leaf nodes. On the other hand, if a previous cycle does not exist (or the selection result in the previous cycle was not one of work element identification models D to F), the movement similarity calculation unit 24 performs a similarity calculation based on the number of leaf nodes. Furthermore, even if the similarity calculation based on the selection result of the previous cycle is less than the threshold, the movement similarity calculation unit 24 performs a similarity calculation based on the number of leaf nodes.

[0080] (Step S49: Product Similarity Calculation Processing) In the second embodiment, the product similarity calculation unit 25 reads index information of selection candidates for the work element identification model 33 and the updated search map 50 from the memory 12. In addition, in the second embodiment, when the product images 45 of the learning video 31 extracted in step S32 are read from the memory 12, not all of the product images 45 are read, which is different from the processing of step S25. In the second embodiment, the order in which similarity calculation is performed for the read product images 45 is determined, and if the similarity is equal to or greater than a threshold, the similarity calculation is terminated and the selection candidates for the work element identification model 33 are narrowed down.

[0081] Specifically, the product similarity calculation unit 25 reads from the memory 12, among the product images 45 of the training video 31 extracted in step S32, product images 45 whose assigned index information matches the selection candidates of the task element identification model 33. However, when referencing the portion where the node was divided using the representative image 49 in step S33, such as when selecting one of node d1, node d2, and node e branching from node a in the updated search map 50, or when selecting one of node d, node e1, and node e2 branching from node b, the product similarity calculation unit 25 reads only the representative image 49 from the memory 12. The product similarity calculation unit 25 determines the order in which to perform similarity calculations by a method in which the skeleton information 42 in step S48 is replaced with the product image 45. In other words, the product similarity calculation unit 25 performs similarity calculations for the product images 45 in the same order as the similarity calculations for the skeleton information 42 in the movement similarity calculation unit 24. If the similarity obtained by the calculation is equal to or greater than a threshold value, the product similarity calculation unit 25 determines, as a selection candidate, the work element specifying model 33 indicated by the index information assigned to the product image 45 used in the calculation. Then, the product similarity calculation unit 25 writes the index information of the work element specifying model 33 determined as a selection candidate and the similarity to the memory 12, and ends the processing of step S49.

[0082] (Step S50: Mode Selection Process) In the second embodiment, if the similarity calculation is skipped in step S47, the mode selection unit 26 selects the same task element identification mode as the previous selection result, which is different from step S28.

[0083] (Step S51: Cycle Switch Determination Process) In the second embodiment, the work element identification unit 27 outputs the reliability of work element identification before outputting the work element name. The work element identification unit 27 determines that the cycle has switched if the reliability of work element identification is less than a threshold value. Then, the work element identification unit 27 treats all work element identification models 33 as selection candidates and writes index information of the selection candidates to the memory 12. Thereafter, the process returns to step S48. If the reliability of work element identification is equal to or greater than the threshold value, the process proceeds to step S52.

[0084] ***Description of Effects*** As described above, the work element identification device 10 according to the second embodiment selects a work element identification model 33 from among the selection candidates narrowed down by the generated updated search map 50 and / or information from the previous cycle. Narrowing down the selection candidates for the work element identification model 33 reduces the amount of calculation for the similarity calculation for the skeleton information 42 and the product image 45, making it possible to speed up processing. Speeding up processing makes it possible to select a work element identification model 33 and identify work elements in real time.

[0085] ***Other Configurations*** <Variation 1> In the second embodiment, the movement similarity calculation unit 24 determines the order in which to perform the similarity calculations, and terminates the similarity calculation when the similarity is equal to or greater than a threshold value. However, the movement similarity calculation unit 24 may perform the similarity calculations using all of the read skeleton information 42 without determining the order in which to perform the similarity calculations, and may select the task element identification model 33 indicated by the index information assigned to the skeleton information 42 with the highest similarity as a selection candidate.

[0086] <Modification 2> In the second embodiment, the work element identification device 10 generates a work element table 47 and a search map 48 as pre-operation processing. Alternatively, the entire pre-operation processing, including the generation of the work element table 47 and the search map 48, may be performed externally. In this case, the work element identification device 10 performs operation processing using the work element table 47 and search map 48 generated externally.

[0087] Although the first and second embodiments have been described above, these two embodiments may be implemented in combination. Alternatively, one of these two embodiments may be implemented in part. Alternatively, these two embodiments may be implemented in part in combination. Furthermore, the configurations and procedures described in these two embodiments may be modified as necessary.

[0088] ***Supplementary Explanation of Hardware Configuration*** Finally, a supplementary explanation of the hardware configuration of the task element identification device 10 will be provided.

[0089] The storage 13 of the work element identification device 10 stores an OS (Operating System) in addition to programs that realize the functional components of the work element identification device 10. At least a portion of the OS is executed by the processor 11. The processor 11 executes programs that realize the functional components of the work element identification device 10 while executing at least a portion of the OS. The processor 11 executes the OS to perform task management, memory management, file management, communication control, and the like. At least one of information, data, signal values, and variable values ​​indicating the results of processing of the functional components of the work element identification device 10 is stored in at least one of the memory 12, the storage 13, and a register and cache memory within the processor 11. The programs that realize the functional components of the work element identification device 10 may be stored on a portable recording medium such as a magnetic disk, a flexible disk, an optical disk, a compact disk, a Blu-ray (registered trademark) disk, or a DVD. Portable recording media on which the programs that realize the functional components of the work element identification device 10 are stored may be distributed.

[0090] Furthermore, the "part" of the functional components of the work element identification device 10 may be read as a "circuit," a "step," a "procedure," a "process," or a "circuitry." In this specification, the higher-level concept of the processor 11 and the electronic circuit 15 is referred to as a "processing circuitry." In other words, the processor 11 and the electronic circuit 15 are each specific examples of a "processing circuitry."

[0091] 10 Work element identification device, 11 Processor, 12 Memory, 13 Storage, 14 Communication interface, 15 Electronic circuit, 21 Video acquisition unit, 22 Skeleton detection unit, 23 Product detection unit, 24 Movement similarity calculation unit, 25 Product similarity calculation unit, 26 Mode selection unit, 27 Work element identification unit, 28 Search map generation unit, 29 Model search unit, 31 Training video, 32 True value of training video, 33 Work element identification model, 41 Worker, 42 Skeleton information, 43 Representative frame, 44 Product, 45 Product image, 46 Input video, 47 Work element table, 48 Search map, 49 Representative image, 50 Updated search map, 110 Detection device, 120 Identification device.

Claims

1. an acquisition unit that acquires worker motion information, which is motion information of a worker when performing work on a work object, and a work object image, which is an image of the work object; a model selection unit that selects, as an estimation model to be used for estimating a work process being performed by a worker on a work object, from a plurality of estimation models that are a plurality of models for estimating a work process being performed among a plurality of work processes, each of which is associated with motion information of a worker and an image of a work object; a similarity calculation unit that calculates a similarity between the worker motion information and motion information associated with each of the plurality of estimation models, and calculates a similarity between the work object image and an image associated with each of the plurality of estimation models, The model selection unit is selecting an estimation model that is a candidate for the used estimation model from among the plurality of estimation models as a first candidate model using the similarity between the worker motion information and motion information associated with each of the plurality of estimation models, calculated by the similarity calculation unit; selecting, as a second candidate model, an estimation model that is a candidate for the used estimation model from among the plurality of estimation models, using the similarity between the work object image and an image associated with each of the plurality of estimation models, calculated by the similarity calculation unit; An information processing device that selects one of the first candidate model and the second candidate model as the utilization estimation model.

2. each of the plurality of estimation models corresponds to any one of a plurality of combinations of a characteristic of a worker and a characteristic of a work object; The model selection unit is comparing a first worker characteristic which is a characteristic of a worker to which the first candidate model corresponds with a second worker characteristic which is a characteristic of a worker to which the second candidate model corresponds, and comparing a first work object characteristic which is a characteristic of a work object to which the first candidate model corresponds with a second work object characteristic which is a characteristic of a work object to which the second candidate model corresponds; selecting, as the used estimation model, a candidate model having a large similarity in motion information from among the first candidate model and the second candidate model when the first worker characteristic and the second worker characteristic match and the first work object characteristic and the second work object characteristic do not match; selecting, as the used estimation model, a candidate model having a large degree of similarity in an image from among the first candidate model and the second candidate model when the first worker characteristic and the second worker characteristic do not match and the first work object characteristic and the second work object characteristic match; 2. The information processing device according to claim 1, wherein when the first worker characteristic does not match the second worker characteristic and the first work object characteristic does not match the second work object characteristic, the candidate model having the larger score obtained by normalizing the similarity for motion information and the similarity for images is selected as the used estimation model from among the first candidate model and the second candidate model.

3. The similarity calculation unit is The information processing apparatus according to claim 1 , wherein in calculating the degree of similarity with the worker motion information and the degree of similarity with the work object image, a priority is given to an estimation model previously selected by the model selection unit as the used estimation model.

4. The similarity calculation unit is The information processing device according to claim 1, wherein priorities are set among the plurality of estimation models by analyzing video of the worker working on the work object, and calculations of the similarity with the worker motion information and the similarity with the work object image are performed in order of priority from the estimation model with the highest priority.

5. Each of the plurality of estimation models corresponds to one of a plurality of work procedures, each of the plurality of work procedures includes two or more of the plurality of work processes, and an execution order of the two or more work processes is specified; The acquisition unit is Obtain a tree-structured search map in which the order of performing two or more work steps is indicated by node connections for each work procedure; The similarity calculation unit is The information processing device according to claim 4 , further comprising: setting priorities among the plurality of estimation models based on the number of leaf nodes branching off from a specific node in the search map corresponding to a specific work process extracted by analysis of the video.

6. An acquisition process in which the computer acquires worker motion information, which is motion information of a worker when working on a work object, and a work object image, which is an image of the work object; a model selection process in which the computer selects, as an estimation model to be used for estimating a work process being performed by the worker on the work object, from a plurality of estimation models, each of which is a plurality of models for estimating a work process being performed among a plurality of work processes, each of which is associated with motion information of a worker and an image of a work object; a similarity calculation process in which the computer calculates a similarity between the worker motion information and motion information associated with each of the plurality of estimation models, and calculates a similarity between the work object image and an image associated with each of the plurality of estimation models; In the model selection process, The computer, selecting an estimation model that is a candidate for the used estimation model from among the plurality of estimation models as a first candidate model using the similarity between the worker motion information and motion information associated with each of the plurality of estimation models calculated in the similarity calculation process; selecting, as a second candidate model, an estimation model that is a candidate for the utilized estimation model from among the plurality of estimation models, using the similarity between the work object image and an image associated with each of the plurality of estimation models calculated in the similarity calculation process; An information processing method comprising the steps of: selecting one of the first candidate model and the second candidate model as the utilization estimation model.

7. An acquisition process for acquiring worker motion information, which is motion information of a worker when working on a work object, and a work object image, which is an image of the work object; a model selection process for selecting, as an estimation model to be used for estimating a work process being performed by a worker on a work object, from a plurality of estimation models, each of which is associated with motion information of a worker and an image of a work object, as an estimation model to be used for estimating a work process being performed by the worker on the work object; an information processing program for causing a computer to execute a similarity calculation process for calculating a similarity between the worker motion information and motion information associated with each of the plurality of estimation models, and calculating a similarity between the work object image and an image associated with each of the plurality of estimation models, In the model selection process, The computer includes: selecting, as a first candidate model, an estimation model that is a candidate for the used estimation model from among the plurality of estimation models, using the similarity between the worker motion information and motion information associated with each of the plurality of estimation models, calculated by the similarity calculation process; selecting, as a second candidate model, an estimation model that is a candidate for the used estimation model from among the plurality of estimation models, using the similarity between the work object image and an image associated with each of the plurality of estimation models, calculated by the similarity calculation process; An information processing program that causes one of the first candidate model and the second candidate model to be selected as the utilization estimation model.