Information processing device, information processing method, and information processing program

Abstract

Acquisition units (22, 23) acquire worker motion information, which is information on motions of a worker working on a workpiece, and a workpiece image, which is an image of the workpiece. A model selection unit (26) uses the worker motion information and the workpiece image to select, as an estimation model to be used, an estimation model to be used for estimating a work process being performed on the workpiece by the worker from among a plurality of estimation models for estimating a work process being performed among a plurality of work processes.

Description

[Technical field]

[0001] The present disclosure relates to a technique for estimating a work process being performed by a worker. [Background technology]

[0002] In the industrial field, there are assembly lines where the characteristics of workers (e.g., the handedness of the workers) and / or the characteristics of the work objects (e.g., the model of the product that is the work object) change depending on the time of day. In such assembly lines, there is a need for a technology that can accurately estimate the work process being performed from video. An operation process is an element that constitutes a work procedure. A single action procedure is composed of multiple operation processes. In the following, an operation process is also called a work element.

[0003] In order to accurately estimate the work process being performed from the video, it is necessary to switch the operation mode (hereinafter, sometimes simply referred to as the mode) according to the characteristics of the worker and the characteristics of the work object. In other words, it is necessary to select an appropriate estimation model according to the characteristics of the worker and / or the characteristics of the work object from among multiple estimation models for estimating the work process being performed. Then, it is necessary to switch to the operation mode corresponding to the selected estimation model and execute the selected estimation model. Currently, such switching of operation modes is performed manually, but this is a laborious task, and there is a demand for automating the switching.

[0004] Techniques related to the present disclosure include the technique disclosed in Patent Document 1 and the technique disclosed in Patent Document 2. Patent Document 1 discloses a method for automatically switching operation modes based on the amount of movement, the moving speed, and the trajectory of a person's fingers acquired from a video. Moreover, Patent Document 2 discloses a method for automatically switching between operation modes by determining the presence or absence of an object from an image. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2004-246814 A [Patent Document 2] JP 2019-215711 A Summary of the Invention [Problem to be solved by the invention]

[0006] However, the techniques of Patent Document 1 and Patent Document 2 have a problem in that they are unable to select an appropriate estimation model depending on a combination of a worker's characteristics (e.g., the worker's handedness) and a work object's characteristics (e.g., the model of the product that is the work object).

[0007] The main objective of the present disclosure is to solve such problems. Specifically, the main objective is to select an appropriate estimation model according to a combination of the characteristics of a worker and the characteristics of a work object. [Means for solving the problem]

[0008] The information processing device according to the present disclosure includes: an acquisition unit that acquires worker motion information, which is motion information of a worker when performing work on a work object, and a work object image, which is an image of the work object; The system has a model selection unit that selects, from a plurality of estimation models which are models for estimating a work process being performed among a plurality of work processes, an estimation model to be used for estimating the work process being performed by the worker on the work object using the worker motion information and the work object image as an estimation model to be used. Effect of the Invention

[0009] According to the present disclosure, an appropriate estimation model can be selected depending on a combination of the characteristics of a worker and the characteristics of a work object. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of a task element identifying device according to a first embodiment. [Diagram 2] 4 is a flowchart showing pre-operation processing according to the first embodiment. [Diagram 3] FIG. 11 is a diagram showing an example of skeleton detection processing in the pre-operation processing according to the first embodiment. [Figure 4] FIG. 11 is a diagram showing an example of product detection processing in pre-operation processing according to the first embodiment. [Diagram 5] 4 is a flowchart showing an operation process according to the first embodiment. [Figure 6] FIG. 4 is a diagram showing an example of skeleton detection processing of the operational processing according to the first embodiment. [Figure 7] FIG. 4 is a diagram showing an example of product detection processing in the operation processing according to the first embodiment. [Figure 8] FIG. 4 is a diagram showing an example of a motion similarity calculation process of the operation process according to the first embodiment. [Figure 9] FIG. 4 is a diagram showing an example of a motion similarity calculation process of the operation process according to the first embodiment. [Figure 10] FIG. 11 is a diagram showing an example of a product similarity calculation process of the operation process according to the first embodiment. [Figure 11] FIG. 4 is a diagram showing an example of a correspondence table between index information and a work element identification model according to the first embodiment. [Figure 12] FIG. 4 is a diagram showing an example of selection criteria for an operation element identification model according to the first embodiment. [Figure 13] FIG. 13 is a diagram showing a configuration example of a task element identifying device according to a second modification of the first embodiment. [Figure 14] FIG. 13 is a diagram showing a configuration example of a task element identifying device according to a third modification of the first embodiment. [Figure 15] FIG. 11 is a diagram showing an example of the configuration of a task element identifying device according to a second embodiment. [Figure 16] 11 is a flowchart showing pre-operation processing according to the second embodiment. [Figure 17] FIG. 11 is a diagram showing an example of a process for generating a work element table according to the second embodiment. [Figure 18] FIG. 11 is a diagram showing an example of a task element table according to the second embodiment. [Figure 19] FIG. 11 is a diagram showing an example of a search map according to the second embodiment. [Figure 20] 13A to 13C are diagrams showing examples of representative images in the second embodiment; [Figure 21] FIG. 11 is a diagram showing an example of an updated search map according to the second embodiment. [Figure 22] 10 is a flowchart showing an operation process according to the second embodiment. [Diagram 23] 10 is a flowchart showing an operation process according to the second embodiment. [Figure 24] FIG. 11 is a diagram showing an example of an updated search map according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, the embodiments will be described with reference to the drawings. In the following description of the embodiments and in the drawings, the same reference numerals denote the same or corresponding parts.

[0012] Embodiment 1 ***Configuration Description*** An example of the configuration of a task element identifying device 10 according to the first embodiment will be described with reference to FIG. The task element identification device 10 corresponds to an information processing device. Moreover, the operation procedure of the task element identification device 10 corresponds to an information processing method. Moreover, a program that realizes the operation of the task element identification device 10 corresponds to an information processing program. As described above, the "task element" identified by the task element identifying device 10 is an element (task process) constituting a task procedure (assembly task in this embodiment). Two or more task elements are combined to form one task procedure.

[0013] The task element identifying device 10 is a computer. The task element identification device 10 includes, as hardware, a processor 11, a memory 12, a storage 13, and a communication interface 14. The processor 11 is connected to other hardware via signal lines and controls the other hardware. The processor 11 is an integrated circuit (IC) that performs processing. The processor 11 is, for example, a central processing unit (CPU), a digital signal processor (DSP), or a graphics processing unit (GPU). The memory 12 is a storage device that temporarily stores data. The memory 12 is, for example, a static random access memory (SRAM) or a dynamic random access memory (DRAM). The storage 13 is a storage device that stores data. The storage 13 is, for example, a HDD (Hard Disk Drive). The storage 13 may also be a portable recording medium such as an SD (registered trademark, Secure Digital) memory card, a CF (registered trademark, CompactFlash), a NAND flash, a flexible disk, an optical disk, a compact disk, a Blu-ray (registered trademark) disk, or a DVD (Digital Versatile Disk). The communication interface 14 is an interface for communicating with an external device, and is, for example, a port for Ethernet (registered trademark), USB (Universal Serial Bus), or HDMI (registered trademark, High-Definition Multimedia Interface).

[0014] The task element identifying device 10 includes, as functional components, an image acquiring unit 21, a skeleton detecting unit 22, a product detecting unit 23, a movement similarity calculating unit 24, a product similarity calculating unit 25, a mode selecting unit 26, and a task element identifying unit 27. The functions of the functional components of the task element identifying device 10 are realized by software. The storage 13 stores a program that realizes each functional component of the task element identification device 10. This program is loaded into the memory 12 by the processor 11 and executed by the processor 11. In this way, each functional component of the task element identification device 10 is realized. The storage 13 also stores a learning video 31 , a true value 32 of the learning video, and an operation element identification model 33 .

[0015] 1, there is one processor 11. However, there may be a plurality of processors 11, and the plurality of processors 11 may cooperate to execute programs that realize the respective functional components.

[0016] The learning video 31 is a video showing one or more workers performing assembly work and a product that is the object of the work. In this embodiment, there are a plurality of learning videos 31. Each learning video 31 corresponds to one of a plurality of work procedures.

[0017] The learning video true value 32 is label information of the task element being performed in each frame of the learning video 31 .

[0018] The task element identification model 33 is a model generated for each learning video 31 . In this embodiment, there are a plurality of work element identification models 33. Each work element identification model 33 corresponds to one of the plurality of learning videos 31. In other words, each work element identification model 33 corresponds to one of the plurality of work procedures. The work element specification model 33 is an example of an estimation model.

[0019] The task element identifying device 10 can estimate the task element being performed by the worker shown in the video by applying the skeletal information of the worker acquired from the video to the task element identifying model 33. The skeletal information is motion information that represents the motion of the body of a worker while working. The skeletal information includes time-series information on the coordinates of multiple joints such as the worker's neck and wrist, or time-series information on the relative positional relationships of multiple joints.

[0020] In this embodiment, as described later, the worker's handedness (right-handed, left-handed) is assumed as the worker's characteristic, and the product model (model i, model ii) is assumed as the work object's characteristic. In this embodiment, there are four learning videos 31 that cover all patterns of worker handedness and product models. That is, in this embodiment, there is a learning video 31 in which a right-handed worker works on a product of model i (hereinafter referred to as "right-handed & model i learning video"). There is also a learning video 31 in which a right-handed worker works on a product of model ii (hereinafter referred to as "right-handed & model ii learning video"). There is also a learning video 31 in which a left-handed worker works on a product of model i (hereinafter referred to as "left-handed & model i learning video"). There is also a learning video 31 in which a left-handed worker works on a product of model ii (hereinafter referred to as "left-handed & model ii learning video").

[0021] The task element identification model 33 is also the same as the learning video 31. That is, there are four task element identification models 33 that cover all patterns of the worker's handedness and the model of the product. Specifically, in this embodiment, there is a task element identification model 33 that is applied when a right-handed worker works on a product of model i (hereinafter referred to as the "right-handed & model i model"). There is also a task element identification model 33 that is applied when a right-handed worker works on a product of model ii (hereinafter referred to as the "right-handed & model ii model"). There is also a task element identification model 33 that is applied when a left-handed worker works on a product of model i (hereinafter referred to as the "left-handed & model i model"). There is also a task element identification model 33 that is applied when a left-handed worker works on a product of model ii (hereinafter referred to as the "left-handed & model ii model").

[0022] Moreover, the learning video true value 32 indicates the task element name corresponding to each frame of the learning video 31 . For example, assume that the work procedure corresponding to the learning video 31 includes work elements a, b, and c. In this case, the true value 32 of the learning video indicates, for example, that frames 1 to 10 are frames of work element a, frames 11 to 20 are frames of work element b, and frames 21 to 30 are frames of work element c.

[0023] The functional components of the task element identifying device 10 will be described in detail later, but each functional component will now be outlined.

[0024] The video acquisition unit 21 acquires a video via the communication interface 14 . Hereinafter, the image acquired by the image acquisition unit 21 via the communication interface 14 will be referred to as an input image 46. The input image 46 is an image of a worker performing work on a product, which is a work target.

[0025] The skeleton detection unit 22 acquires skeleton information, which is motion information of the worker, from the learning video 31 and the input video 46. The skeleton detection unit 22 corresponds to an acquisition unit. Moreover, the process performed by the skeleton detection unit 22 corresponds to an acquisition process. Moreover, the skeleton information acquired by the skeleton detection unit 22 from the input video 46 corresponds to worker motion information.

[0026] In addition, the product detection unit 23 acquires product images from the training video 31 and the input video 46 . The product detection unit 23 also corresponds to an acquisition unit. Moreover, the process performed by the product detection unit 23 also corresponds to an acquisition process. Moreover, the product image acquired by the product detection unit 23 from the input video 46 corresponds to a work object image.

[0027] The movement similarity calculation unit 24 calculates the similarity between the skeleton information acquired from the learning video 31 and the skeleton image acquired from the input video 46 . The movement similarity calculation unit 24 corresponds to a similarity calculation unit.

[0028] The product similarity calculation unit 25 calculates the similarity between the product image acquired from the learning video 31 and the product image acquired from the input video 46 . The product similarity calculation unit 25 also corresponds to a similarity calculation unit.

[0029] The mode selection unit 26 uses the movement similarity calculated by the movement similarity calculation unit 24 and the product similarity calculated by the product similarity calculation unit 25 to select a work element identification model 33 to be used to identify a work element from among the multiple work element identification models 33. The mode selection unit 26 corresponds to a model selection unit. The process performed by the mode selection unit 26 corresponds to a model selection process. The task element specification model 33 selected by the mode selection unit 26 corresponds to a utilization estimation model.

[0030] The task element identifying unit 27 uses the task element identifying model 33 selected by the mode selecting unit 26 to identify the task element being performed by the worker.

[0031] ***Explanation of Operation*** Next, an example of the operation of the task element identifying device 10 according to the first embodiment will be described.

[0032] The task element identification device 10 according to the first embodiment performs pre-operation processing and operation processing. In the pre-operation processing, the task element identification device 10 uses the learning video 31 to prepare for performing the operation processing. In the operation process, the task element identification device 10 analyzes the input video 46 and selects the task element identification model 33 to be used for identifying the task element. First, with reference to FIG. 2, a pre-operation process according to the first embodiment will be described.

[0033] (Step S11: Skeleton detection process) The skeleton detection unit 22 reads out all the learning videos 31 from the storage 13 . As shown in FIG. 3, the skeleton detection unit 22 detects skeleton information 42 of one worker 41 from each learning video 31. 3 shows skeletal information 42 for one cycle of a work procedure, which corresponds to one joint. However, in reality, the skeleton detection unit 22 detects skeletal information 42 for multiple cycles, which corresponds to multiple joints. One cycle of a work procedure includes all work elements.

[0034] The skeleton detection unit 22 determines one task identification target from one or more workers 41 shown in the learning video 31. Then, the skeleton detection unit 22 detects skeleton information 42 of the task identification target. Furthermore, the skeleton detection unit 22 assigns index information of the task element identification model 33 to the skeleton information 42 of the task identification target in association with the learning video 31 from which the skeleton information 42 was detected. Then, the skeleton detection unit 22 writes the skeleton information 42 together with the index information to the memory 12. In this embodiment, the skeleton detection unit 22 detects skeleton information 42 of the task specific target person from each of "right-handed & model i learning video", "right-handed & model ii learning video", "left-handed & model i learning video", and "left-handed & model ii learning video". Then, the skeleton detection unit 22 assigns index information indicating "right-handed & model i model" to the skeleton information 42 detected from "right-handed & model i learning video". The skeleton detection unit 22 similarly detects skeleton information 42 for "right-handed & model ii learning video", "left-handed & model i learning video", and "left-handed & model ii learning video", and assigns index information. The method of detecting the skeletal information 42 may be a method using deep learning, etc. The method of determining the person to be specified for the task may be a method of selecting the person whose Euclidean distance between the detected coordinates of the right shoulder and the left shoulder is the maximum.

[0035] (Step S12: Product detection process) The product detection unit 23 reads all the learning videos 31 from the storage 13 . 4, the product detection unit 23 detects a product 44 to be assembled from a representative frame 43 of each learning video 31, and cuts out a product image 45 showing the product 44 to be assembled. The product detection unit 23 assigns index information of the work element identification model 33 to the cut-out product image 45 in association with the learning video 31 from which the product image 45 was cut out. Then, the product detection unit 23 writes the product image 45 together with the index information into the memory 12.

[0036] As a method for selecting the representative frame 43, there is a method in which the product detection unit 23 selects the representative frame 43 from the learning video 31 at regular time intervals, and a method in which the true value 32 of the learning video is used. In the method using the true value 32 of the learning video, the product detection unit 23 reads the true value 32 of the learning video from the memory 12, and identifies the frame where each work element starts and the frame where each work element ends from the true value 32 of the learning video. Then, the product detection unit 23 selects a central frame or the like from between the frames where each work element starts and ends as a representative frame 43. There may be multiple frames where one work element starts and ends. That is, there may be cases where one work element is performed multiple times in the learning video 31. In this case, the product detection unit 23 may prepare a representative frame 43 each time. The product detection unit 23 may also select multiple representative frames 43 from between the frames where each work element starts and ends. Methods for cutting out the product image 45 include a method using deep learning, a method of cutting out the area around the coordinates of the wrist of the specific task target person detected in step S11, a method of cutting out a predetermined range, and the like.

[0037] The operation process according to the first embodiment will be described with reference to FIG.

[0038] (Step S21: Video acquisition process) The video acquisition unit 21 acquires the input video 46 via the communication interface 14. The video acquisition unit 21 writes the input video 46 into the memory 12. The input video 46 is a video of an arbitrary time that shows one or more workers 41 performing assembly work and a product 44 to be assembled.

[0039] (Step S22: End determination process) The video acquisition unit 21 determines whether or not the input video 46 has been acquired. If the input video 46 cannot be acquired, the operation process ends.

[0040] (Step S23: Skeleton detection process) The skeleton detection unit 22 reads the input video 46 from the memory 12 . As shown in Fig. 6, the skeleton detection unit 22 detects skeleton information 42 of one worker 41 from an input video 46. Fig. 6 shows skeleton information 42 for one joint. However, in reality, the skeleton detection unit 22 detects skeleton information 42 for a plurality of joints. The skeleton detection unit 22 determines one specific task target person from one or more workers 41 shown in the input video 46. Then, the skeleton detection unit 22 detects skeleton information 42 of the specific task target person. Furthermore, the skeleton detection unit 22 writes the skeleton information 42 of the specific task target person to the memory 12. A method for detecting the skeletal information 42 includes a method using deep learning, etc. A method for determining a specific task target person includes a method for selecting a person whose Euclidean distance between the detected coordinates of the right shoulder and the left shoulder is the largest, etc.

[0041] (Step S24: Product detection process) The product detection unit 23 reads the input video 46 from the memory 12 . 7, the product detection unit 23 detects a product 44 to be assembled from a representative frame 43 of an input video 46, and cuts out a product image 45 showing the product 44 to be assembled. The product detection unit 23 writes the cut-out product image 45 to the memory 12. The product detection unit 23 selects one frame, such as a central frame of the input video 46, as the representative frame 43. Methods for cutting out the product image 45 include a method using deep learning, a method of cutting out the area around the coordinates of the wrist of the specific task target person detected in step S23, a method of cutting out a predetermined range, and the like.

[0042] (Step S25: Motion similarity calculation process) The motion similarity calculation unit 24 reads from the memory 12 the skeletal information 42 of all the learning videos 31 detected in step S11 and the skeletal information 42 of the input video 46 detected in step S23. 8, the motion similarity calculation unit 24 calculates the similarity between each piece of skeletal information 42 of the learning video 31 detected in step S11 and the skeletal information 42 of the input video 46 detected in step S23 in a brute force manner. The motion similarity calculation unit 24 writes the calculated similarity in the memory 12.

[0043] 9, when calculating the similarity of the skeleton information 42, the movement similarity calculation unit 24 divides each piece of skeleton information 42 of the learning video 31 detected in step S11 into work elements. Specifically, the movement similarity calculation unit 24 reads the true value 32 of the learning video from the memory 12, and identifies the frame where each work element starts and the frame where each work element ends from the true value 32 of the learning video. Then, the movement similarity calculation unit 24 divides the skeleton information 42 of the learning video 31 into work elements based on the identified frames. Furthermore, the movement similarity calculation unit 24 calculates the similarity between the skeletal information 42 of the divided learning video 31 and the skeletal information 42 of the input video 46 detected in step S23 in a brute force manner. In the examples of Figures 8 and 9, for each of the two pieces of skeletal information 42 shown on the left side of Figure 8, the similarity between the skeletal information 42 of the input video 46 detected in step S23 and each of the work elements shown in Figure 9 is calculated in a brute-force manner. As a method for calculating the similarity of the skeleton information 42, there is a method in which the Euclidean distance between two pieces of skeleton information 42 is calculated for each joint, and the skeleton information 42 with a smaller total Euclidean distance is deemed to have a higher similarity.

[0044] (Step S26: Product similarity calculation process) The product similarity calculation unit 25 reads from the memory 12 all of the product images 45 of the learning video 31 cut out in step S12 and the product images 45 of the input video 46 cut out in step S24. As shown in FIG. 10, the product similarity calculation unit 25 calculates the similarity between each product image 45 of the learning video 31 cut out in step S12 and the product image 45 of the input video 46 cut out in step S24 in a brute-force manner. The method for calculating the similarity of the product images may be a well-known method such as Bag of Features. The product similarity calculation process 24 writes the calculated similarity into the memory 12 .

[0045] (Step S27: Mode selection process) The mode selection unit 26 reads from the memory 12 the skeleton information 42 of the learning video 31 detected in step S11, the product image 45 of the learning video 31 cut out in step S12, and the similarities calculated in steps S25 and S26, respectively. The mode selection unit 26 selects a work element identification model 33 based on the degree of similarity of the skeletal information 42 calculated in step S25, the degree of similarity of the product image 45 calculated in step S26, the index information assigned to the skeletal information 42 of the learning video 31 used in the similarity calculation, and the index information assigned to the product image 45 of the learning video 31. The mode selection unit 26 selects a mode corresponding to the selected work element identification model 33 as the operation mode (work element identification mode) of the work element identification unit 27.

[0046] Here, we consider a case where the mode selection unit 26 selects one of the four types of work element identification models 33, namely, the "right-handed & model I model", "right-handed & model II model", "left-handed & model I model", and "left-handed & model II model" mentioned above. The task element specifying device 10 separately has a correspondence table showing the correspondence between the index information of each task element specifying model 33 and the four types of task element specifying models, as exemplified in FIG. At this time, the mode selection unit 26 selects a task element identification model 33 based on a combination of "handedness", which is a characteristic of the task identification person shown in the input image 46, and "model", which is a characteristic of the product 44 to be assembled.

[0047] First, the mode selection unit 26 selects, on a work element basis, from the skeletal information 42 of the learning video 31 detected in step S11, the skeletal information 42 that has the greatest similarity to the skeletal information 42 of the input video 46 detected in step S23. In other words, the mode selection unit 26 selects, from the work elements of the multiple pieces of skeletal information 42 of the learning video 31, the work element that has the greatest similarity. Then, the mode selection unit 26 specifies the task element specifying model 33 indicated by the index information assigned to the selected task element as a first candidate model.

[0048] Next, the mode selection unit 26 selects, from among the product images 45 in the learning video 31 cut out in step S12, the product image 45 having the greatest similarity to the product image 45 in the input video 46 cut out in step S24. Then, the mode selection unit 26 specifies the operation element specifying model 33 indicated by the index information assigned to the selected product image 45 as a second candidate model.

[0049] The mode selection unit 26 compares a first worker characteristic (handedness) which is a characteristic of a worker to which the first candidate model corresponds with a second worker characteristic (handedness) which is a characteristic of a worker to which the second candidate model corresponds. The mode selection unit 26 also compares a first work object characteristic (model) which is a characteristic of a work object to which the first candidate model corresponds with a second work object characteristic (model) which is a characteristic of a work object to which the second candidate model corresponds. Then, when the first worker characteristic matches the second worker characteristic and the first work object characteristic does not match the second work object characteristic, the mode selection unit 26 selects one of the candidate models by prioritizing the calculation result of the similarity of the skeleton information 42. In other words, the mode selection unit 26 selects the candidate model having the greater similarity in the skeleton information 42 from the first candidate model and the second candidate model. Furthermore, when the first worker characteristic does not match the second worker characteristic and the first work object characteristic matches the second work object characteristic, the mode selection unit 26 selects one of the candidate models by prioritizing the calculation result of the similarity of the product image 45. In other words, the mode selection unit 26 selects the candidate model having the greater similarity with respect to the product image 45 from the first candidate model and the second candidate model. Furthermore, when the first worker characteristic does not match the second worker characteristic and further when the first work object characteristic does not match the second work object characteristic, the mode selection unit 26 normalizes the similarity of the skeleton information 42 and the similarity of the product image 45 to a score between 0 and 1, and selects one of the candidate models by prioritizing the calculation result with the larger total score. In other words, the mode selection unit 26 selects, from the first candidate model and the second candidate model, the candidate model with the larger total score obtained by normalizing the similarity of the skeleton information 42 and the similarity of the product image 45.

[0050] A specific example of the selection criteria for the above-mentioned task element identification model 33 is shown in FIG. If the first worker characteristic matches the second worker characteristic and the first work object characteristic matches the second work object characteristic, i.e., if the first candidate model matches the second candidate model, mode selection unit 26 selects the first candidate model (=second candidate model) as is. When the first candidate model and the second candidate model do not match, and only the first work object characteristics and the second work object characteristics do not match (the product models do not match), such as "right-handed & model i model" and "right-handed & model ii model", the mode selection unit 26 selects the candidate model with the greatest similarity in product image 45. In addition, when only the first operator characteristic and the second operator characteristic do not match, such as between a “right-handed & machine type i model” and a “left-handed & machine type i model”, the mode selection unit 26 selects a candidate model having a high degree of similarity in the skeletal information 42. Furthermore, when the first worker characteristic and the second worker characteristic do not match, such as "right-handed & machine type i model" and "left-handed & machine type ii model", and further when the first work object characteristic and the second work object characteristic do not match, the mode selection unit 26 normalizes the similarity for the skeleton information 42 and the similarity for the product image 45 to scores of 0 to 1. Then, the mode selection unit 26 selects the candidate model with the largest total value of the score for the skeleton information 42 and the score for the product image 45.

[0051] (Step S28: Work element identification process) In the work element identification mode selected in step S27, the work element identification unit 27 uses the selected work element identification model 33 to identify the work element being performed by the worker 41, and outputs the name of the work element.

[0052] ***Advantages of the First Embodiment*** As described above, according to this embodiment, it is possible to select an appropriate task element specifying model 33 according to a combination of the characteristics of the worker and the characteristics of the task object.

[0053] More specifically, the task element identification device 10 according to the present embodiment calculates the similarity of the skeleton information 42 between the input video 46 and the learning video 31, and selects a task element identification mode based on the calculated similarity. The skeleton information 42 is human motion information such as time series information of the coordinates of the human joints and time series information of the relative positional relationship of a plurality of joints. In addition, the similarity can be calculated using a method that uses the Euclidean distance of the skeleton information 42, etc. In the method that uses the Euclidean distance, even if the same task element part of two pieces of skeleton information 42 includes different movements, it can be determined that the similarity is high from the common movement part. Therefore, according to the present embodiment, an appropriate task element identification model 33 can be selected using the human motion information.

[0054] Furthermore, the task element identification device 10 according to this embodiment calculates the similarity of the product image 45 between the input video 46 and the learning video 31, and determines the task element identification mode based on the calculated similarity. The similarity of the product image 45 is high if the products 44 are the same model, and low if the products are different models. Therefore, according to this embodiment, it is possible to select an appropriate task element identification model 33 depending on the model of the product 44.

[0055] ***Other configurations*** <Variation 1> 2 to 4, the task element identification device 10 acquires the skeleton information 42 and the product image 45 from the learning video 31 and the input video 46 as pre-operation processing, and writes them into the memory 12. However, the task element identification device 10 may write the skeleton information 42 and the product image 45 that have been generated externally in advance into the memory 12. In addition, the entire pre-operation process (up to the assignment of index information to the skeleton information 42 and the assignment of index information to the product image 45) may be performed externally. In this case, the work element identification device 10 performs the operation process using the skeleton information 42 and index information, and the product image 45 and index information generated externally.

[0056] <Variation 2> In the first embodiment, the task element identifying device 10 is one device as shown in Fig. 1. However, the task element identifying device 10 may be composed of a plurality of devices. 13, the task element identifying device 10 may be composed of two devices, a detection device 110 and an identifying device 120. The detection device 110 acquires skeleton information 42 and a product image 45 from a learning video 31. The identifying device 120 identifies a task element from an input video 46. In this case, the learning video 31, the true value 32 of the learning video, and the work element identification model 33 may be stored in storage provided outside the detection device 110 and the identification device 120. In addition, the learning video 31, the true value 32 of the learning video, and the work element identification model 33 may be stored in storage of either the detection device 110 or the identification device 120. In FIG. 13, the hardware of the detection device 110 and the identification device 120 is omitted. Similar to the work element identifying device 10, the detection device 110 and the identification device 120 each include a processor, a memory, a storage device, and a communication interface as hardware.

[0057] <Modification 3> In the first embodiment, each functional component of the task element identification device 10 is realized by software. However, as a third modification, each functional component of the task element identification device 10 may be realized by hardware. The third modification will be described below with respect to the differences from the first embodiment. An example of the configuration of a task element identifying device 10 according to the third modification will be described with reference to FIG. When each functional component is realized by hardware, the task element identification device 10 includes an electronic circuit 15 instead of the processor 11, the memory 12, and the storage 13. The electronic circuit 15 is a dedicated circuit for realizing the functions of each functional component, the memory 12, and the storage 13. Possible electronic circuits 15 include a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, a logic IC, a GA (Gate Array), an ASIC (Application Specific Integrated Circuit), and an FPGA (Field-Programmable Gate Array). All of the functional components may be realized by one electronic circuit 15, or the functional components may be distributed among a plurality of electronic circuits 15.

[0058] <Variation 4> As a fourth modification, some of the functional components of the task element identifying device 10 may be realized by hardware, and other functional components may be realized by software.

[0059] <Variation 5> In the first embodiment, the mode selection unit 26 determines whether to prioritize the calculation result of the similarity between the skeletal information 42 or the product image 45 based on the relationship between the first and second candidate models narrowed down based on the similarity between the skeletal information 42 and the product image 45. However, the mode selection unit 26 may prioritize the calculation result of the similarity of the product image 45 if the maximum value of the similarity of the product image 45 calculated in step S26 is equal to or greater than a threshold value, and may prioritize the calculation result of the similarity of the skeletal information 42 if the maximum value is less than the threshold value.

[0060] Embodiment 2 The second embodiment differs from the first embodiment in that a search map for narrowing down the selection candidates of the work element identification model is generated in the pre-operation processing. Also, the second embodiment differs from the first embodiment in that a work element identification model is selected from the selection candidates narrowed down by the search map and / or the information of the previous cycle in the operation processing. In the second embodiment, these different points will be mainly described, and a description of the same points will be omitted.

[0061] Note that a plurality of work procedures are assumed in the second embodiment. In the second embodiment, the work elements included in each work procedure are specified, and the order in which the work elements are performed is also specified. In the second embodiment, it is assumed that one model of product 44 to be assembled is determined for one work procedure. For example, when work procedures A and B exist, model A of product 44 is to be assembled for work procedure A, and model B of product 44 is to be assembled for work procedure B. Furthermore, if workers 41 have different handedness, the work procedures and the work elements that make up the work procedures are treated as different procedures even if the model of product 44 to be assembled is the same. For example, as work procedures for assembling products A and B, there are work procedures A and B for right-handed workers, and work procedures A' and B' for left-handed workers. Furthermore, if work procedure A is made up of work elements a, b, ..., there are work elements a', b', ... that make up work procedure A'. The same applies to the work elements that make up work procedures B and B'. A learning video 31 exists for each work procedure, and a work element identification model 33 exists for each learning video 31. For example, work procedures A, B, A', B' each have work videos A, B, A', B', and work videos A, B, A', B' each have work element identification models A, B, A', B'. In the first embodiment, the dominant hand of the worker (right-handed, left-handed) is assumed as the characteristic of the worker, and the model of the product (model i, model ii) is assumed as the characteristic of the work object. The "right-handed & model i learning video", "right-handed & model ii learning video", "left-handed & model i learning video", and "left-handed & model ii learning video" assumed in the first embodiment each correspond to the work videos A, B, A', and B' in the second embodiment. In addition, the "right-handed & model i model", "right-handed & model ii model", "left-handed & model i model", and "left-handed & model ii model" correspond to the work element identification models A, B, A', and B' in the second embodiment. On the other hand, in this embodiment, it is assumed that the characteristic of the worker is that the worker is either right-handed or left-handed, and the characteristic of the work object is the model of the product (models A to F). Moreover, the second embodiment is premised on the premise that a plurality of learning videos 31 includes at least one common task element. Furthermore, in the second embodiment, it is assumed that the input video starts from the beginning of the work, not from the middle of the work.

[0062] ***Configuration Description*** The configuration of a task element identifying device 10 according to the second embodiment will be described with reference to FIG. 1 in that the work element identification device 10 includes a search map generation unit 28 and a model search unit 29. Also, the second embodiment differs from the first embodiment in that the work element identification model 33 outputs not only the name of the identified work element but also the reliability of the identification. Furthermore, in the second embodiment, the movement similarity calculation unit 24 prioritizes the task element specifying model 33 previously selected by the mode selection unit 26 in calculating the similarity with the skeletal information 42 of the input video 46. Similarly, the product similarity calculation unit 25 prioritizes the task element specifying model 33 previously selected by the mode selection unit 26 in calculating the similarity with the product image 45 of the input video 46. Furthermore, in the second embodiment, the movement similarity calculation unit 24 sets priorities among the work element specifying models 33 by analyzing the input video 46, and calculates the similarity between the input video 46 and the skeletal information 42, starting from the work element specifying model 33 with the highest priority. Similarly, the product similarity calculation unit 25 calculates the similarity between the input video 46 and the product image 45, starting from the work element specifying model 33 with the highest priority.

[0063] ***Explanation of Operation*** The operation of the task element identifying device 10 according to the second embodiment will be described with reference to FIGS. The operation of the task element identification device 10 according to the second embodiment includes pre-operation processing and operation processing.

[0064] The pre-operation process according to the second embodiment will be described with reference to FIG. The processing from step S31 to step S32 is the same as the processing from step S11 to step S12 in Fig. 2. Therefore, the description of these processing will be omitted.

[0065] (Step S33: Search map generation process) The search map generation unit 28 reads the true value 32 of the learning video from the storage 13, and reads from the memory 12 the skeleton information 42 detected in step S31 and the product image 45 cut out in step S32. The search map generating unit 28 generates a task element table 47 using the skeleton information 42 and the true value 32 of the learning video. Furthermore, the search map generating unit 28 generates a search map 48 from the work element table 47, the skipped points obtained from the work element table 47, and the product image 45. Then, the search map generating unit 28 writes the search map 48 into the memory 12. The work element table 47 is a table in which the work elements performed in each learning video 31 are arranged in chronological order. The work element table 47 includes only work elements for one cycle of the work procedure. The search map 48 is a map for narrowing down the work element identification models 33 to be selected when the work element identification device 10 selects a work element identification model 33 from the input video 46.

[0066] The contents of the work element table 47 and the method of generating it will now be specifically described. The search map generation unit 28 identifies the frame where each work element starts and the frame where each work element ends from the learning video true value 32. Then, the search map generation unit 28 divides the skeleton information 42 into work elements based on the identified frames. Furthermore, as shown in FIG. 17, the search map generating unit 28 calculates the similarity for each task element in a brute force manner for the skeleton information 42 from which the learning video 31 from which the detection originates is different. As a method for calculating the similarity of the skeleton information 42, there is a method in which the Euclidean distance between two pieces of skeleton information 42 is calculated for each joint, and the skeleton information 42 with a smaller total Euclidean distance is deemed to have a higher similarity. The search map generating unit 28 treats work elements whose skeletal information 42 has a similarity equal to or greater than a threshold as the same work element, and generates a work element table 47 as shown in FIG. 18, the work element numbers represent the order of the work elements. For example, learning video A corresponds to work procedure A. In work procedure A, the work elements are performed in the order of work elements a, c, d, f, and h. It should be noted that work element numbers 1 to 5 in FIG. 18 constitute one cycle of the work procedure.

[0067] Next, the contents of the search map 48 and the method of generating it will be specifically described. The search map generating unit 28 generates a search map 48 as shown in Fig. 19. Fig. 19 shows the search map 48 generated from the work element table 47 in Fig. 18. The search map 48 is a tree-structured map in which the order of performing two or more work processes for each learning video (work procedure) is indicated by node connections. In FIG. 19, the task element identifying models 33 generated from the learning videos A, B, . . . are represented as task element identifying models A, B, .

[0068] In FIG. 19, for example, when the work element number is 1, if the previously identified work element is a, work element identification models A, B, and C, which are leaf nodes of the subtree rooted at node a, become selection candidates. 19 indicates that it is not necessary to change the work element identifying model 33 from the point when the previous work element is being performed. In other words, (skip) indicates that it is possible to skip the similarity calculation for selecting the work element identifying model 33.

[0069] The search map generation unit 28 determines up to one representative image 49 from among the product images 45 for each of the original learning video 31 and work elements, and generates an updated search map 50 using the representative image 49.

[0070] An example of the representative image 49 is shown in FIG. Fig. 20 corresponds to the work element table 47 in Fig. 18, and is a list in which the product image 45 of the start frame of each work element is set as the representative image 49. Note that the start frame of work element number 1 is the frame before the product is set, so there is no representative image 49. 20 corresponding to each node of the search map 48, the search map generating unit 28 calculates the similarity of the representative images 49 of the node in a brute force manner. A well-known method such as Bag of Features may be used to calculate the similarity of the representative images 49. The search map generator 28 generates an updated search map 50 if the calculated similarity of the representative image 49 is less than a threshold value. For example, node a in FIG. 19 corresponds to A1, B1, and C1 in FIG. 20, but since there is no representative image, the search map generation unit 28 does not perform a similarity calculation. Node d of the subtree rooted at node a in FIG. 19 corresponds to A3 and B3 in FIG. 20. Therefore, the search map generation unit 28 calculates the similarity between representative image A3 and representative image B3. If the similarity is less than the threshold, the search map generation unit 28 splits node d into nodes d1 and d2. An updated search map 50, which is the search map 48 after all node splits have been performed, is shown in FIG. 21.

[0071] The operation process according to the second embodiment will be described with reference to FIGS. The processing from step S41 to step S44 is the same as the processing from step S21 to step S24 in Fig. 5. Moreover, the processing from step S52 is the same as the processing from step S28 in Fig. 5. Therefore, the description of these processing will be omitted.

[0072] (Step S45: Search execution determination process) If the model searching unit 29 has performed step S45 for the second or subsequent time, the process proceeds to step S46. If step S45 is performed for the first time, the model search unit 29 sets all the work element specifying models 33 as selection candidates, and writes the index information of the selection candidates in the memory 12. After that, the process proceeds to step S48.

[0073] (Step S46: Model search process) The model search unit 29 reads the updated search map 50 from the memory 12 . The model search unit 29 determines selection candidates for the work element identification model 33 using an updated search map 50 from the work element number of the input video 46 at that time and the past work elements identified by the work element identification unit 27. In addition, the model search unit 29 determines a search result by the updated search map 50 for the work element number of the input video 46 at that time from the work element number of the input video 46 at that time and the past work elements identified by the work element identification unit 27. The model search unit 29 writes the index information of the selection candidates of the work element identification model 33 and the search results based on the updated search map 50 into the memory 12 .

[0074] (Step S47: Skip determination process) The model search unit 29 reads the search results based on the updated search map 50 from the memory 12 . If the search result is (skip), the model search unit 29 skips the similarity calculation process and proceeds to step S50. On the other hand, if the search result is not (skip), the model search unit 29 proceeds to step S48.

[0075] (Step S48: Motion similarity calculation process) In the second embodiment, the process is different from that in step S25 in that the movement similarity calculation unit 24 reads out the index information of the selection candidates of the work element identification model 33 and the updated search map 50 from the memory 12. Furthermore, in the second embodiment, when skeletal information 42 of the learning video 31 detected in step S31 is read from the memory 12, not all of the skeletal information 42 is read, which is different from the process in step S25. Furthermore, in the second embodiment, the order of performing similarity calculations using the read skeletal information 42 is determined, and if the similarity is equal to or greater than a threshold, the similarity calculations are terminated, and the selection candidates for the work element identification model 33 are narrowed down, which is different from the processing in step S25.

[0076] Specifically, the movement similarity calculation unit 24 reads from the memory 12 the skeletal information 42 of the learning video 31 detected in step S31, the skeletal information 42 having the assigned index information that matches the selection candidate of the work element identification model 33. When the input video 46 shows the work procedure of the second or subsequent cycles, the movement similarity calculation unit 24 performs a first similarity calculation using the read skeletal information 42 whose assigned index information matches the work element identification model 33 selected in the previous cycle. For example, assume that the input video 46 shows the work procedure of the third cycle, and the work element specifying model A is selected for the work procedure of the second cycle. In this case, the movement similarity calculation unit 24 performs similarity calculation using the skeleton information 42 having index information of the work element specifying model A. Work procedures on a product assembly line are often performed not only in one cycle but in multiple cycles. For this reason, it is considered that the probability that the same work element specifying model 33 as in the previous cycle is used is higher than the probability that a work element specifying model 33 different from that in the previous cycle is used. Therefore, the movement similarity calculation unit 24 prioritizes the work element specifying model 33 selected in the previous cycle in calculating the similarity with the skeleton information 42 of the input video 46.

[0077] In addition, the movement similarity calculation unit 24 refers to the updated search map 50 and sets the priority order between the work element identification models 33, i.e., the order of similarity calculation, based on the number of leaf nodes of the work element identification models 33 remaining as selection candidates at that time. For example, assume a situation in which a work process b is extracted by analyzing the input video 46. That is, assume a situation in which one of nodes d and e branching from node b corresponding to work process b is selected in the update search map 50 of FIG. 24. In this case, work element identification models D, E, and F remain as selection candidates. At this time, the subtree rooted at node d has one leaf node D, and the subtree rooted at node e has two leaf nodes E and F. Therefore, the motion similarity calculation unit 24 first performs similarity calculation using skeleton information 42 having index information of the work element identification model E or the work element identification model F. Then, the motion similarity calculation unit 24 later performs similarity calculation using skeleton information 42 having index information of the work element identification model D. Since it is considered that the work element corresponding to the node that is the root of the subtree with a large number of leaf nodes is highly likely to be performed in the input video 46, the motion similarity calculation unit 24 sets a priority order between the work element identification models 33 in this manner.

[0078] When the similarity obtained by the calculation is equal to or greater than a threshold value, the movement similarity calculation unit 24 determines, as a selection candidate, the task element specifying model 33 indicated by the index information assigned to the skeleton information 42 used in the calculation. Then, the movement similarity calculation unit 24 writes the index information of the work element specifying model 33 determined as the selection candidate and the similarity into the memory 12, and ends the process of step S48.

[0079] If a previous cycle exists, the movement similarity calculation unit 24 performs a similarity calculation based on the selection result of the previous cycle. On the other hand, if a previous cycle does not exist, the movement similarity calculation unit 24 performs a similarity calculation based on the number of leaf nodes. Specifically, in the example of FIG. 24, in a situation where one of nodes d and e branching from node b is to be selected, if work element identification model D is selected in the previous cycle, the movement similarity calculation unit 24 performs a similarity calculation for the work element identification model D first. On the other hand, if a previous cycle does not exist (or if the selection result in the previous cycle is not any of the work element identification models D to F), the movement similarity calculation unit 24 performs a similarity calculation based on the number of leaf nodes. Furthermore, when the similarity calculation is performed based on the selection result of the previous cycle and the similarity is less than the threshold, the motion similarity calculation unit 24 also performs similarity calculation based on the number of leaf nodes.

[0080] (Step S49: Product similarity calculation process) In the second embodiment, the process is different from that in step S25 in that the product similarity calculation unit 25 reads out the index information of the selection candidates of the operation element identification model 33 and the updated search map 50 from the memory 12. Furthermore, in the second embodiment, when the product images 45 of the learning video 31 cut out in step S32 are read from the memory 12, some of the product images 45 are read, not all of them, which is different from the process in step S25. In addition, in the second embodiment, the order of performing similarity calculations on the read product images 45 is determined, and if the similarity is equal to or greater than a threshold, the similarity calculations are terminated, and the selection candidates for the work element identification model 33 are narrowed down, which is different from the processing in step S25.

[0081] Specifically, the product similarity calculation unit 25 reads from the memory 12 the product images 45 of the learning video 31 extracted in step S32, the product images 45 whose assigned index information matches the selection candidates of the work element identification model 33. However, when the product similarity calculation unit 25 refers to the portion where the node was divided using the representative image 49 in step S33, such as when selecting one of node d1, node d2, and node e branching off from node a in the updated search map 50, or when selecting one of node d, node e1, and node e2 branching off from node b, the product similarity calculation unit 25 reads only the representative image 49 from the memory 12. The product similarity calculation unit 25 determines the order of similarity calculation by a method in which the skeleton information 42 in step S48 is replaced with the product image 45. That is, the product similarity calculation unit 25 performs similarity calculation for the product image 45 in the same order as the movement similarity calculation unit 24 performs similarity calculation for the skeleton information 42. If the similarity obtained by the calculation is equal to or greater than a threshold value, the product similarity calculation unit 25 determines, as a selection candidate, the work element specifying model 33 indicated by the index information attached to the product image 45 used in the calculation. Then, the product similarity calculation unit 25 writes the index information of the work element specifying model 33 determined as a selection candidate and the similarity into the memory 12, and ends the processing of step S49.

[0082] (Step S50: Mode selection process) In the second embodiment, when the similarity calculation is skipped in step S47, the mode selection unit 26 selects the same task element identification mode as the previously selected mode, which is different from step S28.

[0083] (Step S51: Cycle switch determination process) In the second embodiment, the work element identifying unit 27 outputs the reliability of the work element identification before outputting the work element name. The work element identification unit 27 determines that the cycle has been switched when the reliability of the work element identification is less than the threshold value. Then, the work element identification unit 27 treats all the work element identification models 33 as selection candidates and writes the index information of the selection candidates to the memory 12. Then, the process returns to step S48. If the reliability of the task element identification is equal to or greater than the threshold, the process proceeds to step S52.

[0084] ***Effect Description*** As described above, the work element identifying device 10 according to the second embodiment selects the work element identifying model 33 from among the selection candidates narrowed down by the generated updated search map 50 and / or information on the previous cycle. Narrowing down the selection candidates for the work element identification model 33 reduces the amount of calculation for the similarity calculation for the skeleton information 42 and the similarity calculation for the product image 45, and enables faster processing. The faster processing makes it possible to select the work element identification model 33 and identify the work elements in real time.

[0085] ***Other configurations*** <Variation 1> In the second embodiment, the movement similarity calculation unit 24 determines the order of performing the similarity calculation, and ends the similarity calculation when the similarity is equal to or greater than a threshold value. However, the movement similarity calculation unit 24 may perform the similarity calculation using all of the read skeleton information 42 without determining the order of performing the similarity calculation, and may select the task element specifying model 33 indicated by the index information assigned to the skeleton information 42 with the highest similarity as a selection candidate.

[0086] <Variation 2> In the second embodiment, the work element identification device 10 generates a work element table 47 and a search map 48 as pre-operation processing. Alternatively, the entire pre-operation processing, including the generation of the work element table 47 and the search map 48, may be performed externally. In this case, the work element identification device 10 performs the operation processing using the work element table 47 and the search map 48 generated externally.

[0087] Although the first and second embodiments have been described above, these two embodiments may be combined and implemented. Alternatively, one of these two embodiments may be partially implemented. Alternatively, these two embodiments may be partially combined. Furthermore, the configurations and procedures described in these two embodiments may be modified as necessary.

[0088] ***Additional hardware configuration information*** Finally, a supplementary explanation of the hardware configuration of the task element identification device 10 will be given.

[0089] In the storage 13 of the task element identifying device 10, in addition to the programs that realize the functional components of the task element identifying device 10, an OS (Operating System) is also stored. At least a part of the OS is executed by the processor 11 . The processor 11 executes a program that realizes the functional components of the task element identifying device 10 while executing at least a part of the OS. The processor 11 executes the OS, which performs task management, memory management, file management, communication control, and the like. In addition, at least one of information, data, signal values, and variable values ​​indicating the results of processing of the functional components of the work element identification device 10 is stored in at least one of the memory 12, the storage 13, and a register and cache memory in the processor 11. In addition, the program for realizing the functional components of the task element identification device 10 may be stored in a portable recording medium such as a magnetic disk, a flexible disk, an optical disk, a compact disk, a Blu-ray (registered trademark) disk, a DVD, etc. Then, the portable recording medium storing the program for realizing the functional components of the task element identification device 10 may be distributed.

[0090] Furthermore, the "part" of the functional components of the work element identification device 10 may be read as a "circuit" or a "step" or a "procedure" or a "process" or a "circuitry". In this specification, the generic concept of the processor 11 and the electronic circuit 15 is called "processing circuitry." That is, the processor 11 and the electronic circuit 15 are each specific examples of "processing circuitry." [Explanation of symbols]

[0091] 10 work element identification device, 11 processor, 12 memory, 13 storage, 14 communication interface, 15 electronic circuit, 21 video acquisition unit, 22 skeleton detection unit, 23 product detection unit, 24 movement similarity calculation unit, 25 product similarity calculation unit, 26 mode selection unit, 27 work element identification unit, 28 search map generation unit, 29 model search unit, 31 training video, 32 true value of training video, 33 work element identification model, 41 worker, 42 skeleton information, 43 representative frame, 44 product, 45 product image, 46 input video, 47 work element table, 48 search map, 49 representative image, 50 updated search map, 110 detection device, 120 identification device.

Claims

1. an acquisition unit that acquires worker motion information, which is motion information of a worker when performing work on a work object, and a work object image, which is an image of the work object; a model selection unit that selects, as an estimation model to be used for estimating a work process being performed by a worker on a work object, from a plurality of estimation models that are a plurality of models for estimating a work process being performed among a plurality of work processes, each of which is associated with motion information of a worker and an image of a work object; a similarity calculation unit that calculates a similarity between the worker motion information and motion information associated with each of the plurality of estimation models, and calculates a similarity between the work object image and an image associated with each of the plurality of estimation models, The model selection unit is selecting an estimation model that is a candidate for the used estimation model from among the plurality of estimation models as a first candidate model using the similarity between the worker motion information and motion information associated with each of the plurality of estimation models, calculated by the similarity calculation unit; selecting, as a second candidate model, an estimation model that is a candidate for the utilized estimation model from among the plurality of estimation models, using the similarity between the work object image and an image associated with each of the plurality of estimation models, calculated by the similarity calculation unit; An information processing device that selects one of the first candidate model and the second candidate model as the utilization estimation model.

2. each of the plurality of estimation models corresponds to any one of a plurality of combinations of a characteristic of a worker and a characteristic of a work object; The model selection unit is comparing a first worker characteristic which is a characteristic of a worker to which the first candidate model corresponds with a second worker characteristic which is a characteristic of a worker to which the second candidate model corresponds, and comparing a first work object characteristic which is a characteristic of a work object to which the first candidate model corresponds with a second work object characteristic which is a characteristic of a work object to which the second candidate model corresponds; selecting, as the used estimation model, a candidate model having a large similarity in motion information from among the first candidate model and the second candidate model when the first worker characteristic and the second worker characteristic match and the first work object characteristic and the second work object characteristic do not match; selecting, as the used estimation model, a candidate model having a large degree of similarity in an image from among the first candidate model and the second candidate model when the first worker characteristic and the second worker characteristic do not match and the first work object characteristic and the second work object characteristic match; 2. The information processing device according to claim 1, wherein when the first worker characteristic does not match the second worker characteristic and the first work object characteristic does not match the second work object characteristic, the candidate model having the larger score obtained by normalizing the similarity for motion information and the similarity for images is selected as the used estimation model from among the first candidate model and the second candidate model.

3. The similarity calculation unit is The information processing apparatus according to claim 1 , wherein in calculating the degree of similarity with the worker motion information and the degree of similarity with the work object image, a priority is given to an estimation model previously selected by the model selection unit as the used estimation model.

4. The similarity calculation unit is The information processing device according to claim 1, wherein priorities are set among the plurality of estimation models by analyzing video of the worker working on the work object, and calculations of the similarity with the worker motion information and the similarity with the work object image are performed in order of priority from the estimation model with the highest priority.

5. Each of the plurality of estimation models corresponds to one of a plurality of work procedures, each of the plurality of work procedures includes two or more of the plurality of work processes, and an execution order of the two or more work processes is specified; The acquisition unit is Obtain a tree-structured search map in which the order of performing two or more work steps is indicated by node connections for each work procedure; The similarity calculation unit is The information processing device according to claim 4 , further comprising: setting priorities among the plurality of estimation models based on the number of leaf nodes branching off from a specific node in the search map corresponding to a specific work process extracted by analysis of the video.

6. An acquisition process in which the computer acquires worker motion information, which is motion information of a worker when working on a work object, and a work object image, which is an image of the work object; a model selection process in which the computer selects, as an estimation model to be used for estimating a work process being performed by the worker on the work object, from a plurality of estimation models, each of which is a plurality of models for estimating a work process being performed among a plurality of work processes, each of which is associated with motion information of a worker and an image of a work object; a similarity calculation process in which the computer calculates a similarity between the worker motion information and motion information associated with each of the plurality of estimation models, and calculates a similarity between the work object image and an image associated with each of the plurality of estimation models; In the model selection process, The computer, selecting an estimation model that is a candidate for the used estimation model from among the plurality of estimation models as a first candidate model using the similarity between the worker motion information and motion information associated with each of the plurality of estimation models calculated in the similarity calculation process; selecting, as a second candidate model, an estimation model that is a candidate for the utilized estimation model from among the plurality of estimation models, using the similarity between the work object image and an image associated with each of the plurality of estimation models calculated in the similarity calculation process; An information processing method comprising the steps of: selecting one of the first candidate model and the second candidate model as the utilization estimation model.

7. An acquisition process for acquiring worker motion information, which is motion information of a worker when working on a work object, and a work object image, which is an image of the work object; a model selection process for selecting, as an estimation model to be used for estimating a work process being performed by a worker on a work object, from a plurality of estimation models, each of which is associated with motion information of a worker and an image of a work object, as an estimation model to be used for estimating a work process being performed by the worker on the work object; an information processing program for causing a computer to execute a similarity calculation process for calculating a similarity between the worker motion information and motion information associated with each of the plurality of estimation models, and calculating a similarity between the work object image and an image associated with each of the plurality of estimation models, In the model selection process, The computer includes: selecting, as a first candidate model, an estimation model that is a candidate for the used estimation model from among the plurality of estimation models, using the similarity between the worker motion information and motion information associated with each of the plurality of estimation models, calculated by the similarity calculation process; selecting, as a second candidate model, an estimation model that is a candidate for the used estimation model from among the plurality of estimation models, using the similarity between the work object image and an image associated with each of the plurality of estimation models, calculated by the similarity calculation process; An information processing program that causes one of the first candidate model and the second candidate model to be selected as the utilization estimation model.

Images