Motion recognition device, motion recognition method, and motion recognition program

Abstract

To provide a motion recognition apparatus configured to recognize each of common motions to be performed in different work processes, a motion recognition method, and a motion recognition program.SOLUTION: In a motion recognition system 100, a motion recognition apparatus 10 includes: an acquisition unit 11 which acquires motion information including work elapsed time, common motions included in different work processes, which are to be recognized, and skeleton data of a worker; a conversion unit 12 which calculates a distance from each of coordinates corresponding to body sections included in the skeleton data to an origin coordinate corresponding to an origin section defined as an origin in the body sections, on the basis of the above coordinates, to convert the coordinates included in the skeleton data into the calculated distance; and a learning unit 13 which learns a model that outputs information representing a motion of the worker corresponding to one of the common motions, on the basis of the motion information obtained by converting the coordinates into the distance by means of the conversion unit, as a model for motion recognition.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to an action recognition device, an action recognition method, and an action recognition program. [Background technology]

[0002] Conventionally, cameras, sensors, etc. are installed on production lines in factories, etc., and the movements of workers are recognized and evaluated based on the information on the movements of the workers obtained by these cameras, sensors, etc. For example, in the following Patent Document 1, the presence or absence of non-standard work that deviates from standard work is determined based on information obtained by motion capture, etc.

[0003] In Patent Document 1, an action to be performed in a certain work process is assumed, and it is determined whether or not the action is present and whether or not there is an abnormality in the action. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2018 / 131630 Summary of the Invention [Problem to be solved by the invention]

[0005] Incidentally, the actions performed in the work processes are often different for each work process, even if they belong to the same action type. For example, the process of tightening a screw and the process of fitting a case include a common action of moving the hand to the work object and grabbing (holding) it, but the gripping movements are different. To be more specific, in the process of tightening a screw, for example, one hand grasps a part placed in a parts box, and the other hand grasps an electric screwdriver suspended by a cable from above and in front of the worker. In contrast, in the process of fitting a case, for example, both hands grasp a case placed in a parts box.

[0006] In this way, when targeting common actions that differ for each work process, it is possible to recognize common actions performed in one of the work processes in Patent Document 1. However, in order to recognize common actions performed in other work processes, it is necessary to carry out a separate design so that common actions performed in other work processes can be recognized.

[0007] Therefore, the present invention provides an action recognition device, an action recognition method, and an action recognition program that are capable of recognizing common actions performed in different work processes. [Means for solving the problem]

[0008] An action recognition device according to one aspect of the present disclosure includes an acquisition unit that acquires action information including elapsed time of work, common actions common to different work processes to be recognized, and skeletal data of a worker, a conversion unit that calculates, based on coordinates corresponding to each of a plurality of body parts included in the skeletal data, a distance between each coordinate and an starting coordinate corresponding to an starting point part of the body part that is defined as an starting point, and converts each coordinate included in the skeletal data to the calculated distance, and a learning unit that trains a model for action recognition that outputs information indicating a worker's action corresponding to one of the common actions, based on the action information after the coordinates have been converted into distances by the conversion unit.

[0009] According to this aspect, the distance between the coordinates corresponding to each of the multiple body parts included in the skeleton data and the starting coordinates corresponding to the starting part can be calculated, and each coordinate of the skeleton data can be replaced with the calculated distance. Then, based on the motion information after each coordinate of the skeleton data is replaced with distance data, a motion recognition model can be trained that outputs information indicating a motion of the worker that corresponds to any of the common motions.

[0010] A motion recognition device according to another aspect of the present disclosure includes an acquisition unit that acquires time series information regarding a worker's motion, a conversion unit that calculates, based on coordinates corresponding to each of a plurality of body parts included in skeletal data extracted from the time series information, a distance between each coordinate and an origin coordinate corresponding to an origin part of the body part that is defined as an origin and converts each coordinate included in the skeletal data into the calculated distance, and a motion recognition unit that inputs the distance after conversion by the conversion unit into a trained model and recognizes the worker's motion based on information output from the trained model that indicates a worker's motion that corresponds to one of the common motions commonly included in different work processes to be recognized.

[0011] According to this aspect, the distance between the coordinates corresponding to each of the body parts included in the skeleton data extracted from the time-series information of the worker to be recognized and the starting coordinates corresponding to the starting part is calculated, and each coordinate of the skeleton data can be replaced with the calculated distance. Then, by inputting the replaced distance data into the trained model, the worker's movements that correspond to any of the common movements can be recognized.

[0012] In each of the above aspects, the starting site may be either the neck or the lower back.

[0013] This makes it possible to determine the starting point location near the center of the movement of the worker to be recognized.

[0014] A motion recognition method according to another aspect of the present disclosure is a motion recognition method executed by a processor, and includes acquiring motion information including an elapsed time of work, common motions that are common to different work processes to be recognized, and skeletal data of a worker, calculating a distance between each coordinate and a starting coordinate corresponding to a starting point among the body parts defined as a starting point based on coordinates corresponding to each of a plurality of body parts included in the skeletal data, and converting each coordinate included in the skeletal data to the calculated distance, and training a model that outputs information indicating a motion of the worker that corresponds to one of the common motions based on the motion information after the coordinates have been converted into distance, as a model for motion recognition.

[0015] According to this aspect, the distance between the coordinates corresponding to each of the multiple body parts included in the skeleton data and the starting coordinates corresponding to the starting part can be calculated, and each coordinate of the skeleton data can be replaced with the calculated distance. Then, based on the motion information after each coordinate of the skeleton data is replaced with distance data, a motion recognition model can be trained that outputs information indicating a motion of the worker that corresponds to any of the common motions.

[0016] A motion recognition method according to another aspect of the present disclosure is a motion recognition method executed by a processor, and includes acquiring time series information regarding a worker's motion, calculating a distance between each coordinate corresponding to a plurality of body parts included in skeletal data extracted from the time series information and an origin coordinate corresponding to an origin part of the body part that is defined as an origin, based on the coordinates corresponding to each of the plurality of body parts included in skeletal data extracted from the time series information, and converting each coordinate included in the skeletal data to the calculated distance, inputting the converted distance into a trained model, and recognizing the worker's motion based on information output from the trained model indicating one of the worker's motions that corresponds to one of the common motions commonly included in the different work processes to be recognized.

[0017] According to this aspect, the distance between the coordinates corresponding to each of the body parts included in the skeleton data extracted from the time-series information of the worker to be recognized and the starting coordinates corresponding to the starting part is calculated, and each coordinate of the skeleton data can be replaced with the calculated distance. Then, by inputting the replaced distance data into the trained model, the worker's movements that correspond to any of the common movements can be recognized.

[0018] An action recognition program according to another aspect of the present disclosure causes a computer to function as an acquisition unit that acquires action information including elapsed time of work, common actions common to different work processes to be recognized, and skeletal data of a worker, a conversion unit that calculates, based on coordinates corresponding to each of a plurality of body parts included in the skeletal data, a distance between each coordinate and a starting coordinate corresponding to a starting point part of the body part that is set as a starting point, and converts each coordinate included in the skeletal data into the calculated distance, and a learning unit that learns a model for action recognition that outputs information indicating a worker's action corresponding to one of the common actions, based on the action information after the coordinates have been converted into distances by the conversion unit.

[0019] According to this aspect, the distance between the coordinates corresponding to each of the multiple body parts included in the skeleton data and the starting coordinates corresponding to the starting part can be calculated, and each coordinate of the skeleton data can be replaced with the calculated distance. Then, based on the motion information after each coordinate of the skeleton data is replaced with distance data, a motion recognition model can be trained that outputs information indicating a motion of the worker that corresponds to any of the common motions.

[0020] A motion recognition program according to another aspect of the present disclosure causes a computer to function as an acquisition unit that acquires time series information regarding a worker's motion, a conversion unit that calculates, based on coordinates corresponding to each of a plurality of body parts included in skeletal data extracted from the time series information, a distance between each coordinate and an origin coordinate corresponding to an origin part of the body part that is set as an origin, and converts each coordinate included in the skeletal data into the calculated distance, and a motion recognition unit that inputs the distance after conversion by the conversion unit into a trained model and recognizes the worker's motion based on information output from the trained model that indicates a worker's motion that corresponds to one of the common motions commonly included in different work processes to be recognized.

[0021] According to this aspect, the distance between the coordinates corresponding to each of the body parts included in the skeleton data extracted from the time-series information of the worker to be recognized and the starting coordinates corresponding to the starting part is calculated, and each coordinate of the skeleton data can be replaced with the calculated distance. Then, by inputting the replaced distance data into the trained model, the worker's movements that correspond to any of the common movements can be recognized. Effect of the Invention

[0022] According to the present invention, it is possible to provide an action recognition device, an action recognition method, and an action recognition program that are capable of recognizing common actions performed in different work processes. [Brief description of the drawings]

[0023] [Figure 1] 1 is a diagram illustrating an example of an overview of an action recognition system according to an embodiment of the present invention. [Diagram 2] 11A to 11C are schematic diagrams illustrating an example of a gripping operation in a process of tightening a screw. [Diagram 3] 11A to 11C are schematic diagrams illustrating an example of a gripping operation in a case fitting process. [Figure 4] FIG. 2 is a diagram illustrating a functional configuration of an action recognition system and an action recognition device. [Diagram 5]FIG. 2 is a diagram illustrating an example of action information stored in the action recognition device. [Figure 6] 13 is a diagram showing a schematic diagram of a manner in which distances between coordinates corresponding to the neck in the skeletal data and each of the other coordinates are calculated. FIG. [Figure 7] FIG. 2 is a diagram illustrating an example of a hardware configuration of the action recognition device. [Figure 8] 10 is a flowchart illustrating an example of an operation of the action recognition device in a learning mode. [Figure 9] 9 is a flowchart for explaining the procedure of the conversion process shown in FIG. 8. [Figure 10] 10 is a flowchart illustrating an example of an operation of the action recognition device in a action recognition mode. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0024] Hereinafter, an embodiment according to one aspect of the present invention (hereinafter, referred to as "the present embodiment") will be described with reference to the drawings. In each drawing, the same reference numerals denote the same or similar configurations.

[0025] §1 Examples of application First, an example of a scene to which the present invention is applied will be described with reference to FIG. 1. In the action recognition system 100 according to this embodiment, the action of a worker A performed in a certain work area R is captured by image sensors 20a, 20b, and 20c, and the action recognition device 10 that acquires the captured video recognizes a common action of the worker A using a trained model. Here, a common action is an action that is commonly included in different work processes (action that belongs to the same action type). Such a common action is, for example, an action that is commonly performed in different work processes, such as grasping, carrying, and adjustment.

[0026] Grasping is defined as the action of moving the hand to the work object and grabbing it, for example, the action of going to grab a part or tool. Transporting is defined as the action of moving the work object to the target location, for example, the action of carrying a part or tool to the product being assembled. Adjusting is defined as the action of moving the work to the target state, for example, the action of assembling parts.

[0027] In this embodiment, a case where the common action is grasping will be described as an example, but the present invention can be similarly applied to other common actions such as carrying and adjusting.

[0028] The trained model is a model for motion recognition that is trained to receive motion information including distance data generated based on the skeletal data of a worker as input and output information indicating the motion of the worker. The distance data input during training is data indicating the distance between the coordinates corresponding to each of a plurality of body parts included in the skeletal data of the worker and the starting coordinates corresponding to a starting part of the body parts that is set as a starting point.

[0029] For example, the neck, the waist, etc. can be set as the starting point part. Here, when the upper body of the worker is the recognition target, it is preferable to set the neck as the starting point part, and when the whole body of the worker is the recognition target, it is preferable to set the waist as the starting point part. In this way, it is preferable to set the starting point part near the center of the movement of the worker to be recognized.

[0030] The distance data will be further described with reference to Fig. 2 and Fig. 3. Fig. 2 is a diagram illustrating the movement of worker Aa stretching out his right hand to grab an electric screwdriver D suspended by a cable from above and in front of worker Aa during the process of tightening a screw. Fig. 3 is a diagram illustrating the movement of worker Ab stretching out both hands to grab a case C placed in a parts box during the process of fitting a case.

[0031] In both Fig. 2 and Fig. 3, workers Aa and Ab are performing a gripping motion. However, for example, the right hand of worker Aa and the right hand of worker Ab perform completely different gripping motions. In order to recognize such gripping motions, it has been necessary to generate a learning model or the like for each work process and recognize the gripping motions performed in each work process.

[0032] In contrast, the present invention replaces each coordinate included in the skeletal data of the worker with a distance starting from a certain body part, and learns the characteristics of the distance in association with the grasping action of the worker, thereby making it possible to recognize each grasping action performed in different work processes. A specific description will be given below.

[0033] 2 and 3, when the neck of each worker Aa and Ab is taken as the starting point and the right hand movements of each worker Aa and Ab are analyzed, it can be characterized that the right hand moves in a direction away from the neck in both gripping movements. In addition, in the process of tightening a screw shown in FIG. 2, the movement of worker Aa gripping an electric screwdriver D with his right hand is desirable for all workers to move equally efficiently in the screw tightening process. Similarly, in the process of fitting a case shown in FIG. 3, the movement of worker Ab gripping a case C with his right hand is desirable for all workers to move equally efficiently in the case fitting process.

[0034] Therefore, by learning the distance data of the right hand corresponding to the grasping action performed in each work process in association with the grasping action in each work process, it becomes possible to accurately recognize the movement of the worker grasping the work object with his or her right hand in each work process.

[0035] In this way, the action recognition device 10 according to the present embodiment can recognize common actions such as gripping actions performed in different work processes.

[0036] §2 Configuration example [Function configuration] Next, referring to FIG. 4, an example of the functional configuration of the action recognition system 100 and the action recognition device 10 according to the present embodiment will be described. The action recognition system 100 includes three image sensors 20a, 20b, and 20c, and the action recognition device 10. In the following, the three image sensors 20a, 20b, and 20c will be referred to as the image sensor 20 when there is no need to distinguish them from one another. The action recognition device 10 has, as functional components, an acquisition unit 11, a conversion unit 12, a learning unit 13, an action recognition unit 14, and a storage unit 19, for example. The storage unit 19 stores, for example, a video 19a, action information 19b, and a learned model 19c.

[0037] Here, in this embodiment, a case will be described in which the action recognition device 10 has a function of learning a model for action recognition (learning mode) and a function of recognizing the actions of a worker (action recognition mode), but each function may be distributed and provided in an independent, separate device.

[0038] The functional components of the action recognition system 100 and the action recognition device 10 will be described in detail below.

[0039] <Image sensor> The image sensor 20 is, for example, a general-purpose camera, and captures a video including a scene in which the worker A is performing an action in the working area R. The image sensor 20 has, for example, a detection unit as a functional configuration. The detection unit detects the action of the worker A, and outputs a video showing the action as time-series information.

[0040] Here, the time-series information is not limited to a video, and may be, for example, information on coordinates indicating the motion of the worker A measured by a motion capture device that can be provided instead of the image sensor 20.

[0041] Each of the image sensors 20a, 20b, 20c is disposed so as to capture the entire working area R and the entire body of the worker A. In this case, for example, each of the image sensors 20a, 20b, 20c may be disposed so as to capture the entire working area R and the entire body of the worker A, or each of the image sensors 20a, 20b, 20c may be disposed so as to capture a part of the working area R and the worker A and to cover the entire working area R and the entire body of the worker A by combining the respective videos. Also, each of the image sensors 20a, 20b, 20c may capture the working area R and the worker A at different magnifications. It is not necessary to provide three image sensors 20, and it is sufficient to provide at least one or more image sensors.

[0042] <Acquisition part> The acquisition unit 11 acquires time-series information (video in this embodiment) related to the actions performed by the worker A from the image sensor 20. The time-series information acquired by the acquisition unit 11 is transmitted to the storage unit 19 and stored as video 19a. The acquisition unit 11 also acquires the video 19a stored in the storage unit 19.

[0043] The acquisition unit 11 extracts skeletal data indicating the movement of the worker's skeleton from the images of the video 19a. The skeletal data can be represented by coordinates (x, y) corresponding to each of a plurality of body parts. In this embodiment, the case where the coordinates are two-dimensional coordinates (x, y) will be described, but the same can be applied to the case where the coordinates are three-dimensional coordinates (x, y, z). In addition to the coordinate values, information indicating the accuracy of the coordinate values ​​may be added.

[0044] The skeleton data is transmitted to the storage unit 19 and stored as a part of the motion information 19b. The acquisition unit 11 also acquires the motion information 19b stored in the storage unit 19.

[0045] The motion information 19b will be described with reference to Fig. 5. The motion information 19b is configured to include, as data items, for example, an elapsed time item, a right hand motion item, a left hand motion item, and a skeleton data item. The elapsed time item stores the elapsed time from the reference time when the first process among all the processes to be worked on is started. The interval of the elapsed time can be set arbitrarily, and may be set, for example, in units of frames of a video, or may be set at a predetermined time interval such as every second.

[0046] The right hand action item stores information indicating which of the common actions to be recognized corresponds to a right hand action. The left hand action item stores information indicating which of the common actions to be recognized corresponds to a left hand action. The information stored in the right hand action item and the left hand action item can be registered, for example, by referring to a video, checking each of the right hand and left hand actions at each elapsed time, and inputting the confirmed action content.

[0047] The skeleton data item of the motion information 19b stores skeleton data extracted from a video corresponding to an elapsed time. Note that the skeleton data stored in this skeleton data item is replaced with distance data calculated by a conversion process described later.

[0048] The motion information in the first row shown in FIG. 5 is based on the video at one second after the start of the first process. This motion information stores information indicating that the motion of the worker's right hand does not fall under any of the common motions and that the motion of the worker's left hand falls under the common motion of grasping, as well as skeletal data extracted from the video at that time. The motion information in the last row shown in FIG. 5 is based on the video at two minutes and fifty-three seconds after the start of the first process. This motion information stores information indicating that the motions of the worker's right and left hands fall under the common motion of adjusting, as well as skeletal data extracted from the video at that time.

[0049] <Conversion section> 4 executes a conversion process for converting each coordinate included in the skeleton data into a distance from a starting coordinate. The conversion process is a process for calculating a distance between each coordinate and a starting coordinate corresponding to a starting point based on coordinates corresponding to each of a plurality of body parts included in the skeleton data, and replacing each coordinate included in the skeleton data with the calculated distance.

[0050] Here, the conversion unit 12 has a common function in the learning mode and the motion recognition mode, but the input skeletal data is different. In the learning mode, the conversion unit 12 calculates the distance based on the skeletal data included in the motion information generated using the video of the target of learning. On the other hand, in the motion recognition mode, the conversion unit 12 calculates the distance based on the skeletal data extracted from the video of the worker to be recognized.

[0051] FIG. 6 is a schematic diagram showing an image in which the neck is set as the origin part a and the distances between the origin coordinates corresponding to the neck and the coordinates corresponding to each body part are calculated.

[0052] By converting each coordinate of the skeleton data shown in Fig. 6 into a distance from the origin coordinate, the following recognition becomes possible. For example, if the distance corresponding to the right hand increases over time, it is possible to recognize how much the right hand is moving away from the neck (stretched). On the other hand, if the distance corresponding to the right hand decreases over time, it is possible to recognize how much the right hand is moving toward the neck (shortened).

[0053] Distance d from the origin coordinate i can be calculated, for example, as in the following formula (1). In formula (1), the origin coordinates corresponding to the origin part a are (x a ,y a ), and the coordinates corresponding to any body part i are (x i ,y i ) (i is variable). The coordinates corresponding to any body part i may include the origin coordinates.

[0054] di =(|x i- x a | p +|y i- y a | p ) 1 / p … Formula (1) The p in formula (1) can be arbitrarily designated as, for example, 1 or 2, but is not limited thereto.

[0055] The conversion unit 12 converts the distance d calculated by the formula (1) i Based on this, the coordinates (x i ,y i ), for example (d i ,d i )

[0056] Here, the conversion unit 12 may further include an adjustment unit that performs an adjustment process for adjusting the skeleton data extracted from the video. The adjustment unit included in the conversion unit 12 performs the adjustment process before performing the above-mentioned conversion process.

[0057] The adjustment process includes, for example, a time series complementation process of the skeleton data, a normalization process of the height (physique), a time series smoothing process of the skeleton data, a shift process of the skeleton data, and a noise addition process. Each process included in the adjustment process will be described below.

[0058] The time-series complementation process of the skeleton data is a process of complementing missing data occurring in the skeleton data based on other skeleton data located before and after in time. Missing data may occur, for example, in a part that cannot be estimated because it is hidden by the posture of the worker.

[0059] The normalization process of height (physique) is a process of normalizing skeletal data based on body shape in order to absorb physique differences that occur between men and women, etc. As a normalization process, for example, normalized data can be generated by dividing the skeletal data by the length of the torso (for example, the length from the nose to the waist).

[0060] The time-series smoothing process of the skeleton data is a process for removing noise due to changes in the time axis direction of the skeleton data. As a smoothing process, for example, Gaussian filter processing can be performed on the skeleton data to generate smoothed data.

[0061] The shift process of the skeleton data is a process to reduce the variation of the movement and make the characteristics of the movement easier to recognize by aligning the origin of the skeleton data and unifying the starting point. For example, the entire skeleton data is translated so that the neck joint is located at the origin, and the shifted skeleton data is generated.

[0062] The noise adding process is a process for virtually increasing the skeleton data by adding noise to the skeleton data. The noise to be added to the skeleton data can be generated, for example, by randomly generating values ​​within a range that can be used as the skeleton data.

[0063] Here, it is preferable that the adjustment unit executes each step of the adjustment process described above in the learning mode, and executes each step of the adjustment process described above excluding the noise adding process in the action recognition mode.

[0064] <Study Section> The learning unit 13 shown in Fig. 4 is a function in the learning mode. The learning unit 13 generates (learns) a model for action recognition that outputs information indicating an action of a worker that corresponds to any of the common actions, based on the action information 19b after each coordinate of the skeleton data is converted into a distance by the conversion unit 12.

[0065] The model trained by the training unit 13 is transmitted to the memory unit 19 and stored as a trained model 19c.

[0066] <Motion recognition section> The motion recognition unit 14 is a function in the motion recognition mode. The motion recognition unit 14 converts the coordinates of the skeleton data corresponding to the worker to be recognized by the conversion unit 12 into distance data. Operational information including 19bis input to the trained model 19c, and the worker's action is recognized based on information indicating the worker's action corresponding to one of the common actions output from the trained model 19c. When recognizing the worker's action, for example, a known action recognition method such as ST-GCN (Spatial Temporal Graph Convolutional Networks) can be used.

[0067] [Hardware configuration] Next, an example of the hardware configuration of the action recognition device 10 according to this embodiment will be described with reference to FIG. 7. The action recognition device 10 has a CPU (Central Processing Unit) 10a corresponding to a calculation device, a RAM (Random Access Memory) 10b corresponding to a storage unit 19, a ROM (Read only Memory) 10c corresponding to the storage unit 19, a communication device 10d, an input device 10e, and a display device 10f. These components are connected to each other via a bus so that they can transmit and receive data to each other. Note that, although a case where the action recognition device 10 is configured by one computer will be described in this embodiment, the action recognition device 10 may be realized using multiple computers.

[0068] The CPU 10a executes programs stored in the RAM 10b or the ROM 10c and functions as a control unit that performs calculations and processing of data. The CPU 10a receives various input data from the input device 10e or the communication device 10d, and displays the results of calculations performed on the input data on the display device 10f or stores the results in the RAM 10b or the ROM 10c.

[0069] The RAM 10b is made up of, for example, a semiconductor memory device, and stores rewritable data.The ROM 10c is made up of, for example, a semiconductor memory device, and stores readable but non-rewritable data.

[0070] The communication device 10d is an interface that connects the action recognition device 10 to an external device. The communication device 10d is connected to the image sensor 20 via a communication network such as a LAN (Local Area Network) or the Internet, and receives video from the image sensor 20.

[0071] The input device 10e is an interface that accepts data input from a user, and can include, for example, a keyboard, a mouse, and a touch panel.

[0072] The display device 10f is an interface that visually displays the results of calculations performed by the CPU 10a, and can be configured, for example, by an LCD (Liquid Crystal Display).

[0073] The program may be provided by being stored in a computer-readable storage medium such as RAM 10b or ROM 10c, or may be provided via a communication network connected by communication device 10d. In action recognition device 10, CPU 10a executes the program to perform the operations of acquisition unit 11, conversion unit 12, learning unit 13, and action recognition unit 14 shown in FIG. 4. Note that these physical configurations are merely examples and do not necessarily have to be independent configurations. For example, action recognition device 10 may include an LSI (Large-Scale Integration) in which CPU 10a is integrated with RAM 10b and ROM 10c.

[0074] §3 Example of operation 8 is a flowchart showing an example of an operation in a learning mode in the action recognition device 10 according to the present embodiment. This operation is an operation for learning an action recognition model using action information 19b obtained by converting each coordinate included in skeletal data extracted from a learning video into a distance from a starting point coordinate.

[0075] First, the acquisition unit 11 acquires the motion information 19b generated based on the learning video 19a from the storage unit 19 (step S101). This motion information 19b is generated based on a video of a worker performing a work process including a common motion. In addition, the skeleton data item of the motion information 19b stores skeleton data of the worker corresponding to the video.

[0076] Next, the conversion unit 12 executes a conversion process to convert each coordinate included in the skeleton data into a distance from the origin coordinate (step S102). The procedure of this conversion process will be described later.

[0077] Next, the learning unit 13 trains a model for action recognition that outputs information indicating an action of the worker that corresponds to any of the common actions, based on the action information 19b including the distance data converted in the above step S102 (step S103). Then, this operation is terminated.

[0078] The procedure of the conversion process executed in step S102 will be described with reference to Fig. 9. In this conversion process, it is assumed that the neck is set in advance as the starting point part.

[0079] First, the acquiring unit 11 extracts skeletal data of one body for each frame from an image of the moving image 19a having, for example, F frames (step S201). That is, the acquiring unit 11 extracts skeletal data of F bodies.

[0080] Next, the conversion unit 12 calculates the distance between the coordinates corresponding to each of the multiple body parts included in the skeletal data of one body and the starting point coordinates corresponding to the starting point part, using the above formula (1) (step S202).

[0081] Next, the conversion unit 12 converts each coordinate included in the skeleton data into the distance calculated in step S202 (step S203). As a result, the distance data is stored in the skeleton data item of the motion information 19b.

[0082] Next, the conversion unit 12 judges whether the coordinates of all the skeletal data of the F body have been converted into distances (step S204). If the judgment is NO, the process proceeds to step S202, whereas if the judgment is YES, the conversion process ends.

[0083] 10 is a flowchart showing an example of the operation in the action recognition mode of the action recognition device 10 according to the present embodiment. This operation is premised on the premise that a video 19a capturing the action of a worker to be recognized has already been stored in the storage unit 19.

[0084] First, the acquisition unit 11 acquires a video 19a for motion recognition from the storage unit 19 (step S301). This video 19a is a video of a worker performing a task according to a work process including a common motion. In addition, the skeleton data item of the motion information 19b stores skeleton data of the worker corresponding to the video.

[0085] Next, the conversion unit 12 executes a conversion process to convert each coordinate included in the skeleton data into a distance from the origin coordinate (step S302). The procedure of this conversion process is the same as the procedure in FIG. 9 described above, and therefore the description thereof will be omitted.

[0086] Next, the action recognition unit 14 converts the distance data converted in step S302 into Operational information including 19b is input to the trained model 19c, and the operator's action is recognized based on information indicating the operator's action that corresponds to any of the common actions output from the trained model 19c (step S303). Then, this operation is terminated.

[0087] As described above, the action recognition device 10 according to the present embodiment can calculate the distance between the coordinates corresponding to each of the multiple body parts included in the skeletal data extracted from the video of the worker to be learned and the starting coordinates corresponding to the starting part, and replace each coordinate of the skeletal data with the calculated distance. Then, based on the action information after replacing each coordinate of the skeletal data with distance data, it is possible to train an action recognition model that outputs information indicating the action of the worker that corresponds to any of the common actions.

[0088] In addition, the distance between the coordinates corresponding to each of the body parts included in the skeleton data extracted from the video of the worker to be recognized and the starting coordinates corresponding to the starting part can be calculated, and each coordinate of the skeleton data can be replaced with the calculated distance. Then, by inputting the replaced distance data into the trained model, the worker's movements that correspond to any of the common movements can be recognized.

[0089] Therefore, the action recognition device 10 according to the present embodiment can recognize common actions performed in different work processes.

[0090] The present invention is not limited to the above-described embodiment, and can be implemented in various other forms without departing from the scope of the present invention. For example, the embodiment of the present invention can be described as in the following appendices. However, the embodiment of the present invention is not limited to the form described in the following appendices. Furthermore, the embodiment of the present invention may be in a form in which the descriptions between the appendices are replaced or combined.

[0091] [Appendix 1] an acquisition unit (11) for acquiring motion information (19b) including an elapsed time of a task, a common motion included in common to different task processes to be recognized, and skeletal data of a worker; a conversion unit (12) that calculates a distance between each of the coordinates corresponding to a plurality of body parts included in the skeletal data and an origin coordinate corresponding to an origin part defined as an origin among the body parts, based on the coordinates corresponding to each of the body parts included in the skeletal data, and converts each of the coordinates included in the skeletal data into the calculated distance; a learning unit (13) that learns, as a model for action recognition, a model that outputs information indicating an action of the worker that corresponds to any one of the common actions, based on the action information (19b) after the coordinates have been converted into the distances by the conversion unit (12); and An action recognition device (10) comprising:

[0092] [Appendix 2] An acquisition unit (11) that acquires time-series information (19a) regarding a motion of a worker; a conversion unit (12) that calculates a distance between each of a plurality of body parts included in the skeleton data extracted from the time-series information (19a) and an origin coordinate corresponding to an origin part of the body parts that is determined as an origin, and converts each of the coordinates included in the skeleton data into the calculated distance; an action recognition unit (14) that inputs the distance converted by the conversion unit (12) into a trained model (19c) and recognizes the action of the worker based on information indicating any of the actions of the worker that corresponds to any of the common actions commonly included in different work processes that are targets of recognition, the information being output from the trained model (19c); An action recognition device (10) comprising:

[0093] [Appendix 3] The starting site is either the neck or the lower back. 3. The motion recognition device (10) according to claim 1 or 2.

[0094] [Appendix 4] A motion recognition method executed by a processor (10a), comprising: Acquiring motion information (19b) including an elapsed time of the work, common motions included in different work processes to be recognized, and skeleton data of the worker; calculating a distance between each of the coordinates corresponding to a plurality of body parts included in the skeletal data and a starting point coordinate corresponding to a starting point part of the body parts that is determined as a starting point, based on the coordinates corresponding to each of the body parts included in the skeletal data, and converting each of the coordinates included in the skeletal data into the calculated distance; learning a model that outputs information indicating an action of the worker that corresponds to any one of the common actions based on the action information (19b) after the coordinates are converted into the distance, as a model for action recognition; The action recognition method includes:

[0095] [Appendix 5] A motion recognition method executed by a processor (10a), comprising: Obtaining time series information regarding the motion of a worker; Calculating a distance between each of the coordinates corresponding to a plurality of body parts included in the skeleton data extracted from the time-series information (19a) and a starting point coordinate corresponding to a starting point part of the body parts that is determined as a starting point, and converting each of the coordinates included in the skeleton data into the calculated distance; inputting the converted distance into a trained model (19c), and recognizing the motion of the worker based on information indicating any of the motions of the worker that correspond to any of the common motions commonly included in different work processes to be recognized, the information being output from the trained model (19c); The action recognition method includes:

[0096] [Appendix 6] Computer, an acquisition unit (11) for acquiring motion information (19b) including an elapsed time of a task, a common motion included in common to different task processes to be recognized, and skeletal data of a worker; a conversion unit (12) that calculates a distance between each of the coordinates corresponding to a plurality of body parts included in the skeleton data and a starting point coordinate corresponding to a starting point part of the body parts that is determined as a starting point, based on the coordinates corresponding to each of the body parts included in the skeleton data, and converts each of the coordinates included in the skeleton data into the calculated distance; a learning unit (13) that learns, as a model for action recognition, a model that outputs information indicating an action of the worker that corresponds to any one of the common actions, based on the action information (19b) after the coordinates have been converted into the distances by the conversion unit (12); A motion recognition program that functions as a

[0097] [Appendix 7] Computer, An acquisition unit (11) that acquires time-series information (19a) regarding the motion of a worker; a conversion unit (12) that calculates, based on coordinates corresponding to each of a plurality of body parts included in the skeleton data extracted from the time-series information (19a), a distance between each of the coordinates and an origin coordinate corresponding to an origin part of the body parts that is determined as an origin, and converts each of the coordinates included in the skeleton data into the calculated distance; an action recognition unit (14) that inputs the distance converted by the conversion unit (12) into a trained model (19c) and recognizes the action of the worker based on information output from the trained model (19c) that indicates any of the actions of the worker that are common actions included in common in different work processes that are targets of recognition; A motion recognition program that functions as a [Explanation of symbols]

[0098] 10...motion recognition device, 10a...CPU, 10b...RAM, 10c...ROM, 10d...communication device, 10e...input device, 10f...display device, 11...acquisition unit, 12...conversion unit, 13...learning unit, 14...motion recognition unit, 19...storage unit, 19a...video, 19b...motion information, 19c...trained model, 20a, 20b, 20c...image sensor, 100...motion recognition system, A...worker, R...work area, a...starting point portion

Claims

1. an acquisition unit that acquires from a storage unit motion information generated based on time-series information on a motion of a first worker in a task performed by the first worker to be recognized, the motion information including an elapsed time of the task performed by the first worker, information indicating a common motion commonly performed by all workers although the motion differs in each task process, and skeletal data of the first worker, in association with each other; a conversion unit that calculates, based on coordinates of a coordinate system that represents the skeletal data corresponding to each of a plurality of body parts included in the skeletal data of the acquired motion information, a distance between each of the coordinates and a starting point coordinate corresponding to a starting point part of the body part that is defined as a starting point near the center of the worker's movement, and replaces each of the coordinates included in the skeletal data with the calculated distance; an action recognition unit that inputs action information including the distance after the replacement by the conversion unit into a trained model, and recognizes the action of the first worker based on information indicating a worker's action corresponding to any of the common actions output from the trained model; and Equipped with The trained model is a model trained to receive input of motion information after the coordinates included in the skeletal data of the motion information generated based on time-series information on the motion of the second worker in the work performed by the second worker to be trained are replaced with the distance by the conversion unit, and to output information indicating a motion of the worker corresponding to any of the common motions. Action recognition device.

2. The starting site is either the neck or the lower back. The action recognition device according to claim 1.

3. 1. A processor-implemented method for action recognition, comprising: acquiring from a storage unit motion information generated based on time-series information on motions of a first worker in a task performed by the first worker to be recognized, the motion information storing, in association with each other, information on an elapsed time of the task performed by the first worker, information on common motions commonly performed by all workers although the movements are different in different task steps, and skeletal data of the first worker; calculating a distance between each of the coordinates of a coordinate system representing the skeletal data corresponding to each of a plurality of body parts included in the skeletal data of the acquired motion information and a starting point coordinate corresponding to a starting point part of the body part that is defined as a starting point near the center of the worker's movement, and replacing each of the coordinates included in the skeletal data with the calculated distance; inputting motion information including the replaced distance into a trained model, and recognizing the motion of the first worker based on information indicating a motion of the worker corresponding to any of the common motions output from the trained model; Including, The trained model is a model trained to receive input of motion information after the coordinates included in the skeletal data of the motion information generated based on time-series information on the motion of the second worker in the work performed by the second worker to be trained are replaced with the distance, and to output information indicating a motion of the worker corresponding to any of the common motions. Action recognition method.

4. Computer, an acquisition unit that acquires from a storage unit motion information generated based on time-series information on a motion of a first worker in a task performed by the first worker to be recognized, the motion information storing, in association with each other, information on an elapsed time of the task performed by the first worker, information on common motions commonly performed by all workers although the movements are different in different task steps, and skeletal data of the first worker; a conversion unit that calculates, based on coordinates of a coordinate system that represents the skeletal data corresponding to each of a plurality of body parts included in the skeletal data of the acquired motion information, a distance between each of the coordinates and a starting point coordinate corresponding to a starting point part of the body parts that is defined as a starting point near the center of the worker's movement, and replaces each of the coordinates included in the skeletal data with the calculated distance; an action recognition unit that inputs action information including the distance after the replacement by the conversion unit into a trained model, and recognizes the action of the first worker based on information indicating a worker's action corresponding to any of the common actions output from the trained model; Function as a The trained model is a model trained to receive input of motion information after the coordinates included in the skeletal data of the motion information generated based on time-series information on the motion of the second worker in the work performed by the second worker to be trained are replaced with the distance by the conversion unit, and to output information indicating a motion of the worker corresponding to any of the common motions. Motion recognition program.

Images