Motion analysis system and computer-readable recording medium

Abstract

A motion analysis device (200) receives a plurality of object image data obtained by continuously photographing an object person (111) performing a series of actions. The motion analysis device detects the posture of the object person reflected in the object image of each object image data, and generates object posture data representing the detected posture. The motion analysis device extracts two or more object posture data from the plurality of object posture data corresponding to the plurality of object image data as object trajectory data representing the transition of the posture of the object person. The motion analysis device obtains an action label for identifying the action of the object person based on the object trajectory data.

Description

Technical Field

[0001] The present invention relates to techniques for motion analysis. Background Art

[0002] In the industrial field, manual processing such as cycle measurement and work content analysis has become the mainstream.

[0003] The cycle time is the time it takes for an operator to assemble a product.

[0004] The analysis of work contents is performed to detect omissions of work or irregular work. Irregular work is work that is not performed routinely.

[0005] Manual processing costs a lot of manpower. In addition, in manual processing, only a limited range can be set as the object of processing.

[0006] If it is easy to analyze the actions of the worker, it is considered that it is easy to perform processes such as measuring the cycle and analyzing the content of the work.

[0007] Automatic motion analysis is described in Patent Document 1. To perform motion analysis, a camera and a three-dimensional sensor are attached to a person's head, and feature quantities of the person's motion are extracted using the camera and the three-dimensional sensor.

[0008] Prior art literature

[0009] Patent Literature

[0010] Patent Document 1: Japanese Patent Application Publication No. 2016-099982 Summary of the invention

[0011] Problems to be solved by the invention

[0012] In Patent Document 1, a camera and a three-dimensional sensor are attached to a person's head in order to perform automatic motion analysis.

[0013] When the method of Patent Document 1 is applied to analyze the actions of the worker, an object that is not necessary for the work is attached to the body of the worker. Furthermore, the object attached to the body of the worker may hinder the work.

[0014] An object of the present invention is to enable analysis of a worker's movements without interfering with the work.

[0015] Means for solving problems

[0016] The motion analysis system of the present invention comprises: an object image receiving unit, which receives a plurality of object image data obtained by continuously photographing an object person performing a series of actions; an object posture detection unit, which detects the posture of the object person reflected in the object image of each object image data, and generates object posture data representing the detected posture; an object trajectory extraction unit, which extracts two or more object posture data including object posture data at an object moment and object posture data at at least one of a previous moment continuous with the object moment and a subsequent moment continuous with the object moment, from a plurality of object posture data corresponding to the plurality of object image data, as object trajectory data representing the transition of the posture of the object person; and an action determination unit, which obtains an action label for identifying the action of the object person based on the object trajectory data.

[0017] Effects of the Invention

[0018] According to the present invention, even if unnecessary objects such as cameras and three-dimensional sensors are not attached to the body of the operator (an example of the subject), the operator's movements can be identified. In other words, the operator's movements can be analyzed without interfering with the work. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a configuration diagram of the motion analysis system 100 in the first embodiment.

[0020] Figure 2 This is a configuration diagram of the motion analysis device 200 in the first embodiment.

[0021] Figure 3 This is a flowchart of the motion analysis method in implementation mode 1.

[0022] Figure 4 This is an explanatory diagram of the object posture detection process ( S120 ) in the first embodiment.

[0023] Figure 5 This is a flowchart of the object trajectory extraction process ( S130 ) in the first embodiment.

[0024] Figure 6 This is an explanatory diagram of the object trajectory extraction process ( S130 ) in the first embodiment.

[0025] Figure 7 This is a diagram showing the structure of the action database 291 in the first embodiment.

[0026] Figure 8 This is a flowchart of the action determination process ( S140 ) in the first embodiment.

[0027] Fig. 9 This is a flowchart of the comparison process ( S144 ) in the first embodiment.

[0028] Fig.10 This is an explanatory diagram of the use of the dynamic planning method in the first embodiment.

[0029] Fig.11 This is a configuration diagram of the motion analysis system 100 in the first embodiment.

[0030] Fig.12 This is a diagram showing the configuration of the registration unit 220 in the first embodiment.

[0031] Fig.13 This is a flowchart of the registration process in the first embodiment.

[0032] Fig.14 This is an explanatory diagram of the motion posture detection process (S103) in the first embodiment.

[0033] Fig.15 This is a diagram showing the configuration of the registration unit 220 in the second embodiment.

[0034] Fig.16 This is a flowchart of the registration process in the second embodiment.

[0035] Fig.17 This is a flowchart of the motion posture generation process ( S250 ) in the second embodiment.

[0036] Fig.18 This is a diagram showing a specific example of sampling in Implementation Example 2.

[0037] Fig.19 This is a configuration diagram of the motion analysis device 200 in the third embodiment.

[0038] Fig. 20 This is a diagram showing the configuration of the learning unit 230 in the third embodiment.

[0039] Fig.21 This is a flowchart of the action analysis method in implementation mode 3.

[0040] Fig. 22 This is a flowchart of the learning process in the third embodiment.

[0041] Fig.23 2 is a hardware configuration diagram of the motion analysis device 200 according to the embodiment. DETAILED DESCRIPTION

[0042] In the embodiments and the drawings, the same reference numerals are used to denote the same elements or corresponding elements. The description of elements with the same reference numerals as those already described is appropriately omitted or simplified. The arrows in the drawings mainly indicate data flows or process flows.

[0043] Implementation Method 1

[0044] according to Figure 1 to Figure 14 A motion analysis system 100 for automatically analyzing human motion will be described.

[0045] ***Description of the structure***

[0046] according to Figure 1 The configuration of the motion analysis system 100 will be described.

[0047] The motion analysis system 100 includes a camera 101 and a motion analysis device 200 .

[0048] The camera 101 captures a subject 111. The camera 101 may capture the subject 111 from the front or from an angle. The camera 101 may capture the entire body of the subject 111 or a portion of the subject 111 (for example, the upper body).

[0049] The motion analysis device 200 analyzes the motion of the subject person 111 .

[0050] The subject person 111 is a person who is the subject of motion analysis. A specific example of the subject person 111 is a person who performs work (worker).

[0051] according to Figure 2 The structure of the motion analysis device 200 will be described.

[0052] The motion analysis device 200 is a computer having hardware such as a processor 201, a memory 202, an auxiliary storage device 203, a communication device 204, and an input / output interface 205. These hardware are connected to each other via signal lines.

[0053] The processor 201 is an IC that performs calculations and controls other hardware. For example, the processor 201 is a CPU, a DSP, or a GPU.

[0054] IC is the abbreviation of Integrated Circuit.

[0055] CPU is the abbreviation of Central Processing Unit.

[0056] DSP is the abbreviation of Digital Signal Processor.

[0057] GPU is the abbreviation of Graphics Processing Unit.

[0058] The memory 202 is a volatile or nonvolatile storage device. The memory 202 is also called a main storage device or a main memory. For example, the memory 202 is a RAM. The data stored in the memory 202 is stored in the auxiliary storage device 203 as needed.

[0059] RAM is the abbreviation of Random Access Memory.

[0060] The auxiliary storage device 203 is a nonvolatile storage device. For example, the auxiliary storage device 203 is a ROM, a HDD, or a flash memory. The data stored in the auxiliary storage device 203 is loaded into the memory 202 as needed.

[0061] ROM is the abbreviation of Read Only Memory.

[0062] HDD is the abbreviation of Hard Disk Drive.

[0063] The communication device 204 is a receiver and a transmitter. For example, the communication device 204 is a communication chip or a NIC.

[0064] NIC is the abbreviation of Network Interface Card.

[0065] The input / output interface 205 is a port to which an input device and an output device are connected. For example, the input / output interface 205 is a USB terminal, the input device is a keyboard and a mouse, and the output device is a display.

[0066] USB is the abbreviation of Universal Serial Bus.

[0067] The motion analysis device 200 includes elements such as a specifying unit 210 and a registering unit 220. The specifying unit 210 includes elements such as an object image accepting unit 211, an object posture detecting unit 212, an object trajectory extracting unit 213, a motion specifying unit 214, and a result outputting unit 215. These elements are realized by software.

[0068] The auxiliary storage device 203 stores a motion analysis program for causing the computer to function as the identification unit 210 and the registration unit 220. The motion analysis program is loaded into the memory 202 and executed by the processor 201.

[0069] The auxiliary storage device 203 also stores an OS. At least a part of the OS is loaded into the memory 202 and executed by the processor 201 .

[0070] The processor 201 executes the motion analysis program while executing the OS.

[0071] OS is the abbreviation of Operating System.

[0072] The input and output data of the motion analysis program are stored in the storage unit 290 .

[0073] The memory 202 functions as the storage unit 290 . However, a storage device such as the auxiliary storage device 203 , a register in the processor 201 , or a cache memory in the processor 201 may function as the storage unit 290 instead of or together with the memory 202 .

[0074] The storage unit 290 stores an action database 291 and the like.

[0075] The motion analysis device 200 may include a plurality of processors instead of the processor 201. The plurality of processors share the function of the processor 201.

[0076] The motion analysis program can be recorded (stored) in a computer-readable form on a nonvolatile recording medium such as an optical disk or a flash memory.

[0077] ***Description of the action***

[0078] The order of operations of the motion analysis system 100 corresponds to the motion analysis method. In addition, the order of operations of the motion analysis device 200 corresponds to the order of processing based on the motion analysis program.

[0079] according to Figure 3 The motion analysis method is described.

[0080] In step S110 , the target image accepting unit 211 accepts a plurality of target image data.

[0081] The plurality of target image data are obtained by continuously photographing the target person 111 performing a series of actions.

[0082] Each object image data represents an object image. An object person 111 is reflected in the object image.

[0083] For example, a plurality of target image data are input by a user to the motion analysis device 200. The target image receiving unit 211 receives the input plurality of target image data.

[0084] For example, a plurality of target image data are transmitted from the camera 101 and received by the motion analysis device 200. The target image receiving unit 211 receives the received plurality of target image data.

[0085] In step S120 , the subject posture detection unit 212 detects the posture of the subject person 111 reflected in the subject image of each subject image data, and generates subject posture data indicating the detected posture.

[0086] Thereby, a plurality of object posture data corresponding to a plurality of object image data are obtained.

[0087] according to Figure 4 The object posture detection process (S120) is described below. [m-1], [m], and [m+1] represent three consecutive shooting times. The shooting time is the time when the shooting is performed.

[0088] The object image data 121 at each imaging time point indicates the object image at each imaging time point. The object image at each imaging time point shows the object person 111 at each imaging time point.

[0089] The subject posture detection unit 212 performs image processing on the subject image at each imaging time, thereby detecting the posture of the subject person 111 at each imaging time.

[0090] Specifically, the subject posture detection unit 212 detects the skeleton of the subject person 111. The posture of the subject person 111 is specified based on the skeleton of the subject person 111.

[0091] More specifically, the subject posture detection unit 212 determines the position of each joint of the subject 111. The position of each joint can be an absolute position in a specific coordinate system or a relative position relative to other joints. The position of each joint is identified by a coordinate value. The skeleton is determined based on multiple joints such as the joints of the head and the joints of the shoulders.

[0092] The subject posture detection unit 212 generates subject posture data 122 indicating the posture of the subject person 111 at each imaging time.

[0093] In the subject posture data 122 at each imaging time, each black circle indicates the position of a joint of the subject person 111 .

[0094] return Figure 3 , and continue with the description after step S130.

[0095] Step S130 and step S140 are performed for each shooting time of the shooting time period. The shooting time period is a time period in which shooting is performed.

[0096] In step S130 , the object trajectory extraction unit 213 extracts object trajectory data from a plurality of object posture data corresponding to a plurality of object image data.

[0097] The object trajectory data is composed of two or more object posture data, and represents the posture transition of the subject 111. In the object trajectory data, the two or more object posture data are arranged in time series. However, the object trajectory data may also be a polynomial function that approximately represents the two or more object posture data.

[0098] The object trajectory data is supplemented. The trajectory data is data at continuous moments. The object trajectory data is object posture data at continuous moments. Continuous moments are the object moment and moments continuous with the object moment. When the object moment is [t], the continuous moments are [t-1], [t] and [t+1].

[0099] according to Figure 5 The object trajectory extraction process ( S130 ) will be described.

[0100] In step S131 , the object trajectory extraction unit 213 selects one unselected shooting time from the shooting time period.

[0101] The shooting time period is a time period during which shooting was performed.

[0102] The selected shooting time is referred to as “target time”.

[0103] In step S132 , the object trajectory extraction unit 213 determines a shooting time group for the object time.

[0104] The imaging time group is composed of two or more consecutive imaging times, including the target time. The number of imaging times included in the imaging time group is predetermined.

[0105] The target time is represented by “m.” For example, the target trajectory extraction unit 213 determines a shooting time group consisting of shooting time (m-1), shooting time (m), and shooting time (m+1) as the shooting time group for the target time.

[0106] The determined shooting time group is referred to as a “target time group”.

[0107] In step S133 , the object trajectory extraction unit 213 extracts object trajectory data corresponding to the object time from the plurality of object posture data.

[0108] For example, when the object time group is composed of time (m-1), time (m), and time (m+1), the object trajectory extraction unit 213 extracts the object posture data at time (m-1), the object posture data at time (m), and the object posture data at time (m+1). The three extracted object posture data constitute the object trajectory data corresponding to the object time.

[0109] according to Figure 6The object trajectory extraction process (S130) will be described. [m-1], [m], and [m+1] represent three consecutive imaging times.

[0110] Assume that the target time group consists of [m-1], [m], and [m+1].

[0111] The object trajectory extraction unit 213 extracts the object posture data 122 of [m−1], the object posture data 122 of [m], and the object posture data 122 of [m+1].

[0112] The extracted three object posture data 122 constitute object trajectory data 123 of [m].

[0113] return Figure 3 , the description continues from step S140.

[0114] In step S140 , the action identification unit 214 obtains an action label of the action of the recognition target person 111 based on the object trajectory data.

[0115] Specifically, the motion identification unit 214 extracts the motion label of the motion trajectory data that matches the object trajectory data from the motion database 291 .

[0116] The motion database 291 is a database in which one or more pairs of motion labels and motion trajectory data are registered.

[0117] Action tags identify actions.

[0118] The motion trajectory data indicates the transition of the posture of a person performing the motion recognized by the motion label.

[0119] according to Figure 7 The following describes the structure of the action database 291. [t-1], [t], and [t+1] represent three consecutive imaging times.

[0120] In the motion database 291 , a plurality of time points, a plurality of motion posture data, and a plurality of motion labels are associated with each other.

[0121] The motion trajectory data 133 is composed of two or more motion posture data 132. Specifically, the motion trajectory data 133 is composed of two or more motion posture data 132 corresponding to two or more consecutive time points.

[0122] The action posture data 132 indicates the posture of a person performing an action.

[0123] The motion trajectory data 133 of [t] is composed of the motion posture data 132 of [t-1], the motion posture data 132 of [t], and the motion posture data 132 of [t+1]. The motion trajectory data 133 also includes data of consecutive time points around [t].

[0124] The motion trajectory data 133 at “t” indicates the transition of the posture of a person performing the motion recognized by the motion label “task A” at time t.

[0125] according to Figure 8 The action determination process ( S140 ) will be described.

[0126] In step S141, the action identification unit 214 selects one unselected time from the "time" in the action database 291. Specifically, the action identification unit 214 selects the time in ascending order.

[0127] The selected time is referred to as a "comparison time".

[0128] In step S142 , the operation determination unit 214 determines a comparison time group for comparison time.

[0129] The comparison time group is composed of two or more consecutive times and includes the comparison time. The number of times included in the comparison time group is predetermined.

[0130] The comparison time is represented by “t.” For example, the operation determination unit 214 determines a time group consisting of time (t-1), time (t), and time (t+1) as a comparison time group for the comparison time.

[0131] In step S143 , the motion identification unit 214 extracts the motion trajectory data corresponding to the comparison time from the motion database 291 .

[0132] For example, when the comparison time group consists of time (t-1), time (t), and time (t+1), the action determination unit 214 extracts the action posture data at time (t-1), the action posture data at time (t), and the action posture data at time (t+1). The three extracted action posture data constitute the action trajectory data corresponding to the comparison time.

[0133] In step S144, the motion determination unit 214 compares the motion trajectory data with the object trajectory data. The object trajectory data is obtained in step S130 (see Figure 3 ) extracted from

[0134] Specifically, the motion identification unit 214 calculates the distance between the motion trajectory data and the object trajectory data.

[0135] The distance between the motion trajectory data and the object trajectory data corresponds to the magnitude of the difference between the posture transition represented by the object trajectory data and the posture transition represented by the motion trajectory data.

[0136] according to Fig. 9 The comparison process (S144) will be described.

[0137] The object posture data included in the object trajectory data is recorded as I(m) or I(m+k).

[0138] The motion posture data included in the motion trajectory data is recorded as R(t) or R(t+k).

[0139] “k” is an integer greater than or equal to −i and less than or equal to +i. “i” is an integer greater than or equal to 1. When the number of object posture data included in the object trajectory data is “3”, “i” is “1”.

[0140] In step S1441, the motion identification unit 214 calculates the distance between the object posture data I(m+k) and the object posture data I(m). The calculated distance is referred to as I'(m+k).

[0141] The distance I′(m+k) corresponds to the size of the difference between the posture represented by the object posture data I(m) and the posture represented by the object posture data I(m+k).

[0142] The distance I'(m+k) can be expressed by equation (1).

[0143] I'(m+k)=I(m+k)-I(m) (1)

[0144] Specifically, the action determination unit 214 calculates the distance I′(m+k) as follows.

[0145] The object pose data represents the positions of each of the multiple joints.

[0146] The action determination unit 214 calculates the distance between the position of the joint in the object posture data I(m) and the position of the joint in the object posture data I(m+k) for each joint. The action determination unit 214 may also weight each joint and calculate the distance of each joint based on the weight of each joint. In this case, the distance of the important joint becomes a larger value.

[0147] Alternatively, the action determination unit 214 calculates the difference between the distance between the joints in the object posture data I(m) and the distance between the joints in the object posture data I(m+k) as the distance for the joint group for each joint group. The action determination unit 214 may also weight each joint group and calculate the distance for each joint group according to the weight of each joint group. In this case, the distance for an important joint group becomes a larger value.

[0148] The operation determination unit 214 calculates the total of the calculated distances. The calculated total becomes the distance I'(m+k).

[0149] In step S1442, the motion identification unit 214 calculates the distance between the motion posture data R(t+k) and the motion posture data R(t). The calculated distance is recorded as R'(t+k).

[0150] The distance R′(t+k) corresponds to the size of the difference between the posture represented by the motion posture data R(t) and the posture represented by the motion posture data R(t+k).

[0151] The distance R'(t+k) can be expressed by equation (2).

[0152] R'(t+k)=R(t+k)-R(t) (2)

[0153] Specifically, the action determination unit 214 calculates the distance R′(t+k) as follows.

[0154] The motion posture data represents the positions of each of the multiple joints.

[0155] The action determination unit 214 calculates the distance between the position of the joint in the action posture data R(t) and the position of the joint in the action posture data R(t+k) for each joint. The action determination unit 214 may also weight each joint and calculate the distance of each joint based on the weight of each joint. In this case, the distance of the important joint becomes a larger value.

[0156] Alternatively, the action determination unit 214 calculates the difference between the distance between the joints in the action posture data R(m) and the distance between the joints in the action posture data I(m+k) as the distance for the joint group for each joint group. The action determination unit 214 may also weight each joint group and calculate the distance for each joint group according to the weight of each joint group. In this case, the distance for an important joint group becomes a larger value.

[0157] The operation determination unit 214 calculates the total of the calculated distances. The calculated total becomes the distance R'(t+k).

[0158] In step S1443 , the operation determination unit 214 calculates the total of the differences between the distance I′(m+k) and the distance R′(t+k).

[0159] The calculated total becomes the distance between the motion trajectory data and the object trajectory data.

[0160] The distance between the motion trajectory data and the object trajectory data can be expressed by equation (3).

[0161]

[0162] The distance between the motion trajectory data and the object trajectory data can be calculated by methods such as dynamic planning or hidden Markov model, etc. These methods dynamically scale the time series to calculate the minimum distance.

[0163] In these methods, for example, when the distance obtained by I'(m+k)-R'(m+(k+a)) is smaller than the distance obtained by I'(m+k)-R'(t+k), the distances of "k" and "k+a" are added.

[0164] according to Fig.10 The use of the dynamic planning method is explained.

[0165] The numbers in the table represent the values ​​of |I'(m+a)-R'(t+b)|, where -2≤a, b≤2. The solid arrows represent the paths (correspondence between I' and R').

[0166] In the dynamic planning method, "a" and "b" are changed to calculate the distance of each grid, and then a path is selected in which the sum of the distances becomes small (see the solid arrow).

[0167] return Figure 8 , the description continues from step S145.

[0168] In step S145 , the operation specifying unit 214 determines whether or not there is an unselected comparison time.

[0169] If there is an unselected comparison time, the process proceeds to step S141.

[0170] When there is no unselected comparison time, the process proceeds to step S146.

[0171] In step S146 , the motion identification unit 214 selects motion trajectory data that matches the target trajectory data based on the comparison result between each motion trajectory data and the target trajectory data.

[0172] Specifically, the motion identification unit 214 selects motion trajectory data that matches the object trajectory data based on the distance of each motion trajectory data with respect to the object trajectory data.

[0173] The motion trajectory data that matches the object trajectory data is determined as follows.

[0174] The motion identification unit 214 selects the smallest distance among the distances calculated in step S144. Then, the motion identification unit 214 selects motion trajectory data corresponding to the selected distance. The selected motion trajectory data is motion trajectory data that matches the target trajectory data.

[0175] However, the motion identification unit 214 may determine that there is no motion trajectory data that matches the target trajectory data when the selected distance (the smallest distance) is greater than a threshold value.

[0176] In step S147 , the motion identification unit 214 extracts the motion label of the motion trajectory data that matches the object trajectory data from the motion database 291 .

[0177] Specifically, the motion identification unit 214 selects the comparison time corresponding to the motion trajectory data selected in step S146 and extracts the motion label corresponding to the selected comparison time. The extracted motion label is the motion label of the motion trajectory data that matches the target trajectory data.

[0178] For example, assuming that the motion trajectory data 133 of [t] is selected in step S146 (refer to Figure 7 ). In this case, the action identification unit 214 extracts the action label “job A” corresponding to [t] from the action database 291 .

[0179] However, if it is determined that there is no motion trajectory data that matches the target trajectory data (see step S146 ), the motion label is not extracted. If the motion label is not extracted, the motion identification unit 214 assigns a label indicating a task that is not usually performed (unusual task).

[0180] return Figure 3 , the processing after step S130 and step S140 is explained.

[0181] Steps S130 and S140 are performed for each shooting moment in the shooting time period, thereby obtaining an action label for each shooting moment.

[0182] After step S130 and step S140 are executed for each shooting time point in the shooting time period, the process proceeds to step S150 .

[0183] In step S150 , the result output unit 215 outputs the result data.

[0184] For example, the result output unit 215 displays the result data on a display.

[0185] The resulting data contains the action labels for each capture moment.

[0186] ***Pre-treatment Supplement***

[0187] A method of generating the action database 291 will be described.

[0188] Fig.11 The configuration of the motion analysis system 100 when the motion database 291 is generated is shown.

[0189] The motion analysis system 100 includes a camera 102 and a motion analysis device 200 .

[0190] The camera 102 captures the action person 112. The camera 102 can be connected to the camera 101 (see Figure 1 ) can also be the same as camera 101.

[0191] The actor 112 is a person who performs an action (target action) that is a registration target for the action database 291. The actor 112 may be a person who is similar to the target person 111 (see Figure 1 ) may be the same person or may be a different person from the subject 111.

[0192] When the actor 112 is a worker, the actor 112 performs a work that is usually performed (routine work) as a series of actions. However, the actor 112 may perform a series of actions including a work that is not usually performed (irregular work).

[0193] Fig.12 The configuration of the registration unit 220 included in the motion analysis device 200 is shown.

[0194] The registration unit 220 includes elements such as a motion label accepting unit 221 , a motion image accepting unit 222 , a motion posture detecting unit 223 , and a motion registering unit 224 .

[0195] according to Fig.13 The registration process performed by the registration unit 220 will be described.

[0196] The action database 291 is generated by the registration process (see Figure 7 ).

[0197] In step S101 , the action label accepting unit 221 accepts a plurality of action labels to which a plurality of times are added.

[0198] For example, the action label accepting unit 221 accepts an action label list. The action label list indicates a plurality of sets of time and action labels.

[0199] For example, the action label list is input by the user to the action analysis device 200. The action label accepting unit 221 accepts the input action label list.

[0200] In addition, a label indicating that the action is not included in the regular work is given.

[0201] In step S102, the motion image receiving unit 222 receives a plurality of motion image data to which a plurality of times are added. The plurality of times added to the plurality of motion image data coincide with the plurality of times added to the plurality of motion tags.

[0202] The plurality of motion image data are obtained by continuously photographing the agent 112 performing a series of motions.

[0203] Each piece of motion image data represents a motion image, in which an actor 112 is shown.

[0204] For example, the motion image receiving unit 222 receives a motion image file. The motion image file includes a plurality of sets of time and motion image data.

[0205] For example, a motion image file is input by a user to the motion analysis device 200. The motion image receiving unit 222 receives the input motion image file.

[0206] In step S103 , the motion posture detection unit 223 detects the posture of the agent 112 shown in the motion image of each motion image data, and generates motion posture data representing the detected posture.

[0207] Thereby, a plurality of motion posture data corresponding to a plurality of motion image data are obtained.

[0208] according to Fig.14 The motion posture detection process (S103) is described below. [t-1], [t], and [t+1] represent three consecutive time points.

[0209] The motion image data 131 at each time point indicates the motion image at each time point. The actor 112 at each time point is shown in the motion image at each time point.

[0210] The action posture detection unit 223 performs image processing on the action image at each time, thereby detecting the posture of the actor 112 at each time.

[0211] Specifically, the action posture detection unit 223 detects the skeleton of the actor 112. The posture of the actor 112 is specified based on the skeleton of the actor 112.

[0212] More specifically, the action posture detection unit 223 determines the position of each joint of the actor 112. The position of each joint can be expressed by a coordinate value or a relative position to other joints. The skeleton is determined from a plurality of joints such as the joints of the head and the joints of the shoulders.

[0213] The motion posture detection unit 223 generates the motion posture data 132 indicating the posture of the agent 112 at each time.

[0214] In the action posture data 132 at each time point, each black circle indicates the position of the joint of the agent 112 .

[0215] return Fig.13 , step S104 is explained.

[0216] In step S104 , the motion registration unit 224 registers a plurality of times, a plurality of motion posture data, and a plurality of motion labels in the motion database 291 in association with each other.

[0217] ***Description of the Examples***

[0218] The action determination unit 214 may also determine a time period for performing routine work and a time period for performing non-routine work.

[0219] Specifically, the action determination unit 214 measures the time period between the moments assigned with the label indicating irregularity. The measured time period is the time period for irregular operations. Then, the action determination unit 214 determines the time period other than the time period for irregular operations. The determined time period is the time period for regular operations.

[0220] ***Effects of Implementation Method 1***

[0221] According to the first embodiment, even if objects not necessary for the work, such as cameras and three-dimensional sensors, are not attached to the body of the worker (an example of the subject), the worker's movements can be identified. In other words, the worker's movements can be analyzed without interfering with the work.

[0222] Furthermore, it is possible to identify the time period in which the subject person 111 performs each action. That is, it is possible to identify the time period in which the subject person 111 performs routine work and the time period in which the subject person performs non-routine work.

[0223] In addition, by comparing trajectory data (continuous moment data) such as object trajectory data and action trajectory data, the following effect is obtained. Sometimes a job includes multiple actions. Therefore, instead of comparing actions at a single moment, continuous actions, i.e., trajectory data, are compared with each other, thereby enabling a more appropriate job judgment.

[0224] Implementation Method 2

[0225] Regarding the method for improving the accuracy of motion analysis, the main Figure 15 to Figure 18 The differences from Implementation Example 1 are described.

[0226] ***Description of the structure***

[0227] The structure of the motion analysis system 100 is similar to that of the first embodiment (see Figure 1 and Fig.11 )same.

[0228] However, in Fig.11 In the video, the actor 112 performs a series of actions multiple times, and the camera 102 captures the series of actions performed by the actor 112 multiple times.

[0229] The structure of the motion analysis device 200 is similar to that of the first embodiment (see Figure 2 )same.

[0230] However, a part of the configuration of the registration unit 220 is different from the configuration in the first embodiment (see Fig.12 )different.

[0231] according to Fig.15 The configuration of the registration unit 220 will be described.

[0232] The registration unit 220 includes elements such as a motion label accepting unit 221 , a motion image accepting unit 222 , a temporary posture detecting unit 225 , a temporary table generating unit 226 , a motion posture generating unit 227 , and a motion registering unit 224 .

[0233] ***Description of the action***

[0234] according to Fig.16 The registration process performed by the registration unit 220 will be described.

[0235] The action database 291 is generated by the registration process (see Figure 7 ).

[0236] Step S210 to Step S240 are executed for each of the multiple series of actions. That is, when the series of actions are performed three times, Step S210 to Step S240 are executed three times.

[0237] In step S210 , the action label accepting unit 221 accepts an action label group.

[0238] An action tag group is a plurality of action tags to which a plurality of time points are attached.

[0239] Step S210 is equivalent to step S101 in the first embodiment (see Fig.13 ).

[0240] In step S220 , the motion image receiving unit 222 receives the motion image data group.

[0241] The action image data set is a plurality of action image data to which a plurality of times are added, obtained by photographing an actor performing a series of actions.

[0242] Step S220 is equivalent to step S202 in the first embodiment (see Fig.13 ).

[0243] In step S230 , the temporary posture detection unit 225 detects the posture of the agent 112 shown in the motion image of each motion image data in the motion image data group, and generates temporary posture data indicating the detected posture.

[0244] Thus, a temporary gesture data group corresponding to the motion image data group is obtained. The temporary gesture data group is a plurality of temporary gesture data corresponding to a plurality of motion image data.

[0245] The method of generating temporary posture data is the same as the method of generating action posture data in the first embodiment (refer to Fig.13 Step S103).

[0246] In step S240 , the temporary table generation unit 226 generates a temporary table.

[0247] The temporary table is a table that associates the time group, the temporary posture data group, and the action label group with each other.

[0248] A time group is a plurality of time points added to a plurality of action tags and a plurality of action images.

[0249] Steps S210 to S240 are respectively executed for a series of actions a plurality of times, thereby generating a plurality of temporary tables.

[0250] After steps S210 to S240 are executed for a series of actions a plurality of times, the process proceeds to step S250 .

[0251] In step S250 , the motion posture generating unit 227 generates two or more motion posture data corresponding to two or more times for each motion included in a series of motions.

[0252] Two or more action posture data corresponding to two or more time points are generated as follows.

[0253] The action posture generating unit 227 extracts two or more pieces of temporary posture data, that is, temporary trajectory data corresponding to two or more action labels identifying the same action, from each of the plurality of temporary tables.

[0254] The motion posture generating unit 227 generates two or more motion posture data corresponding to two or more time points based on the plurality of temporary trajectory data extracted from the plurality of temporary tables.

[0255] according to Fig.17 The motion posture generation process ( S250 ) will be described.

[0256] Steps S251 to S253 are executed for each action included in a series of actions.

[0257] In step S251, the motion posture generation unit 227 extracts temporary trajectory data from each of the plurality of temporary tables. In this way, a plurality of temporary trajectory data are extracted.

[0258] In step S252, the motion posture generation unit 227 determines the number of motion posture data constituting the motion trajectory data. The determined number is referred to as "the number of constituting data".

[0259] For example, the number of constituent data is the average number of temporary gesture data in a plurality of temporary trajectory data. Assume that the number of temporary gesture data constituting the first temporary trajectory data is "3", the number of temporary gesture data constituting the second temporary trajectory data is "4", and the number of temporary gesture data constituting the third temporary trajectory data is "5". In this case, the number of constituent data is "4".

[0260] In step S253 , the motion posture generation unit 227 samples each temporary trajectory data.

[0261] Thus, the number of temporary posture data constituting each temporary trajectory data becomes the same as the number of constituent data.

[0262] Fig.18 A specific example of sampling is shown.

[0263] “A n " represents the temporary posture data contained in the n-th temporary trajectory data before sampling.

[0264] “A n '" represents the temporary posture data contained in the n-th temporary trajectory data after sampling.

[0265] "*" means multiplication and "+" means addition.

[0266] Each temporary posture data indicates the position of each of the plurality of joints. For each joint, the coordinate value for identifying the position of the joint is multiplied and added.

[0267] Assume that the number of constituent data is "4".

[0268] Before sampling, the first temporary trajectory data consists of three temporary posture data A1 (1 to 3). In addition, the second temporary trajectory data consists of four temporary posture data A2 (1 to 4). In addition, the third temporary trajectory data consists of five temporary posture data A3 (1 to 5).

[0269] By sampling the first temporary trajectory data, four temporary posture data A1 ′ ( 1 to 4 ) are generated based on the three temporary posture data A1 ( 1 to 3 ).

[0270] By sampling the second temporary trajectory data, four temporary posture data A2 ′ ( 1 to 4 ) are generated based on the four temporary posture data A2 ( 1 to 4 ).

[0271] By sampling the provisional trajectory data for the third time, four provisional posture data A3 ′ ( 1 to 4 ) are generated based on the five provisional posture data A3 ( 1 to 5 ).

[0272] After sampling, the first temporary trajectory data consists of four temporary posture data A1' (1 to 4). In addition, the second temporary trajectory data consists of four temporary posture data A2' (1 to 4). In addition, the third temporary trajectory data consists of four temporary posture data A3' (1 to 4).

[0273] return Fig.17 , step S254 is explained.

[0274] In step S254 , the motion posture generating unit 227 generates the same number of motion posture data as the number of component data.

[0275] The motion posture generating unit 227 generates motion posture data at each time point as follows.

[0276] The motion posture generation unit 227 calculates the sum of a plurality of temporary posture data corresponding to the same time. The sum of a plurality of temporary posture data is calculated for each joint for the coordinate value of the position of the identified joint.

[0277] The motion posture generation unit 227 divides the sum of the plurality of temporary posture data by the number of temporary posture data. Thus, the average of the plurality of temporary posture data corresponding to the same time is calculated. The average of the plurality of temporary posture data becomes the motion posture data.

[0278] However, the motion posture generating unit 227 may calculate the variance of a plurality of temporary posture data together with the average of the plurality of temporary posture data.

[0279] return Fig.16 , step S260 is explained.

[0280] In step S260 , the action registration unit 224 registers two or more times, two or more action posture data, and two or more action labels identifying the same action in the action database 291 in association with each other for each action included in a series of actions.

[0281] ***Effects of Implementation Method 2***

[0282] According to Embodiment 2, the motion database 291 can be generated based on a series of motions performed a plurality of times. As a result, the accuracy of motion analysis can be improved.

[0283] Implementation 3

[0284] Regarding the method of using learning models for motion analysis, it is mainly based on Figure 19 to Figure 22The differences from the first and second embodiments are described.

[0285] ***Description of the structure***

[0286] The structure of the motion analysis system 100 is similar to that of the first embodiment (see Figure 1 and Fig.11 )same.

[0287] according to Fig.19 The structure of the motion analysis device 200 will be described.

[0288] The motion analysis device 200 includes elements such as a determination unit 210 and a learning unit 230. These elements are implemented by software.

[0289] The motion analysis program causes the computer to function as the identification unit 210 and the learning unit 230 .

[0290] The learning unit 230 generates a learning model 292 .

[0291] The storage unit 290 stores a learning model 292 .

[0292] The learning model 292 outputs an action label in response to the input of action trajectory data.

[0293] according to Fig. 20 The configuration of the learning unit 230 will be described.

[0294] The learning unit 230 includes elements such as a motion label accepting unit 231 , a motion image accepting unit 232 , a motion posture detecting unit 233 , and a motion learning unit 234 .

[0295] ***Description of the action***

[0296] according to Fig.21 The motion analysis method is described.

[0297] In step S310 , the target image accepting unit 211 accepts a plurality of target image data.

[0298] Step S310 is similar to step S110 in Embodiment 1 (see Figure 3 )same.

[0299] In step S320 , the subject posture detection unit 212 detects the posture of the subject person 111 reflected in the subject image of each subject image data, and generates subject posture data indicating the detected posture.

[0300] Thereby, a plurality of object posture data corresponding to a plurality of object image data are obtained.

[0301] Step S320 is similar to step S120 in Embodiment 1 (see Figure 3 )same.

[0302] In step S330 , the object trajectory extraction unit 213 extracts object trajectory data from the plurality of object posture data corresponding to the plurality of object image data.

[0303] Step S330 is similar to step S130 in Embodiment 1 (see Figure 3 )same.

[0304] In step S340 , the action identification unit 214 obtains an action label of the action of the recognition target person 111 based on the object trajectory data.

[0305] The action labels are obtained as follows.

[0306] The motion identification unit 214 inputs the object trajectory data into the learning model.

[0307] The learning model outputs an action label of the learning trajectory data that is consistent with the input object trajectory data.

[0308] The action identification unit 214 obtains action labels from the learning model.

[0309] Step S330 and step S340 are performed for each shooting time in the shooting time period, thereby obtaining an action label for each shooting time.

[0310] After step S330 and step S340 are executed for each shooting time of the shooting time period, the process proceeds to step S350.

[0311] In step S350 , the result output unit 215 outputs the result data.

[0312] Step S350 is similar to step S150 in Embodiment 1 (see Figure 3 )same.

[0313] ***Pre-processing instructions***

[0314] according to Fig. 22 The learning process of the learning unit 230 will be described.

[0315] A learning model 292 is generated through the learning process.

[0316] In step S301 , the action label accepting unit 231 accepts a plurality of action labels to which a plurality of times are added.

[0317] Step S301 is similar to step S101 in Embodiment 1 (see Fig.13 )same.

[0318] In step S302 , the motion image receiving unit 232 receives a plurality of motion image data to which a plurality of times are added.

[0319] Step S302 is similar to step S102 in Embodiment 1 (see Fig.13 )same.

[0320] In step S303 , the motion posture detection unit 233 detects the posture of the agent 112 shown in the motion image of each motion image data, and generates motion posture data representing the detected posture.

[0321] Thereby, a plurality of motion posture data corresponding to a plurality of motion image data are obtained.

[0322] Step S303 is similar to step S103 in Embodiment 1 (see Fig.13 )same.

[0323] In step S304 , the motion learning unit 234 performs learning using a set of motion posture data and a motion label as learning data for each time point.

[0324] For example, the action learning unit 234 performs machine learning using an existing machine learning model. Specifically, the action learning unit 234 inputs the learning data at each moment into the machine learning model. Then, the machine learning model performs machine learning on the learning data at each moment. However, the action learning unit 234 may also perform learning using other methods.

[0325] Thus, the learning model 292 is generated. If the learning model 292 exists, the learning model 292 is updated.

[0326] ***Effects of Implementation 3***

[0327] According to the third embodiment, it is possible to perform motion analysis using the learning model 292. As a result, the accuracy of motion analysis can be improved.

[0328] ***Supplementary implementation methods***

[0329] according to Fig.23 The hardware configuration of the motion analysis device 200 will be described.

[0330] The motion analysis device 200 includes a processing circuit 209 .

[0331] The processing circuit 209 is hardware that realizes the determination section 210 , the registration section 220 , and the learning section 230 .

[0332] The processing circuit 209 may be dedicated hardware, or may be the processor 201 that executes a program stored in the memory 202 .

[0333] In the case where the processing circuit 209 is dedicated hardware, the processing circuit 209 is, for example, a single circuit, a complex circuit, a programmed processor, a parallel programmed processor, an ASIC, an FPGA, or a combination thereof.

[0334] ASIC is the abbreviation of Application Specific Integrated Circuit.

[0335] FPGA is the abbreviation of Field Programmable Gate Array.

[0336] The motion analysis device 200 may include a plurality of processing circuits instead of the processing circuit 209. The function of the processing circuit 209 is shared by the plurality of processing circuits.

[0337] In the processing circuit 209 , part of the functions may be implemented by dedicated hardware, and the remaining functions may be implemented by software or firmware.

[0338] In this way, each function of the motion analysis device 200 can be implemented by hardware, software, firmware, or a combination thereof.

[0339] Each embodiment is an illustration of a preferred mode and is not intended to limit the technical scope of the present invention. Each embodiment can be implemented in part or in combination with other modes. The order of explanation using flowcharts, etc. can also be appropriately changed.

[0340] The element "unit" of the motion analysis device 200 may be rewritten as "processing" or "step".

[0341] The motion analysis device 200 may also be implemented by multiple devices.

[0342] Camera 101 and camera 102 may also be replaced with depth sensors or other sensors, through which three-dimensional image data can be obtained.

[0343] The learning model 292 may be implemented by hardware or by a combination of software and hardware.

[0344] Description of symbols

[0345] 100: motion analysis system; 101: camera; 102: camera; 111: subject; 112: subject; 121: object image data; 122: object posture data; 123: object trajectory data; 131: action image data; 132: action posture data; 133: action trajectory data; 200: motion analysis device; 201: processor; 202: memory; 203: auxiliary storage device; 204: communication device; 205: input / output interface; 209: processing circuit; 210: determination unit; 211: object image receiving unit; 212: object posture detection unit Detection unit; 213: Object trajectory extraction unit; 214: Action determination unit; 215: Result output unit; 220: Registration unit; 221: Action label acceptance unit; 222: Action image acceptance unit; 223: Action posture detection unit; 224: Action registration unit; 225: Temporary posture detection unit; 226: Temporary table generation unit; 227: Action posture generation unit; 230: Learning unit; 231: Action label acceptance unit; 232: Action image acceptance unit; 233: Action posture detection unit; 234: Action learning unit; 290: Storage unit; 291: Action database; 292: Learning model.

Claims

1. A motion analysis system, comprising: a subject image receiving unit that receives a plurality of subject image data obtained by continuously photographing a subject performing a series of actions; a subject posture detection unit for detecting a posture of the subject person reflected in the subject image of each subject image data, and generating subject posture data representing the detected posture; an object trajectory extraction unit that extracts, from a plurality of object posture data corresponding to the plurality of object image data, two or more object posture data including object posture data at an object time and object posture data at at least one of a previous time continuous with the object time and a subsequent time continuous with the object time, as object trajectory data indicating a transition of the posture of the object person; and an action determination unit, which obtains an action label for identifying the action of the subject from an action database according to the object trajectory data, In the action database, multiple moments, multiple action posture data and multiple action tags are associated with each other. Each action tag identifies the action, Each action posture data represents the transition of the posture of a person performing the action identified by the corresponding action tag. Each action trajectory data is composed of two or more action posture data. The object trajectory extraction unit determines an object time group including the object time, extracts two or more object posture data corresponding to two or more shooting times constituting the object time group as the object trajectory data, The action determination unit determines a comparison moment group including a comparison moment at each comparison moment, extracts two or more action posture data corresponding to the two or more comparison moments constituting the determined comparison moment group from the action database, compares the action trajectory data constituted by the extracted two or more action posture data with the object trajectory data, selects the action trajectory data consistent with the object trajectory data based on a comparison result between each action trajectory data and the object trajectory data, and extracts an action label corresponding to the comparison moment corresponding to the selected action trajectory data as an action label consistent with the object trajectory data.

2. The motion analysis system according to claim 1, wherein: The motion analysis system comprises: an action label receiving unit configured to receive the plurality of action labels to which the plurality of time points are attached; a motion image receiving unit that receives a plurality of motion image data obtained by continuously photographing an actor performing a series of motions and to which the plurality of time points are added; an action posture detection unit for detecting a posture of the actor shown in the action image of each action image data, and generating action posture data representing the detected posture; as well as The motion registration unit registers the plurality of time points, the plurality of motion posture data corresponding to the plurality of motion image data, and the plurality of motion labels in the motion database in association with each other.

3. The motion analysis system according to claim 1, wherein: The motion analysis system comprises: an action label receiving unit that receives a plurality of action label groups, wherein the action label groups are a plurality of action labels to which a plurality of time points are attached; a motion image receiving unit that receives a plurality of motion image data groups, the motion image data groups being a plurality of motion image data to which a plurality of time points are attached, obtained by photographing an actor performing a series of motions; a temporary posture detection unit that detects the posture of the actor reflected in the motion image of each motion image data in the motion image data group for each received motion image data group, and generates temporary posture data representing the detected posture; a temporary table generating unit for generating a plurality of temporary tables for associating the time groups, the temporary posture data groups corresponding to the action image data groups, and the action label groups with each other; an action posture generating unit, which extracts two or more temporary posture data corresponding to two or more action tags identifying the same action from a plurality of temporary tables, i.e., temporary trajectory data, according to each action included in the series of actions, and generates two or more action posture data corresponding to two or more moments based on the plurality of temporary trajectory data extracted from the plurality of temporary tables; and The action registration unit registers, for each action included in the series of actions, two or more time points, two or more action posture data, and two or more action labels for identifying the same action in a action database in association with each other.

4. The motion analysis system according to any one of claims 1 to 3, wherein: Each object posture data represents the position of each of a plurality of joints of the object person.

5. A motion analysis system, comprising: a subject image receiving unit that receives a plurality of subject image data obtained by continuously photographing a subject performing a series of actions; a subject posture detection unit for detecting a posture of the subject person reflected in the subject image of each subject image data, and generating subject posture data representing the detected posture; an object trajectory extraction unit that extracts, from a plurality of object posture data corresponding to the plurality of object image data, two or more object posture data including object posture data at an object time and object posture data at at least one of a previous time continuous with the object time and a subsequent time continuous with the object time, as object trajectory data representing a transition of the posture of the object person; A motion identification unit that inputs the subject trajectory data into a learning model that outputs a motion label identifying a motion in response to input of motion trajectory data representing a transition of a person's posture, and obtains a motion label identifying a motion of the subject person from the learning model; An action label receiving unit that receives a plurality of action labels to which a plurality of time points are attached; a motion image receiving unit that receives a plurality of motion image data obtained by continuously photographing an actor performing a series of motions and to which the plurality of time points are added; an action posture detection unit for detecting a posture of the actor shown in the action image of each action image data, and generating action posture data representing the detected posture; as well as The motion learning unit generates the learning model by learning a set of motion posture data and motion labels as learning data at each time.

6. The motion analysis system according to claim 5, wherein: Each object posture data represents the position of each of a plurality of joints of the object person.

7. A computer-readable recording medium having a motion analysis program recorded thereon, the motion analysis program being configured to cause a computer to execute the following processing: A subject image receiving process for receiving a plurality of subject image data obtained by continuously photographing a subject who performs a series of actions; object posture detection processing, detecting the posture of the subject person reflected in the object image of each object image data, and generating object posture data representing the detected posture; object trajectory extraction processing, extracting, from a plurality of object posture data corresponding to the plurality of object image data, two or more object posture data including object posture data at an object time and object posture data at at least one of a previous time continuous with the object time and a subsequent time continuous with the object time, as object trajectory data representing a posture transition of the object person; and Action determination processing, obtaining an action label for identifying the action of the subject from an action database according to the object trajectory data, wherein: In the action database, multiple moments, multiple action posture data and multiple action tags are associated with each other. Each action tag identifies the action, Each action posture data represents the transition of the posture of a person performing the action identified by the corresponding action tag. Each action trajectory data is composed of two or more action posture data. In the object trajectory extraction process, an object time group including the object time is determined, and two or more object posture data corresponding to two or more shooting times constituting the object time group are extracted as the object trajectory data. In the action determination processing, a comparison moment group including a comparison moment is determined according to each comparison moment, and two or more action posture data corresponding to the two or more comparison moments constituting the determined comparison moment group are extracted from the action database, and the action trajectory data constituted by the extracted two or more action posture data are compared with the object trajectory data, and based on the comparison result of each action trajectory data with the object trajectory data, the action trajectory data consistent with the object trajectory data is selected, and the action label corresponding to the comparison moment corresponding to the selected action trajectory data is extracted as the action label consistent with the object trajectory data.

8. A computer-readable recording medium having a motion analysis program recorded thereon, the motion analysis program being configured to cause a computer to execute the following processing: A subject image receiving process for receiving a plurality of subject image data obtained by continuously photographing a subject who performs a series of actions; object posture detection processing, detecting the posture of the subject person reflected in the object image of each object image data, and generating object posture data representing the detected posture; object trajectory extraction processing, extracting, from a plurality of object posture data corresponding to the plurality of object image data, two or more object posture data including object posture data at an object moment and object posture data at at least one of a previous moment continuous with the object moment and a subsequent moment continuous with the object moment, as object trajectory data representing a posture transition of the object person; Action identification processing, inputting the subject trajectory data to a learning model that outputs an action label identifying an action in response to an input of action trajectory data representing a transition of a person's posture, and obtaining an action label identifying an action of the subject person from the learning model; Action tag acceptance processing, accepting multiple action tags with multiple moments attached; Action image receiving processing for receiving a plurality of action image data to which the plurality of time points are attached, obtained by continuously photographing an actor performing a series of actions; Action posture detection processing, detecting the posture of the actor reflected in the action image of each action image data, and generating action posture data representing the detected posture; as well as The motion learning process generates the learning model by learning a set of motion posture data and a motion label as learning data at each time.

Images