Action recognition method

Abstract

To appropriately recognize an action of a worker as a target of action recognition.SOLUTION: In an action recognition method, a computer acquires skeletal information for a worker as a time series from video data (S12). The computer acquires position coordinates of each part of the worker included in the acquired skeletal information (S21). When the computer acquires skeletal information for a plurality of workers, the computer identifies a worker to be a target of action recognition based on the position coordinates for each worker (S24). The computer recognizes an action of the worker based on the time series of the skeletal information for the identified worker (S43). The action recognition method may also recognize an action of a worker using a trained model that has been trained in advance by machine learning.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an action recognition method. [Background technology]

[0002] The behavior recognition system of Patent Document 1 includes a behavior recognition device and a camera. The behavior recognition device acquires video data from the camera. The behavior recognition device also acquires skeletal information of a worker in time series from the video data. The behavior recognition device then recognizes the behavior of the worker based on the acquired skeletal information in time series. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2022-003434 A Summary of the Invention [Problem to be solved by the invention]

[0004] In a behavior recognition system such as that of Patent Document 1, multiple workers may appear in the video data acquired from the camera. In this case, the behavior recognition device may acquire skeletal information of the multiple workers. Since the behavior recognition device of Patent Document 1 does not focus on the case where skeletal information of multiple workers is acquired as described above, there is a risk that the behavior of a worker that should be the target of behavior recognition may not be recognized as expected. [Means for solving the problem]

[0005] A behavior recognition method for solving the above problem includes a computer acquiring skeletal information of a worker in chronological order from video data, acquiring position coordinates of each part of the worker contained in the acquired skeletal information, and, when the skeletal information of a plurality of workers is acquired, identifying the worker to be targeted for behavior recognition based on the position coordinates of each worker, and recognizing the behavior of the worker based on the chronological order of the skeletal information of the identified worker. Effect of the Invention

[0006] According to the above configuration, when skeletal information of multiple workers is acquired, the worker to be the target of behavior recognition is identified. Then, based on the time series of the skeletal information of the identified worker, the behavior of the target worker can be appropriately recognized. In other words, it is possible to reduce the influence of the behavior of a worker who is not the target of behavior recognition on the recognition of the behavior of the actor who is the target of behavior recognition. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a schematic configuration diagram of the behavior recognition system. [Diagram 2] FIG. 2 is a flowchart showing the recognition control. [Diagram 3] FIG. 3 is an explanatory diagram regarding skeleton information of a worker. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] <Outline of behavior recognition system> Hereinafter, one embodiment of the present invention will be described with reference to Figures 1 to 3. First, a schematic configuration of a behavior recognition system 10 will be described.

[0009] As shown in FIG. 1, the behavior recognition system 10 includes a computer 20, an input device 30, a camera 40, and a display 50. The input device 30 includes, for example, a keyboard and a pointing device. The camera 40 detects video data DP, which is an image of a subject. In this embodiment, the camera 40 is located near the location where a worker who is the subject of behavior recognition works. The camera 40 is also located at a location where the appearance of the worker who is the subject of behavior recognition can be captured from the front of the worker. The camera 40 faces the location where the worker who is the subject of behavior recognition works. An example of a worker who is the subject of behavior recognition is a worker who performs assembly work for automobile parts. The display 50 is capable of displaying various information.

[0010] The computer 20 includes an execution device 21 and a storage device 22. An example of the execution device 21 is a CPU. The storage device 22 includes a ROM that can only be read, a volatile RAM that can be read and written, and a non-volatile storage that can be read and written. The storage device 22 stores various programs and various data in advance. The storage device 22 also stores a behavior recognition program 22A in advance as one of the various programs. The storage device 22 also stores mapping data 22B in advance as one of the various data. The mapping data 22B is a mapping that is a relationship between a predetermined input variable and an output variable that indicates the behavior of the worker, described in a format executable by the execution device 21. The mapping described in the mapping data 22B is learned in advance by machine learning. A specific description of the mapping data 22B will be given later. The execution device 21 executes the behavior recognition program 22A stored in the storage device 22 to realize various processes in the behavior recognition method. An example of the computer 20 is a so-called personal computer.

[0011] The computer 20 acquires signals from the input device 30 and the camera 40. That is, the computer 20 acquires video data DP from the camera 40. At this time, the execution unit 21 stores the video data DP in the storage device 22. The computer 20 also outputs a control signal to the display 50, thereby displaying various information on the display 50.

[0012] <Perception Control> Next, the recognition control executed by the computer 20 will be described with reference to Fig. 2. This recognition control is a control for recognizing the behavior of a worker who is a target of behavior recognition. In this embodiment, the computer 20 executes the recognition control at each predetermined control cycle.

[0013] 2, when the execution device 21 of the computer 20 starts recognition control, it executes the process of step S11. In step S11, the execution device 21 acquires from the storage device 22 video data DP from the processing point of step S11 to a predetermined specified period PR. An example of the specified period PR is several seconds to a dozen seconds. After step S11, the execution device 21 advances the process to step S12.

[0014] In step S12, the execution device 21 acquires skeletal information IS for the workers included in the video data DP in time series from the video data DP for the specified period PR. Specifically, the execution device 21 acquires a plurality of frames included in the video data DP for the specified period PR. Then, the execution device 21 acquires skeletal information IS for the workers included in each frame by performing a process called skeleton detection or the like. At this time, the execution device 21 acquires skeletal information IS for all workers included in the frames. Note that, for example, the technology described in JP 2022-003434 A and JP 2019-071008 A can be used as the process for acquiring the skeletal information IS. After step S12, the execution device 21 advances the process to step S21.

[0015] In step S21, the execution device 21 acquires position coordinates PC of the body parts included in the skeletal information IS of the worker for the time series of the skeletal information IS of the worker, that is, for the skeletal information IS included in a plurality of frames. As shown in FIG. 3, the body parts of the worker included in the skeletal information IS are a total of nine body parts, including the head A, right shoulder B, right elbow C, right hand D, right hip E, left shoulder F, left elbow G, left hand H, and left hip I. The position coordinates PC are expressed in an orthogonal coordinate system of X and Y axes. In this embodiment, the X axis is an axis perpendicular to the vertical direction. The Y axis is an axis along the vertical direction. The execution device 21 acquires position coordinates PC of the body parts included in the skeletal information IS of each worker for all workers. As shown in FIG. 2, after step S21, the execution device 21 advances the process to step S22.

[0016] As shown in FIG. 2, in step S22, the execution unit 21 calculates the worker speed SW, which is the moving speed of the worker corresponding to the position coordinate PC, based on the time series of the position coordinate PC. For example, the execution unit 21 calculates the worker speed SW based on the absolute value of the amount of change per unit time of the value indicating the position on the X-axis of the position coordinate PC corresponding to the head A. Here, it is assumed that there are a total of N sets of data as the time series of the skeletal information IS of a certain worker acquired in step S12. Note that "N" is an integer of 2 or more. In addition, the N pieces of data are, in order from oldest to newest, the data at the first time point, the data at the second time point, ..., the data at the Nth time point. In this case, in step S22, the execution unit 21 calculates the absolute value of the difference between the values ​​indicating the position on the X-axis of the position coordinate PC corresponding to the head A for the first time point and the second time point as the absolute value of the amount of change per unit time for the first time point and the second time point. Similarly, the execution unit 21 calculates the absolute value of the difference between the values ​​indicating the positions on the X-axis of the position coordinate PC corresponding to the head A for the second time point and the third time point as the absolute value of the amount of fluctuation per unit time for the second time point and the third time point. Also, the execution unit 21 calculates the absolute value of the difference between the values ​​indicating the positions on the X-axis of the position coordinate PC corresponding to the head A for the N-1th time point and the Nth time point as the absolute value of the amount of fluctuation per unit time for the N-1th time point and the Nth time point. Then, the execution unit 21 calculates the average value of the calculated absolute values ​​of the amount of fluctuation per unit time as the worker speed SW. For example, it is assumed that the position coordinate PC corresponding to the head A could not be acquired at the N-1th time point. In this case, the absolute value of the amount of fluctuation per unit time from the N-2th time point to the N-1th time point and the absolute value of the amount of fluctuation per unit time from the N-1st time point to the Nth time point cannot be calculated. Therefore, in the above case, the absolute value of the amount of fluctuation per unit time from the N-2th time point to the N-1th time point and the absolute value of the amount of fluctuation per unit time from the N-1th time point to the Nth time point are not used in calculating the average value. As a result, for example, the worker speed SW of a worker who is walking tends to be higher than the worker speed SW of a worker who is working at a certain point. Note that the execution unit 21 calculates the worker speed SW of each worker for all workers. After step S22, the execution unit 21 advances the process to step S23.

[0017] In step S23, the execution device 21 calculates the accumulated period PA based on the time series of the position coordinate PC. Specifically, the execution device 21 calculates the accumulated value of the period during which the value indicating the position on the X-axis of the position coordinate PC corresponding to the right shoulder B is smaller than the value indicating the position on the X-axis of the position coordinate PC corresponding to the left shoulder F, as the accumulated period PA. Therefore, as shown in FIG. 3, the longer the state in which the worker's body faces the camera 40, the larger the accumulated period PA. In this embodiment, the accumulated period PA is a value indicating the positional relationship between the position coordinate PC corresponding to the right shoulder B and the position coordinate PC corresponding to the left shoulder F. As shown in FIG. 2, after step S23, the execution device 21 advances the process to step S24.

[0018] As shown in FIG. 2, in step S24, the executing device 21 identifies a worker to be the target of behavior recognition based on the position coordinates PC of each worker. For example, it is assumed that the executing device 21 acquires skeletal information IS of multiple workers in step S12. In this case, the executing device 21 extracts workers from the multiple workers, excluding workers whose worker speed SW is higher than a predetermined specified worker speed SWA. The specified worker speed SWA is a threshold value that is predetermined to exclude workers who are moving by walking or the like. The specified worker speed SWA can be determined through experiments and simulations. Furthermore, the executing device 21 identifies a worker with the longest cumulative period PA among the extracted workers as a worker to be the target of behavior recognition. In other words, the executing device 21 identifies a worker whose body faces the camera 40 for the longest period among the multiple workers, excluding workers who are moving, as a worker to be the target of behavior recognition. After step S24, the executing device 21 advances the process to step S31.

[0019] In step S31, the execution device 21 calculates the right hand speed SSD, which is the moving speed of the right hand D, based on the position coordinate PC corresponding to the right hand D included in the skeletal information IS of the worker identified in step S24. Here, it is assumed that there are a total of N sets of data as the time series of the skeletal information IS of the identified worker. It is to be noted that "N" is an integer of 2 or more. In addition, the N pieces of data are arranged in order from the oldest to the oldest, namely, the data at the first time point, the data at the second time point, ..., the data at the Nth time point. In this case, the execution device 21 calculates the right hand speed SSD, for example, as follows. The execution device 21 determines the Euclidean distance between the position coordinate PC corresponding to the right hand D at the first time point and the position coordinate PC corresponding to the right hand D at the second time point to be the right hand speed SSD at the second time point. Similarly, the execution device 21 determines the Euclidean distance between the position coordinate PC corresponding to the right hand D at the second time point and the position coordinate PC corresponding to the right hand D at the third time point to be the right hand speed SSD at the third time point. In the same manner as above, the execution unit 21 calculates the right hand speed SSD up to the Nth point in time.

[0020] The execution device 21 also calculates the left hand velocity SSH, which is the moving speed of the left hand H, based on the position coordinate PC corresponding to the left hand H included in the skeletal information IS of the worker identified in step S24. For example, the execution device 21 determines the Euclidean distance between the position coordinate PC corresponding to the left hand H at the first time point and the position coordinate PC corresponding to the left hand H at the second time point as the left hand velocity SSH at the second time point. Similarly, the execution device 21 determines the Euclidean distance between the position coordinate PC corresponding to the left hand H at the second time point and the position coordinate PC corresponding to the left hand H at the third time point as the left hand velocity SSH at the third time point. In the same manner as above, the execution device 21 calculates the left hand velocity SSH up to the Nth time point. In this embodiment, each of the right hand velocity SSD and the left hand velocity SSH corresponds to a specific part velocity, which is the moving speed of a predetermined specific part. After step S31, the execution device 21 advances the process to step S32.

[0021] In step S32, the execution device 21 specifies the right hand noise data DND based on the right hand speed SSD calculated in step S31. Specifically, the execution device 21 extracts the right hand speed SSD higher than a predetermined specified right hand speed from the right hand speed SSD from the second time point to the Nth time point. Then, the execution device 21 specifies the position coordinate PC corresponding to the extracted right hand speed SSD as the right hand noise data DND. For example, if the extracted right hand speed SSD is the right hand speed SSD at the Nth time point, the execution device 21 specifies the position coordinate PC corresponding to the right hand D at the Nth time point as the right hand noise data DND. That is, the right hand noise data DND is the position coordinate PC corresponding to the right hand speed SSD higher than a predetermined specified right hand speed from the time series of the position coordinate PC corresponding to the right hand D. The specified right hand speed is a threshold value for extracting the right hand speed SSD exceeding the assumption in the work process performed by the worker.

[0022] The execution device 21 also specifies the left hand noise data DNH based on the left hand speed SSH calculated in step S31. Specifically, the execution device 21 extracts the left hand speed SSH higher than a predetermined specified left hand speed from the left hand speeds SSH from the second time point to the Nth time point. Then, the execution device 21 specifies the position coordinate PC corresponding to the extracted left hand speed SSH as the left hand noise data DNH. For example, when the extracted left hand speed SSH is the left hand speed SSH at the Nth time point, the execution device 21 specifies the position coordinate PC corresponding to the left hand H at the Nth time point as the left hand noise data DNH. That is, the left hand noise data DNH is the position coordinate PC corresponding to the left hand speed SSH higher than a predetermined specified left hand speed from the time series of the position coordinate PC corresponding to the left hand H. The specified left hand speed is a threshold value for extracting the left hand speed SSH that exceeds the assumption in the work process performed by the worker. In this embodiment, each of the right hand noise data DND and the left hand noise data DNH is noise data. Moreover, each of the prescribed right hand speed and the prescribed left hand speed is a prescribed part speed. After step S32, the execution unit 21 advances the process to step S33.

[0023] In step S33, the execution device 21 generates right hand alternative data DAD based on the position coordinate PC corresponding to the right hand D. For example, it is assumed that the right hand noise data DND identified in step S32 is the position coordinate PC corresponding to the right hand D at the Nth time point. In this case, the execution device 21 generates the average value of the position coordinate PC corresponding to the right hand D at the N-1th time point and the position coordinate PC corresponding to the right hand D at the N-2th time point as the right hand alternative data DAD at the Nth time point. Therefore, the right hand alternative data DAD corresponds to the position coordinate PC corresponding to the right hand D whose right hand speed SSD is equal to or lower than a predetermined specified right hand speed. For example, if the right hand noise data DND identified in step S32 is the position coordinate PC corresponding to the right hand D at the second time point, the execution device 21 generates the same as the position coordinate PC corresponding to the right hand D at the first time point as the right hand alternative data DAD at the second time point.

[0024] Further, the execution device 21 generates the left hand alternative data DAH based on the position coordinate PC corresponding to the left hand H. For example, it is assumed that the left hand noise data DNH specified in step S32 is the position coordinate PC corresponding to the left hand H at the Nth time point. In this case, the execution device 21 generates the average value of the position coordinate PC corresponding to the left hand H at the N-1th time point and the position coordinate PC corresponding to the left hand H at the N-2th time point as the left hand alternative data DAH at the Nth time point. Therefore, the left hand alternative data DAH corresponds to the position coordinate PC corresponding to the left hand H whose left hand speed SSH is equal to or lower than a predetermined specified left hand speed. For example, when the left hand noise data DNH specified in step S32 is the position coordinate PC corresponding to the left hand H at the second time point, the execution device 21 generates the same as the position coordinate PC corresponding to the left hand H at the first time point as the left hand alternative data DAH at the second time point. In this embodiment, each of the right hand alternative data DAD and the left hand alternative data DAH is alternative data. After step S33, the execution unit 21 advances the process to step S34.

[0025] In step S34, the execution device 21 generates the recognition target information IR based on the skeletal information IS, the right hand alternative data DAD, and the left hand alternative data DAH of the worker identified in step S24. Specifically, the execution device 21 generates information by removing the right hand noise data DND from the skeletal information IS of the worker identified in step S24 and adding the right hand alternative data DAD instead of the right hand noise data DND. Furthermore, the execution device 21 generates information by removing the left hand noise data DNH from the information to which the right hand alternative data DAD has been added and adding the left hand alternative data DAH instead of the left hand noise data DNH as the recognition target information IR. After step S34, the execution device 21 advances the process to step S41.

[0026] In step S41, the execution device 21 generates the recognition target information IR as an input variable to the mapping described in the mapping data 22B. Here, it is assumed that there are a total of N sets of data as a time series of the skeletal information IS included in the recognition target information IR. Here, "N" is an integer of 2 or more. Among the N pieces of data, the oldest data is the data at the first time point, the data at the second time point, ..., the data at the Nth time point. As described above, the skeletal information IS included in the recognition target information IR includes a total of nine parts. The position coordinates PC corresponding to the parts include a value indicating a position on the X axis and a value indicating a position on the Y axis. Therefore, each piece of skeletal information IS includes a total of 18 kinds of numerical values. In step S41, the execution device 21 sequentially assigns the 18 kinds of numerical values ​​included in the skeletal information IS at the first time point to the input variables x(1) to x(18) one by one. Similarly, the execution device 21 assigns the 18 types of numerical values ​​contained in the skeletal information IS at the second time point to the input variables x(19) to x(36) one by one in order. By generating input variables in the same manner as above, the execution device 21 generates input variables x(1) to x(18×N). In the following, the number of types of input variables generated in step S41 will be referred to as "Z". After step S41, the execution device 21 advances the process to step S42.

[0027] In step S42, the execution unit 21 calculates the value of the output variable y(i) by inputting the input variables x(1) to x(Z) and the input variable x(0) as a bias parameter to the mapping described in the mapping data 22B.

[0028] An example of the mapping described in the mapping data 22B is a function approximator, which is a fully connected forward propagation type neural network with one intermediate layer. Specifically, in the mapping, the input variable x(1) to the input variable x(Z) and the input variable x(0) as a bias parameter are transformed by a linear mapping defined by coefficients wFjk (j=1 to m, k=0 to Z), and each of the "m" values ​​is substituted into the activation function f. As a result, the values ​​of the nodes in the intermediate layer are determined. Also, the values ​​of the nodes in the intermediate layer transformed by the linear mapping defined by coefficients wSij (i=1 to P) are substituted into the activation function g, and the output variables y(1) to y(P) are determined. The number of types of output variables in step S42 is described as "P". Here, the number of types of output variables, "P", is the same as the number of types of work processes performed by the worker. That is, for example, if the work processes performed by the worker include the 1st process to the 10th process, "P" is "10." In this embodiment, an example of the activation function f is the ReLU function. Also, an example of the activation function g is the softmax function. Therefore, the output variables y(1) to y(P) indicate the probability that the corresponding work processes are being performed.

[0029] The mapping described in the mapping data 22B is learned in advance, for example, as follows. Here, for example, it is assumed that the work steps performed by the worker are the first step to the tenth step. In this case, first, in learning to recognize the first step, the input variable x(1) to the input variable x(Z) are generated in the same manner as above while the worker performs the work of the first step in the correct procedure. At this time, the generated input variable x(1) to the input variable x(Z) are used for learning. In addition, the first step is set as the correct label. That is, the value of the output variable y(1) is set to "1", while the values ​​of the output variable y(2) to the output variable y(P) are set to "0". By inputting such data into the mapping, the mapping is learned by machine learning. Similarly, in learning to recognize the second step, the input variable x(1) to the input variable x(Z) are generated in the same manner as above while the worker performs the work of the second step in the correct procedure. At this time, the generated input variable x(1) to input variable x(Z) are used as data to be used for learning. Also, the second step is used as the correct label. That is, the value of the output variable y(2) is set to "1", while the values ​​of the output variable y(1) and the output variables y(3) to y(P) are set to "0". By inputting such data into the mapping, the mapping is learned by machine learning. In the same manner as above, the mapping is also learned by machine learning for the third step to the tenth step. After step S42, the execution unit 21 advances the process to step S43.

[0030] In step S43, the execution device 21 identifies the work process performed by the worker to be the target of behavior recognition based on the output variables y(1) to y(P). Specifically, the execution device 21 identifies the maximum value among the output variables y(1) to y(P). Then, the execution device 21 identifies the work process corresponding to the maximum value as the work process performed by the worker. For example, when the output variable y(1) is the maximum value, the execution device 21 identifies the first process as the work process performed by the worker. The execution device 21 stores the identified work process in the storage device 22. In this embodiment, the process of step S43 is a process of recognizing the behavior performed by the worker to be the target of behavior recognition. As described above, the input variables to the mapping described in the mapping data 22B are based on the time series of the skeleton information IS included in the recognition target information IR, so the execution device 21 recognizes the behavior performed by the worker to be the target of behavior recognition based on the time series of the skeleton information IS. After step S43, the execution device 21 advances the process to step S51.

[0031] In step S51, the execution device 21 determines whether or not a predetermined normal condition is satisfied. In this embodiment, the normal condition is that either of the following requirements (1) and (2) is satisfied.

[0032] Requirement (1): The work process identified in this recognition control is the same as the work process identified in the previous recognition control. Requirement (2): The work process identified in the current recognition control is the next work process following the work process identified in the previous recognition control.

[0033] In step S51, if the execution unit 21 determines that the normal condition is satisfied (S51: YES), the execution unit 21 advances the process to step S61. In step S61, the execution unit 21 determines that the work process of the worker is correct. After step S61, the execution unit 21 ends the current recognition control. Then, the execution unit 21 advances the process to step S11 again.

[0034] On the other hand, if the execution unit 21 determines in step S51 that the normal condition is not satisfied (S51: NO), the execution unit 21 proceeds to step S71. Note that, for example, if the work process identified in the previous recognition control is the first process and the work process identified in the current recognition control is the third process, the execution unit 21 determines that the normal condition is not satisfied.

[0035] In step S71, the execution unit 21 determines that the work process of the worker is incorrect. After step S71, the execution unit 21 advances the process to step S72. In step S72, the execution unit 21 outputs a control signal to the display 50, thereby displaying on the display 50 that the work process of the worker is incorrect. After step S72, the execution unit 21 ends the current recognition control. Then, the execution unit 21 advances the process to step S11 again.

[0036] <Action of this embodiment> As shown in FIG. 2, in step S12, the execution device 21 acquires, from the video data DP of a specified period PR, skeletal information IS of the worker included in the video data DP in time series. In addition, in step S21, the execution device 21 acquires position coordinates PC of parts included in each piece of skeletal information IS. In step S22, the execution device 21 calculates a worker speed SW, which is the moving speed of the worker corresponding to the position coordinate PC, based on the time series of the position coordinate PC. Furthermore, in step S23, the execution device 21 calculates an accumulated period PA indicating the positional relationship between the position coordinate PC corresponding to the right shoulder B and the position coordinate PC corresponding to the left shoulder F, based on the time series of the position coordinate PC. Then, in step S24, the execution device 21 identifies a worker to be a target of behavior recognition based on the position coordinate PC of each worker, specifically based on the worker speed SW and the accumulated period PA.

[0037] <Effects of this embodiment> (1) Even if the execution device 21 acquires skeletal information IS of multiple workers in step S12, there should be differences in the worker speed SW and cumulative period PA of each worker depending on whether the worker is a target of behavior recognition or not. Therefore, based on the time series of position coordinates PC included in the skeletal information IS of each worker, it is possible to identify the worker to be the target of behavior recognition in step S24. This makes it possible to appropriately recognize the work process performed by the worker to be the target of behavior recognition in steps S41 to S43. In other words, it is possible to prevent the behavior of a worker who is not a target of behavior recognition from affecting the recognition of the behavior of a worker who is a target of behavior recognition.

[0038] (2) In step S12, the execution unit 21 may acquire not only skeletal information IS of a target worker working at a certain point, but also skeletal information IS of a worker who is moving by walking or the like near the target worker. In this case, the worker speed SW of the moving worker will be higher than the worker speed SW of a worker working at a certain point without moving.

[0039] In this regard, in step S24, the execution device 21 specifies workers to be the targets of behavior recognition after excluding workers whose worker speeds SW are higher than a predetermined specified worker speed SWA from among the multiple workers. In this way, by excluding moving workers, it is possible to more reliably specify workers to be the targets of behavior recognition.

[0040] (3) If the positional relationship between the worker whose behavior is to be recognized and the camera 40 used to acquire the video data DP of that worker is the same, there is a high possibility that the body orientation of the worker whose behavior is to be recognized will be approximately the same on the video data DP.

[0041] In this regard, in step S23, the execution device 21 calculates an accumulated period PA based on the time series of the position coordinate PC. Here, the accumulated period PA is an accumulated value of a period during which the value indicating the position on the X-axis of the position coordinate PC corresponding to the right shoulder B is smaller than the value indicating the position on the X-axis of the position coordinate PC corresponding to the left shoulder F during the specified period PR. Then, in step S24, the execution device 21 identifies, among the multiple workers, the worker with the longest accumulated period PA as the worker to be targeted for behavior recognition. This makes it possible to more reliably identify workers whose body orientation is approximately the same on the video data DP, i.e., workers to be targeted for behavior recognition, based on the positional relationship between the position coordinate PC corresponding to the right shoulder B and the position coordinate PC corresponding to the left shoulder F.

[0042] (4) A worker who is the subject of behavior recognition may perform an unnecessary action, such as touching his / her head A with his / her right hand D. If the work steps performed by the worker are recognized in steps S41 to S43 based on skeletal information IS that includes the unnecessary action as described above, there is a possibility that the work steps cannot be recognized appropriately.

[0043] In this regard, in step S32, the execution device 21 specifies, as right hand noise data DND, a position coordinate PC corresponding to a right hand speed SSD higher than a predetermined specified right hand speed, from the time series of position coordinates PC corresponding to the right hand D. In addition, in step S34, the execution device 21 generates recognition target information IR after removing the right hand noise data DND from the skeletal information IS of the worker specified in step S24. Then, in steps S41 to S43, the execution device 21 recognizes the work process performed by the worker based on the recognition target information IR. Thereby, by removing unnecessary actions of the worker who is the target of behavior recognition, the work process performed by the worker can be more appropriately recognized.

[0044] (5) In step S33, the execution unit 21 generates right hand alternative data DAD based on position coordinates PC corresponding to the right hand D. This right hand alternative data DAD is the position coordinates PC corresponding to the right hand D whose right hand speed SSD is equal to or lower than a predetermined specified right hand speed. Then, in step S34, the execution unit 21 generates recognition target information IR using information obtained by removing the right hand noise data DND from the skeletal information IS of the worker identified in step S24 and adding the right hand alternative data DAD instead of the right hand noise data DND. This makes it possible to prevent a situation in which the work process performed by the worker cannot be recognized due to the removal of the right hand noise data DND.

[0045] <Example of change> This embodiment can be modified as follows: This embodiment and the following modifications can be combined with each other to the extent that there is no technical contradiction.

[0046] In the above embodiment, the recognition control may be changed. For example, in step S21, the body parts of the worker for which position coordinates PC are acquired may be changed. As a specific example, the execution device 21 may acquire position coordinates PC of other body parts of the worker from the skeletal information IS instead of or in addition to the head A, right shoulder B, right elbow C, right hand D, right hip E, left shoulder F, left elbow G, left hand H, and left hip I. Also, as a specific example, the execution device 21 may acquire position coordinates PC of eight or less body parts, or ten or more body parts from the skeletal information IS, instead of a total of nine body parts.

[0047] For example, the way of calculating the worker speed SW in step S22 may be changed. As a specific example, the execution device 21 may calculate the worker speed SW based on the absolute value of the amount of change per unit time of the value indicating the position on the X-axis of the position coordinate PC corresponding to the right hip E, instead of or in addition to the position coordinate PC corresponding to the head A.

[0048] For example, in step S23, the method of calculating the cumulative period PA may be changed. As a specific example, the camera 40 is positioned in a location where it can capture an image of the worker, whose behavior is to be recognized, from behind the worker. In this case, the execution device 21 may calculate, as the cumulative period PA, the cumulative value of the period during which the value indicating the position on the X-axis of the position coordinate PC corresponding to the right shoulder B is greater than the value indicating the position on the X-axis of the position coordinate PC corresponding to the left shoulder F, during the specified period PR.

[0049] For example, in step S24, the way in which the worker to be the target of behavior recognition is specified may be changed. As a specific example, the execution device 21 may specify, among multiple workers, the worker with the longest cumulative period PA as the worker to be the target of behavior recognition, regardless of the worker speed SW. In this case, the process of step S22 can be omitted.

[0050] As a specific example, the execution device 21 may specify, among multiple workers, the worker with the lowest worker speed SW as the worker to be targeted for behavior recognition, regardless of the cumulative period PA. In this case, the process of step S23 can be omitted.

[0051] For example, in step S31, the execution device 21 may calculate the speed of another body part as the specific body part speed in addition to or instead of the right hand speed SSD and the left hand speed SSH. As a specific example, the execution device 21 may calculate the right elbow speed, which is the moving speed of the right elbow C, based on the position coordinate PC corresponding to the right elbow C included in the skeletal information IS of the worker identified in step S24. In this case, in step S32, the execution device 21 may specify noise data based on the right elbow speed calculated in step S31. That is, the noise data specified in step S32 may be changed in accordance with the specific body part speed calculated in step S31. Similarly, the alternative data generated in step S33 may be changed in accordance with the specific body part speed calculated in step S31.

[0052] For example, in step S34, the way of generating the recognition target information IR may be changed. As a specific example, the executing device 21 may generate the recognition target information IR by removing the right hand noise data DND from the skeletal information IS of the worker identified in step S24, but without adding the right hand alternative data DAD. In this case, for example, when the right hand noise data DND at the Nth time point is removed from the skeletal information IS at the Nth time point, the executing device 21 may generate information from the skeletal information IS at the 1st time point to the skeletal information IS at the N-1th time point as the recognition target information IR. That is, the executing device 21 does not necessarily have to add the right hand alternative data DAD. Similarly, the executing device 21 does not necessarily have to add the left hand alternative data DAH. In this case, the processing of step S33 can be omitted.

[0053] As a specific example, the execution device 21 may generate the skeletal information IS of the worker identified in step S24 as the recognition target information IR. That is, the execution device 21 does not necessarily have to remove the right hand noise data DND. Similarly, the execution device 21 does not necessarily have to remove the left hand noise data DNH. In this case, the processing of step S32 can be omitted.

[0054] For example, the manner of notification may be changed in step S72. As a specific example, the execution device 21 may execute a notification by generating a sound from a speaker, instead of or in addition to the notification displayed on the display 50.

[0055] In the above embodiment, the configuration of the behavior recognition system 10 may be changed. For example, the activation function of the mapping described in the mapping data 22B is an example, and the activation function of the mapping may be changed.

[0056] For example, the mapping described in the mapping data 22B is exemplified as a neural network with one hidden layer, but the number of hidden layers may be two or more. For example, a fully connected forward propagation type neural network has been exemplified as the neural network of the mapping described in the mapping data 22B, but this is not limiting. As a specific example, the neural network may be a recurrent connection type neural network. Also, for example, the function approximator as the mapping is not limited to a neural network. As a specific example, the mapping may be a regression formula without a hidden layer.

[0057] For example, the computer 20 may be modified. As a specific example, the computer 20 is not limited to a computer that executes software processing by a CPU. As an example, the computer 20 may be equipped with a dedicated hardware circuit such as an ASIC that performs hardware processing of at least a part of what was processed by software in the above embodiment. That is, the computer 20 may have any of the following configurations (a) to (c). (a) A processing device that executes all of the above processing according to a program, and a program storage device such as a ROM that stores the program. (b) A processing device and a program storage device that execute a part of the above processing according to a program, and a dedicated hardware circuit that executes the remaining processing. (c) A dedicated hardware circuit that executes all of the above processing. Here, there may be a plurality of software execution devices equipped with a processing device and a program storage device, and a plurality of dedicated hardware circuits. [Explanation of symbols]

[0058] DP...video data IS...skeletal information 10...behavior recognition system 20...computer 21...execution device 22...storage device 22A...behavior recognition program 22B...map data 30...input device 40...camera 50...display

Claims

1. The computer Acquiring skeletal information of a worker in time series from video data; acquiring position coordinates of each part of the worker included in the acquired skeletal information; When the skeleton information of a plurality of the workers is acquired, the worker to be subjected to behavior recognition is identified based on the position coordinates of each of the workers; Recognizing a behavior of the identified worker based on a time series of the skeletal information of the worker; Run Behavior recognition method.

2. The computer, calculating a worker speed, which is a moving speed of each worker, for each worker based on a time series of the position coordinates of the worker when the skeletal information of the plurality of workers is acquired; Identifying the target worker after excluding the worker whose calculated worker speed is higher than a predetermined specified worker speed; Run The activity recognition method according to claim 1 .

3. The computer, when the skeletal information of a plurality of the workers is acquired, identifying a target worker based on a positional relationship between the position coordinates of a right shoulder included in the acquired skeletal information and the position coordinates of a left shoulder included in the acquired skeletal information; Run The behavior recognition method according to claim 1 .

4. The computer, calculating a specific part speed, which is a moving speed of a predetermined specific part included in the skeletal information, based on a time series of the skeletal information of the identified worker; When the position coordinates of the specific body part, in which the calculated specific body part speed is higher than a predetermined specified body part speed, are regarded as noise data, generating recognition target information by removing the noise data from the time series of the skeleton information of the identified worker; Recognizing an action performed by the identified worker based on the recognition target information of the worker; Run The behavior recognition method according to claim 1 .

5. The computer, generating, as substitute data, the position coordinates of the specific part whose calculated specific part speed is equal to or less than the prescribed part speed; generating information obtained by removing the noise data and adding the alternative data in place of the noise data as the recognition target information; Run The behavior recognition method according to claim 4 .

Images