Movement classification device, movement classification method, and movement classification program

Abstract

To provide a movement classification device, a movement classification method, and a movement classification program capable of classifying a person movement even if the body part of the person in an image is hidden.SOLUTION: A movement classification device 10 includes a person detection part 102 to detect the person from the multiple images generated in chronological order, a preceding image which is the image the target person before the part of the target person body is hidden when the part of the detected target person body is hidden, a trajectory calculation part 104 to calculate at least one trajectory of the hidden fingertip and the hidden toe of the target person using the subsequent images which is the image of the target person after the hidden body part of the target person is re-exposed, a skeleton information generation part 105 to generate the skeleton information representing the target person skeleton from the generated trajectory, and a movement classification part 109 to classify the target person movement using the generated skeleton information.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an action classification device, an action classification method, and an action classification program for classifying human actions. [Background technology]

[0002] Various techniques for classifying the actions of people, such as workers, have been proposed in the past. As one example of such a technique, Patent Document 1 discloses a behavior recognition device that recognizes the actions of a person or other target based on time-series data of feature amounts of the target's actions extracted from data in which the target's actions are recorded in chronological order. This behavior recognition device recognizes actions at each time point by tracking the time-series data of feature amounts in chronological order, and counts a series of identical actions that are not distinguished as one action. After recognizing a predetermined number of actions in chronological order, the device recognizes the action at the current recognition time point based on time-series data of feature amounts corresponding to multiple actions that are in chronological order from a time point going back a predetermined number of actions to the current recognition time point. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2017 / 150211 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the behavior recognition device disclosed in Patent Document 1 has a problem in that it cannot classify the behavior of a person in an image when part of the person's body is hidden.

[0005] The present disclosure provides an action classification device, an action classification method, and an action classification program that are capable of classifying the actions of a person appearing in an image even if part of the person's body is hidden. [Means for solving the problem]

[0006] The motion classification device according to the present disclosure comprises: a person detection unit that detects a person from a plurality of images generated in time series; a trajectory calculation unit that calculates the trajectory of at least one of the hidden hand and foot tips of the target person when a part of the body of the target person detected by the person detection unit is hidden, using a preceding image that is an image of the target person before the part of the body of the target person is hidden and a subsequent image that is an image of the target person after the hidden part of the body of the target person is exposed again; a skeletal information generation unit that acquires coordinates of at least one of the hand tips and the foot tips of the target person from the trajectory calculated by the trajectory calculation unit, and generates skeletal information indicating the skeleton of the target person by solving inverse kinematics using the acquired coordinates; The system includes a movement classifying unit that classifies the movement of the target person using the skeleton information generated by the skeleton information generating unit.

[0007] In addition, the trajectory calculation unit Identifying preceding coordinates, which are coordinates in the preceding image of the hidden hand or foot of the target person corresponding to the hidden body part of the target person, and subsequent coordinates, which are coordinates in the subsequent image of the hidden hand or foot; Using the preceding and succeeding coordinates, calculate intermediate coordinates, which are coordinates of hidden fingers or toes in one or more intermediate images between the preceding and succeeding images, by an interpolation method; The line connecting the preceding coordinate, the calculated intermediate coordinate, and the succeeding coordinate can be calculated as the trajectory of the hidden hand or foot.

[0008] Furthermore, the action classification device may include an image processing unit that processes an image in which the target person is detected by the person detection unit, the image processing unit deletes a background image of the target person from the image in which the target person is detected, or cuts out an image including the target person from the image in which the target person appears, and if the size of the cut-out image is smaller than a predetermined size, enlarges the size of the cut-out image to the predetermined size; The skeleton information generating unit can generate skeleton information using the image processed by the image processing unit.

[0009] The motion classification method for classifying a person's motion according to the present disclosure includes: Detect people from multiple images generated in time series, If a part of the body of the detected target person is hidden, a preceding image is an image of the target person before the part of the body of the target person is hidden, and a subsequent image is an image of the target person after the hidden part of the body of the target person is exposed again, and a trajectory of at least one of the hand and foot of the target person is calculated; obtaining coordinates of at least one of the hand and foot of the target person from the calculated trajectory; By solving inverse kinematics using the acquired coordinates, skeletal information indicating the skeleton of the target person is generated; The generated skeletal information is used to classify the movements of the target person.

[0010] The motion classification program for classifying a person's motion according to the present disclosure includes: detecting a person from a plurality of images generated in time series; When a part of the body of the detected target person is hidden, calculating a trajectory of at least one of the hand and foot of the target person using a preceding image which is an image of the target person before the part of the body of the target person is hidden and a subsequent image which is an image of the target person after the hidden part of the body of the target person is exposed again; acquiring coordinates of at least one of the hand and foot of the target person from the calculated trajectory; generating skeletal information indicating the skeleton of the target person by solving inverse kinematics using the acquired coordinates; A step of classifying the motion of the target person using the generated skeletal information is executed. [Effects of the Invention]

[0011] The present disclosure provides an action classification device, an action classification method, and an action classification program that are capable of classifying the actions of a person appearing in an image even if part of the person's body is hidden. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a block diagram illustrating a configuration of an action classification device according to the present disclosure. [Figure 2] FIG. 10 is a diagram illustrating an example of processing executed by the action classification device according to the present disclosure. [Figure 3] FIG. 10 is a diagram illustrating an example of processing executed by the action classification device according to the present disclosure. [Figure 4] FIG. 10 is a diagram illustrating an example of processing executed by the action classification device according to the present disclosure. [Figure 5] 10A and 10B are diagrams illustrating image processing performed by the action classification device according to the present disclosure. [Figure 6] 1A and 1B are diagrams showing examples of an image of a person before a part of the body is hidden, an image of the person with a part of the body hidden, and an image of the person with the hidden part of the body exposed again. DETAILED DESCRIPTION OF THE INVENTION

[0013] An embodiment will be described below with reference to the drawings. FIG. 1 is a block diagram showing the configuration of an action classification device 10 according to the present disclosure. The action classification device 10 is a device for classifying human actions. Specific examples of the action classification device 10 include information processing devices such as a PC (Personal Computer) and a server. The action classification device 10 corresponds to a computer. The action classification device 10 includes a processor 100, a communication interface (I / F) 110, and a storage device 120.

[0014] The processor 100 is a processor such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit) that controls electronic circuits and devices included in the action classification device 10. The processor 100 executes the action classification method according to the present disclosure by reading and executing a program from the storage device 120. The processor 100 executes the following programs: an image acquisition unit 101, a person detection unit 102, an image processing unit 103, a trajectory calculation unit 104, a skeleton information generation unit 105, a skeleton information determination unit 106, a flag processing unit 107, a flag determination unit 108, and an action classification unit 109. The processor 100 corresponds to a computer.

[0015] The communication interface 110 is an interface for data communication between the action classification device 10 and other devices. The storage device 120 is a storage device for storing various data such as programs executed by the action classification device 10 and multiple images of people.

[0016] This is a program in which the image acquisition unit 101 acquires images of people that have been generated in time series from the storage device 120. Specifically, the image acquisition unit 101 acquires, for example, images of people that have been photographed by a fixed camera.

[0017] The person detection unit 102 is a program that detects people from the image acquired by the image acquisition unit 101. As a method for detecting people, for example, YOLO, YOLX, or the like, which is a type of trained model, can be adopted.

[0018] The image processing unit 103 is a program that performs various image processing on an image. Specifically, as shown in Fig. 5, the image processing unit 103 cuts out image areas 51 and 52 for each detected person from an image 50 acquired by the image acquisition unit 101. The shape of the image area can be any shape, such as a rectangle, a circle, or an ellipse. The image processing unit 103 also deletes the background image of the person from the cut-out image.

[0019] Furthermore, the image processing unit 103 changes the size of the image areas 51 and 52 of each person from which the background image has been deleted to a default size (w, l), as shown in Fig. 5. The default size is an arbitrary size smaller than a reference size (W, L) described below. Then, the image processing unit 103 generates images 53 and 54 of a standard size, including the image areas 51 and 52 that have been changed to the default size, as shown in Fig. 5. Here, the image processing unit 103 sets a predetermined pixel value, for example, 0 (black), to the image areas of the images 53 and 54 of the standard size other than the image areas 51 and 52.

[0020] The trajectory calculation unit 104 is a program that calculates the trajectory of the hands and / or feet of a detected person when a part of the person's body is hidden. When calculating the trajectory of the hands and / or feet of the detected person that are not hidden, the trajectory calculation unit 104 identifies the positions of the hands and / or feet that are not hidden using an image of the person before the part of the person's body is hidden (hereinafter referred to as a "preceding image"), an image of the person after the hidden part of the person's body is exposed again (hereinafter referred to as a "subsequent image"), and an image between the preceding image and the subsequent image (hereinafter referred to as an "intermediate image"). For example, as shown in images 61 to 63 in FIG. 6, when a part of the detected person's body is hidden, the trajectory calculation unit 104 identifies the coordinates of the hands and / or feet of the person using image 60, which is the preceding image, image 64, which is the subsequent image, and images 61 to 63 between them. Next, the trajectory calculation unit 104 can calculate the trajectory of the hands and / or feet of the person that are not hidden by generating a line connecting the identified coordinates. The number of intermediate images is not limited to that shown in FIG. 6, but is determined by an arbitrary frame rate.

[0021] When calculating the trajectory of a hand and / or toe hidden in an intermediate image (hereinafter simply referred to as "hidden part"), the trajectory calculation unit 104 calculates the trajectory of the hidden part using the preceding image and the subsequent image. Specifically, the trajectory calculation unit 104 identifies the coordinates of the hidden part in the preceding image (hereinafter referred to as "preceding coordinates") and the coordinates of the hidden part in the subsequent image (hereinafter referred to as "subsequent coordinates"). Next, the trajectory calculation unit 104 calculates the trajectory of the hidden part by various interpolation methods using the preceding coordinates and the subsequent coordinates. In this embodiment, spline interpolation based on Equation 1 can be adopted as the interpolation method.

number

[0022] When N is 2 or more, the trajectory calculation unit 104 calculates the coordinates of the hidden part in one or more intermediate images (hereinafter referred to as "intermediate coordinates"). Then, the trajectory calculation unit 104 can calculate the line connecting the preceding coordinates, the intermediate coordinates, and the succeeding coordinates as the trajectory of the hidden part.

[0023] As a specific example of spline interpolation, cubic spline interpolation will be described. Equation 2 defines spline interpolation when N=3, that is, cubic spline interpolation.

number

number

number

number

number

[0024] The skeletal information generating unit 105 is a program that generates skeletal information indicating the skeleton of a detected person. The skeletal information is the absolute coordinates in the image of points necessary for determining the posture of a person, such as the person's fingers, elbows, toes, knees, sternum, and pelvis, or the relative coordinates of each point as viewed from a specific reference part, such as the pelvis. Note that the skeletal information may include not only the coordinates of each point but also the angle of each point.

[0025] If the body of a person in an image is not hidden, the skeletal information generation unit 105 can generate skeletal information of the person from an image in which the person appears, using various trained models that estimate the coordinates of the person's skeleton, such as OpenPose.

[0026] On the other hand, if a part of the body of a person in an image is hidden, the skeletal information generation unit 105 cannot generate skeletal information from the image. In this case, the skeletal information generation unit 105 can estimate skeletal information by acquiring the coordinates of the hand and / or toe from the trajectory calculated by the trajectory calculation unit 104 and solving inverse kinematics using the acquired coordinates. For example, when estimating the coordinate of an elbow from the coordinate of the hand, the skeletal information generation unit 105 acquires the coordinates of the hand at multiple times (from t = 0 to t = T). Then, the skeletal information generation unit 105 can estimate the coordinate of the elbow by solving inverse kinematics using the coordinate of the hand. Here, to estimate the coordinate of the elbow, the lengths of the upper arm and the forearm are required, and these lengths can be obtained from the coordinate of the hand, the coordinate of the elbow, and the coordinate of the shoulder in the preceding image. An example of inverse kinematics for obtaining the coordinate of the elbow is 7-degree-of-freedom inverse kinematics, which is an inverse kinematics modeled after the joints of a human shoulder, elbow, and wrist. The skeleton information generating unit 105 may estimate the skeleton information by solving inverse kinematics using the coordinates of the hand and / or the toe at a single time.

[0027] The skeletal information determination unit 106 is a program that determines whether skeletal information has been generated for each person. As described above, if a part of the body of a person in an image is hidden, the skeletal information generation unit 105 cannot generate skeletal information from the image. Therefore, if a part of the body of a person in an image is hidden, the skeletal information determination unit 106 determines that skeletal information has not been generated for each person.

[0028] The flag processing unit 107 is a program that processes a queue flag that indicates whether or not an image is stored in the queue of the storage device 120. In this embodiment, "1" is used as information that an image is stored in the queue, and "0" is used as information that an image is not stored in the queue.

[0029] This is a program that determines the information set in the queue flag by the flag determination unit 108. Specifically, the flag determination unit 108 determines whether or not information indicating that an image is stored in the queue is set in the queue flag.

[0030] The movement classification unit 109 is a program that classifies the movements of a person using the skeletal information generated by the skeletal information generation unit 105. Specifically, the movement classification unit 109 can classify the movements of a person using a trained model that uses skeletal information as input information and the type of movement of a person as output information. The trained model is a program that can be trained by machine learning. This model can learn skeletal information corresponding to a specific movement of a person and information indicating the type of the specific movement as training data. This skeletal information can be generated by the skeletal information generation unit 105 using an image of a predetermined size. This predetermined size corresponds to the above-mentioned reference size.

[0031] This trained model can be trained using skeletal information at a single time or multiple times. By training the model using skeletal information at multiple times, the estimation accuracy can be improved. For example, when training the model on the action of throwing an object, skeletal information obtained from a series of actions of throwing an object can be used as training data. When using a model trained using skeletal information at multiple times, the action classification unit 109 classifies the person's action using the skeletal information at multiple times generated by the skeletal information generation unit 105. On the other hand, when using a model trained using skeletal information at a single time, the action classification unit 109 classifies the person's action using the skeletal information at a single time generated by the skeletal information generation unit 105.

[0032] 2 is a diagram showing an example of processing executed by the action classification device 10. In step S1, the image acquisition unit 101 acquires one image from a plurality of images generated in chronological order. Here, the image acquisition unit 101 acquires images in order of generation date and time.

[0033] In step S2, the person detection unit 102 detects people from the image acquired in step S1. If multiple people appear in the image, the person detection unit 102 detects each person. In step S3, the person detection unit 102 assigns identification information (ID) to each person detected in step S2.

[0034] In step S4, the image processing unit 103 cuts out an image area for each detected person from the image acquired in step S1. In step S5, the image processing unit 103 deletes the background image of the person from the image cut out in step S4.

[0035] In step S6, the image processing unit 103 changes the size of the image of each person from which the background image has been deleted to a default size. Specifically, if the image of each person from which the background image has been deleted is smaller than the default size, the image processing unit 103 enlarges the size of the image to the default size. On the other hand, if the image of each person from which the background image has been deleted is larger than the default size, the image processing unit 103 reduces the size of the image to the default size. When enlarging or reducing the image, it is preferable that the image processing unit 103 fix the aspect ratio of the image of each person from which the background image has been deleted.

[0036] In step S7, the image processing unit 103 generates a standard-size image including an image of each person resized to a predetermined size. In step S8, the image processing unit 103 associates the standard-size image generated in step S7 with the identification information of the people appearing in the image and stores it in the storage device 120. In step S9, the process of FIG. 3 is executed for each detected person.

[0037] In step S10, the image acquisition unit 101 determines whether or not all of the multiple images generated in time series have been acquired. If it is determined that all of the multiple images have not been acquired (NO), the process returns to step S1, and the image acquisition unit 101 acquires other images that have not been acquired. On the other hand, if it is determined that all of the multiple images have been acquired (YES), the process of FIG. 2 ends.

[0038] In step S20 of FIG. 3, the skeleton information generation unit 105 executes a skeleton information generation process for each person using the standard-sized image generated in step S7. In step S21, the skeleton information determination unit 106 determines whether or not skeleton information has been generated for each person. If it is determined that skeleton information has been generated (YES), in step S22 the flag determination unit 108 determines whether or not the queue flag is set to "0," which indicates that the image is stored in the queue. If it is determined that the queue flag is set to "0" (YES), in step S23 the action classification unit 109 classifies the action using the generated skeleton information.

[0039] If it is determined in step S21 that skeletal information has not been generated (NO), in other words, if part of the body of the person in the image is hidden, in step S24, the image acquisition unit 101 acquires the previous image from the storage device 120 and stores it in a queue in the storage device 120. The previous image is the image immediately before the image acquired in step S1. The queue can store the preceding image and one or more intermediate images. In step S25, the flag processing unit 107 sets the queue flag to "1", which indicates that an image is stored in the queue.

[0040] If it is determined in step S22 that the queue flag is not set to "0" (NO), in other words, if the queue flag is set to "1", then in step S26 the image acquisition unit 101 acquires the images stored in the queue and the current image generated in step S7.

[0041] In step S27, the trajectory calculation unit 104 calculates the trajectory of the person's hand and / or foot. Here, when calculating the trajectory of the person's non-occluded hand and / or foot, the trajectory calculation unit 104 uses the preceding image and intermediate image, which are images stored in the queue, and the current image, which corresponds to the subsequent image. On the other hand, when calculating the trajectory of the person's occluded hand and / or foot, the trajectory calculation unit 104 uses the preceding image, which is an image stored in the queue, and the current image, which corresponds to the subsequent image.

[0042] In step S28, the skeleton information generation unit 105 generates skeleton information using information indicating the trajectories of the person's hands and / or feet calculated by the trajectory calculation unit 104. In step S29, the queue in the storage device 120 is emptied. In step S30, the flag processing unit 107 sets the queue flag to "0".

[0043] In the above-described embodiment, the person detection unit 102 detects a person from multiple images generated in time series. When a body part of the target person detected by the person detection unit 102 is hidden, the trajectory calculation unit 104 calculates the trajectory of at least one of the hidden hand and foot of the target person using a preceding image, which is an image of the target person before the hidden body part of the target person is hidden, and a subsequent image, which is an image of the target person after the hidden body part of the target person is exposed again. The skeleton information generation unit 105 generates skeleton information indicating the skeleton of the target person from the trajectory calculated by the trajectory calculation unit 104. Then, the movement classification unit 109 classifies the movement of the target person using the skeleton information generated by the skeleton information generation unit 105.

[0044] By adopting this configuration, even if part of a person's body is hidden in the image, the trajectory of the person's hands and feet can be calculated, making it possible to classify the person's movements.

[0045] In the above-described embodiment, the image processing unit 103 deletes a background image of the target person from an image in which the target person is detected. The image processing unit 103 also cuts out an image including the target person from an image in which the target person appears, and if the size of the cut-out image is smaller than a predetermined size, enlarges the size of the cut-out image to the predetermined size. The skeleton information generation unit 105 generates skeleton information using the image processed by the image processing unit 103 in this manner.

[0046] By deleting the background image in this way, the influence of the background image when generating skeletal information can be suppressed. Also, when multiple people are captured in one image, people who are relatively far away from the camera may appear small. Even in such cases, by enlarging the size of the images cut out for each person, which are smaller than the default size, to the default size, skeletal information for the people who are far away can be generated and the movements of those people can be classified.

[0047] In the above examples, the program includes instructions (or software code) that, when loaded into a computer, cause the computer to perform one or more functions described in the embodiments. The program may be stored in a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, computer-readable media or tangible storage media include random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technology, CD-ROM, digital versatile disk (DVD), Blu-ray® disc or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device. The program may be transmitted on a transient computer-readable medium or a communication medium. By way of example and not limitation, transient computer-readable media or communication media include electrical, optical, acoustic, or other forms of propagated signals. Computers include various devices such as personal computers (PCs), servers, CPUs, MPUs, field-programmable gate arrays (FPGAs), and application-specific integrated circuits (ASICs).

[0048] The present disclosure is not limited to the above-described embodiments, and can be modified as appropriate within the scope of the present disclosure. [Explanation of symbols]

[0049] 10: Action classification device, 100: processor, 101: image acquisition unit, 102: person detection unit, 103: image processing unit, 104: trajectory calculation unit, 105: skeleton information generation unit, 106: skeleton information determination unit, 107: flag processing unit, 108: flag determination unit, 109: action classification unit, 110: communication interface, 120: storage device

Claims

1. a person detection unit that detects a person from a plurality of images generated in time series; a trajectory calculation unit that, when a part of the body of the target person detected by the person detection unit is hidden, calculates a trajectory of at least one of the hidden hand and foot of the target person using a preceding image that is an image of the target person before the part of the body of the target person is hidden and a subsequent image that is an image of the target person after the hidden part of the body of the target person is exposed again; a skeletal information generation unit that acquires coordinates of at least one of a hand tip and a foot tip of the target person from the trajectory calculated by the trajectory calculation unit, and generates skeletal information indicating a skeleton of the target person by solving inverse kinematics using the acquired coordinates; a motion classification unit that classifies motions of the target person using the skeleton information generated by the skeleton information generation unit; A motion classifier including:

2. The trajectory calculation unit Identifying previous coordinates, which are coordinates in the previous image of a hidden hand or foot of the target person that corresponds to a hidden body part of the target person, and subsequent coordinates, which are coordinates in the subsequent image of the hidden hand or foot; Using the preceding coordinates and the subsequent coordinates, calculate intermediate coordinates, which are coordinates of the hidden fingertip or toe in one or more intermediate images between the preceding image and the subsequent image, by an interpolation method; The motion classification device according to claim 1 , wherein a line connecting the preceding coordinate, the calculated intermediate coordinate, and the succeeding coordinate is calculated as the trajectory of the hidden hand or foot.

3. further including an image processing unit that processes an image in which the target person is detected by the person detection unit; the image processing unit deletes a background image of the target person from an image in which the target person is detected, or cuts out an image including the target person from an image in which the target person appears, and if the size of the cut-out image is smaller than a default size, enlarges the size of the cut-out image to the default size; The action classification device according to claim 1 , wherein the skeleton information generating unit generates the skeleton information using the image processed by the image processing unit.

4. A motion classification method for classifying a person's motion, comprising: Detect people from multiple images generated in time series, When a part of the body of the detected target person is hidden, a preceding image is an image of the target person before the part of the body of the target person is hidden, and a subsequent image is an image of the target person after the hidden part of the body of the target person is exposed again, and a trajectory of at least one of the hand and foot of the target person is calculated; obtaining coordinates of at least one of the hand and foot of the target person from the calculated trajectory; generating skeletal information indicating the skeleton of the target person by solving inverse kinematics using the acquired coordinates; classifying the motion of the target person using the generated skeletal information; Behavior classification method.

5. A motion classification program for classifying a person's motion, the program comprising: detecting a person from a plurality of images generated in time series; When a part of the body of the detected target person is hidden, a step of calculating a trajectory of at least one of the hand and foot of the target person using a preceding image which is an image of the target person before the part of the body of the target person is hidden and a subsequent image which is an image of the target person after the hidden part of the body of the target person is exposed again; acquiring coordinates of at least one of a hand tip and a foot tip of the target person from the calculated trajectory; generating skeletal information indicative of the skeleton of the target person by solving inverse kinematics using the acquired coordinates; classifying the motion of the target person using the generated skeletal information; A behavior classification program that executes the following.

Images