Motion classification device, motion classification method, and motion classification program

Abstract

To provide a movement classification device, a movement classification method, and a movement classification program capable of classifying a person movement even if the body part of the person in an image is hidden.SOLUTION: A movement classification device 10 includes a person detection part 102 to detect the person from the multiple images generated in chronological order, a preceding image which is the image the target person before the part of the target person body is hidden when the part of the detected target person body is hidden, a trajectory calculation part 104 to calculate at least one trajectory of the hidden fingertip and the hidden toe of the target person using the subsequent images which is the image of the target person after the hidden body part of the target person is re-exposed, a skeleton information generation part 105 to generate the skeleton information representing the target person skeleton from the generated trajectory, and a movement classification part 109 to classify the target person movement using the generated skeleton information.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to a motion classification device, a motion classification method, and a motion classification program for classifying motions of a person. [Background Art]

[0002] Conventionally, various techniques for classifying motions of a person such as a worker have been proposed. As an example of such a technique, Patent Document 1 discloses a behavior recognition device that recognizes a behavior of a target such as a person based on time-series data of feature amounts of the target behavior extracted from data in which the behavior of the target is recorded in time series. This behavior recognition device recognizes the behavior at each time point by tracking time-series data of feature amounts in time series, regards a sequence of the same non-distinguishable behavior as one behavior, and after completing recognition of a predetermined number of behaviors arranged in time series, recognizes the behavior at the current recognition time point based on time-series data of feature amounts corresponding to a plurality of behaviors arranged in time series from a time point going back a predetermined number of behaviors before to the current recognition time point. [Prior Art Documents] [Patent Documents]

[0003] [Patent Document 1] International Publication No. 2017 / 150211 [Summary of the Invention] [Problem to be Solved by the Invention]

[0004] However, the behavior recognition device disclosed in Patent Document 1 has a problem that when a part of the body of the person captured in an image is hidden, the motion of the person cannot be classified.

[0005] The present disclosure provides a motion classification device, a motion classification method, and a motion classification program capable of classifying the motion of a person captured in an image even when a part of the body of the person is hidden. [Means for Solving the Problem]

[0006] The operation classification device relating to this disclosure is A person detection unit that detects a person from multiple images generated in a time series, If a part of the body of a person detected by the person detection unit is hidden, the trajectory calculation unit calculates the trajectory of at least one of the hidden hands and feet of the person using a preceding image which is an image of the person before the part of the person's body was hidden, and a succeeding image which is an image of the person after the part of the person's body that was hidden is exposed again. A skeletal information generation unit obtains the coordinates of at least one of the subject person's fingertips and toes from the trajectory calculated by the trajectory calculation unit, and generates skeletal information indicating the subject person's skeleton by solving the inverse kinematics using the obtained coordinates. It includes a motion classification unit that classifies the movements of a target person using the skeletal information generated by the skeletal information generation unit.

[0007] Furthermore, the orbit calculation unit, Identify the preceding coordinates, which are the coordinates in the preceding image of the hidden hand or foot of the subject, corresponding to the hidden part of the subject's body, and the succeeding coordinates, which are the coordinates in the subsequent image of the hidden hand or foot. Using the preceding and succeeding coordinates, an interpolation method is used to calculate the intermediate coordinates, which are the coordinates of the hidden fingertips or feet in one or more intermediate images between the preceding and succeeding images. The line connecting the preceding coordinates, the calculated intermediate coordinates, and the succeeding coordinates can be calculated as the trajectory of the hidden hand or foot.

[0008] Furthermore, the motion classification device may include an image processing unit that processes images in which the target person has been detected by the person detection unit. The image processing unit removes the background image of the person from the image in which the person was detected, or extracts an image containing the person from the image containing the person, and if the size of the extracted image is smaller than the default size, it enlarges the size of the extracted image to the default size. The skeletal information generation unit can generate skeletal information using the image processed by the image processing unit.

[0009] The behavior classification method for classifying the actions of a person relating to this disclosure is a computer that, Detecting people from multiple images generated in a time series, If part of the detected person's body is obscured, the trajectory of at least one of the person's hands and feet is calculated using a preceding image, which is an image of the person before the part of their body was obscured, and a succeeding image, which is an image of the person after the previously obscured part of their body has been re-exposed. From the calculated trajectory, obtain the coordinates of at least one of the subject's fingertips and toes. By solving the inverse kinematics using the acquired coordinates, skeletal information representing the skeleton of the subject is generated. The generated skeletal information is used to classify the movements of the subject.

[0010] The motion classification program for classifying the actions of a person relating to this disclosure is provided to a computer, Steps include detecting a person from multiple images generated in a time series, If part of the detected person's body is obscured, the process involves calculating the trajectory of at least one of the person's hands and feet using a preceding image, which is an image of the person before the part of their body was obscured, and a succeeding image, which is an image of the person after the previously obscured part of their body has been re-exposed. A step of obtaining the coordinates of at least one of the subject person's fingertips and toes from the calculated trajectory, The process involves generating skeletal information representing the skeleton of the subject by solving the inverse kinematics using the acquired coordinates, and The generated skeletal information is used to perform a step that classifies the movements of the target person. [Effects of the Invention]

[0011] This disclosure provides a motion classification device, a motion classification method, and a motion classification program that can classify the movements of a person even when part of the person's body is obscured in an image. [BRIEF DESCRIPTION OF DRAWINGS]

[0012] [Figure 1] FIG. 1 is a block diagram showing the configuration of an motion classification device according to the present disclosure. [Figure 2] FIG. 2 is a diagram showing an example of processing executed by the motion classification device according to the present disclosure. [Figure 3] FIG. 3 is a diagram showing an example of processing executed by the motion classification device according to the present disclosure. [Figure 4] FIG. 4 is a diagram showing an example of processing executed by the motion classification device according to the present disclosure. [Figure 5] FIG. 5 is a diagram for explaining image processing performed by the motion classification device according to the present disclosure. [Figure 6] FIG. 6 is a diagram showing an example of an image of a person before a part of the body is hidden, an image of a person with a part of the body hidden, and an image of a person with the hidden part of the body exposed again. DESCRIPTION OF EMBODIMENTS

[0013] Hereinafter, an embodiment will be described with reference to the drawings. FIG. 1 is a block diagram showing the configuration of a motion classification device 10 according to the present disclosure. The motion classification device 10 is a device for classifying motions of a person. Specific examples of the motion classification device 10 include information processing devices such as PCs (Personal Computers) and servers. The motion classification device 10 corresponds to a computer. The motion classification device 10 includes a processor 100, a communication interface (I / F) 110, and a storage device 120.

[0014] The processor 100 is a processor such as a CPU (Central Processing Unit) or MPU (Micro Processing Unit) that controls the electronic circuits and devices of the motion classification device 10. The processor 100 executes the motion classification method according to this disclosure by reading and executing a program from the storage device 120. The processor 100 executes a program consisting of an image acquisition unit 101, a person detection unit 102, an image processing unit 103, a trajectory calculation unit 104, a skeletal information generation unit 105, a skeletal information determination unit 106, a flag processing unit 107, a flag determination unit 108, and a motion classification unit 109. The processor 100 is equivalent to a computer.

[0015] The communication interface 110 is an interface for data communication between the motion classification device 10 and other devices. The storage device 120 is a storage device that stores various data such as programs executed by the motion classification device 10 and multiple images of people.

[0016] The image acquisition unit 101 is a program that acquires images of people that have been generated in chronological order from the storage device 120. Specifically, the image acquisition unit 101 acquires images of people taken by a fixed-point camera, for example.

[0017] The person detection unit 102 is a program that detects people from images acquired by the image acquisition unit 101. As a method for detecting people, for example, a type of pre-trained model such as YOLO or YOLX can be used.

[0018] The image processing unit 103 is a program that performs various image processing operations on an image. Specifically, as shown in Figure 5, the image processing unit 103 extracts image regions 51 and 52 from the image 50 acquired by the image acquisition unit 101 for each detected person. The shape of the image region can be any shape, such as a rectangle, circle, or ellipse. The image processing unit 103 also removes the background image of the person from the extracted image.

[0019] Furthermore, as shown in Figure 5, the image processing unit 103 changes the size of the image regions 51 and 52 of each person from which the background image has been removed to a default size (w, l). The default size is any size smaller than the reference size (W, L) described later. Then, as shown in Figure 5, the image processing unit 103 generates reference-sized images 53 and 54 that include the image regions 51 and 52 that have been changed to the default size. Here, the image processing unit 103 sets the image regions other than image regions 51 and 52 in the reference-sized images 53 and 54 to a predetermined pixel value, for example, 0 (black).

[0020] The trajectory calculation unit 104 is a program that calculates the trajectory of a person's hands and / or feet when part of the detected person's body is hidden. When calculating the trajectory of the hands and / or feet of a detected person that are not hidden, the trajectory calculation unit 104 uses an image of the person before part of their body is hidden (hereinafter referred to as the "preceding image"), an image of the person after part of their body is exposed again (hereinafter referred to as the "successive image"), and an image between the preceding image and the succeeding image (hereinafter referred to as the "intermediate image") to determine the position of the hands and / or feet that are not hidden. For example, as shown in images 61 to 63 of Figure 6, when part of the detected person's body is hidden, the trajectory calculation unit 104 uses the preceding image 60, the succeeding image 64, and the images 61 to 63 in between to determine the coordinates of the person's hands and / or feet. Then, the trajectory calculation unit 104 can calculate the trajectory of the hands and / or feet of the person that are not hidden by generating a line connecting the determined coordinates. Note that the number of intermediate images is not limited to those shown in Figure 6, but is determined by any frame rate.

[0021] When calculating the trajectory of the hand and / or foot (hereinafter simply referred to as the "hidden part") hidden in the intermediate image, the trajectory calculation unit 104 uses the preceding image and the succeeding image to calculate the trajectory of the hidden part. Specifically, the trajectory calculation unit 104 identifies the coordinates of the hidden part in the preceding image (hereinafter referred to as the "preceding coordinates") and the coordinates of the hidden part in the succeeding image (hereinafter referred to as the "successing coordinates"). Then, the trajectory calculation unit 104 uses the preceding coordinates and the succeeding coordinates to calculate the trajectory of the hidden part using various interpolation methods. In this embodiment, spline interpolation based on Equation 1 can be used as the interpolation method.

number

[0022] If N is 2 or greater, the trajectory calculation unit 104 calculates the coordinates of the hidden portion in one or more intermediate images (hereinafter referred to as "intermediate coordinates"). The trajectory calculation unit 104 can then calculate the trajectory of the hidden portion by drawing a line connecting the preceding coordinates, the intermediate coordinates, and the succeeding coordinates.

[0023] As a concrete example of spline interpolation, we will explain cubic spline interpolation. Equation 2 defines spline interpolation when N=3, i.e., cubic spline interpolation.

number

number

number

number

number

[0024] The skeletal information generation unit 105 is a program that generates skeletal information indicating the skeleton of the detected person. Skeletal information refers to the absolute coordinates in the image of points necessary for determining a person's posture, such as the fingertips, elbows, toes, knees, sternum, and pelvis, or the relative coordinates of each point as viewed from a specific reference part, such as the pelvis. Note that the skeletal information may include not only the coordinates of each point but also the angles of each point.

[0025] If the body of a person in the image is not hidden, the skeletal information generation unit 105 can generate skeletal information of the person from the image containing the person using various trained models for estimating the coordinates of the person's skeleton, such as OpenPose.

[0026] On the other hand, if part of a person's body is hidden in the image, the skeletal information generation unit 105 cannot generate skeletal information from the image. In this case, the skeletal information generation unit 105 can estimate skeletal information by obtaining the coordinates of the fingertips and / or toes from the trajectory calculated by the trajectory calculation unit 104 and solving inverse kinematics using the obtained coordinates. For example, when estimating the coordinate of the elbow from the coordinate of the fingertips, the skeletal information generation unit 105 obtains the coordinates of the fingertips at multiple time points (from t=0 to t=T). Then, the skeletal information generation unit 105 can estimate the coordinate of the elbow by solving inverse kinematics using these fingertips coordinates. Here, the lengths of the upper arm and forearm are needed to estimate the coordinate of the elbow, but these lengths can be obtained from the coordinates of the fingertips, elbow, and shoulder in the preceding image. An example of inverse kinematics for obtaining the coordinate of the elbow is the 7-degree-of-freedom inverse kinematics, which is an inverse kinematics modeled after the joints of the human shoulder, elbow, and wrist. The skeletal information generation unit 105 may also estimate skeletal information by solving inverse kinematics using the coordinates of the fingertips and / or toes at a single time point.

[0027] The skeletal information determination unit 106 is a program that determines whether or not skeletal information has been generated for each person. As mentioned above, if part of a person's body is hidden in the image, the skeletal information generation unit 105 cannot generate skeletal information from the image. Therefore, if part of a person's body is hidden in the image, the skeletal information determination unit 106 determines that skeletal information has not been generated for each person.

[0028] The flag processing unit 107 is a program that processes a queue flag indicating whether or not an image is stored in the queue of the storage device 120. In this embodiment, "1" is used as information that an image is stored in the queue, and "0" is used as information that an image is not stored in the queue.

[0029] The flag determination unit 108 is a program that determines the information set in the queue flag. Specifically, the flag determination unit 108 determines whether or not information indicating that an image is stored in the queue is set in the queue flag.

[0030] The motion classification unit 109 is a program that classifies human motion using skeletal information generated by the skeletal information generation unit 105. Specifically, the motion classification unit 109 can classify human motion using a trained model that takes skeletal information as input information and the type of human motion as output information. This trained model is a program that can be trained by machine learning. This model can be trained using skeletal information corresponding to a specific human motion and information indicating the type of that specific motion as training data. This skeletal information can be generated by the skeletal information generation unit 105 using an image of a predetermined size. This predetermined size corresponds to the standard size mentioned above.

[0031] This trained model can be trained using skeletal information from a single time point or multiple time points. By training the model using skeletal information from multiple time points, its estimation accuracy can be improved. For example, when training the model to perform the action of throwing an object, skeletal information obtained from a series of throwing actions can be used as training data. When using a model trained using skeletal information from multiple time points, the motion classification unit 109 classifies the person's actions using the skeletal information from multiple time points generated by the skeletal information generation unit 105. On the other hand, when using a model trained using skeletal information from a single time point, the motion classification unit 109 classifies the person's actions using the skeletal information from a single time point generated by the skeletal information generation unit 105.

[0032] Figure 2 shows an example of the process performed by the motion classification device 10. In step S1, the image acquisition unit 101 acquires one image from a plurality of images generated in a time series. Here, the image acquisition unit 101 acquires images in order from the earliest generation date and time.

[0033] In step S2, the person detection unit 102 detects people from the image acquired in step S1. If there are multiple people in the image, the person detection unit 102 detects each person. In step S3, the person detection unit 102 assigns identification information (ID) to each person detected in step S2.

[0034] In step S4, the image processing unit 103 extracts an image region for each detected person from the image acquired in step S1. In step S5, the image processing unit 103 removes the background image of the person from the image extracted in step S4.

[0035] In step S6, the image processing unit 103 resizes the images of each person from whom the background image has been removed to a default size. Specifically, if the image of each person from whom the background image has been removed is smaller than the default size, the image processing unit 103 enlarges the image to the default size. On the other hand, if the image of each person from whom the background image has been removed is larger than the default size, the image processing unit 103 reduces the image to the default size. When enlarging or reducing an image, it is preferable for the image processing unit 103 to maintain a fixed aspect ratio for each person from whom the background image has been removed.

[0036] In step S7, the image processing unit 103 generates a standard-size image containing images of each person that have been resized to a predetermined size. In step S8, the image processing unit 103 associates the standard-size image generated in step S7 with the identification information of the people depicted in the image and saves it to the storage device 120. In step S9, the process shown in Figure 3 is executed for each detected person.

[0037] In step S10, the image acquisition unit 101 determines whether or not it has acquired all of the multiple images generated in time series. If it is determined that not all of the multiple images have been acquired (NO), the process returns to step S1, and the image acquisition unit 101 acquires the other images that have not been acquired. On the other hand, if it is determined that all of the multiple images have been acquired (YES), the process shown in Figure 2 ends.

[0038] In step S20 of Figure 3, the skeletal information generation unit 105 performs skeletal information generation processing for each person using the standard-size image generated in step S7. In step S21, the skeletal information determination unit 106 determines whether or not skeletal information has been generated for each person. If it is determined that skeletal information has been generated (YES), in step S22, the flag determination unit 108 determines whether or not the information "0" indicating that an image is stored in the queue is set as the queue flag. If it is determined that "0" is set as the queue flag (YES), in step S23, the motion classification unit 109 classifies the motion using the generated skeletal information.

[0039] If it is determined in step S21 that skeletal information was not generated (NO), in other words, if part of the person's body in the image is hidden, in step S24 the image acquisition unit 101 acquires the previous image from the storage device 120 and saves it to the queue of the storage device 120. The previous image is the image immediately preceding the image acquired in step S1. This queue may store the preceding image and one or more intermediate images. In step S25, the flag processing unit 107 sets the queue flag to "1", indicating that an image is stored in the queue.

[0040] If it is determined in step S22 that the queue flag is not set to "0" (NO), in other words, if the queue flag is set to "1", then in step S26 the image acquisition unit 101 acquires the image stored in the queue and the current image generated in step S7.

[0041] In step S27, the trajectory calculation unit 104 calculates the trajectories of the person's fingertips and / or feet. When calculating the trajectory of the person's unobstructed fingertips and / or feet, the trajectory calculation unit 104 uses the preceding image and intermediate image, which are images stored in the queue, and the current image, which corresponds to the succeeding image. On the other hand, when calculating the trajectory of the person's hidden fingertips and / or feet, the trajectory calculation unit 104 uses the preceding image, which is an image stored in the queue, and the current image, which corresponds to the succeeding image.

[0042] In step S28, the skeletal information generation unit 105 generates skeletal information using the information indicating the trajectories of the person's fingertips and / or toes calculated by the trajectory calculation unit 104. In step S29, the queue of the storage device 120 is emptied. In step S30, the flag processing unit 107 sets the queue flag to "0".

[0043] In the embodiment described above, the person detection unit 102 detects a person from a plurality of images generated in a time series. If part of the body of the target person detected by the person detection unit 102 is hidden, the trajectory calculation unit 104 uses a preceding image, which is an image of the target person before part of their body is hidden, and a subsequent image, which is an image of the target person after part of their body is exposed again, to calculate the trajectory of at least one of the hidden hands and feet of the target person. The skeleton information generation unit 105 generates skeleton information indicating the skeleton of the target person from the trajectory calculated by the trajectory calculation unit 104. Then, the motion classification unit 109 classifies the motion of the target person using the skeleton information generated by the skeleton information generation unit 105.

[0044] By adopting this configuration, even if part of a person's body is hidden in the image, the trajectory of the person's hands and feet can be calculated, making it possible to classify the person's movements.

[0045] Furthermore, in the above-described embodiment, the image processing unit 103 removes the background image of the target person from the image in which the target person was detected. The image processing unit 103 also extracts an image containing the target person from the image in which the target person is photographed, and if the size of the extracted image is smaller than a predetermined size, it enlarges the size of the extracted image to the predetermined size. The skeletal information generation unit 105 generates skeletal information using the image processed in this way by the image processing unit 103.

[0046] By removing the background image in this way, the influence of the background image on the generation of skeletal information can be suppressed. Also, when multiple people are in one image, those who are relatively far from the camera may appear small. Even in such cases, by enlarging the size of the individual images, which are smaller than the default size, to the default size, skeletal information for the distant individuals can be generated, and their movements can be classified.

[0047] In the examples described above, the program includes a set of instructions (or software code) that, when loaded into a computer, cause the computer to perform one or more of the functions described in the embodiments. The program may be stored in a non-temporary computer-readable medium or a physical storage medium. Examples, but not limited to, include random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technologies, CD-ROM, digital versatile disk (DVD), Blu-ray® disc or other optical disc storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices. The program may be transmitted over a temporary computer-readable medium or a communication medium. Examples, but not limited to, include temporary computer-readable medium or a communication medium that includes propagating signals of electrical, optical, acoustic or other forms. Computers include various devices such as PCs (Personal Computers), servers, CPUs, MPUs, FPGAs (Field-Programmable Gate Arrays), and ASICs (Application Specific Integrated Circuits).

[0048] This disclosure is not limited to the embodiments described above, and may be modified as appropriate without departing from the spirit of this disclosure. [Explanation of Symbols]

[0049] 10: Motion classification device, 100: Processor, 101: Image acquisition unit, 102: Person detection unit, 103: Image processing unit, 104: Trajectory calculation unit, 105: Skeleton information generation unit, 106: Skeleton information determination unit, 107: Flag processing unit, 108: Flag determination unit, 109: Motion classification unit, 110: Communication interface, 120: Storage device

Claims

1. A person detection unit that detects a person from multiple images generated in a time series, If a part of the body of a target person detected by the person detection unit is hidden, a trajectory calculation unit calculates the trajectory of at least one of the hidden hands and feet of the target person using a preceding image which is an image of the target person before the part of the body of the target person is hidden, and a succeeding image which is an image of the target person after the hidden part of the body of the target person is exposed again. A skeletal information generation unit obtains the coordinates of at least one of the fingertips and toes of the person in question from the trajectory calculated by the trajectory calculation unit, and generates skeletal information indicating the skeleton of the person in question by solving the inverse kinematics using the obtained coordinates. A motion classification unit classifies the movements of the target person using the skeletal information generated by the skeletal information generation unit. A motion classification device that includes [a specific type of device].

2. The aforementioned orbit calculation unit, Identify the preceding coordinates, which are the coordinates in the preceding image of the hidden hand or foot of the person, corresponding to the part of the body of the person being hidden, and the succeeding coordinates, which are the coordinates in the succeeding image of the hidden hand or foot. Using the preceding and succeeding coordinates, an interpolation method is used to calculate intermediate coordinates, which are the coordinates of the hidden fingertip or foottip in one or more intermediate images between the preceding image and the succeeding image. The motion classification device according to claim 1, wherein the line connecting the preceding coordinates, the calculated intermediate coordinates, and the succeeding coordinates is calculated as the trajectory of the hidden hand or foot.

3. The system further includes an image processing unit that processes images in which the target person has been detected by the person detection unit, The image processing unit removes the background image of the person from the image in which the person was detected, or extracts an image containing the person from the image in which the person was photographed, and if the size of the extracted image is smaller than a predetermined size, enlarges the size of the extracted image to the predetermined size. The motion classification device according to claim 1 or 2, wherein the skeletal information generation unit generates the skeletal information using the image processed by the image processing unit.

4. A method for classifying human actions, wherein a computer... Detecting people from multiple images generated in a time series, If a part of the detected person's body is obscured, the trajectory of at least one of the person's hands and feet is calculated using a preceding image, which is an image of the person before the part of their body was obscured, and a subsequent image, which is an image of the person after the previously obscured part of their body has been re-exposed. From the calculated trajectory, obtain the coordinates of at least one of the fingertips and toes of the person in question. By solving the inverse kinematics using the acquired coordinates, skeletal information representing the skeleton of the subject person is generated. Using the generated skeletal information, the movements of the person in question are classified. Behavior classification method.

5. A motion classification program for classifying human actions, which allows a computer to perform the following actions: Steps include detecting a person from multiple images generated in a time series, If a part of the body of the detected target person is hidden, the process involves calculating the trajectory of at least one of the target person's fingertips and toes using a preceding image which is an image of the target person before the part of their body was hidden, and a subsequent image which is an image of the target person after the hidden part of their body has been exposed again. A step of obtaining the coordinates of at least one of the fingertips and toes of the person in question from the calculated trajectory, The steps include: generating skeletal information representing the skeleton of the person in question by solving the inverse kinematics using the acquired coordinates; The steps include classifying the movements of the person in question using the generated skeletal information, and A behavior classification program that executes actions.

Images