Posture analysis device, posture analysis method, and program

Abstract

The posture analysis device (2000) acquires a target image (10) and target person information (20). The target image includes a plurality of persons performing arbitrary actions such as performing, exercising, playing a musical instrument, etc. The posture analysis device (2000) estimates the posture for each person and calculates a posture score representing the quality of the person's posture. The posture analysis device (2000) detects one or more reference persons whose posture scores are higher than the posture score of the target person, and outputs reference information (30) indicating the reference persons.

Description

Technical Field

[0001] The present invention generally relates to a posture analysis device, a posture analysis method, and a non-transitory computer-readable storage medium.

Background Art

[0002] There is a technology for analyzing images of people. Patent Document 1 discloses a system that analyzes images of students in a class and determines the current class situation such as concentration. The class situation is determined by comparing features (for example, postures) obtained from pre-stored class situation sample images with the features of the students in the class imaged in the image.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Patent Document 1 does not disclose a technique for handling an image in which a plurality of people are imaged. An object of the present disclosure is to provide a new technique for analyzing the postures of people using an image in which a plurality of people are imaged.

Means for Solving the Problems

[0005] The posture analysis device provided by the present disclosure includes at least one storage element configured to store instructions, and at least one processor. The at least one processor is configured to execute the instructions to obtain a target image in which two or more persons are imaged, obtain target person information indicating a target person, estimate the posture of each person imaged in the target image, calculate a posture score representing the quality of the posture of each person, detect one or more reference persons having a posture of higher quality than the quality of the posture of the target person, and output reference information indicating the reference persons.

[0006] The posture analysis method further provided by the present disclosure is executed by a computer. The posture analysis method includes obtaining a target image in which two or more persons are imaged, obtaining target person information indicating a target person, estimating the posture of each person imaged in the target image, calculating a posture score representing the quality of the posture of each person, detecting one or more reference persons having a posture of higher quality than the quality of the posture of the target person, and outputting reference information indicating the reference persons.

[0007] The non-transitory computer-readable storage medium further provided by the present disclosure stores a program. The program causes the computer to obtain a target image in which two or more persons are imaged, obtain target person information indicating a target person, estimate the posture of each person imaged in the target image, calculate a posture score representing the quality of the posture of each person, detect one or more reference persons having a posture of higher quality than the quality of the posture of the target person, and output reference information indicating the reference persons.

Advantages of the Invention

[0008] According to the present disclosure, a novel technique for analyzing the postures of persons using an image in which a plurality of persons are imaged is provided.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, an example of an embodiment according to the present disclosure will be described with reference to the drawings. In the drawings, the same elements are denoted by the same reference numerals, and redundant descriptions will be omitted as necessary. Also, predetermined information (for example, a predetermined value or a predetermined threshold) is stored in advance in a storage device accessible by a computer that uses the information, unless otherwise specified.

[0011] Embodiment 1 <Overview> FIG. 1 is a diagram showing an overview of the posture analysis device 2000 according to Embodiment 1. Note that the overview shown in FIG. 1 shows an example of the operation of the posture analysis device 2000 so as to facilitate understanding of the posture analysis device 2000, and does not limit or narrow the scope of the operation of the posture analysis device 2000.

[0012] The posture analysis device 2000 is configured to detect a reference person with a higher-quality posture than the quality of the posture of the target person from the target image 10. The target person is a person designated by the user of the posture analysis device 2000. The target image 10 is image data (for example, an RGB image or a grayscale image) including a plurality of persons in a visible manner.

[0013] The persons captured in the target image 10 do anything. For example, they perform a performance such as figure skating or dance. In other examples, the persons perform exercises such as yoga. In other examples, the persons play musical instruments such as guitars or pianos. In other examples, the persons attend school classes. In other examples, the persons perform work such as assembling parts in a factory or patrolling inside a building.

[0014] To detect the reference person, the posture analysis device 2000 can operate as follows. The posture analysis device 2000 acquires the target image 10 and target person information 20 indicating the target person. The posture analysis device 2000 estimates the posture of each person captured in the target image 10 and calculates a posture score for each person. The posture score of a specific person represents how high the quality of that person's posture is. The posture analysis device 2000 detects a person with a posture score greater than the posture score of the target person as the reference person. The posture analysis device 2000 outputs reference information 30 indicating the reference person.

[0015] Note that the posture analysis device 2000 may handle a plurality of target images 10 including different persons in parallel. In this case, a plurality of cameras may be installed to image different regions (for example, different areas of a lesson room where a person is receiving a performance lesson), and each camera may be configured to generate a target image 10. The posture analysis device 2000 may analyze each of these target images 10 to detect one or more persons and calculate a posture score for each detected person.

[0016] For the sake of simplicity, it is assumed that there is one camera that generates the target image 10. The posture analysis device 2000 corresponding to the case where there are two or more cameras that generate the target image 10 can operate in the same manner as the posture analysis device 2000 corresponding to the case where there is one camera that generates the target image 10.

[0017] <Examples of effects> According to the posture analysis device 2000 of Embodiment 1, a target image in which a plurality of people are imaged is acquired, the posture of each person is estimated, the posture score of each person is calculated, and a reference person whose posture quality is higher than that of the target person is detected. Thereby, a new technique for analyzing the posture of a person using an image in which a plurality of people are imaged is provided.

[0018] In addition, the posture analysis device 2000 outputs reference information 30 indicating the reference person. The information indicating the reference person is effective and useful in various aspects. Briefly explained, the viewer of the reference information 30 can easily and naturally identify the reference person (that is, a person with a higher posture quality than the target person) from the other people imaged in the target image 10.

[0019] In some embodiments, in an environment where the person imaged in the target image 10 is a practitioner such as a performer and the user of the posture analysis device 2000 is one of the practitioners, the posture analysis device 2000 can be used. In this case, it is effective and useful for the user to improve the user's posture by referring to a person with a higher posture quality than the user. However, depending on the situation, it may be difficult for the user to recognize which practitioner has a better posture than the user.

[0020] According to the posture analysis device 2000, the reference person is automatically detected and the reference information 30 indicating the reference person is provided. As a result, it becomes easier for the user to notice a person with a higher posture quality than the user. Therefore, the user can easily refer to the posture of the reference person and improve the user's own posture.

[0021] Hereinafter, a more detailed description of the posture analysis device 2000 will be given.

[0022] <Functional configuration example> FIG. 2 is a block diagram showing an example of the functional configuration of the posture analysis device 2000 according to Embodiment 1. The posture analysis device 2000 includes an acquisition unit 2020, an estimation unit 2040, a calculation unit 2060, a detection unit 2080, and an output unit 2100. The acquisition unit 2020 acquires the target image 10 and the target person information 20. The estimation unit 2040 estimates the posture of the person imaged in the target image 10. The calculation unit 2060 calculates a posture score for each person. The detection unit 2080 detects a person whose posture score is higher than the posture score of the target person as a reference person. The output unit 2100 outputs the reference information 30.

[0023] <Hardware configuration example> The posture analysis device 2000 may be realized by one or more computers. Each of the one or more computers may be a dedicated computer manufactured to realize the posture analysis device 2000, or may be a general-purpose computer such as a personal computer (PC), a server machine, or a mobile device.

[0024] The posture analysis device 2000 may also be realized by installing an application on one or more computers. The application is realized by a program that causes one or more computers to function as the posture analysis device 2000. In other words, the program is implemented with the functional components of the posture analysis device 2000 illustrated in FIG. 2.

[0025] FIG. 3 is a block diagram showing an example of the hardware configuration of the computer 1000 that realizes the posture analysis device 2000. In FIG. 3, the computer 1000 includes a bus 1020, a processor 1040, a memory 1060, a storage device 1080, an input / output (I / O) interface 1100, and a network interface 1120.

[0026] Bus 1020 is a data transmission path for the processor 1040, the memory 1060, the storage device 1080, the input / output interface 1100, and the network interface 1120 to transmit and receive data from each other. The processor 1040 is a processor such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), or an FPGA (Field-Programmable Gate Array). The memory 1060 is a main memory element such as a RAM (Random Access Memory) or a ROM (Read Only Memory). The storage device 1080 is an auxiliary storage element such as a hard disk, an SSD (Solid State Drive), or a memory card. The input / output interface 1100 is an interface between the computer 1000 and peripheral devices (such as a keyboard, a mouse, or a display device). The network interface 1120 is an interface between the computer 1000 and a network. The network may be a LAN (Local Area Network) or a WAN (Wide Area Network).

[0027] The hardware configuration of the computer 1000 is not limited to that shown in FIG. 3. For example, as described above, the posture analysis device 2000 may be realized by a plurality of computers. In this case, those computers may be connected to each other via a network.

[0028] <Flow of processing> FIG. 4 is a flowchart showing an example of the flow of processing executed by the posture analysis apparatus 2000 according to Embodiment 1. The acquisition unit 2020 acquires the target image 10 (S102). The acquisition unit 2020 acquires the target person information 20 (S104). The estimation unit 2040 estimates the posture of the person imaged in the target image 10 (S106). The calculation unit 2060 calculates a posture score of the person (S108). The detection unit 2080 detects a reference person based on the posture score of the person (S110). The output unit 2100 outputs reference information (S112).

[0029] Note that the flow of processing shown in FIG. 4 is an example, and there can be various variations in the flow of processing performed by the posture analysis apparatus 2000. For example, the acquisition of the target image 10 (S102) and the acquisition of the target person information 20 (S104) may be executed in an order reverse to that shown in FIG. 4, or may be executed in parallel.

[0030] <Acquisition of the target image 10: S102> The acquisition unit 2020 acquires the target image 10 (S102). As described above, the target image 10 includes one or more persons. In some embodiments, the target image 10 is one (video frame) of a plurality of time-series images constituting video data. Hereinafter, this video data is referred to as "target video". In this case, the acquisition unit 2020 can acquire one or more video frames constituting the target video and use the acquired video frames as the target image 10.

[0031] Note that it is not necessary to use all the video frames of the target video as the target image 10. For example, the acquisition unit 2020 can acquire video frames (for example, every 10 frames) at predetermined intervals from the target video as the target image 10. In another example, the acquisition unit 2020 can divide the target video into a plurality of sections and acquire one or more video frames from each section as the target image 10.

[0032] In some embodiments, the target video can be divided into intervals based on the length of time. Specifically, the target video can be divided into intervals of a predetermined time length. In other examples, the acquisition unit 2020 recognizes a plurality of scenes captured in the target video and divides the target video into intervals representing the recognized scenes.

[0033] Suppose a figure skating performance is captured in the target video. In this case, since the target video may include scenes such as jumps, spins, and steps, the acquisition unit 2020 divides the target video into intervals such as jumps, spins, and steps.

[0034] Note that there are various techniques for recognizing scenes from video data, and any of these techniques can be applied to the acquisition unit 2020 to recognize scenes from the target video.

[0035] There are various methods for acquiring the target image 10. In some embodiments, the target image 10 is pre-stored in the storage device in a state where the posture analysis device 2000 can acquire it. In this case, the acquisition unit 2020 can access the storage device to acquire the target image 10. In other embodiments, the target image 10 may be transmitted from another computer such as a camera that generates the target image 10 to the posture analysis device 2000. In this case, the acquisition unit 2020 can receive and acquire the target image 10. When the target image 10 is transmitted by a camera, the acquisition unit 2020 can acquire the target image 10 in real time.

[0036] When the acquisition unit 2020 acquires the target video, the target video may be acquired in the same manner as the target image 10. The acquisition unit 2020 may acquire the target video in real time. Specifically, the video camera that generates the target video can repeatedly execute "capturing the surrounding scenery to generate video frames of the target video and outputting the generated video frames to the posture analysis device 2000". In this case, the acquisition unit 2020 receives the video frames sequentially transmitted from the video camera and uses the time series of the received video frames as the target video.

[0037] <Acquisition of target person information 20: S104> The acquisition unit 2020 acquires target person information 20 indicating the target person (S104). The target person information 20 indicates the target person so that the posture analysis device 2000 can detect the target person from the target image 10 based on the target person information 20.

[0038] For example, the acquisition unit 2020 may acquire target person information 20 including a sample image of the target person in which a part (e.g., face) or the whole of the target person is imaged. As another example, the acquisition unit 2020 may acquire target person information 20 including feature amounts extracted from the sample image of the target person. As another example, when the relative positions of the respective persons imaged in the target image 10 are predefined, the target person information 20 may indicate the position of the target person. Suppose the target image 10 includes four persons and the target person is always imaged in the upper left region of the target image. In this case, the target person information 20 indicates "upper left" as the position of the target person.

[0039] The above target person information 20 can be acquired by the same method as the method for acquiring the target image 10.

[0040] As another example, the acquisition unit 2020 may prompt the user of the posture analysis device 2000 to input the target person information 20. In this case, the acquisition unit 2020 may output the target image 10 and allow the user to select from the persons imaged in the target image 10. Specifically, the acquisition unit 2020 outputs the target image 10 to a display device and causes the display device to display the target image 10. The target image 10 is displayed on the display device so that the user can select from the persons imaged in the target image 10. When the user selects from the displayed persons, the acquisition unit 2020 acquires, as the target person information 20, information (e.g., coordinates on the target image 10 specified by the user) that allows the posture analysis device 2000 to determine which person was selected by the user.

[0041] <Estimation of posture: S106> The estimation unit 2040 estimates the posture of the person imaged in the target image 10 (S106). There are various methods for posture estimation, and any of these methods can be applied to the estimation unit 2040. For example, the estimation unit 2040 detects the positions of characteristic parts of the human body (such as the neck, eyes, shoulders, etc.) from the target image 10 as key points. Then, the estimation unit 2040 divides the key points into groups (referred to as key point groups) that include key points belonging to the same person, and estimates the posture of each person based on the key point group corresponding to that person.

[0042] The posture of a person can be classified into any of a predetermined type of postures (such as a jump, spin, or step in figure skating). In this case, the posture of a specific person is represented by a pair of the key point group of that person and a type label indicating the type of the posture of that person. To recognize the type of a person's posture, the estimation unit 2040 can include a classification model configured to obtain a set of key points of a person (i.e., a key point group) and output a type label indicating the type of the person's posture. The classification model can be implemented by a machine learning-based model such as a neural network.

[0043] As will be described later, the calculation unit 2060 may use the time series of a person's posture instead of a single posture of a person to calculate the posture score of the person. In this case, the estimation unit 2040 uses the time series of the target image 10 to estimate the posture of the person from each target image 10, thereby obtaining the time series of the posture of each person. Note that the time series of postures can also be referred to as "motion". Therefore, when calculating the posture score using the time series of a person's posture, it can be said that the posture analysis device 2000 calculates a posture score indicating how high the quality of the person's motion is.

[0044] <Calculation of posture score: S108> The calculation unit 2060 calculates a posture score for each person (S108). An example of the method for calculating the posture score will be described below.

[0045] <<Example 1>> In some embodiments, the calculation unit 2060 calculates a posture score representing the similarity between the posture of a person and a predetermined sample posture. The sample posture can be defined by a series of key points representing an ideal posture. In this case, it can be said that the higher the similarity between the posture of the person and the sample posture, the higher the quality of the posture.

[0046] There are various methods for quantifying the similarity between two postures, and one of these methods can be applied to the calculation unit 2060 to calculate the posture score. Briefly speaking, the similarity between the posture of a person and the sample posture can be represented by the similarity between the spatial arrangement of the key points within the key point group of that person and the spatial arrangement of the key points of the sample posture.

[0047] In some embodiments, the calculation unit 2060 includes a machine learning-based feature extractor such as a neural network, which is configured to obtain a key point group as an input and output a feature amount of the posture represented by the key point group (for example, a feature amount of the spatial arrangement of the key points in the key point group). In this case, the calculation unit 2060 inputs the key point group of the person into the feature extractor to obtain the feature amount of the person's posture. Also, the calculation unit 2060 inputs the key point group of the sample posture into the feature extractor to obtain the feature amount of the sample posture. Then, the calculation unit 2060 calculates a value representing the similarity between the feature amount of the person's posture and the feature amount of the sample posture as the posture score. Note that there are various methods for quantifying the similarity between two feature amounts, and one of them can be applied to the calculation unit 2060 to quantify the similarity between the feature amount of the person's posture and the feature amount of the sample posture.

[0048] Note that in order to calculate the posture score, the calculation unit 2060 may obtain information (for example, a key point group representing the sample posture) called "sample information" indicating the sample posture. The sample information can be stored in advance in the storage device so that the posture analysis device 2000 can obtain the sample information.

[0049] A plurality of sample postures may be prepared. Hereinafter, these sample postures are referred to as "candidate sample postures". In this case, the calculation unit 2060 selects one of these candidate sample postures as the sample posture to be used for calculating the similarity score. For example, the user of the posture analysis device 2000 designates a candidate sample posture to be used as the sample posture. In this case, the calculation unit 2060 may acquire the candidate sample posture designated by the user as the sample posture and calculate the similarity score.

[0050] As another example, the calculation unit 2060 may select any one of the candidate sample postures based on the similarity between the posture of the target person and each candidate sample posture. Specifically, the calculation unit 2060 can calculate a candidate score representing the similarity between the candidate sample posture and the posture of the target person for each candidate sample posture. Then, the calculation unit 2060 selects the candidate sample posture with the largest candidate score as the sample posture and can calculate the posture score. Note that the method for calculating the candidate score is the same as the method for calculating the posture score based on the sample posture described above.

[0051] In some embodiments, the calculation unit 2060 may calculate the posture score using the time series (motion) of the person's posture. In this case, a time series of sample postures called "sample motions" may be prepared in advance. A sample motion can be represented by a time series of groups of keypoints each representing a sample posture at a certain point in time. The calculation unit 2060 calculates a posture score representing the similarity between the motion of each person and the sample motion.

[0052] There are various methods for quantifying the similarity between two motions, and one of these methods can be applied to the calculation unit 2060. Briefly speaking, the similarity between the motion of a person and the sample motion can be represented by the similarity between the time series of the spatial arrangement of the keypoints of that person and the time series of the spatial arrangement of the keypoints of the sample motion.

[0053] In some embodiments, the calculation unit 2060 includes a machine learning-based feature extractor such as a neural network, which is configured to obtain a time series of a key point group as an input and output a feature amount of an action represented by the key point group. In this case, the calculation unit 2060 inputs the time series of the key point group of a person into the feature extractor to obtain the feature amount of the action of that person. Further, the calculation unit 2060 inputs the time series of the key point group of the sample action into the feature extractor to obtain the feature amount of the sample action. Then, the calculation unit 2060 calculates a value representing the similarity between the feature amount of the person's action and the feature amount of the sample action as a posture score.

[0054] In addition, when a plurality of candidates for sample actions called "candidate sample actions" are prepared, the calculation unit 2060 may select any one of those candidate sample actions as the sample action for calculating the posture score. This selection can be performed in the same manner as the method of selecting a sample action from candidate sample actions.

[0055] For example, the calculation unit 2060 can obtain sample information for specifying a candidate sample action and select the candidate sample action specified by the sample information as the sample action for calculating the posture score. Further, the calculation unit 2060 can calculate a candidate score for each candidate sample action and select the candidate sample action with the largest candidate score as the sample action for calculating the posture score. In this case, the candidate score can be calculated as a value representing the similarity between the candidate sample action and the action of the target person.

[0056] <<<Consideration of the Trajectory of Representative Key Points>>> In addition to the action of a person, the calculation unit 2060 may use the time series (in other words, the trajectory) of the representative key points of the person. The representative key point of a person can be, for example, one of the key points included in the key point group of the person, such as the key point of the right ankle. Note that multiple types (for example, the key point of the right ankle and the key point of the left waist) of representative key points can be used for calculating the posture score.

[0057] In addition, when the calculation unit 2060 calculates the posture score using the trajectory of the representative key points, the similarity between the motion of the person and the sample motion may be calculated as the first score S1, and the similarity between the trajectory of the representative key points of the person and the trajectory of the representative key points of the sample motion may be calculated as the second score S2. Then, the first score S1 and the second score S2 are aggregated to obtain the posture score S. For example, as follows, the weighted sum of the first score S1 and the second score S2 can be calculated as the posture score S. [Number]

[0058] In addition, when a plurality of types of representative key points are defined, the similarity between the trajectory of the representative key points of the person and the trajectory of the representative key points of the sample motion is calculated for each type of representative key point, and aggregated into the second score S2. This aggregation can be performed in the same manner as the method of aggregating the first score and the second score into the posture score.

[0059] When different types of postures are captured in the target image 10, since there is an ideal posture for each type of posture, it is preferable to prepare a sample posture for each type of posture. Also, the target person may want to refer to a person taking the same type of posture as the target person.

[0060] In this case, the estimation unit 2040 estimates the posture of each person by generating a key point group of the person and determining the type label of the person. Then, the calculation unit 2060 identifies a person whose type label is the same as that of the target person, and calculates the posture score of each identified person based on the sample posture corresponding to the type label of the target person. The sample posture is prepared in advance in association with the type label.

[0061] In addition, since the pose score is calculated only for a person whose type label is the same as that of the target person, the detection unit 2080 detects a reference person from among the persons for whom the pose score has been calculated (persons whose type label is the same as that of the target person).

[0062] <<Example 2>> In some embodiments, the calculation unit 2060 can calculate a pose score by dividing the persons imaged on the target image 10 into a plurality of clusters based on the similarity between poses. This means assigning persons with similar poses to the same cluster and persons with different poses to different clusters. In other words, each cluster represents a group of persons with similar poses.

[0063] The calculation unit 2060 calculates the pose score of each person based on the size of the cluster to which the person belongs. If it is known in advance that most persons take high-quality poses, the cluster of persons taking high-quality poses may include more persons than the cluster of persons taking low-quality poses. Therefore, the calculation unit 2060 can assign a larger pose score to a person as the number of persons in the cluster to which the person belongs is larger. Specifically, the calculation unit 2060 can calculate the ratio of the persons included in the cluster to which a specific person belongs as the pose score of that person.

[0064] On the other hand, if it is known in advance that most persons take low-quality poses, the cluster of persons taking low-quality poses may include more persons than the cluster of persons taking high-quality poses. Therefore, the calculation unit 2060 can assign a larger pose score to a person as the number of persons in the cluster to which the person belongs is smaller. Specifically, the calculation unit 2060 can calculate the reciprocal of the ratio of the persons included in the cluster to which a specific person belongs as the pose score of that person.

[0065] To divide the persons into clusters, the calculation unit 2060 can perform clustering such as k-means clustering on the key point groups of the persons. Note that the key point group can be represented by multi-dimensional data (for example, an array of the positions of body parts), and there are various methods for performing clustering on a set of multi-dimensional data. Therefore, one of those methods can be applied to the calculation unit 2060 to perform clustering on the set of key point groups. Note that the number of clusters may be defined in advance or may be dynamically determined as a result of clustering.

[0066] <Detection of reference person: S108> The detection unit 2080 detects a reference person from the target image 10 (S108). First, the detection unit 2080 determines which person is the target person based on the target person information 20. There are various methods for detecting a specific person from an image based on the information (for example, feature amount) of the person, and one of those methods can be applied to the detection unit 2080 to detect the target person from the target image 10. Note that when the calculation unit 2060 has already detected the target person from the target image 10, it is not necessary for the detection unit 2080 to detect the target person from the target image 10.

[0067] The detection unit 2080 determines whether there is one or more persons whose pose score is greater than the pose score of the target person. When there is no person whose pose score is greater than the pose score of the target person, the detection unit 2080 determines that no reference person is detected from the target image 10. On the other hand, when there is one or more persons whose pose score is greater than the pose score of the target person, the detection unit 2080 can identify one or more of them as the reference person.

[0068] In some embodiments, the number of reference persons may be predefined. When the number of reference persons is one, the detection unit 2080 identifies the person with the largest pose score as the reference person. When the number of reference persons is defined as N (N>1), the detection unit 2080 may identify the persons from the 1st to the nth in descending order of pose score as the reference persons.

[0069] In other embodiments, the number of reference persons may not be predefined. For example, the detection unit 2080 may divide the persons captured in the target image 10 into a plurality of groups called "pose groups" based on the pose scores, and may identify one or more of these pose groups as the group of reference persons. Hereinafter, the group of reference persons is referred to as the "reference group". The reference group is a pose group that includes persons with a pose score greater than the pose score of the target person.

[0070] The pose groups may be associated with different pose score ranges called "score ranges". The score ranges are defined so as not to overlap with each other. Assume that the entire range of the pose score S is 0 <= S <= 100 and three pose groups GP1, GP2, and GP3 are defined. In this case, the pose groups GP1, GP2, and GP3 may be defined as the pose group GP1 having a score range of 0 <= S < 33, the pose group GP2 having a score range of 33 <= S < 66, and the pose group GP3 having a score range of 66 <= S <= 100.

[0071] The detection unit 2080 identifies, for each person, any score range including the person's posture score, and assigns the person to a posture group corresponding to the identified score range. Assume that there are the three posture groups GP1, GP2, and GP3 described above. Also assume that there are six persons: person P1 with a posture score of 20, person P2 with a posture score of 70, person P3 with a posture score of 60, person P4 with a posture score of 45, person P5 with a posture score of 10, and person P6 with a posture score of 80. In this case, persons P1 and P5 are assigned to the posture group GP1, persons P3 and P4 are assigned to the posture group GP2, and persons P2 and P6 are assigned to the posture group GP3.

[0072] The detection unit 2080 identifies a posture group whose posture score is higher than that of the posture group to which the target person belongs. Assume that the target person is person P3 in the above example. In this case, the posture group to which the target person belongs is the posture group GP2. Therefore, since the posture score of the posture group GP3 (i.e., 66 <= S < 100) is higher than the posture score of the posture group GP2 (i.e., 33 <= S < 66), the detection unit 2080 identifies the posture group GP3 as the reference group.

[0073] When there are a plurality of posture groups whose posture scores are higher than that of the posture group to which the target person belongs, the detection unit 2080 may identify all or part of those posture groups as the reference group. For example, the detection unit 2080 may identify the posture group with the highest posture score as the reference group.

[0074] When the detection unit 2080 divides the people into clusters by the calculation unit 2060 and calculates the posture scores, each cluster can be treated as a posture group. In this case, the detection unit 2080 can calculate statistical values (such as the average value) of the posture scores of the people belonging to the posture group for each posture group. Then, the detection unit 2080 can identify the posture group whose statistical value of the posture score is greater than the posture score of the target person as the reference group. Also, the detection unit 2080 may identify the posture group whose statistical value of the posture score is greater than the statistical value of the posture score of the posture group to which the target person belongs as the reference group.

[0075] <Output of reference information 20: S110> The output unit 2100 outputs the reference information 30 (S110). The reference information 30 includes one or more pieces of information about one or more reference persons. In some embodiments, the output unit 2100 modifies the target image 10 so that the viewer of the modified target image 10 can grasp one or more reference persons, and includes the modified target image 10 (hereinafter, "output image") in the reference information 30.

[0076] For example, the output image includes marks such as a bounding box above or around the reference person. When two or more reference persons are detected, the output image can be generated by modifying the target image 10 so as to add a common mark such as a bounding box of the same color to the reference persons above or around them. The output image preferably also shows the target person so that the target person can be distinguished from the reference persons.

[0077] FIG. 5 shows an example of the output image. The output image 60 includes a mark 70 indicating the target person and marks 80-1 and 80-2 indicating the reference persons. The mark 70 is a solid-line bounding box, and the mark 80 is a dotted-line bounding box. Since the types of these lines are different, the viewer of the output image 60 can easily and naturally distinguish the target person from the reference persons.

[0078] If multiple reference groups are detected, the output image can include a common mark for each reference group. This means that reference persons belonging to the same reference group are associated with the same type of mark (for example, bounding boxes of the same color as each other, or bounding boxes with the same type of line as each other). On the other hand, reference persons belonging to different reference groups are associated with different types of marks (for example, bounding boxes with different colors from each other, or bounding boxes with different types of lines from each other).

[0079] The output unit 2100 can generate an output image in which the pose of the reference person is superimposed on the target person. FIG. 6 shows an image of the target person with the pose of the reference person superimposed. In FIG. 6, the pose of the reference person is indicated by the key points 40 and the links 50, and is superimposed on the image 90 of the target person. The link 50 represents the connection between adjacent key points such as the neck and the right shoulder, or the left hip and the left knee.

[0080] When superimposing the pose of the reference person on the target person, the output unit 2100 can adjust the pose of the reference person to match the target person. For example, the output unit 2100 can match the size of the reference person to the size of the target person. In another example, the output unit 2100 can match the orientation of the reference person to the orientation of the target person.

[0081] The reference information 30 can also include information indicating one or more features of the target person, the reference person, or both (for example, the similarity score of the person, the accuracy of the pose estimation for the person, or the appropriate or inappropriate points of the person). These information may be included inside or outside the output image.

[0082] The accuracy of the pose estimation of a specific person represents how accurately the estimation unit 2040 estimated the pose of that person. In this case, the estimation unit 2040 is configured to calculate the accuracy of the pose estimation for each person while estimating the pose of that person.

[0083] To calculate the accuracy of pose estimation, the estimation unit 2040 may be configured to include a machine learning-based pose estimator such as a neural network that can calculate the accuracy of pose estimation. Specifically, the pose estimator may be configured to acquire an image region in which a specific person is imaged and output, for each predetermined part of the human body, a key point of that part and the accuracy of that key point. The accuracy of pose estimation of a specific person may be represented by a list of the accuracies of the key points of that person or by a statistical value (such as an average value) of the accuracies of the key points of that person.

[0084] As described above, other examples of the characteristics of a person that may be included in the reference information 30 are the appropriate points, inappropriate points, or both of that person. When these pieces of information are included in the reference information 30, the calculation unit 2060 may further calculate a score for each key point (referred to as a "point score") indicating the degree of similarity between the key point of the target person and the key point of the sample pose.

[0085] For example, the output unit 2100 may identify one or more key points of the target person whose point scores are greater than a predetermined threshold Th1 as appropriate key points. Then, the output unit 2100 generates the reference information 30 indicating the appropriate key points of the target person.

[0086] Similarly, the output unit 2100 may identify one or more key points of the target person whose point scores are less than a predetermined threshold Th2 (Th2 < Th1) as inappropriate key points. Then, the output unit 2100 generates the reference information 30 indicating the inappropriate key points of the target person.

[0087] The posture analysis device 2000 may store history information including the history of inappropriate keypoints for each person. In this case, the output unit 2100 can determine whether the inappropriate keypoints of the currently detected target person are included in the history information and determine the content of the reference information 30. If the currently detected inappropriate keypoints are included in the history information, it can be said that the user should pay attention to the part corresponding to this inappropriate keypoint and improve the posture. Therefore, the output unit 2100 may generate the reference information 30 that emphasizes the inappropriate keypoints included in the history information more than the inappropriate keypoints not included in the history information.

[0088] In addition, the reference information 30 may not only point out the inappropriate keypoints but also include a method for improving the posture of the target person. For example, it is assumed that the position of the keypoint of the right knee of the target person is determined to be an inappropriate keypoint because it is lower than the ideal keypoint (that is, the position of the right knee keypoint of the sample posture). In this case, the posture of the target person can be improved by raising the right knee more. Therefore, the output unit 2100 generates the reference information 30 including a message indicating that the target person should raise the right knee more.

[0089] The reference information 30 may include, in addition to or instead of, the posture scores of individual persons, statistical values related to the posture scores. An example of the statistical value related to the posture score is the number of persons whose posture scores are greater than the posture score of the target person (in other words, the number of persons whose posture quality is higher than the posture quality of the target person).

[0090] When the reference group is generated, the reference information 30 may include one or more statistical values related to the posture scores for each reference group (for example, the average, variance, maximum, or minimum of the posture scores of the reference persons within the reference group).

[0091] In addition, the posture analysis device 2000 may acquire the target image 10 that constitutes the target video and output the reference information 30 in real time. In this case, the viewer can easily grasp the reference person in real time.

[0092] For example, assume that a camera is installed in a lesson room where a plurality of trainees receive performance lessons, the target image 10 that constitutes the target video is generated, and is transmitted to the posture analysis device 2000. Further, the posture analysis device 2000 receives the target image 10, detects the reference person, and outputs a sequence of output images (referred to as an output video) to the display device in real time.

[0093] In this case, the user of the posture analysis device 2000 (for example, one of the trainees) can easily refer to the performance of other trainees with higher quality than himself / herself. Thereby, the user can compare his / her own performance with the performance of the reference person and recognize the differences and points to be improved.

[0094] Note that there are various methods for the output unit 2100 to output the reference information 30. In some embodiments, the reference information 30 can be stored in a storage device, displayed on a display device, or transmitted to other computers such as the PC or smartphone of the user of the posture analysis device 2000.

[0095] Embodiment 2 <Overview> FIG. 7 shows an overview of the posture analysis device 2000 according to Embodiment 2. Note that the overview in FIG. 7 shows an operation example of the posture analysis device 2000 according to Embodiment 2 to facilitate understanding of the posture analysis device 2000 according to Embodiment 2, and does not limit or narrow the scope of the operation of the posture analysis device 2000 according to Embodiment 2.

[0096] The posture analysis device 2000 of Embodiment 2 is configured to calculate a posture score based on a sample motion and the trajectory of representative key points of the sample motion. As described in Embodiment 1, multiple types of representative key points can be used to calculate the posture score.

[0097] For example, as illustrated in Embodiment 1, the posture analysis device 2000 of Embodiment 2 can calculate the similarity between the motion of a person and the sample motion as a first score S1, and calculate the similarity between the trajectory of the representative key points of the person and the trajectory of the representative key points of the sample motion as a second score S2. Then, the first score S1 and the second score S2 are aggregated to obtain a posture score S.

[0098] Note that the posture analysis device 2000 of Embodiment 2 acquires the target person information 20 and does not need to detect a reference person from the person captured in the target image 10.

[0099] Note that the posture analysis device 2000 can use this posture score to generate and output information called "output information". For example, the output unit 2100 of Embodiment 2 can divide a person into posture groups based on the posture score and generate output information indicating one or more posture groups. Also, in order to indicate the posture groups, an output video generated by modifying the target image 10 may be included in the output information. Specifically, a common mark may be added to the persons belonging to the same posture group in the output image.

[0100] In another example, the output information may include one or more statistical values calculated based on the posture score. For example, the statistical values can include the average, variance, maximum, minimum, etc. of the posture score. These statistical values may be calculated for each posture group or for all the persons captured in the target image 10. In another example, the statistical values can include the number or ratio of persons in each posture group.

[0101] <Examples of advantageous effects> According to the posture analysis device 2000 of Embodiment 2, a time series of target images in which a plurality of persons are imaged is acquired, and the postures of each person are estimated. Then, based on the sample motion and the trajectory of the representative key points of the sample motion, the posture score of each person is calculated. Thereby, a novel technique for analyzing the posture of a person using an image in which a plurality of persons are imaged is provided.

[0102] In addition, not only the sample motion and the person's motion are compared, but also the trajectory of the representative key points of the sample motion and the trajectory of the representative key points of the person are compared to calculate the posture score. Therefore, the posture analysis device 2000 of Embodiment 2 can evaluate the person's motion more accurately than when evaluating the person's motion by comparing only the sample motion and the person's motion.

[0103] Hereinafter, a more detailed description of the posture analysis device 2000 will be given.

[0104] <Example of functional configuration> FIG. 8 is a block diagram showing an example of the functional configuration of the posture analysis device 2000 of Embodiment 2. The posture analysis device 2000 includes an acquisition unit 2020, an estimation unit 2040, and a calculation unit 2060. The acquisition unit 2020 acquires the time series of the target image 10. Note that the acquisition unit 2020 of Embodiment 2 does not need to acquire the target person information 20. The estimation unit 2040 estimates the motion of the person imaged in the target image 10. The calculation unit 2060 calculates the posture score of the person based on the sample motion and the trajectory of the representative key points of the sample motion.

[0105] Although not shown in FIG. 8, the posture analysis device 2000 of Embodiment 2 may further include an output unit 2100 that generates and outputs the above output information.

[0106] <Example of hardware configuration> The posture analysis device 2000 of Embodiment 2 can be realized in the same manner as the posture analysis device 2000 of Embodiment 1. For example, the posture analysis device 2000 of Embodiment 2 is realized by the computer 1000 shown in FIG. 3. However, the storage device 1080 of Embodiment 2 includes a program for realizing the functions of the posture analysis device 2000 of Embodiment 2.

[0107] <Flow of processing> FIG. 9 is a flowchart showing an example of the flow of processing executed by the posture analysis device 2000 of Embodiment 2. The acquisition unit 2020 acquires the time series of the target image 10 (S202). The estimation unit 2040 estimates the movement of the person imaged in the target image 10 (S204). The calculation unit 2060 calculates the posture score of each person based on the sample motion and the trajectory of the representative keypoints of the sample motion (S206).

[0108] The program can be stored using various types of non-transitory computer readable media and provided to a computer. Non-transitory computer readable media include various types of tangible storage media. Examples of non-transitory computer readable media include magnetic recording media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (e.g., magneto-optical disks), CD-ROM, CD-R, CD-R / W, semiconductor memories (e.g., mask ROM, PROM (Programmable ROM), EPROM (Erasable PROM), flash ROM, RAM). Also, the program may be provided to the computer by various types of transitory computer readable media. Examples of transitory computer readable media include electrical signals, optical signals, and electromagnetic waves. The transitory computer readable media can supply the program to the computer via wired communication paths such as electric wires and optical fibers, or wireless communication paths.

[0109] The present disclosure has been described with reference to the embodiments, but the present disclosure is not limited to the above-described embodiments. Various changes that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the invention.

Explanation of Signs

[0110] 10 Target image 20 Target person information 30 Reference information 40 Key point 50 Link 60 Output image 70 Mark 80 Mark 90 Image 1000 Computer 1020 Bus 1040 Processor 1060 Memory 1080 Storage device 1100 Input / output interface 1120 Network interface 2000 Posture analysis device 2020 Acquisition unit 2040 Estimation unit 2060 Calculation unit 2080 Detection unit 2100 Output unit

Claims

1. At least one storage element configured to store commands, and at least one processor, and the at least one processor executes the commands to acquire a target image in which two or more persons are imaged, acquire target person information indicating a target person, estimate the posture of each person imaged in the target image, calculate a posture score representing the quality of the posture of each person, detect one or more reference persons having a posture of higher quality than the posture of the target person, and output reference information indicating the reference persons, a posture analysis device configured as such.

2. The calculation of the posture score includes acquiring a sample posture, and for each person, calculating the similarity between the posture of the person and the sample posture to calculate the posture score, the posture analysis device according to Claim 1.

3. The calculation of the posture score includes, for each person, calculating a first score and a second score, and aggregating the first score and the second score into the posture score, the first score representing the similarity between the posture of the person and the sample posture, and the second score representing the similarity between the trajectory of the representative keypoints of the person and the trajectory of the representative keypoints in the sample posture, the posture analysis device according to Claim 2.

4. The acquisition of the sample posture includes, for each candidate sample posture, calculating a candidate score representing the similarity between the candidate sample posture and the posture of the target person, and selecting the candidate sample posture having the maximum candidate score as the sample posture, the posture analysis device according to Claim 2 or 3.

5. The calculation of the posture score includes performing clustering on the person based on the posture of the person to divide the person into two or more clusters, and for each person, calculating the posture score based on the size of the cluster to which the person belongs, the posture analysis device according to Claim 1.

6. The reference information includes an output image generated by changing the target image so that a mark indicating the reference person is shown, the posture analysis device according to any one of Claims 1 to 5.

7. In the output image, the posture of the reference person is superimposed on the target person, the posture analysis device according to Claim 6.

8. Obtaining a target image in which two or more persons are imaged; Obtaining target person information indicating a target person; Estimating the posture of each of the persons imaged in the target image; Calculating, for each of the persons, a posture score representing the quality of the posture of the person; Detecting one or more reference persons whose postures are of higher quality than the quality of the posture of the target person; Outputting reference information indicating the reference persons, which is a posture analysis method executed by a computer.

9. The calculation of the posture score includes: Obtaining a sample posture; Calculating, for each of the persons, the similarity between the posture of the person and the sample posture to calculate the posture score, which is the posture analysis method according to claim 8.

10. The calculation of the posture score includes: Calculating a first score and a second score for each of the persons, and aggregating the first score and the second score into the posture score. The first score represents the similarity between the posture of the person and the sample posture, and the second score represents the similarity between the trajectory of the representative keypoints of the person and the trajectory of the representative keypoints in the sample posture, which is the posture analysis method according to claim 9.

11. The obtaining of the sample posture includes: Calculating, for each candidate sample posture, a candidate score representing the similarity between the candidate sample posture and the posture of the target person; Selecting, as the sample posture, the candidate sample posture having the maximum candidate score, which is the posture analysis method according to claim 9 or 10.

12. The calculation of the posture score includes: Performing clustering on the person based on the posture of the person to divide the person into two or more clusters; Calculating, for each person, the posture score based on the size of the cluster to which the person belongs, which is the posture analysis method according to claim 8.

13. The reference information includes an output image generated by changing the target image so that a mark indicating the reference person is shown, which is the posture analysis method according to any one of claims 8 to 12.

14. In the output image, the posture of the reference person is superimposed on the target person, which is the posture analysis method according to claim 13.

15. Obtaining a target image in which two or more persons are imaged; Obtaining target person information indicating the target person; Estimating the posture of each person captured in the target image; Calculating a posture score representing the quality of the posture of each person; Detecting one or more reference persons having a posture of higher quality than the quality of the posture of the target person; Outputting reference information indicating the reference person; A non-transitory computer-readable storage medium storing a program for causing a computer to execute.

16. The calculation of the posture score includes: Obtaining a sample posture; Calculating the similarity between the posture of each person and the sample posture for each person, and calculating the posture score. The storage medium according to claim 15.

17. The calculation of the posture score includes: Calculating a first score and a second score for each person, and aggregating the first score and the second score into the posture score. The first score represents the similarity between the posture of the person and the sample posture, and the second score represents the similarity between the trajectory of the representative keypoints of the person and the trajectory of the representative keypoints in the sample posture. The storage medium according to claim 16.

18. The obtaining of the sample posture includes: Calculating a candidate score representing the similarity between each candidate sample posture and the posture of the target person for each candidate sample posture; Selecting the candidate sample posture having the maximum candidate score as the sample posture. The storage medium according to claim 16 or 17.

19. The calculation of the posture score includes: Performing clustering on the person based on the posture of the person to divide the person into two or more clusters; Calculating the posture score for each person based on the size of the cluster to which the person belongs. The storage medium according to claim 15.

20. The reference information includes an output image generated by changing the target image so that a mark indicating the reference person is shown. The storage medium according to any one of claims 15 to 19.

21. In the output image, the posture of the reference person is superimposed on the target person. The storage medium according to claim 20.

Images