Posture estimation device, learning device, posture estimation method and program

Abstract

To provide a posture estimation device, a program and a learning device for improving the accuracy of posture estimation.SOLUTION: A posture estimation device 10 comprises: a person region image information generation unit 11 for extracting a person region from inside of an image and generating person region image information on the basis of the extracted person region image; a joint point information generation unit 12 for extracting a person's joint point from inside of an image and generating joint point information on the basis of the extracted joint point; a feature quantity information generation unit 13 for generating feature quantity information on the basis of both of the person region image information and the joint point information; and an estimation unit 14 for estimating the posture of the person included in the image, on the basis of an estimation model that accepts the feature quantity information as input and delivers the result of posture estimation as output.SELECTED DRAWING: Figure 11

Description

[Technical field]

[0001] The present invention relates to a posture estimation device, a learning device, a posture estimation method, and a program. [Background technology]

[0002] A technology related to the present invention is disclosed in Patent Document 1. Patent Document 1 discloses a technology for integrating the scores of each of a plurality of classes calculated by an engine that estimates the behavior of a person included in an image by image analysis and the scores of each of a plurality of classes calculated by an engine that estimates the behavior of a person included in an image based on joint point information to calculate an integrated score for each of a plurality of classes.

[0003] Non-Patent Document 1 is a document about Transformer, which is an estimation model equipped with a self-attention mechanism. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2019-144830 A [Non-patent literature]

[0005] [Non-Patent Document 1] Ashish Vaswani et al., "Attention Is All You Need", [online], [Retrieved January 22, 2021], Internet <URL: https: / / papers.nips.cc / paper / 2017 / file / 3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf> Summary of the Invention [Problem to be solved by the invention]

[0006] In the case of the technology disclosed in Patent Document 1, after classifying based on image information and classifying based on joint point information separately, the results of each classifying are integrated. As will be shown in the following embodiment, the accuracy improvement rate is low in the case of processing in which the results of classifying separately are simply integrated in this way. An object of the present invention is to improve the accuracy of posture estimation. [Means for solving the problem]

[0007] According to the present invention, a person area image information generating means for extracting a person area from an image and generating person area image information based on an image of the extracted person area; a joint point information generating means for extracting joint points of a person from the image and generating joint point information based on the extracted joint points; a feature amount information generating means for generating feature amount information based on both the person area image information and the joint point information; an estimation means for estimating a posture of a person included in the image based on an estimation model that receives the feature amount information as an input and outputs a posture estimation result; A pose estimation apparatus is provided having the following:

[0008] Further, according to the present invention, The computer Extracting a person area from the image, and generating person area image information based on the image of the extracted person area; extracting joint points of a person from the image, and generating joint point information based on the extracted joint points; generating feature information based on both the person area image information and the joint point information; There is provided a posture estimation method for estimating a posture of a person included in the image based on an estimation model that receives the feature amount information as an input and outputs a posture estimation result.

[0009] Further, according to the present invention, Computer, a person area image information generating means for extracting a person area from an image and generating person area image information based on an image of the extracted person area; a joint point information generating means for extracting joint points of a person from the image and generating joint point information based on the extracted joint points; a feature amount information generating means for generating feature amount information based on both the person area image information and the joint point information; an estimation means for estimating a posture of a person included in the image based on an estimation model that receives the feature amount information as an input and outputs a posture estimation result; A program is provided to function as a

[0010] Further, according to the present invention, a person area image information generating means for extracting a person area from an image and generating person area image information based on an image of the extracted person area; a joint point information generating means for extracting joint points of a person from the image and generating joint point information based on the extracted joint points; a feature amount information generating means for generating feature amount information based on both the person area image information and the joint point information; a learning means for learning an estimation model in which the feature information is input and a posture estimation result is output; A learning device is provided having the following: Effect of the Invention

[0011] According to the present invention, the accuracy of pose estimation is improved. [Brief description of the drawings]

[0012] [Figure 1] FIG. 2 is a diagram showing an overall view of the processing executed by the learning device of the present embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of a learning device and a posture estimation device according to the present embodiment. [Diagram 3] FIG. 2 is an example of a functional block diagram of a learning device according to the present embodiment. [Figure 4] 10A to 10C are diagrams for explaining the processing of a person area image information generating unit according to the present embodiment. [Diagram 5] 11A to 11C are diagrams for explaining the processing of a joint point information generating unit according to the present embodiment. [Figure 6] 11A to 11C are diagrams for explaining the processing of a joint point information generating unit according to the present embodiment. [Figure 7] 11 is a flowchart showing an example of a processing flow of the learning device of the present embodiment. [Figure 8] FIG. 2 is a diagram showing an overall view of the processing executed by the learning device of the present embodiment. [Figure 9] 11A to 11C are diagrams for explaining the processing of a joint point information generating unit according to the present embodiment. [Figure 10] FIG. 2 is a diagram showing an overall view of the processing executed by the learning device of the present embodiment. [Figure 11] FIG. 2 is an example of a functional block diagram of the posture estimation device according to the present embodiment. [Figure 12] 10 is a flowchart showing an example of a processing flow of the posture estimation device of the present embodiment. [Figure 13] 10A to 10C are diagrams for explaining the effects of the learning device and the posture estimation device according to the present embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In all the drawings, the same components are given the same reference numerals and the description will be omitted as appropriate.

[0014] <First embodiment> "overview" The present embodiment relates to a learning device that learns an estimation model for estimating the posture of a person included in an image to be processed.

[0015] FIG. 1 shows an overview of the processing executed by the learning device of this embodiment. As shown in the figure, the learning device of this embodiment A process of extracting a person area from the image to be processed and generating person area image information based on the image of the extracted person area ((1) in the figure); A process of extracting human joint points from the image to be processed and generating joint point information based on the extracted joint points ((2) in the figure); A process of convolving the person area image information and the joint point information to generate feature information ((3) in the figure). The feature information is learned using a Transformer equipped with a self-attention mechanism (see (4) in the figure). Execute.

[0016] In this way, the learning device of this embodiment performs a characteristic process of generating feature information by convolving person area image information and joint point information, and learning the feature information using a Transformer equipped with a self-attention mechanism.

[0017] "Configuration of learning device" First, an example of the hardware configuration of the learning device will be described. Fig. 2 is a diagram showing an example of the hardware configuration of the learning device. Each functional unit of the learning device is realized by any combination of hardware and software, centering on a central processing unit (CPU) of any computer, memory, programs loaded into the memory, a storage unit such as a hard disk for storing the programs (programs stored beforehand at the stage of shipping the device, as well as programs downloaded from storage media such as a compact disc (CD) or a server on the Internet, can be stored), and a network connection interface. Those skilled in the art will understand that there are various variations in the methods and devices for realizing the above.

[0018] As shown in FIG. 2, the learning device has a processor 1A, a memory 2A, an input / output interface 3A, a peripheral circuit 4A, and a bus 5A. The peripheral circuit 4A includes various modules. The learning device does not have to have the peripheral circuit 4A. The learning device may be composed of multiple devices that are physically and / or logically separated, or may be composed of a single device that is physically and logically integrated. In the former case, each of the multiple devices that make up the learning device can have the above hardware configuration.

[0019] The bus 5A is a data transmission path for the processor 1A, memory 2A, peripheral circuit 4A, and input / output interface 3A to transmit and receive data to each other. The processor 1A is an arithmetic processing device such as a CPU or a GPU (Graphics Processing Unit). The memory 2A is a memory such as a RAM (Random Access Memory) or a ROM (Read Only Memory). The input / output interface 3A includes an interface for acquiring information from an input device, an external device, an external server, an external sensor, etc., and an interface for outputting information to an output device, an external device, an external server, etc. Examples of the input device include a keyboard, a mouse, a microphone, etc. Examples of the output device include a display, a speaker, a printer, a mailer, etc. The processor 1A can issue commands to each module and perform calculations based on the results of those calculations.

[0020] Next, a functional configuration of the learning device will be described. Fig. 3 shows an example of a functional block diagram of the learning device 20. As shown in the figure, the learning device 20 has a person area image information generation unit 21, a joint point information generation unit 22, a feature amount information generation unit 23, and a learning unit 24.

[0021] The person area image information generating unit 21 extracts a person area from the image to be processed, and generates person area image information based on the image of the extracted person area.

[0022] A person region is a region in which a person exists. The extraction of the person region may be realized by using the result of a well-known person detection process (image analysis process) that detects external features such as a face from within an image, or may be realized by using the result of a joint point extraction process generated by the joint point information generating unit 22. Fig. 4 shows an example of a person region extracted from an image to be processed. The region surrounded by a frame is the extracted person region.

[0023] Next, the person area image information will be described. Each pixel of the image of the extracted person area has RGB information (color information). The person area image information is composed of an R image indicating R (red) information of the image of the extracted person area, a G image indicating G (green) information of the image of the extracted person area, and a B image indicating B (blue) information of the image of the extracted person area. The R image, G image, and B image have the same vertical and horizontal sizes, which are predetermined sizes. The image size is, for example, 256×256, but is not limited to this. If the size of the image of the extracted person area is different from the above-mentioned predetermined size, the image size can be adjusted by a well-known image correction process such as enlargement / reduction.

[0024] The joint point information generating unit 22 extracts joint points of a person from the image to be processed, and generates joint point information based on the extracted joint points. The process of analyzing an image and extracting joint points of a person can be realized by utilizing any conventional technology (Open Pose, etc.). According to the joint point information generating unit 22, for example, 18 joint points as shown in FIG. 5 are extracted. Note that the number of joint points to be extracted is a design matter. FIG. 6 shows an example of joint points extracted from the image to be processed. The extracted joint points are indicated by black circles.

[0025] The joint point information is composed of joint point position images corresponding to each of the multiple joint points extracted. When an engine that extracts M joint points is used, the joint point information is composed of M joint point position images. In FIG. 1, it is assumed that an engine that extracts 18 joint points is used, and therefore the joint point information is shown to be composed of 18 joint point position images. However, M=18 is merely an example, and the present invention is not limited to this.

[0026] Each joint point position image corresponding to each joint point indicates the position of each joint point, more specifically, the position of each joint point in the image of the extracted person area. A first joint point position image corresponding to a first joint point indicates the position of the first joint point. The first joint point position image does not indicate the positions of other joint points. Similarly, a second joint point position image corresponding to a second joint point indicates the position of the second joint point. The second joint point position image does not indicate the positions of other joint points.

[0027] Here, an example of a method for generating an articulation point position image will be described. First, the articulation point information generating unit 22 determines a score for each of a plurality of coordinates in the image of the extracted person area. In one example, the score of the coordinate corresponding to the position of the articulation point and the scores of the other coordinates are defined in advance as fixed values. For example, the score of the coordinate corresponding to the position of the articulation point is "1", and the scores of the other coordinates are "0". When generating a first articulation point position image corresponding to a first articulation point, the score of the coordinate corresponding to the position of the first articulation point is "1", and the scores of the other coordinates are "0". Then, when generating a second articulation point position image corresponding to a second articulation point, the score of the coordinate corresponding to the position of the second articulation point is "1", and the scores of the other coordinates are "0".

[0028] Then, the joint point information generating unit 22 generates a joint point position image in which the score of each coordinate is expressed as a heat map. As a modified example of this process, a Gaussian distribution or the like may be used to gradually bring the scores of the coordinates surrounding the coordinate corresponding to the position of the joint point closer to "0." The closer the coordinate is to the coordinate corresponding to the position of the joint point, the closer the value is to "1."

[0029] In addition, among engines for extracting joint points such as Open Pose, there are some that output the above-mentioned joint point position image as an intermediate product. When using such an engine, the joint point information generating unit 22 may acquire the intermediate product (joint point position image) as the joint point information.

[0030] The size of the joint point position image is the same as that of the R image, G image, and B image. However, in the process of generating feature amount information described below, when the person area image information and the joint point position image information are input to different convolutional neural networks, the sizes do not need to be the same.

[0031] The feature amount information generating unit 23 generates feature amount information based on both the person area image information (R image, G image, and B image) and the joint point information (M joint point position images). Specifically, the feature amount information generating unit 23 convolutes the R image, G image, B image, and M joint point position images to generate a feature amount map (feature amount information). As a result, for example, as shown in FIG. 1, the 3×256×256 person area image information and the 18×256×256 joint point information are convolved to generate a 256×16×16 feature amount map. Note that this example is merely an example.

[0032] The feature amount information generating unit 23 may, for example, input the R image, the G image, the B image, and the M joint point position images into one convolutional neural network (for example, Resnet-50, etc.) to generate feature amount information. Alternatively, the feature amount information generating unit 23 may input the R image, the G image, and the B image into one convolutional neural network to generate feature amount information, and separately input the M joint point position images into one convolutional neural network to generate other feature amount information, and then integrate these two pieces of feature amount information by any means to generate one piece of feature amount information.

[0033] The learning unit 24 learns an estimation model that receives feature information as input and outputs an estimation result of a posture. The estimation model is a Transformer equipped with a self-attention mechanism. Details of the estimation model are disclosed in Non-Patent Document 1, and therefore will not be described here. The estimation model outputs the confidence level of each of N predefined postures (classes) (the confidence level that a person included in the image to be processed is taking each posture) as an estimation result. Examples of postures include, but are not limited to, turning on the light, crouching, sitting, standing, walking, holding the head, pointing hands, and waving arms.

[0034] The learning unit 24 inputs the feature information generated by the feature information generating unit 23 to the estimation model to obtain an estimation result (classification result). Then, the learning unit 24 adjusts the parameters of the estimation model based on the result of matching the estimation result with the correct label. The learning process can be realized based on conventional technology.

[0035] Next, an example of the flow of processing by the learning device 20 will be described with reference to the flowchart of FIG.

[0036] When the learning device 20 acquires an image to be processed (S10), it extracts a person area from the image to be processed, and generates person area image information (R image, G image, and B image) based on the image of the extracted person area (S11). The learning device 20 also extracts joint points of the person from the image to be processed, and generates joint point information (M joint point position images) based on the extracted joint points (S12). The processing order of S11 and S12 may be the order shown in FIG. 7, or may be reversed. The processing order of S11 and S12 may also be performed in parallel.

[0037] Next, the learning device 20 generates feature information based on both the person area image information generated in S11 and the joint point information generated in S12 (S13). Specifically, the learning device 20 convolves the R image, the G image, the B image, and M joint point position images to generate a feature map (feature information).

[0038] Next, the learning device 20 learns an estimation model for estimating a posture using the feature amount information generated in S13 as learning data (S14). Specifically, the learning device 20 inputs the feature amount information generated in S13 to the estimation model and obtains an estimation result (classification result). Then, the learning device 20 adjusts parameters of the estimation model based on a result of matching the estimation result with the correct label.

[0039] Thereafter, the learning device 20 repeats the same process.

[0040] "Action and effect" The learning device 20 of this embodiment executes a characteristic process of generating feature information by convolving person area image information and joint point information, and learning the feature information using a Transformer equipped with a self-attention mechanism. With such a learning device 20, the accuracy of pose estimation is improved, as shown in the following verification results.

[0041] <Second embodiment> 8 shows an overall view of the processing executed by the learning device 20 of this embodiment. As shown in the figure, this embodiment differs from the first embodiment in the content of the joint point information. The other contents are the same as those of the first embodiment.

[0042] Here, an example of a method for generating an articulation point position image of this embodiment will be described. First, the articulation point information generating unit 22 determines a score for each of a plurality of coordinates in the image of the extracted person area. In this embodiment, an arithmetic expression is defined in advance for calculating the score (fixed value) of the coordinate corresponding to the position of the articulation point, and the score of other coordinates from the value of the coordinate corresponding to the position of the articulation point. An example of the arithmetic expression is shown in FIG. 9.

[0043] In the figure, (px, py) is the coordinate value corresponding to the position of the joint point. (X-direction encoding) and (Y-direction encoding) are the scores of each coordinate. The values ​​of a and b are fixed values ​​that are determined in advance.

[0044] In this example, the formula for calculating the scores of the other coordinates is: A first arithmetic expression for calculating scores of other coordinates from an x-coordinate value of a coordinate corresponding to the position of the joint point; A second formula for calculating scores of other coordinates from the y-coordinate value of the coordinate corresponding to the position of the joint point; has.

[0045] Then, the joint point information generating unit 22 a process of calculating scores of other coordinates based on the x-coordinate value of the coordinates corresponding to the position of the joint point and the first arithmetic expression, and generating a first joint point position image in which the scores of each coordinate are represented as a heat map as a joint point position image; and a process of calculating scores of other coordinates based on the y-coordinate value of the coordinate corresponding to the position of the joint point and a second arithmetic expression, and generating a second joint point position image in which the scores of each coordinate are represented as a heat map as a joint point position image; Execute both.

[0046] That is, in this embodiment, the joint point information generating unit 22 generates two joint point position images (a first joint point position image and a second joint point position image) corresponding to one joint point. As shown in Fig. 8, the joint point information generated in this embodiment is, for example, 18 x 2 x 256 x 256.

[0047] Although not shown in Fig. 9, in the first joint point position image, the scores of coordinates having the same y coordinate value are the same. And, in the second joint point position image, the scores of coordinates having the same x coordinate value are the same. In the example shown in Fig. 9, the joint point information generating unit 22 determines the score of each coordinate based on the illustrated arithmetic expression and the condition.

[0048] The other configurations of the learning device 20 of this embodiment are similar to those of the first embodiment.

[0049] According to the learning device 20 of this embodiment, the accuracy of pose estimation is improved compared to the learning device 20 of the first embodiment, as shown in the following verification results.

[0050] Furthermore, the learning device 20 of this embodiment is superior to the learning device 20 of the first embodiment in the following points.

[0051] (1) "The relationship between any skeletal points can be referenced from the beginning of the network, regardless of the relative positions of the joint points." In the case of the method of the first embodiment, if the distance between the joint points is large, there is a problem that the relationship between the two cannot be referenced unless the convolution process of ResNet-50 goes to a later layer (where both are within the receptive field of the network). This complicates learning, and may lead to difficulty in learning and reduced accuracy. In contrast, by giving a predetermined score based on the coordinate values ​​of the joint points to coordinates that do not correspond to the positions of the joint points as in the present embodiment, it becomes possible to reference the relationship between any skeleton points from the beginning of the network, regardless of the positional relationship between the joint points. As a result, the inconvenience of the method of the first embodiment can be reduced.

[0052] (2) "No differential calculations are required to reference the positional relationship between joint points." The relative position of a joint point viewed from a certain point is essentially two-dimensional information, such as angle and distance, or Δx and Δy. In the case of the method of the first embodiment, since only one-dimensional information is available for each pixel, a differential operation is required to extract two-dimensional information from the information. This complicates learning, which may lead to difficulty in learning and a decrease in accuracy. In contrast, by giving a predetermined score based on the coordinate value of a joint point to a coordinate that does not correspond to the position of the joint point as in the present embodiment, the relative position of the joint point viewed from a certain point can be grasped based on the score. In other words, the relative position of the joint point viewed from a certain point can be grasped without troublesome differential calculation. As a result, the inconvenience of the method of the first embodiment can be reduced.

[0053] (3) "Articulation point position images can be generated quickly" In one example of the first embodiment, a joint point position image is generated using, for example, a Gaussian distribution. In this case, the calculation process becomes complicated, and the time required for image generation increases. In contrast, in the case of this embodiment, for example, as shown in FIG. 9, a joint point position image can be generated based on the calculation result of a linear expression. Therefore, the inconvenience of the method of the first embodiment can be reduced.

[0054] (4) "Accuracy is achieved even if the size of the joint point position image is small" In the method of the first embodiment, since information is extracted by differential calculation, the accuracy of the information decreases when the image size of the joint point position image is reduced. In contrast, in the case of this embodiment, the necessary information is already decoded, so the accuracy is less likely to decrease even with small images such as 64 x 64 or 32 x 32, making it possible to reduce calculation resources and increase speed.

[0055] <Third embodiment> FIG. 10 shows an overall view of the process executed by the learning device 20 of this embodiment. As shown in the figure, in this embodiment, joint point coordinate information indicating the coordinate values ​​of the joint points of a person extracted by the joint point information generating unit 22 is used in learning the estimation model. Specifically, feature information is generated using the joint point coordinate information as well (by integrating the joint point coordinate information with the output from Resnet-50), and the feature information is input to the Transformer. Then, in addition to the estimation result of the class classification, the Transformer further outputs an estimation result of the coordinate values ​​of the joint points. Then, in adjusting the parameters of the estimation model, in addition to the result of matching the estimation result of the class classification with the correct label, the estimation result of the coordinate values ​​of the joint points and the correct label are further used. This will be described in detail below.

[0056] The feature amount information generating unit 23 generates feature amount information based on the person area image information, the joint point information, and the joint point coordinate information. The person area image information and the joint point information are as described in the first and second embodiments. Although Fig. 10 shows the joint point information generated by the method described in the first embodiment, the joint point information generated by the method described in the second embodiment may also be used.

[0057] The joint point coordinate information indicates the coordinate values ​​of the joint points of the person extracted by the joint point information generating unit 22, more specifically, the coordinate values ​​of each joint point in the image of the person area extracted by the person area image information generating unit 21. Note that the joint point information and the joint point coordinate information are common in that they indicate the positions of the joint points, but differ from each other in that the former is imaged information and the latter is information indicating coordinate values.

[0058] The learning unit 24 receives feature information as input and learns an estimation model that outputs posture estimation results and coordinate values ​​of joint points. The estimation model is a Transformer equipped with a self-attention mechanism. Details of the estimation model are disclosed in Non-Patent Document 1, so a detailed explanation is omitted here. The estimation model outputs the confidence level of each of N predefined postures (classes) (the confidence level that a person included in the image to be processed is taking each posture) as an estimation result. The estimation model also outputs the coordinate values ​​of the joint points as an estimation result.

[0059] The learning unit 24 inputs the feature information generated by the feature information generating unit 23 to the estimation model, and obtains an estimation result (classification result and coordinate value of the joint point).The learning unit 24 then adjusts the parameters of the estimation model based on both the result of matching the class classification result (estimation result) with the correct label and the result of matching the coordinate value of the joint point (estimation result) with the correct label.The learning process can be realized based on conventional technology.

[0060] The other configurations of the learning device 20 of this embodiment are similar to those of the first and second embodiments.

[0061] According to the learning device 20 of this embodiment, the same operational effects as those of the first and second embodiments are achieved. Moreover, according to the learning device 20 of this embodiment, which utilizes the estimation results of the coordinate values ​​of the joint points in learning the estimation model, the estimation accuracy is improved.

[0062] <Fourth embodiment> Pose estimation device 10 of this embodiment has a function of estimating the pose of a person included in an image to be processed, using an estimation model trained by learning device 20 described in the first to third embodiments.

[0063] 11 shows an example of a functional block diagram of pose estimation device 10. As shown in the figure, pose estimation device 10 has a person area image information generation unit 11, a joint point information generation unit 12, a feature amount information generation unit 13, and an estimation unit 14.

[0064] The person area image information generation unit 11 executes the same process as the person area image information generation unit 21 described in the first to third embodiments. The joint point information generation unit 12 executes the same process as the joint point information generation unit 22 described in the first to third embodiments. The feature amount information generation unit 13 executes the same process as the feature amount information generation unit 23 described in the first to third embodiments.

[0065] The estimation unit 14 estimates the posture of the person included in the image to be processed based on the estimation model learned by the learning device 20 described in the first to third embodiments. By inputting the feature information generated by the feature information generation unit 13 to the estimation model, the certainty of each of N predefined postures (classes) (the certainty that the person included in the image to be processed has each posture) is obtained as an estimation result. The estimation unit 14 estimates the posture of the person included in the image to be processed based on this estimation result. For example, the estimation unit 14 may estimate the posture with the highest certainty as the posture of the person included in the image to be processed, or may estimate it using another method.

[0066] Next, an example of the flow of processing by posture estimation device 10 will be described with reference to the flowchart in FIG.

[0067] When pose estimation device 10 acquires an image to be processed (S20), it extracts a person area from the image to be processed, and generates person area image information (R image, G image, and B image) based on the image of the extracted person area (S21). The image to be processed is a still image, an image for one frame of a moving image, or the like. In addition, pose estimation device 10 extracts joint points of the person from the image to be processed, and generates joint point information (M joint point position images) based on the extracted joint points (S22). The processing order of S21 and S22 may be the order shown in FIG. 12, or may be reversed. The processing of S21 and S22 may be performed in parallel.

[0068] Next, pose estimation device 10 generates feature information based on both the person area image information generated in S21 and the joint point information generated in S22 (S23). Specifically, pose estimation device 10 convolutes the R image, the G image, the B image, and M joint point position images to generate a feature map (feature information).

[0069] Next, the posture estimation device 10 estimates the posture of the person included in the image to be processed based on the feature amount information generated in S23 and the estimation model learned by the learning device 20 described in the first to third embodiments (S24). Specifically, the posture estimation device 10 inputs the feature amount information generated in S23 to the estimation model. The estimation model outputs the confidence level of each of N postures (classes) defined in advance (the confidence level that the person included in the image to be processed has each posture) as an estimation result. Examples of postures include, but are not limited to, falling, crouching, sitting, standing, walking, holding the head, pointing the hands, and waving the arms. The posture estimation device 10 estimates the posture of the person included in the image to be processed based on the estimation result. For example, the posture estimation device 10 may estimate the posture with the highest confidence level as the posture of the person included in the image to be processed, or may estimate it using other methods.

[0070] Although not shown, the posture estimation result may be displayed on a display device such as a display. The display device may display the posture estimation result as well as an image / video captured by a camera, an image of a person area, an image showing extracted joint points, a heat map, etc. Furthermore, the posture estimation result may be superimposed on the image / video captured by a camera, an image of a person area, an image showing extracted joint points, a heat map, etc.

[0071] Next, an example of the hardware configuration of posture estimation device 10 will be described. Fig. 2 is a diagram showing an example of the hardware configuration of posture estimation device 10. Each functional unit of posture estimation device 10 is realized by any combination of hardware and software, centering on a CPU (Central Processing Unit) of any computer, memory, programs loaded into the memory, a storage unit such as a hard disk for storing the programs (programs stored in advance at the stage of shipping the device, as well as programs downloaded from storage media such as CDs (Compact Discs) or servers on the Internet, can be stored), and a network connection interface. Those skilled in the art will understand that there are various modified examples of the realization method and device.

[0072] As shown in FIG. 2, posture estimation device 10 has a processor 1A, a memory 2A, an input / output interface 3A, a peripheral circuit 4A, and a bus 5A. The peripheral circuit 4A includes various modules. Posture estimation device 10 does not have to have the peripheral circuit 4A. Note that posture estimation device 10 may be configured from a plurality of physically and / or logically separated devices, or may be configured from a single device that is physically and logically integrated. In the former case, each of the plurality of devices that configure posture estimation device 10 may have the above hardware configuration.

[0073] As described above, posture estimation device 10 of the present embodiment executes a characteristic process of generating feature information by convolving person area image information and joint point information before executing class classification, and then executing class classification using the feature information. With such posture estimation device 10, the accuracy of posture estimation is improved, as shown in the following verification results.

[0074] <Verification results> 13 shows the verification results of Examples 1 and 2 and Comparative Examples 1 to 3. The horizontal axis represents the number of learners (the number of images learned), and the vertical axis represents the recognition accuracy (%).

[0075] Example 1 is an example in which an estimation model is learned by the method described in the first embodiment. Example 2 is an example in which an estimation model is learned by the method described in the second embodiment. Comparative Example 1 is an example in which an estimation model is trained only with person area image information, without using joint point information. Comparative Example 2 is an example in which an estimation model is trained only with joint point information without using person area image information. Comparative Example 3 is an example corresponding to the method disclosed in Patent Document 1. Specifically, this is an example in which a class classification result obtained by an estimation model trained only with person area image information without using joint point information is integrated with a class classification result obtained by an estimation model trained only with joint point information without using person area image information.

[0076] 13, regardless of the number of learners, Examples 1 and 2 have higher recognition accuracy than Comparative Examples 1 to 3. And, in Comparative Examples 1 to 3, the recognition accuracy drops significantly when the number of learners is small, but Examples 1 and 2 have a relatively high recognition accuracy even when the number of learners is small. And, the difference in recognition accuracy between Examples 1 and 2 and Comparative Examples 1 to 3 when the number of learners is small is remarkable.

[0077] Also, from FIG. 13, it can be seen that the second embodiment provides a higher recognition accuracy than the first embodiment.

[0078] Although the embodiments of the present invention have been described above with reference to the drawings, these are merely examples of the present invention, and various configurations other than those described above can also be adopted.

[0079] In this specification, "acquisition" includes at least one of the following: "the device itself goes to retrieve data stored in another device or storage medium (active acquisition)" based on user input or based on program instructions, such as receiving data by making a request or inquiry to another device, or accessing another device or storage medium and reading it, and "the device itself inputs data output from another device (passive acquisition)" based on user input or based on program instructions, such as receiving data that is distributed (or transmitted, push notification, etc.), and selecting and acquiring data from among the received data or information, and "editing data (converting it to text, rearranging data, extracting some data, changing the file format, etc.) to generate new data and acquiring the new data."

[0080] A part or all of the above-described embodiments can be described as, but are not limited to, the following supplementary notes. 1. A person area image information generating means for extracting a person area from an image and generating person area image information based on the image of the extracted person area; a joint point information generating means for extracting joint points of a person from the image and generating joint point information based on the extracted joint points; a feature amount information generating means for generating feature amount information based on both the person area image information and the joint point information; an estimation means for estimating a posture of a person included in the image based on an estimation model that receives the feature amount information as an input and outputs a posture estimation result; A posture estimation device having the following configuration. 2. The joint point information generating means Extract M joint points of a person, The posture estimation device according to claim 1, wherein a plurality of joint point position images, each of which corresponds to one of the M joint points and each of which indicates the position of one of the M joint points, is generated as the joint point information. 3. The score of the coordinate corresponding to the position of the joint point and the scores of other coordinates are defined in advance as fixed values; The joint point information generating means 3. The posture estimation device according to claim 2, further comprising: a joint point position image that represents the score for each coordinate as a heat map. 4. An arithmetic expression for calculating a score of a coordinate corresponding to the position of the joint point and a score of other coordinates from the value of the coordinate corresponding to the position of the joint point is defined in advance; The joint point information generating means Calculating scores for the other coordinates based on the coordinate values ​​corresponding to the positions of the joint points and the calculation formula; 3. The posture estimation device according to claim 2, further comprising: a joint point position image that represents the score for each coordinate as a heat map. 5. The above formula is: a first calculation formula for calculating a score of other coordinates from an x-coordinate value of the coordinates corresponding to the position of the joint point, and a second calculation formula for calculating a score of other coordinates from a y-coordinate value of the coordinates corresponding to the position of the joint point, The joint point information generating means calculating scores for the other coordinates based on an x-coordinate value of a coordinate corresponding to the position of the joint point and the first calculation formula, and generating a first joint point position image as the joint point position image, the first joint point position image being a heat map of the scores for each coordinate; 5. The posture estimation device according to claim 4, further comprising: a y-coordinate value of a coordinate corresponding to the position of the joint point; a score for the other coordinates is calculated based on the second arithmetic formula; and a second joint point position image in which the score for each coordinate is represented as a heat map is generated as the joint point position image. 6. The object region image information generating means 6. The posture estimation device according to any one of 1 to 5, wherein the person area is extracted from the image using a result of a person detection process or a result of a joint point extraction process. 7. A posture estimation device according to any one of 1 to 6, wherein the estimation model includes a self-attention mechanism. 8. The computer: Extracting a person area from the image, and generating person area image information based on the image of the extracted person area; extracting joint points of a person from the image, and generating joint point information based on the extracted joint points; generating feature information based on both the person area image information and the joint point information; A posture estimation method for estimating a posture of a person included in the image based on an estimation model that uses the feature information as an input and outputs a posture estimation result. 9. Computer a person area image information generating means for extracting a person area from an image and generating person area image information based on an image of the extracted person area; a joint point information generating means for extracting joint points of a person from the image and generating joint point information based on the extracted joint points; a feature amount information generating means for generating feature amount information based on both the person area image information and the joint point information; an estimation means for estimating a posture of a person included in the image based on an estimation model that receives the feature amount information as an input and outputs a posture estimation result; A program that functions as a 10. A person area image information generating means for extracting a person area from an image and generating person area image information based on the image of the extracted person area; a joint point information generating means for extracting joint points of a person from the image and generating joint point information based on the extracted joint points; a feature amount information generating means for generating feature amount information based on both the person area image information and the joint point information; a learning means for learning an estimation model in which the feature information is input and a posture estimation result is output; A learning device having the above configuration. [Explanation of symbols]

[0081] 10 Posture estimation device 11 Human area image information generation unit 12 Joint point information generation unit 13 Feature information generation unit 14 Estimation part 20 Learning Device 21 Human area image information generation unit 22 Joint point information generation unit 23 Feature information generation unit 24 Learning Department 1A Processor 2A Memory 3A input / output I / F 4A Peripheral circuit 5A Bus

Claims

1. a person area image information generating means for extracting a person area from an image and generating person area image information based on an image of the extracted person area; a joint point information generating means for extracting joint points of a person from the image and generating joint point information based on the extracted joint points; a feature amount information generating means for generating feature amount information based on both the person area image information and the joint point information; an estimation means for estimating a posture of a person included in the image based on an estimation model including a self-attention mechanism, the estimation means receiving the feature information as an input and a posture estimation result as an output; A posture estimation device having the following configuration.

2. a person area image information generating means for extracting a person area from an image and generating person area image information based on an image of the extracted person area; a joint point information generating means for extracting joint points of a person from the image and generating joint point information based on the extracted joint points; a feature amount information generating means for generating feature amount information based on both the person area image information and the joint point information; an estimation means for estimating a posture of a person included in the image based on an estimation model that receives the feature amount information as an input and outputs a posture estimation result; having The joint point information generating means Extract M joint points of a person; generating a plurality of joint point position images as the joint point information, each of which corresponds to one of the M joint points and indicates a position of each of the M joint points; A calculation formula for calculating a score of the coordinate corresponding to the position of the joint point and a score of other coordinates from the value of the coordinate corresponding to the position of the joint point is defined in advance; The joint point information generating means Calculating scores for the other coordinates based on the coordinate values ​​corresponding to the positions of the joint points and the calculation formula; generating a joint point position image in which the scores of each coordinate are represented as a heat map; The above-mentioned calculation formula is: a first calculation formula for calculating a score of other coordinates from an x-coordinate value of the coordinates corresponding to the position of the joint point, and a second calculation formula for calculating a score of other coordinates from a y-coordinate value of the coordinates corresponding to the position of the joint point, The joint point information generating means calculating scores for the other coordinates based on an x-coordinate value of a coordinate corresponding to the position of the joint point and the first calculation formula, and generating a first joint point position image as the joint point position image, the first joint point position image being a heat map of the scores for each coordinate; a y-coordinate value of a coordinate corresponding to the position of the joint point and a score of the other coordinates based on the second arithmetic formula, and generates a second joint point position image as the joint point position image, in which the score of each coordinate is represented as a heat map.

3. The joint point information generating means Extract M joint points of a person; The posture estimation device according to claim 1 , wherein a plurality of joint point position images, each of which corresponds to one of M joint points and indicates a position of each of the M joint points, are generated as the joint point information.

4. A score of the coordinate corresponding to the position of the joint point and a score of other coordinates are defined in advance as a fixed value, The joint point information generating means The posture estimation device according to claim 2 or 3, wherein the joint point position image is generated by expressing the score of each coordinate as a heat map.

5. The person area image information generating means The posture estimation device according to claim 1 , wherein the human region is extracted from the image using a result of a human detection process or a result of a joint point extraction process.

6. The computer Extracting a person area from the image, and generating person area image information based on the image of the extracted person area; extracting joint points of a person from the image, and generating joint point information based on the extracted joint points; generating feature information based on both the person area image information and the joint point information; A pose estimation method that uses the feature information as input, uses a pose estimation result as output, and estimates the pose of a person included in the image based on an estimation model including a self-attention mechanism.

7. The computer Extracting a person area from the image, and generating person area image information based on the image of the extracted person area; extracting joint points of a person from the image, and generating joint point information based on the extracted joint points; generating feature information based on both the person area image information and the joint point information; estimating a posture of a person included in the image based on an estimation model that uses the feature amount information as an input and outputs a posture estimation result; In the process of generating the joint point information, Extract M joint points of a person; generating a plurality of joint point position images as the joint point information, each of which corresponds to one of the M joint points and indicates a position of each of the M joint points; A calculation formula for calculating a score of the coordinate corresponding to the position of the joint point and a score of other coordinates from the value of the coordinate corresponding to the position of the joint point is defined in advance; In the process of generating the joint point information, Calculating scores for the other coordinates based on the coordinate values ​​corresponding to the positions of the joint points and the calculation formula; generating a joint point position image in which the scores of each coordinate are represented as a heat map; The above-mentioned calculation formula is: a first calculation formula for calculating a score of other coordinates from an x-coordinate value of the coordinates corresponding to the position of the joint point, and a second calculation formula for calculating a score of other coordinates from a y-coordinate value of the coordinates corresponding to the position of the joint point, In the process of generating the joint point information, calculating scores for the other coordinates based on an x-coordinate value of a coordinate corresponding to the position of the joint point and the first calculation formula, and generating a first joint point position image as the joint point position image, the first joint point position image being a heat map of the scores for each coordinate; a y-coordinate value of a coordinate corresponding to the position of the joint point and a score of the other coordinates based on the second arithmetic formula, and a second joint point position image in which the score of each coordinate is represented as a heat map is generated as the joint point position image.

8. Computer, a person area image information generating means for extracting a person area from an image and generating person area image information based on an image of the extracted person area; a joint point information generating means for extracting joint points of a person from the image and generating joint point information based on the extracted joint points; a feature amount information generating means for generating feature amount information based on both the person area image information and the joint point information; an estimation means for estimating a posture of a person included in the image based on an estimation model including a self-attention mechanism, the feature information being input and a posture estimation result being output; A program that functions as a

9. Computer, a person area image information generating means for extracting a person area from an image and generating person area image information based on an image of the extracted person area; a joint point information generating means for extracting joint points of a person from the image and generating joint point information based on the extracted joint points; a feature amount information generating means for generating feature amount information based on both the person area image information and the joint point information; an estimation means for estimating a posture of a person included in the image based on an estimation model that receives the feature amount information as an input and outputs a posture estimation result; Function as a The joint point information generating means Extract M joint points of a person; generating a plurality of joint point position images as the joint point information, each of which corresponds to one of the M joint points and indicates a position of each of the M joint points; A calculation formula for calculating a score of the coordinate corresponding to the position of the joint point and a score of other coordinates from the value of the coordinate corresponding to the position of the joint point is defined in advance; The joint point information generating means Calculating scores for the other coordinates based on the coordinate values ​​corresponding to the positions of the joint points and the calculation formula; generating a joint point position image in which the scores of each coordinate are represented as a heat map; The above-mentioned calculation formula is: a first calculation formula for calculating a score of other coordinates from an x-coordinate value of the coordinates corresponding to the position of the joint point, and a second calculation formula for calculating a score of other coordinates from a y-coordinate value of the coordinates corresponding to the position of the joint point, The joint point information generating means calculating scores for the other coordinates based on an x-coordinate value of a coordinate corresponding to the position of the joint point and the first calculation formula, and generating a first joint point position image as the joint point position image, the first joint point position image being a heat map of the scores for each coordinate; a program for calculating scores for the other coordinates based on a y coordinate value of a coordinate corresponding to the position of the joint point and the second arithmetic formula, and generating a second joint point position image as the joint point position image, in which the scores for each coordinate are represented as a heat map.

10. a person area image information generating means for extracting a person area from an image and generating person area image information based on an image of the extracted person area; a joint point information generating means for extracting joint points of a person from the image and generating joint point information based on the extracted joint points; a feature amount information generating means for generating feature amount information based on both the person area image information and the joint point information; A learning means for learning an estimation model including a self-attention mechanism by using the feature information as an input and a posture estimation result as an output; A learning device having the above configuration.

11. a person area image information generating means for extracting a person area from an image and generating person area image information based on an image of the extracted person area; a joint point information generating means for extracting joint points of a person from the image and generating joint point information based on the extracted joint points; a feature amount information generating means for generating feature amount information based on both the person area image information and the joint point information; a learning means for learning an estimation model in which the feature information is input and a posture estimation result is output; having The joint point information generating means Extract M joint points of a person; generating a plurality of joint point position images as the joint point information, each of which corresponds to one of the M joint points and indicates a position of each of the M joint points; A calculation formula for calculating a score of the coordinate corresponding to the position of the joint point and a score of other coordinates from the value of the coordinate corresponding to the position of the joint point is defined in advance; The joint point information generating means Calculating scores for the other coordinates based on the coordinate values ​​corresponding to the positions of the joint points and the calculation formula; generating a joint point position image in which the scores of each coordinate are represented as a heat map; The above-mentioned calculation formula is: a first calculation formula for calculating a score of other coordinates from an x-coordinate value of the coordinates corresponding to the position of the joint point, and a second calculation formula for calculating a score of other coordinates from a y-coordinate value of the coordinates corresponding to the position of the joint point, The joint point information generating means calculating scores for the other coordinates based on an x-coordinate value of a coordinate corresponding to the position of the joint point and the first calculation formula, and generating a first joint point position image as the joint point position image, the first joint point position image being a heat map of the scores for each coordinate; a learning device that calculates scores for the other coordinates based on a y coordinate value of the coordinates corresponding to the position of the joint point and the second arithmetic formula, and generates a second joint point position image as the joint point position image, in which the scores for each coordinate are represented as a heat map.

Images