Information processor, method for estimating posture, program, and storage medium

Abstract

To precisely estimate a posture at which at least three or more points of a human body are in contact with a floor surface or a ground.SOLUTION: An information processor includes: acquisition means for acquiring an image; detection means for detecting the whole body of a human from the image acquired by the acquisition means; bone structure estimation means for estimating the detected bone structure of the whole body of the human and generating bone structure information on the whole body of the human; first extraction means for extracting a first feature amount on the basis of the bone structure information generated by the estimation means; second extraction means for extracting a second feature amount on the basis of a cutoff image including the whole body of the human detected by the detection means; and posture estimation means for estimating the posture of the whole body of the human detected by the detection means on the basis of a third feature amount which connects the first feature amount and the second feature amount to each other.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present invention relates to an information processing device, a posture estimation method, a program, and a storage medium. [Background technology]

[0002] In recent years, technology that estimates a person's whole-body posture from surveillance camera footage has been applied in the fields of customer protection in stores and urban surveillance.

[0003] Conventionally, a method has been proposed for estimating the posture of the entire body by estimating the skeletal information (joints) of the entire body in a video and estimating the posture based on the joints. However, when different postures appear to be the same from the joints, estimating the posture is difficult.

[0004] Patent Document 1 discloses a technology that detects objects (such as chairs) around a person and distinguishes between a "squatting posture" and a "sitting on a chair posture" based on the distance between the target person's joints and the objects. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] WO2015 / 186436 Summary of the Invention [Problem to be solved by the invention]

[0006] However, the technology described in Patent Document 1 cannot distinguish between a "sitting on the floor posture" and a "squatting posture" because they are not related to objects in the background.

[0007] The problem to be solved by the present invention is to accurately estimate a posture in which three or more points of a human body part are in contact with a floor or ground surface. [Means for solving the problem]

[0008] In order to solve the above problem, an information processing device according to one embodiment of the present invention has an acquisition means for acquiring an image, a detection means for detecting the entire body of a human body from the image acquired by the acquisition means, and a skeleton estimation means for estimating the skeleton of the entire body of the detected human body and generating skeletal information of the entire body of the human body, and is characterized in using a neural network including: a first extraction means for extracting a first feature based on the skeletal information generated by the estimation means; a second extraction means for extracting a second feature based on a cut-out image including the entire body of the human body detected by the detection means; and a posture estimation means for estimating the posture of the entire body of the human body detected by the detection means based on a third feature that combines the first feature and the second feature. Effect of the Invention

[0009] According to the present invention, it is possible to accurately estimate a posture in which three or more points of a human body are in contact with a floor or ground surface. [Brief description of the drawings]

[0010] [Figure 1] FIG. 2 is a diagram illustrating an example of the hardware configuration of the whole-body posture estimation apparatus according to the present embodiment. [Diagram 2] FIG. 2 is a diagram showing a functional configuration of the whole-body posture estimation device according to the present embodiment. [Diagram 3] 13 is an example of a skeleton estimation result. [Figure 4] 4 is a flowchart showing a processing flow of the whole-body posture estimation device of the present embodiment. [Diagram 5] FIG. 2 is a diagram showing the structure of a neural network of a posture estimation means. [Figure 6] This is an example of the whole body and background of a cut-out image in a standing position and a three or more points on the ground position. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, the embodiment for carrying out the present invention will be described in detail with reference to the attached drawings. The embodiment described below is an example of a means for realizing the present invention, and should be appropriately modified or changed depending on the configuration of the device to which the present invention is applied and various conditions, and the present invention is not limited to the following embodiment. In addition, a configuration may be made by appropriately combining parts of each embodiment described later.

[0012] <Embodiment 1> 1 is a block diagram showing the hardware configuration of an information processing device according to this embodiment. A CPU 101 executes instructions according to programs stored in a ROM 102 or a RAM 103. The ROM 102 is a non-volatile memory, and stores the program of the present invention and other programs and data necessary for control. The RAM 103 is a volatile memory, and stores temporary data such as frame image data and pattern discrimination results.

[0013] The auxiliary storage device 104 is a rewritable auxiliary storage device such as a hard disk drive or flash memory, and stores image information, programs, various settings, etc. This information is transferred to the RAM 103, and the CPU 101 executes the programs and uses the data.

[0014] The imaging device 105 is composed of an imaging lens, an imaging sensor such as a CCD or CMOS, a video signal processing unit, etc., and captures an image. The input device 106 is a keyboard, a mouse, etc., and enables input from the user. The display device 107 is a cathode ray tube (CRT) or a liquid crystal display, etc., and displays processing results to the user. The network I / F 108 is a modem or LAN that connects to a network such as the Internet or an intranet. The bus 109 connects these devices and performs data input and output between them. Software that implements processes corresponding to each step of the flowchart described later is read from the auxiliary storage device 104, RAM 103, etc., and executed by the CPU 101.

[0015] 2 is a diagram showing the functional configuration of an information processing device according to this embodiment. Reference numeral 201 denotes an image acquisition means, which is composed of an imaging device 105 and is a means for acquiring an image (image). Reference numeral 202 denotes a person detection means, which is a means for detecting an area of ​​a person (human body) from the image acquired by the image acquisition means 201. Note that in this embodiment, the entire body area of ​​the human body is detected.

[0016] Reference numeral 203 denotes a skeleton estimation means, which estimates the skeleton of the entire human body from the whole body area detected by the person detection means 202, and generates joint points of the entire body as skeleton information of the entire human body. A joint point indicates a connection position between parts of the human body. In this embodiment, the joint points are the shoulders (301, 302), elbows (303, 304), wrists (305, 306), waists (307, 308), knees (309, 310), and ankles (311, 312) shown as dots in FIG. 3. At the same time, the eyes (313, 314), nose (317), and ears (315, 316), which are facial organs, are also treated as joint points.

[0017] Reference numeral 204 denotes a posture estimation means for estimating a posture in which three or more points of detected human body parts, such as a lying or sitting position, are on the ground (hereinafter referred to as a three or more point contact posture) based on the skeleton information estimated by the skeleton estimation means 203 and the whole body area detected by the person detection means 202.

[0018] Reference numeral 205 denotes a user notification determination means, which is a means for determining whether or not a notification to the user is necessary based on the result of the posture estimation means 204. Reference numeral 206 denotes a display means, which is composed of the display device 107, and is a means for displaying a notification to the user when the user notification determination means 205 determines that a user notification is necessary.

[0019] Next, details of the processing of the whole body posture estimation device in this embodiment will be described with reference to the flowchart in Fig. 4. First, the image acquisition means 201 acquires an image from the imaging device 105 in units of frame images (step S401). Next, the person detection means 202 detects the whole body area of ​​a person from within the frame images (step S402).

[0020] A specific method for detecting a person is a method using a Convolutional Neural Network (CNN). Note that any method can be used as long as it can detect the whole body area. For example, a method may be used in which the contour of a subject is detected and the whole body area is detected by comparing the edge information with information on the contour of a person stored in advance.

[0021] The whole body region is expressed by the x and y coordinates of the two top left and bottom right points of a rectangle surrounding the person, with the top left corner of the frame image as the origin. Next, the skeleton estimation means 203 estimates the skeleton of the person from the whole body region and outputs a likelihood map of joint points corresponding to the whole body region (step S403).

[0022] A specific method for estimating the skeleton is to use CNN. For each joint, a likelihood (value between 0 and 1, closer to 1 it is more likely to be a joint) indicating the likelihood of a joint at each pixel in the whole body region is calculated and used as a likelihood map of the joint points. In this embodiment, 17 joints are estimated, so 17 joint likelihood maps are created. The most likely coordinates of each likelihood map are candidates for the joint point coordinates. Although it is possible to output and use the joint point coordinates, in this embodiment, 17 likelihood maps are output.

[0023] It should be noted that the method is not limited to the method using CNN as long as the coordinates of the joint points can be estimated. For example, a method of estimating from the relative positions of the joint points based on the outline of a person and a general human body structure may be used. Note that, although the present embodiment is configured to perform skeletal estimation on a whole body area obtained by person detection from the entire screen, it is also possible to perform skeletal estimation on the entire screen to obtain joints first, and then to organize them into person units based on the relationship between each joint.

[0024] Next, the posture estimation means 204 estimates whether the person is in a three or more points grounding posture from the joint likelihood map and the image of the whole body region (step S404). A likelihood indicating the likelihood of a three or more points grounding posture is output in association with the whole body region. The likelihood takes values ​​from 0 to 1, and the closer it is to 1, the more likely it is that the posture is estimated to be three points grounding. The specific method of this means will be described later.

[0025] The processing from step S403 onwards is repeated for all people (whole body area) included in the current frame image (step S405).

[0026] Next, the user notification determination means 205 determines whether to issue an alert to the user based on the likelihood output by the posture estimation means 204 (step S406). For all whole body regions detected from the frame image, the likelihood of a three or more point grounding posture is compared with a predetermined threshold, and when the threshold is exceeded, it is determined that an alert is necessary, and the (rectangular coordinates of) the whole body region and the likelihood of a three or more point grounding posture are output.

[0027] Next, the display means 206 displays the result of step S406 to the user (step S407). For example, a message such as "There is a person lying down" may be displayed, or a rectangle surrounding the person lying down may be superimposed on the camera image. A numerical value indicating the likelihood of a three or more point grounding posture, or a color / bar graph according to the magnitude of the numerical value may be displayed.

[0028] The above is the flow of processing performed by the whole-body posture estimation device. Note that, although the flow of processing up to displaying the posture estimation result has been described in order, it is assumed that all steps are constantly repeated until the information processing device is terminated.

[0029] The following describes in detail the posture estimation means 204. The posture estimation means 204 is realized by a neural network. First, the structure of the network and the process flow will be described with reference to FIG.

[0030] Reference numeral 501 denotes the entire network constituting the posture estimation means 204 .

[0031] Below, each part constituting the network will be explained in the order of processing.

[0032] Reference numeral 502 denotes a skeleton feature extraction means (first extraction means), which is composed of a plurality of convolution layers and pooling layers. A skeleton feature map (first feature amount) is extracted from the joint likelihood map output by the skeleton estimation means 203.

[0033] The reason for inputting the joint likelihood map is that the posture of the whole body is determined by the skeleton, and it is also useful as information for distinguishing the whole body from the background in the whole body cutout image. The purpose is to reflect the relationship between the whole body and the background in the image features described later.

[0034] Other networks may be used as long as they can extract skeletal feature maps. For example, a residual net may be used in which multiple residual blocks are connected in series, with the input of the convolutional layer skipped and connected (added) to the output. In addition, a method such as Visual Transformer (ViT) may be used, which divides an image into patches, tokenizes (vectorizes) it, and extracts the degree of association between tokens.

[0035] An image feature extraction means (second extraction means) 503 is composed of multiple convolution layers and pooling layers, etc. Based on the whole body area detected by the person detection means 202, an image feature map (second feature amount) is extracted by inputting a cut-out image including the whole body and the background around the whole body.

[0036] In order to obtain an image feature map that reflects the relationship information between the whole body and the background, it is desirable that the cut-out image contains a sufficient amount of the background around the whole body. For this reason, the whole body area detected by the person detection means 202 may be input as a cut-out image with a predetermined margin added both vertically and horizontally.

[0037] As with the skeleton feature extraction means 502, any network may be used as long as it can extract an image feature map, such as a residual network or ViT.

[0038] Reference numeral 504 denotes a feature linking means for linking the skeleton feature map with the image feature map.

[0039] In this embodiment, the joint likelihood maps input to the skeleton feature extraction means 502 are resized and padded to a predetermined height H x width W for all joints, and a skeleton feature map of H / K x W / K (K is a predetermined constant) x C (number of channels) is output. The image feature extraction means 503 also resizes and pads the input cut-out image to H x W, obtaining an image feature map of H / K x W / K x C. These are linked by the feature linking means 504 to obtain a linked feature map of H / K x W / K x 2 x C.

[0040] In this embodiment, the skeleton feature map and the image feature map are the same size, but if they are different, the feature linking means 504 can link them by downsampling the larger feature map to match the size.

[0041] Reference numeral 505 denotes a posture likelihood estimation unit, which estimates the likelihood of a posture in which three or more points of the body parts are on the ground from the linked feature map. This is configured with a Multi Layer Perceptron (MLP) and handles it as a multi-class problem by defining a binary classification or a posture class other than the posture in which three or more points are on the ground, such as a standing position.

[0042] Reference numeral 506 denotes a whole body / background segmentation means (relationship information estimation means), which is configured with a Fully Convolutional Network (FCN) and calculates the likelihood (0 to 1, the closer to 1 the higher the likelihood) of the segmentation category ("whole body", "background") for each pixel in the cut-out image. By selecting the most likely category for each pixel, it is possible to segment the whole body and the background. Note that it is also effective to segment the categories in more detail. This will be described later.

[0043] Next, a method for training the neural network of the posture estimation means 204 will be described.

[0044] First, the learning data will be described. A combination of whole-body cutout images that will be input to the posture estimation means 204 and a skeletal feature map acquired from the whole-body cutout images using the skeleton estimation means 203 is prepared. As correct answer data, a label indicating whether each whole-body cutout image and skeletal feature map is in a grounded posture with three or more points, and a label image in which each pixel of the whole-body cutout image is labeled as either the whole body or the background are prepared.

[0045] The whole body cut-out image and skeletal feature map are input to the posture estimation network 501, and the task of estimating postures with three or more ground contact points is learned using the output of the posture likelihood estimation unit 505 and the correct label of the posture with three or more ground contact points. Of course, the image feature extraction means 503 may be pre-trained in advance with a general image classification task.

[0046] In this embodiment, in addition to the task of estimating the three or more points grounding posture, a task of dividing the whole body from the background is added, and by performing multitask learning, relationship information between the person and the background is reflected in the image feature map extracted by the image feature extraction means 503. The following will be explained with reference to FIG. 6.

[0047] FIG. 6 shows a cutout image 601 in a standing position and a cutout image 608 in a three or more points grounded position (lying position). 602 and 609 respectively show the outline of a person (boundary with the background). 603 and 610 shown by circles and dotted lines are schematic diagrams of skeletal information estimated by the skeleton estimation means 203. The circles indicate the positions of the peaks of each joint likelihood map, and the dotted lines indicate the parts connecting them. It is difficult to distinguish whether the person is standing or lying down only from the skeleton information (603 and 610). However, information that is useful for distinguishing between the two postures is included around each whole body. 604, 605, 611, and 612 indicate walls, and 606 and 613 indicate the floor. In the standing posture (601), the floor exists only around the lower body (606), whereas in the lying posture (608), the floor exists around the whole body (613).

[0048] Also, 607 and 614 show shadows. In the standing posture (601), there is a shadow only around the feet (607), but in the lying posture (608), there is a shadow around the whole body (614).

[0049] By training the task of dividing the whole body into the whole body and the background, the image feature extraction means 503 is able to extract the features of the floor surface and shadows around the whole body when the body is in a supine position, and the posture likelihood estimation unit 505 is able to estimate the likelihood of a posture with three or more points on the ground.

[0050] In addition, the division category "background" may be replaced with "floor" and "background other than floor" so that the background can be divided into the floor and other areas. This makes it possible to clearly capture the features of the floor in particular in the image feature extraction means 503. Also, by adding "shadow" to the division category so that the shadow area can be divided, it becomes possible to capture the shape of the shadow due to differences in posture.

[0051] Also, by setting the division category to "floor" and "wall" so as to divide the background into the floor and the wall, information related to the perspective (shooting angle) can be obtained. As shown in FIG. 6, even if the image is shot with the same downward angle, the appearance (shape) of the whole body differs between standing and lying positions. With a downward angle of view, the upper body appears larger than the lower body compared to when shot from the horizontal direction as in 602, but in 609 it appears the same as when shot from the horizontal direction. Therefore, by dividing the floor, wall, and whole body, the relationship between the shooting direction and the change in the appearance of the person (the ratio of the upper body to the lower body) is reflected in the image feature map output from the image feature extraction means 503. Note that if the whole body region is further divided into the upper body and the lower body (the division category "whole body" is changed to "upper body" and "lower body"), the above features are reflected more clearly in the image feature map.

[0052] In addition, to handle cases where a person is sitting facing forward in a "sitting on a chair" posture, "chair" may be included in the division category. When the characteristics of a "chair" are found around a person in the image feature map, learning is possible so that the likelihood of a posture with three or more points on the ground becomes small.

[0053] By using posture estimation network 501 trained as described above, a three or more point ground contact posture can be accurately estimated. Since whole body / background division means 506 is a necessary part for learning, an omitted network may be used in posture estimation means 204.

[0054] In this embodiment, a skeleton feature map is input to the posture estimation means 204 as skeleton information, but this is not limited to this. For example, peaks of each joint may be detected from the skeleton feature map and input as joint point coordinates. Also, a skeleton image in which joint points are connected by lines may be generated and input.

[0055] In this embodiment, the image segmentation task is added by the whole body / background segmentation means 506, so that the relationship information between the whole body and the background is reflected in the image feature map output from the image feature extraction means 503 when estimating the three or more points grounding posture. However, the added task is not limited to image segmentation as long as it can reflect the relationship information between the whole body and the background. For example, it may be a task that determines the shooting angle itself by regression. This allows the relationship between the shooting angle and the appearance of the whole body to be reflected in the image feature map as described above. Also, it may be configured to estimate depth information for each pixel. This allows the depth difference (relationship information) between each part of the whole body and the floor surface to be reflected in the image feature map.

[0056] As described above, according to this embodiment, since image features reflecting relationship information between the whole body and the background can be extracted from a cut-out image including the background around the whole body, it becomes possible to estimate the whole body posture even when a background object does not exist or cannot be detected. This makes it possible to improve the estimation accuracy of postures with three or more points on the ground, such as lying down or sitting down.

[0057] <Other Examples> In this embodiment, the processing result is displayed to the user, but further, a video recording means may be used to start recording the video when a three or more point ground contact posture is detected, or meta information indicating the detection time in the recorded video may be added. By adding the meta information, it becomes possible to search for the video at the time when a three or more point ground contact posture was recorded.

[0058] In this embodiment, all functions are incorporated into a single device, but the image acquired from the image acquisition means may be transmitted to the cloud, and processing from the person detection means to the posture estimation means and user notification determination means may be performed on the cloud.

[0059] The object of this embodiment may be achieved by the following method. A recording medium (or storage medium) on which program code of software for realizing the functions of the above-mentioned embodiment is recorded is supplied to a system or device. Then, a computer (or a CPU, MPU, or GPU) of the system or device reads and executes the program code stored in the recording medium. In this case, the program code itself read from the recording medium realizes the functions of the above-mentioned embodiment, and the recording medium on which the program code is recorded constitutes this embodiment.

[0060] Furthermore, the functions of the above-described embodiments are not only realized by the computer reading and executing the program code, but also include an operating system (OS) running on the computer performing all or part of the actual processing based on the instructions of the program code.

[0061] Furthermore, the functions of the above-described embodiments may be realized by the following method: Program code read from a recording medium is written into a memory provided in a function expansion card inserted into a computer or a function expansion unit connected to a computer. Then, based on the instructions of the program code, a CPU provided in the function expansion card or function expansion unit performs some or all of the actual processing.

[0062] When this embodiment is applied to the above-mentioned recording medium, the recording medium stores program code corresponding to the flowcharts described above. [Explanation of symbols]

[0063] 101 CPU 102 ROM 103 RAM 104 Auxiliary storage 105 Imaging device 106 Input Devices 107 Display device 108 Network Interface 109 Bus 201 Video Acquisition Method 202 Person detection means 203 Skeleton Estimation Method 204 Posture estimation means 205 User notification determination means 206 Display means 501 Pose Estimation Network 502 Skeleton feature extraction means 503 Image feature extraction means 504 Feature Linking Means 505 Pose Likelihood Estimation Unit 506 Whole body / background division means

Claims

1. Means for acquiring images, A detection means for detecting the entire human body from the image acquired by the acquisition means, An information processing apparatus having a skeleton estimation means for estimating the entire skeleton of the detected human body and generating skeletal information of the entire human body, A first extraction means for extracting a first feature quantity based on the skeletal information generated by the skeletal estimation means, A second extraction means extracts a second feature quantity based on the cropped image including the entire human body detected by the detection means, An information processing device characterized by using a neural network comprising: a posture estimation means that estimates whether the posture of the entire human body detected by the detection means is a posture in which three or more points of the human body are in contact with the ground, based on a third feature obtained by concatenating the first feature and the second feature.

2. The information processing apparatus according to claim 1, further comprising a neural network that estimates relational information indicating the relationship between the entire human body and the background obtained by removing the entire human body from the cropped image, based on the third feature quantity.

3. The information processing apparatus according to claim 2, characterized in that the relational information estimation means estimates information that divides the entire human body from the cropped image into the background as relational information.

4. The information processing apparatus according to claim 2, characterized in that the relational information estimation means divides the entire human body and the floor surface in the cropped image.

5. The information processing apparatus according to claim 2, characterized in that the relational information estimation means estimates information that divides the entire human body and the shadow of the human body in the cropped image as the relational information.

6. The information processing apparatus according to claim 2, characterized in that the relational information estimation means divides the extracted image into the upper body of the human body, the lower body of the human body, the floor surface, and the wall surface.

7. The information processing apparatus according to claim 2, characterized in that the relational information estimation means estimates the shooting angle of the cropped image as relational information.

8. The information processing apparatus according to claim 2, characterized in that the relational information estimation means estimates depth information in the cropped image.

9. The process of acquiring images, A detection step for detecting the entire human body from the acquired image, A skeletal estimation step that estimates the entire skeleton of the detected human body and generates skeletal information of the entire human body, A first extraction step of extracting a first feature quantity based on the generated skeletal information, A second extraction step is performed to extract a second feature quantity based on the detected cropped image including the entire human body, A posture estimation step is performed to estimate whether the detected posture of the entire human body is one in which three or more points of the human body are in contact with the ground, based on a third feature obtained by concatenating the first feature and the second feature. A sharp posture estimation method.

10. The posture estimation method according to claim 9, further comprising a relational information estimation step of estimating relational information showing the relationship between the entire human body and the background obtained by removing the entire human body from the cutout, based on the third feature quantity.

11. The posture estimation method according to claim 10, characterized in that the relational information estimation step estimates information that divides the entire human body from the cropped image into the background as the relational information.

12. The posture estimation method according to claim 10, characterized in that the relational information estimation step estimates information that divides the entire human body from the floor surface from the cropped image as the relational information.

13. The posture estimation method according to claim 10 is characterized in that the relational information estimation step estimates information that separates the entire human body from the shadow of the human body in the cropped image as the relational information. method.

14. The posture estimation method according to claim 10, characterized in that the relationship information estimation step estimates the shooting angle of the cropped image as the relationship information.

15. The posture estimation method according to claim 10, characterized in that the relational information estimation step estimates the depth information in the cropped image as the relational information.

16. A program for causing a computer to execute the posture estimation method described in claim 9.

17. A computer-readable storage medium storing the program described in claim 16.

Images