Image processing apparatus, method for controlling image processing apparatus, and program

Abstract

To provide an image processing apparatus, a method for controlling an image processing apparatus, and a program that accurately estimate the posture of a person.SOLUTION: A method includes detecting an area of a human body from an acquired image, creating a cutout image for joint point detection from a visible detection frame including a visible area of the human body, and when the difference between the visible detection frame and an invisible detection frame is equal to or more than a predetermined value, creating a cutout image for posture estimation from the invisible detection frame. The method further includes creating a joint point likelihood map from the cutout image for joint point detection, resizing, rotating, and adding a margin to the created joint point likelihood map so as to match its space information with that of a cutout image for posture detection, and estimating the posture of the human body on the basis of the joint point likelihood map for posture estimation and the cutout image for posture detection matched with each other in terms of space information.SELECTED DRAWING: Figure 10

Description

[Technical field]

[0001] In particular, the present invention relates to an image processing device suitable for use in estimating a posture of a person, and a control method and program for the image processing device. [Background technology]

[0002] In recent years, technology has been proposed to estimate a person's posture from images captured by security cameras, and such technology is being applied in the fields of customer protection in stores and urban surveillance. Examples of applications of technology to estimate a person's posture include the need to confirm whether a store was at fault when a customer falls and injures themselves and sues the store, and the need to detect homeless people sitting or sleeping at ATMs and notify security guards.

[0003] Therefore, a method has been proposed as a technique for estimating a posture, in which the joint positions of a person in a video are detected and the posture is estimated based on the joint positions. Patent Document 1 discloses a technique for estimating a posture of a person by inputting an image and a joint point position image. Patent Document 2 discloses a technique for rotating an input image so that the head of the human body stands upright in the vertical direction, detecting the joint point positions, and then performing reverse rotation processing to recognize behavior. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2022-131397 A [Patent Document 2] JP 2019-121045 A [Non-patent literature]

[0005] [Non-Patent Document 1] J.Redmon, "You Only Look Once: Unified, Real-Time Object Detection", CVPR2015. [Non-Patent Document 2] K. Sun, "Deep High-Resolution Representation Learning for Human Pose Estimation", CVPR2019. [Non-Patent Document 3] R. Ranftl, "Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer", TPAMI 2020 Summary of the Invention [Problem to be solved by the invention]

[0006] However, since a person has various postures, there are cases where the positions of the joint points cannot be accurately detected or the posture cannot be accurately determined based on the positions of the joint points alone depending on the posture. In the method described in Patent Document 1, the posture estimation accuracy decreases when the camera is tilted or the person falls over. In addition, the method described in Patent Document 2 can accurately estimate the positions of the joint points, but the posture estimation accuracy may decrease when the distinction between postures that appear to be the same based on the positions of the joint points alone is insufficient or the camera is tilted.

[0007] In view of the above problems, an object of the present invention is to enable the estimation of a person's posture with high accuracy. [Means for solving the problem]

[0008] The image processing device of the present invention is characterized by having human body detection means for detecting a human body region from an image, image generation means for generating from the image a first cut-out image including the human body region detected by the human body detection means and a second cut-out image including the human body region and different from the first cut-out image, joint point detection means for detecting joint points of the human body from the first cut-out image and generating joint point information of the human body, conversion means for converting the joint point information to correspond to spatial information of the second cut-out image, and posture estimation means for estimating a posture of the human body based on the joint point information converted by the conversion means and the second cut-out image. Effect of the Invention

[0009] According to the present invention, the posture of a person can be estimated with high accuracy. [Brief description of the drawings]

[0010] [Figure 1] FIG. 2 is a block diagram illustrating an example of a hardware configuration of the image processing device. [Diagram 2] 1 is a block diagram showing an example of a functional configuration of an image processing apparatus according to a first embodiment; [Diagram 3] 11 is a diagram for explaining an example of a human body detection result; FIG. [Figure 4] 11A and 11B are diagrams for explaining an example of a human body detection result in a fallen posture. [Diagram 5] FIG. 11 is a diagram showing an example of an image cut out based on a human body detection result. [Figure 6] FIG. 11 is a diagram for explaining an example of a joint point detection result. [Figure 7] 13 is a diagram for explaining an example of a result of a conversion process into a joint point likelihood map for posture estimation. FIG. [Figure 8] FIG. 13 is a diagram illustrating an example of weight information applied to a joint point likelihood map. [Figure 9] FIG. 1 is a diagram illustrating an example of a structure of a deep learning model for pose estimation. [Figure 10]6 is a flowchart showing an example of a processing procedure for estimating an attitude according to the first embodiment. [Figure 11] FIG. 11 is a block diagram illustrating an example of a functional configuration of an image processing apparatus according to a second embodiment. [Figure 12] 11 is a diagram for explaining an example of a relationship between a depth map and an estimated value of a tilt angle; [Figure 13] 11A and 11B are diagrams for explaining an example in which a clipping region for orientation detection is corrected using an object detection result. [Figure 14] 13 is a flowchart illustrating an example of a processing procedure for estimating an attitude according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, the present invention will be described in detail based on preferred embodiments thereof with reference to the accompanying drawings. Note that the configurations shown in the following embodiments are merely examples, and the present invention is not limited to the configurations shown in the drawings.

[0012] (First embodiment) FIG. 1 is a block diagram showing an example of the hardware configuration of an image processing device 100 according to this embodiment. The image processing device 100 includes an input unit 101, a display unit 102, a network I / F unit 103, a CPU 104, a RAM 105, a ROM 106, a HDD 107, and a data bus .

[0013] The CPU 104 reads out a control computer program stored in the ROM 106, loads it into the RAM 105, and executes various control processes. The RAM 105 is used as a temporary storage area for programs executed by the CPU 104, a work memory, etc. The network I / F unit 103 is an interface for transmitting and receiving various information to and from other devices via a network. The HDD 107 stores image data, setting parameters, various programs, etc. Image data, etc. received from an external device via the network I / F unit 103 is transmitted and received to the CPU 104, RAM 105, and ROM 106 via a data bus 108.

[0014] Image processing of image data is realized by CPU 104 executing an image processing program stored in ROM 106 or HDD 107. Also, it is assumed that HDD 107 is capable of inputting data from an external device via network I / F unit 103. Input unit 101 is, for example, a keyboard, a mouse, a touch panel, etc., and display unit 102 includes a display device such as a liquid crystal display, and input unit 101 and display unit 102 are also connected to data bus 108.

[0015] 2 is a block diagram showing an example of the functional configuration of the image processing device 100 according to this embodiment. The image processing device 100 includes an image acquisition unit 201, a human body detection unit 202, an image generation unit 203, a joint point detection unit 204, a conversion unit 205, and a posture estimation unit 206.

[0016] The image acquisition unit 201 acquires an image. The image to be acquired may be an image captured by a network camera or the like, an image recorded in the HDD 107 or the like, or an image received from the network I / F unit 103 via a network such as the Internet. The image acquired by the image acquisition unit 201 is sent to the human body detection unit 202.

[0017] The human body detection unit 202 detects a human body from the image acquired by the image acquisition unit 201, and generates a list of human body detection results. Here, the human body detection results include information on a visible detection frame (center position, size) that includes the visible area of ​​the human body, information on an invisible detection frame (center position, size) that estimates the invisible area of ​​the human body in addition to the visible area, a detection score that indicates the reliability of detection, and an amount of image rotation.

[0018] FIG. 3 is a diagram for explaining an example of a human body detection result. A person 301 is shown in an image 300, and a visible detection frame 302 and an invisible detection frame 303 are obtained. Since the lower half of the person 301 is hidden by a shelf and cannot be seen, the lower end of the visible detection frame 302 is at the position of the person 301's waist. In contrast, the lower end of the invisible detection frame 303 is at a position that estimates the position of the person 301's feet. The human body detection unit 202 generates images by rotating the image by 0 degrees, 90 degrees, 180 degrees, and 270 degrees, detects a human body in each image, and generates a list of human body detection results. In this embodiment, the human body detection unit 202 generates images rotated in 90-degree increments, but images rotated in 45-degree increments or images rotated in 30-degree increments may also be generated. In other words, the rotation amount and rotation direction are not limited to a specific amount, and it is sufficient if the image can be rotated by various amounts.

[0019] For example, the method described in Non-Patent Document 1 is used for detecting human bodies. All obtained human body detection results are rotated inversely based on the image rotation amount to return them to coordinates on the original image. In addition, in order to identify which image rotation amount each human body detection result was obtained from, information on the image rotation amount is also included in the human body detection result. Finally, the overlap rate of the visible detection frame is calculated for all pairs of human body detection results, and for pairs with an overlap rate of a certain level or more, the pair with the highest detection score is left as the human body detection result, and the other pair is excluded from the list of human body detection results. The list of human body detection results obtained by the human body detection unit 202 is sent to the image generation unit 203 together with the image.

[0020] The image generation unit 203 generates a cut-out image for joint point detection, a cut-out image for posture estimation, and image generation information from the image and the list of human body detection results. First, the image generation unit 203 generates a cut-out image for joint point detection for each person based on each human body detection result. As a specific procedure, the image and visible detection frame are rotated based on the image rotation amount included in the human body detection result. Next, the visible detection frame is converted into a cut-out region for joint point detection according to the following equations (1) to (5).

[0021]

number

[0022] In the formula, ltx', lty', width', and height' respectively represent the upper left x coordinate, upper left y coordinate, width, and height of the visible detection frame. cx, cy, width, and height respectively represent the center coordinate in the x direction, center coordinate in the y direction, width, and height of the rectangle that is the cut-out region for joint point detection. Then, based on the cut-out region for joint point detection, an image is cut out from the rotated image to be used as the cut-out image for joint point detection.

[0023] The image generating unit 203 also generates a clipped image for posture estimation for each person based on each human body detection result. As a specific procedure, first, the visible detection frame and invisible detection frame included in the human body detection result are compared to confirm whether there is a difference of a predetermined value or more in width and height. If there is a difference of a predetermined value or more, it is determined that occlusion has occurred, and if not, it is determined that occlusion has not occurred. If this determination shows that occlusion has occurred, the invisible detection frame is set as the selected detection frame, and if not, the visible area frame is set as the selected detection frame. Next, the selected detection frame is converted into a clipped region for posture estimation according to the following equations (6) to (10).

[0024]

number

[0025] In the formula, "ltx", "lty", "width", and "height" respectively represent the upper left x-coordinate, upper left y-coordinate, width, and height of the selection detection frame. pose ,cy pose ,width pose ,height pose represent the center coordinate in the X direction, the center coordinate in the Y direction, the width, and the height of a rectangle that is a cut-out region for pose estimation. Then, an image is cut out based on the cut-out region for pose estimation to obtain a cut-out image for pose estimation.

[0026] The image generation unit 203 performs the above processing on each human body detection result, sends the cut-out image for joint point detection to the joint point detection unit 204, and sends the cut-out image for posture estimation to the posture estimation unit 206. Furthermore, the image generation unit 203 sends information on the size of the original image, the amount of image rotation, the cut-out region for joint point detection, and the cut-out region for posture estimation to the conversion unit 205 as image generation information.

[0027] Fig. 4 is a diagram for explaining an example of a human body detection result in a fallen posture, and Fig. 5 is a diagram showing an example of an image cut out based on the human body detection result 401 in Fig. 4. In Fig. 4, a fallen person is captured in an image 400, and the human body detection result 401 is obtained. In this case, the human body detection result 401 is assumed to have been obtained from an image rotated 90 degrees clockwise.

[0028] FIG. 5(a) is a diagram showing an example of a cut-out image for joint point detection obtained from an image 400 based on a human body detection result 401. The image shown in FIG. 5(a) is an image cut out based on a cut-out region obtained from an image rotated 90 degrees clockwise based on the human body detection result 401 according to the above-mentioned formulas (1) to (5). FIG. 5(b) is a diagram showing an example of a cut-out image for posture estimation obtained from an image 400 based on the human body detection result 401. The image shown in FIG. 5(b) is an image cut out based on a cut-out region obtained from the image 400 based on the human body detection result 401 according to the above-mentioned formulas (6) to (10).

[0029] The joint point detection unit 204 generates a joint point likelihood map as joint point information based on the cut-out image for joint point detection generated by the image generation unit 203. Here, the joint point likelihood map is information in which the position information of each joint point is expressed by a heat map, and in this embodiment, the joint points are 17 points including the nose, both eyes, both ears, both shoulders, both elbows, both hips, both wrists, both knees, and both ankles. The joint point likelihood map can be generated by the method described in Non-Patent Document 2, and is output in a size reduced by a certain magnification from the input cut-out image for joint point detection. This reduced magnification is set as the joint point likelihood map magnification. Note that, in this embodiment, the joint points detected by the joint point detection unit 204 are defined as 17 points, but the definition of the joint points may be any part of the human body, and is not limited thereto.

[0030] FIG. 6 is a diagram for explaining an example of a joint point detection result. FIG. 6(a) is a diagram showing an example of a joint point likelihood map. The joint point likelihood map is made up of 17 pieces of heat map information, and the higher the position where the corresponding joint point is more likely to exist, the higher the value held. By acquiring the peak position from this joint point likelihood map, it is possible to obtain the 17 joint detection results. FIG. 6(b) shows an example in which the joint point detection results are superimposed on an image. The joint point detection unit 204 compiles the joint likelihood map and the joint point likelihood map magnification as joint point likelihood map information, and sends it to the conversion unit 205.

[0031] The conversion unit 205 generates a joint point likelihood map for pose estimation based on the image generation information and the joint point likelihood map information. Specifically, first, the joint point likelihood map generated by the joint point detection unit 204 is resized by the inverse magnification of the joint point likelihood map magnification. Next, the joint point likelihood map is rotated inversely based on the image rotation amount included in the image generation information, with the center coordinates of the joint likelihood map as a reference. Finally, margins are provided above, below, left, and right of the joint point likelihood map based on the difference between the cut-out region for joint point detection and the cut-out region for pose estimation. At this time, the likelihood of the margin region is filled as 0. The values ​​for providing the margin are as follows.

[0032]

number

[0033] In the formula, Margin left,right represents the amount of margin to be applied on the left and right, and Margin top,bottom represents the amount of margin to be applied above and below. keypoint ,height keypoint respectively represent the width and height of the cutout region for joint point detection on the original image.

[0034] FIG. 7 is a diagram for explaining an example of a result of a conversion process to a joint point likelihood map for posture estimation. FIG. 7(a) to FIG. 7(c) respectively show an example of a joint point likelihood map before conversion, a joint point likelihood map after conversion, and a cut-out image for posture estimation. As shown in FIG. 7(b), the spatial information of the joint point likelihood map after conversion matches the cut-out image for posture estimation through the processes of resizing, rotating, and adding a margin. Note that in the example of FIG. 7, the spatial information is matched through three conversion processes of resizing, rotating, and adding a margin, but it is not necessary to perform all three conversion processes as long as it is possible to match the spatial information through at least one of these conversion processes.

[0035] Furthermore, the conversion unit 205 weights the converted joint point likelihood map for each joint point. The weighting of the joint points is performed based on weight information for each joint point that is determined in advance for each rotation amount. FIG. 8 shows an example of weight information applied to the joint point likelihood map. As shown in FIG. 8, the weight information includes weights for each of the image rotation amounts of 0 degrees, 90 degrees, 180 degrees, and 270 degrees. The conversion unit 205 sends the thus weighted joint point likelihood map to the posture estimation unit 206 as a joint point likelihood map for posture estimation.

[0036] The posture estimation unit 206 outputs a posture estimation result based on the clipped image for posture estimation and the joint point likelihood map for posture estimation. Here, the posture estimation result is output as score information in which the higher the likelihood that the person is in a posture classified into "standing", "sitting", and "lying down", the larger the corresponding value becomes. The posture estimation process is executed using a deep learning model.

[0037] FIG. 9 is a diagram showing an example of the structure of a deep learning model for posture estimation. The deep learning model has a first feature extraction layer 900, a second feature extraction layer 901, a combination layer 902, and a discrimination layer 903. The first feature extraction layer 900 extracts features from a joint point likelihood map for posture estimation generated by the conversion unit 205, and the second feature extraction layer 901 extracts features from a cut-out image for posture estimation generated by the image generation unit 203. In particular, the output sizes of the first feature extraction layer 900 and the second feature extraction layer 901 are both M×N-dimensional numerical vectors, so that the spatial information is input to the discrimination layer 903 in a matched state. The discrimination layer 903 has a structure for outputting a posture estimation result based on the feature matrix combined in the combination layer 902.

[0038] Next, the processing of the image processing device 100 according to this embodiment will be described with reference to the flowchart of Fig. 10. Fig. 10 is a flowchart showing an example of a processing procedure for estimating a posture by the image processing device 100 according to this embodiment.

[0039] First, in S1000, the image acquisition unit 201 performs an image acquisition process. Then, in S1001, the image acquisition unit 201 determines whether or not the image has been acquired. If the result of this determination is that the image has not been acquired, the process proceeds to S1008 and ends. On the other hand, if the image has been acquired, the process proceeds to S1002.

[0040] Next, in S1002, the human body detection unit 202 detects all people in the acquired image and generates a list of human body detection results. Then, in S1003, the image generation unit 203 generates a cut-out image for joint point detection from the image and the list of human body detection results. After that, in S1004, the joint point detection unit 204 performs a joint point detection process on the cut-out image for joint point detection. In this process, a joint point likelihood map is generated.

[0041] Next, in S1005, image generation unit 203 generates a cut-out image for posture estimation from the image and the list of human body detection results. At this time, image generation unit 203 also generates image generation information. Next, in S1006, conversion unit 205 converts the joint point likelihood map generated in S1004 into a joint point likelihood map for posture estimation, using the image generation information generated in S1005. Then, in S1007, posture estimation unit 206 performs posture estimation processing using the cut-out image and joint point likelihood map for posture estimation. Then, the process returns to S1000 and repeats the process.

[0042] As described above, according to the present embodiment, the joint point likelihood map generated from the cutout image for joint point detection is converted to correspond to the spatial information of the cutout image for posture estimation by processing of resizing, rotation, and adding margin. This allows accurate correspondence between the joint point detection result and the posture of the person, thereby further improving the estimation accuracy of the posture of the person. In addition, when generating the cutout image for posture estimation, if there is a difference of a predetermined value or more in width and height compared with the visible detection frame and the invisible detection frame, the cutout image for posture estimation is generated based on the invisible detection frame that is wider than the visible detection frame. This allows the posture to be estimated taking into account important information such as surrounding objects, so that the posture can be estimated with high accuracy.

[0043] In this embodiment, the calculation method of the cutout region for joint detection and the cutout region for posture estimation has been described using formulas (1) to (10), but any calculation method may be used as long as the periphery of the person region can be cut out, and the present invention is not limited to this. In addition, in this embodiment, an example has been described in which the spatial information of the joint point likelihood map is matched with the cutout image for posture estimation by processing of resizing, rotation, and adding margins, but other methods may be used. Specifically, any transformation that matches the cutout image for posture estimation with the spatial information, such as an affine transformation, may be used, and the present invention is not limited to this.

[0044] Furthermore, in the present embodiment, an example has been given in which score information having a value for each posture classification is output as a posture estimation result, but the most probable classification information may be output as a label. The posture estimation result is not limited to this, and may be any information indicating what posture the person in question is in.

[0045] Second Embodiment In this embodiment, an example will be described in which object detection or floor estimation is performed from an image, and image generation conditions and conversion conditions are controlled using object information and depth information. Below, the same parts as in the first embodiment will be omitted, and only the differences will be described.

[0046] Fig. 11 is a block diagram showing an example of the functional configuration of an image processing device 1000 according to this embodiment. In this embodiment, an object detection unit 207 and a depth estimation unit 208 are further added, as compared with the configuration of Fig. 2 described in the first embodiment.

[0047] The object detection unit 207 detects objects in the image. Here, the object refers to a chair, a desk, a shelf, etc., and the technology described in Non-Patent Document 1 is used for object detection. The object detection result includes information on a detection frame that contains the object and an object label that indicates what the object is. The object detection unit 207 sends the object detection result to the image generation unit 203 and the conversion unit 205.

[0048] The depth estimation unit 208 estimates depth information of the shooting environment from the image. Here, the depth information is distance information from the camera to the subject expressed as a depth map, and is bitmap data in which each pixel is indicated by a depth value. Note that the technology described in Non-Patent Document 3 is used for depth estimation. The depth estimation unit 208 sends the depth information to the image generation unit 203 and the conversion unit 205.

[0049] In addition to the processing described in the first embodiment, the image generation unit 203 changes the cutout region for pose estimation based on the human body detection result, the object detection result, and the depth information. First, the cutout region for pose estimation is obtained using the method of the first embodiment. Next, for each human body detection result, the cutout region for pose estimation is corrected so that it is horizontal to the floor surface due to lens distortion and camera tilt. Finally, based on the object detection result, the cutout region for pose estimation is corrected so that it includes an object region related to a person.

[0050] Here, a procedure for correcting the cutout region for pose estimation so that it is horizontal to the floor surface will be described. First, if the difference between the bottom y coordinates of the visible detection frame and the invisible detection frame of the target person is equal to or greater than a predetermined value, it is determined that occlusion has occurred. If it is determined that occlusion has occurred, correction of the cutout region is not performed. On the other hand, if it is determined that occlusion has not occurred, a depth map corresponding to the peripheral part of the bottom y coordinate of the visible detection frame is cut out from the depth map of the entire image and used as a local depth map. If the local depth map is operated in the x-axis direction and the depth value is flat, it is determined that the inclination is 0. If the depth value gradually decreases, it can be determined that the inclination is negative, and if the depth value gradually increases, it can be determined that the inclination is positive. The inclination angle can be determined according to the amount of change in the depth value. For example, if the increase in the depth value is an average of +8 degrees, the inclination angle is -15 degrees, and if the increase in the depth value is an average of +16 degrees, the inclination angle is -30 degrees, and so on. It is determined that the greater the amount of change, the greater the inclination angle.

[0051] 12 is a diagram for explaining an example of the relationship between the depth map and the estimated value of the tilt angle. The amount of rotation is calculated based on the tilt angle obtained using the local depth map, and the cut-out region is resized and rotated so as to include the original cut-out region, thereby correcting the cut-out region.

[0052] Although the rotation amount of the cutout region is calculated using depth information as described above, the rotation amount may be calculated by other methods. For example, the rotation amount of the cutout region may be calculated by directly measuring the rotation amount due to lens distortion from the internal parameters of the camera. Also, the tilt of the camera may be directly obtained from the external parameters of the camera, and the rotation amount of the cutout region may be calculated by regarding the tilt of the camera as the tilt angle. As another method, the rotation amount of the cutout region may be determined by measuring the degree of lens distortion from edge information of the shooting environment. Furthermore, the floor surface may be specified by area division using machine learning, the degree of tilt may be obtained from the shape of the floor surface area, and the rotation amount of the cutout region may be calculated from the degree of tilt. As described above, as long as it is possible to obtain the amount of rotation that makes the cutout region horizontal to the floor surface, the method is not limited to a specific method.

[0053] Next, we will explain the procedure for correcting the cutout region to include an object region related to a person. First, we list up from the object detection results those whose object detection frames overlap with the visible detection frame of the target person. If the object label of one of these matches a predetermined label, we correct the cutout region for pose estimation so that the object detection frame is also included in the cutout region.

[0054] FIG. 13 is a diagram for explaining an example of correcting a clipping region for posture detection using an object detection result. In the example of FIG. 13, a person in an image is sitting on a chair, and the lower half of the person's body is hidden by the chair and cannot be seen. First, a clipping region for posture estimation 1302 is calculated from a human body detection result 1300 by calculation using equations (6) to (10). In this embodiment, a clipping region for posture estimation 1303 including an object region is obtained by further performing correction using an object detection result 1301. An image is clipped based on the clipping region for posture estimation corrected by the above procedure, and is used as a clipped image for posture estimation.

[0055] The conversion unit 205 generates a joint point likelihood map for posture estimation based on the image generation information, the joint point likelihood map information, and the depth information. First, a process equivalent to that of the first embodiment is performed to generate a joint point likelihood map for posture estimation. Next, based on the image generation information, depth information is cut out from the depth information of the entire image so that the cut-out image for posture estimation and the spatial information match, and is set as person area depth information. Then, the cut-out person area depth information and the joint point likelihood map for posture estimation are combined. At this time, the depth information may be combined as a new channel, or the numerical value of a pixel whose likelihood of a joint point is equal to or greater than a certain value for each channel of the joint point likelihood map may be replaced with the value of the person area depth information, and the numerical value of the other pixels may be replaced with 0.

[0056] Although an example has been described in which depth information is directly replaced with a value when combining the joint point likelihood map with depth information, background depth information may be acquired in advance and replaced with a distance from a background object by taking a difference. Also, pseudo three-dimensional joint position information may be generated by taking a difference with depth information of a specific joint point. As described above, the method is not limited to a specific method as long as it is a method for converting a joint point likelihood map into information related to three-dimensional space.

[0057] Next, the processing of the image processing device 1000 according to this embodiment will be described with reference to the flowchart of Fig. 14. Fig. 14 is a flowchart showing an example of a processing procedure for estimating a posture by the image processing device 1000 according to this embodiment.

[0058] First, the processes of S1400 to S1402 and S1410 are respectively the same as S1000 to S1002 and S1008 in Fig. 10 described in the first embodiment. Then, in S1403, the object detection unit 207 detects an object from within the acquired image. Then, in S1404, the depth estimation unit 208 estimates the depth of the shooting environment from the acquired image and generates depth information. The processes of S1405 and S1406 are respectively the same as S1003 and S1004 in Fig. 10.

[0059] Next, in S1407, the image generation unit 203 generates a cut-out image for posture estimation from the image, the human body detection result, the object detection result, and the depth information. In this process, a cut-out region for posture estimation is calculated in the same procedure as in the first embodiment, and then the cut-out region is corrected in the procedure described above. Then, an image is cut out based on the corrected cut-out region for posture estimation, and a cut-out image for posture estimation is generated. The image generation unit 203 also generates image generation information.

[0060] Next, in S1408, the conversion unit 205 converts the joint point likelihood map generated in S1406 into a joint point likelihood map for pose estimation using the image generation information generated in S1407, and combines it with the depth information. The process of S1409 is the same as that of S1007 in Fig. 10, and then returns to S1400.

[0061] As described above, according to this embodiment, the clipping region for pose estimation is corrected based on the depth map, so that the pose can be estimated with higher accuracy even for an image that is not horizontal to the floor surface due to lens distortion or camera tilt. In addition, the clipping region for pose estimation is corrected to include an object region related to a person, so that important information such as surrounding objects is also included in the clipped image for pose estimation, so that the pose can be estimated with higher accuracy.

[0062] (Other embodiments) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0063] The disclosure of this embodiment includes the following configuration, method, and program.

[0064] (Configuration 1) A human body detection means for detecting a human body area from an image; an image generating means for generating, from the image, a first clipped image including a region of the human body detected by the human body detecting means, and a second clipped image including the region of the human body and different from the first clipped image; a joint point detection means for detecting joint points of the human body from the first extracted image and generating joint point information of the human body; a conversion means for converting the joint point information so as to correspond to spatial information of the second extracted image; a posture estimation means for estimating a posture of the human body based on the joint point information converted by the conversion means and the second extracted image; 13. An image processing device comprising:

[0065] (Configuration 2) the joint point information is a map indicating positions of the joint points as likelihoods, The image processing device according to configuration 1, characterized in that the conversion means performs at least one conversion process of rotating, resizing, and adding a margin on the map to convert it into a map corresponding to spatial information of the second cropped image. (Configuration 3) 3. The image processing device according to claim 1, wherein the conversion means further converts the information into weighted information for each joint point detected by the joint point detection means. (Configuration 4) the human body detection means detects a visible detection frame that includes a visible area of ​​the human body and an invisible detection frame that includes the visible area and invisible area of ​​the human body, The image processing device according to any one of configurations 1 to 3, wherein the image generation means generates the first cropped image based on the position and size of the visible detection frame, and generates the second cropped image based on the position and size of the invisible detection frame.

[0066] (Configuration 5) The image processing device according to any one of configurations 1 to 4, wherein the posture estimation means estimates the posture of the human body using a deep learning model including a first feature extraction layer that outputs a first feature from the joint point information converted by the conversion means, a second feature extraction layer that outputs a second feature from the second cropped image, a combination layer that combines the first feature and the second feature, and a classification layer that estimates a posture using the combined feature. (Configuration 6) the human body detection means detects a region of the human body while rotating the image; 6. The image processing device according to any one of configurations 1 to 5, wherein the image generating means generates the first clipped image from an image in which the region of the human body is rotated by the detected rotation amount.

[0067] (Configuration 7) The apparatus further includes a depth estimation means for estimating a depth of a photographing environment from the image, 7. The image processing device according to any one of configurations 1 to 6, wherein the conversion means further converts the joint point information based on the depth estimated by the depth estimation means. (Configuration 8) The apparatus further includes a depth estimation means for estimating a depth of a photographing environment from the image, 8. The image processing device according to any one of configurations 1 to 7, wherein the image generating means further generates the second extracted image based on the depth estimated by the depth estimating means. (Configuration 9) The method further comprises: detecting an object from the image; The image processing device according to any one of configurations 1 to 8, wherein the image generating means generates the second cut-out image so as to include an area of ​​an object related to the human body among the objects detected by the object detecting means.

[0068] (method) a human body detection step of detecting a human body region from the image; an image generating step of generating, from the image, a first clipped image including a region of the human body detected in the human body detecting step, and a second clipped image including the region of the human body and different from the first clipped image; a joint point detection step of detecting joint points of the human body from the first extracted image and generating joint point information of the human body; a conversion step of converting the articulation point information to correspond to spatial information in the second extracted image; a posture estimation step of estimating a posture of the human body based on the joint point information converted in the conversion step and the second extracted image; 13. A method for controlling an image processing apparatus comprising the steps of:

[0069] (program) a human body detection step of detecting a human body region from the image; an image generating step of generating, from the image, a first clipped image including a region of the human body detected in the human body detecting step, and a second clipped image including the region of the human body and different from the first clipped image; a joint point detection step of detecting joint points of the human body from the first extracted image and generating joint point information of the human body; a conversion step of converting the articulation point information so as to correspond to spatial information of the second extracted image; a posture estimation step of estimating a posture of the human body based on the joint point information converted in the conversion step and the second extracted image; A program for causing a computer to execute the following. [Explanation of symbols]

[0070] 202 human body detection unit, 203 image generation unit, 204 joint point detection unit, 205 conversion unit, 206 posture estimation unit

Claims

1. A human body detection means for detecting the area of ​​the human body from an image, Image generation means that generates from the aforementioned image a first cropped image including the region of the human body detected by the human body detection means, and a second cropped image including the region of the human body but different from the first cropped image, A joint point detection means for detecting joint points of the human body from the first cropped image and generating joint point information of the human body, A conversion means for converting the joint point information to correspond to the spatial information of the second cropped image, A posture estimation means for estimating the posture of the human body based on the joint point information converted by the conversion means and the second cropped image, It has, The posture estimation means is an image processing device that estimates the posture of a human body using a deep learning model comprising: a first feature extraction layer that outputs a first feature quantity from joint point information converted by the conversion means; a second feature extraction layer that outputs a second feature quantity from the second cropped image; a coupling layer that combines the first feature quantity and the second feature quantity; and a discrimination layer that estimates the posture using the combined feature quantity.

2. The aforementioned joint point information is a map that shows the position of the joint point as a likelihood, The image processing apparatus according to claim 1, characterized in that the conversion means performs at least one conversion process on the map, such as rotation, resizing, and adding margins, to convert it into a map corresponding to the spatial information of the second cropped image.

3. The image processing apparatus according to claim 1, characterized in that the conversion means further converts the information into weighted information for each joint point detected by the joint point detection means.

4. The human body detection means detects a visible detection frame that encompasses the visible region of the human body and an invisible detection frame that encompasses both the visible and invisible regions of the human body. The image processing apparatus according to claim 1, characterized in that the image generation means generates a first cropped image based on the position and size of the visible detection frame, and generates a second cropped image based on the position and size of the invisible detection frame.

5. The human body detection means detects the region of the human body while rotating the image, The image processing apparatus according to claim 1, characterized in that the image generation means generates the first cropped image from an image in which the region of the human body has been rotated by the detected amount of rotation.

6. The system further includes a depth estimation means for estimating the depth of the shooting environment from the aforementioned image, The image processing apparatus according to claim 1, wherein the conversion means further converts the joint point information based on the depth estimated by the depth estimation means.

7. The system further includes a depth estimation means for estimating the depth of the shooting environment from the aforementioned image, The image processing apparatus according to claim 1, wherein the image generation means further generates the second cropped image based on the depth estimated by the depth estimation means.

8. The system further includes object detection means for detecting the region of an object from the aforementioned image, The image processing apparatus according to claim 1, characterized in that the image generation means generates a second cropped image that includes the region of an object related to the human body among the objects detected by the object detection means.

9. A human body detection process that detects the area of ​​the human body from an image, Image generation step to generate from the aforementioned image a first cropped image including the region of the human body detected in the human body detection step, and a second cropped image including the region of the human body but different from the first cropped image, A joint point detection step that detects the joint points of the human body from the first cropped image and generates joint point information of the human body, A conversion step of converting the joint point information to correspond to the spatial information in the second cropped image, A posture estimation step is performed to estimate the posture of the human body based on the joint point information converted in the conversion step and the second cropped image, It has, The control method for an image processing apparatus is characterized in that the posture estimation step estimates the posture of the human body using a deep learning model comprising: a first feature extraction layer that outputs a first feature quantity from joint point information converted in the conversion step; a second feature extraction layer that outputs a second feature quantity from the second cropped image; a coupling layer that combines the first feature quantity and the second feature quantity; and a discrimination layer that estimates the posture using the combined feature quantity.

10. A human body detection process that detects the area of ​​the human body from an image, Image generation step to generate from the aforementioned image a first cropped image including the region of the human body detected in the human body detection step, and a second cropped image including the region of the human body but different from the first cropped image, A joint point detection step that detects the joint points of the human body from the first cropped image and generates joint point information of the human body, A conversion step of converting the joint point information to correspond to the spatial information of the second cropped image, A posture estimation step is performed to estimate the posture of the human body based on the joint point information converted in the conversion step and the second cropped image, Have the computer run it, The posture estimation step is a program for estimating the posture of a human body using a deep learning model comprising: a first feature extraction layer that outputs a first feature quantity from the joint point information converted in the conversion step; a second feature extraction layer that outputs a second feature quantity from the second cropped image; a coupling layer that combines the first feature quantity and the second feature quantity; and a discrimination layer that estimates the posture using the combined feature quantity.

Images