Attitude estimation device, imaging device, attitude estimation method, and program

Abstract

To solve the problem in which the conventional method for a joint connection problem in posture estimation requires map calculations on a neural net for the number of joints, which requires a lot of calculations and since a determination is made from an image, there is a possibility of miscalculation when the image is confusing.SOLUTION: A posture estimation device includes depth detection means for detecting depth in the depth direction from an image, joint detection means for detecting joints or organs from the image, and joint connection means for connecting the joints or the organs that are detected. The depth detection means detects depths at locations of the plurality of detected joints or organs. The joint connection means selects the joints or organs to be connected by a combination of joints or organs that are determined to be of the same person by evaluating the depth on the path between the joints or organs.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to a technique for detecting multiple parts of an object and estimating the posture of the object by connecting these parts. [Background technology]

[0002] Advances in machine learning technology have led to the proposal of methods for detecting people in images and estimating their poses with high accuracy. One such method is Human-Pose-Estimation (hereafter referred to as "pose estimation"), which estimates each of a person's joints and their connections from an image and expresses the person's pose using a simple skeleton. This pose estimation method uses a bottom-up approach to extract all of the joints of all people in the image, and then assembles each joint individually to estimate the pose. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 9-237348 [Non-patent literature]

[0004] [Non-Patent Document 1] Associative-Embedding(NIPS 2017) Summary of the Invention [Problem to be solved by the invention]

[0005] As an example of a bottom-up approach, Paper 1 discloses a method for connecting the joints of multiple individuals by creating an evaluation value map called an Embedding map and determining the connections between each joint based on the similarity of the evaluation value. In this case, the Embedding map is output for each joint to be connected. Generally, the computational load required to output map-like data in neural networks is high. Therefore, preparing map-like data for each joint requires a significant amount of computation. While this is not a major problem for PCs or supercomputers equipped with GPUs, it is impractical to implement this on battery-powered embedded devices due to the processing power requirements. Furthermore, when creating an Embedding map, it can be difficult to distinguish the joints of the same individual in visually confusing images. For example, if the head of a person behind the person accidentally appears on the shoulder, the joints of the person in front may be connected to the head of the person behind.

[0006] Patent Document 1 discloses a method for calculating the distance between joints in three dimensions by adding depth information to a normal two-dimensional image. This method estimates posture by comparing with a stick model of the human body and connecting joints with matching distances. In this case, three-dimensional distance calculations are required for each joint, which imposes a computational load, although not as heavy as the neural network described above.

[0007] Furthermore, highly accurate depth information is required to connect joints, so high distance measurement accuracy is required to distinguish joints from those of other people and to determine the joint distance.

[0008] Therefore, an object of the present invention is to provide a posture estimation device that can improve the accuracy of joint connection while reducing the amount of calculation by using distance differences and the depth of paths between joints in joint connection for posture estimation. [Means for solving the problem]

[0009] In order to achieve the above object, the posture estimation device according to the present invention includes a depth detection unit that detects a depth map in the depth direction from an image, and a joint detection unit that detects a joint map in the depth direction from the image. ora joint detection means for detecting an organ; and the detected joint. or and an articulation means for connecting the organs, the joint detection means detects representative points of a plurality of joints or organs for each subject; The articulation means comprises: When a representative point of a connection source joint or organ has a plurality of representative points of connection destination joints or organs, a path between the representative points of the connection source and the connection destination joints or organs is identified on the depth map for each of the connection destinations; On the route Each point Depth Variance Evaluation value of calculation death, The connection destination with the smallest variance evaluation value among these variance evaluation values ​​is selected as the connection source joint or organ. Select joints or organs that should be connected to the same person By doing so, the process of repeating connections between a plurality of representative points is repeated the number of times equal to the number of heads detected by the joint detection means. It is characterized by do. [Effects of the Invention]

[0010] The posture estimation device of the present invention can reduce the amount of calculation required for joint connection in posture estimation. Similarly, joint connection accuracy can be improved even for ambiguous subjects by using distance differences and depth along the path between joints. [Brief explanation of the drawings]

[0011] [Figure 1] Interchangeable lens camera system block diagram [Figure 2] Pixel arrangement diagram for image-plane phase-detection AF [Figure 3] Flowchart of posture estimation process [Figure 4] Depth map and graph examples [Figure 5] Joint detection example [Figure 6] Flowchart of joint connection processing [Figure 7] Flowchart of joint connection processing [Figure 8] Flowchart of joint connection processing [Figure 9] Depth map and graph examples DETAILED DESCRIPTION OF THE INVENTION

[0012] The best mode for carrying out the present invention will be described in detail below with reference to the accompanying drawings. Note that the embodiment described below is an example of a means for realizing the present invention, and should be appropriately modified or changed depending on the configuration of the device to which the present invention is applied and various conditions, and the present invention is not limited to the following embodiment. Furthermore, although multiple features are described in the embodiment, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined as desired.

[0013] [First embodiment] FIG. 1 is a block diagram showing an example of the functional configuration of an interchangeable lens camera as an example of an imaging apparatus according to a first embodiment of the present invention.

[0014] The imaging device of this embodiment is composed of an interchangeable lens unit 10 and a camera body 20. A lens control unit 106 that controls the overall operation of the lens and a camera control unit 30 that controls the overall operation of the camera system including the lens unit 10 can communicate with each other via terminals provided on the lens mount.

[0015] First, the configuration of lens unit 10 will be described. Fixed lens 101, aperture 102, and focus lens 103 constitute the imaging optical system. Aperture 102 is driven by aperture drive unit 104 and controls the amount of light incident on image sensor 201, which will be described later. Focus lens 103 is driven by focus lens drive unit 105, and the focal length of the imaging optical system changes depending on the position of focus lens 103. Aperture drive unit 104 and focus lens drive unit 105 are controlled by lens control unit 106, which determines the opening size of aperture 102 and the position of focus lens 103.

[0016] The lens operation unit 107 switches between AF (autofocus) and MF (manual focus) modes, adjusts the position of the focus lens using MF, sets the operating range of the focus lens, and sets the image stabilization mode. In other words, the lens operation unit 107 is a group of input devices that allow the user to make settings related to the operation of the lens unit 10. When the lens operation unit 107 is operated, the lens control unit 106 performs control in accordance with the operation.

[0017] The lens control unit 106 controls the aperture driving unit 104 and the focus lens driving unit 105 in accordance with control commands and control information received from the camera control unit 30, which will be described later, and also transmits lens control information to the camera control unit 30.

[0018] Next, a description will be given of the configuration of the camera body 20. The camera body 20 is configured so that it can acquire an image signal from a light beam that has passed through the imaging optical system of the lens unit .

[0019] The image sensor 201 is composed of a CCD or CMOS sensor. A light beam incident from the photographing optical system of the lens unit 10 forms an image on the light receiving surface of the image sensor 201 and is converted into a signal charge corresponding to the amount of incident light by photodiodes provided in pixels arranged on the image sensor 201. The signal charge accumulated in each photodiode is sequentially read out from the image sensor 201 as a voltage signal corresponding to the signal charge, in response to a drive pulse output by a timing generator 214 in accordance with a command from the camera control unit 30.

[0020] Each pixel of the image sensor 201 used in this embodiment is composed of two (a pair) photodiodes A and B and one microlens provided for the pair of photodiodes A and B. Each pixel divides the incident light into pupils using the microlens to form a pair of optical images on the pair of photodiodes A and B, and outputs a pair of pixel signals (signals A and B) used as an AF signal (described later) from the pair of photodiodes A and B. Furthermore, an imaging signal (signal A+B) can be obtained by adding the outputs of the pair of photodiodes A and B. The image sensor used in the present invention is not limited to this, and may be composed of two or more photodiodes, such as four photodiodes corresponding to each pixel (each macrolens). Furthermore, although each pixel (each microlens) corresponds to one photodiode, a configuration may also be used in which multiple photodiodes are provided between different pixels to receive light beams from different pupil regions of the imaging optical system.

[0021] The multiple A signals output from the multiple pixels are combined with each other, and the multiple B signals are combined with each other (to form an image signal). This provides a pair of image signals as AF signals used for AF using an image plane phase difference detection method (hereinafter referred to as image plane phase difference AF), i.e., focus detection signals. The AF signal processing unit 204, which will be described later, performs a correlation calculation on the pair of image signals to calculate the phase difference (hereinafter referred to as image shift amount), which is the amount of shift between the pair of image signals, and further calculates the defocus amount (and defocus direction) of the photographic optical system from the image shift amount.

[0022] FIG. 2(a) shows a pixel configuration that does not support image-plane phase-difference AF, while FIG. 2(b) shows a pixel configuration that supports image-plane phase-difference AF. Both figures use a Bayer array, with R representing a red color filter, B representing a blue color filter, and Gr and Gb representing green color filters. In the pixel configuration shown in FIG. 2(b) that supports image-plane phase-difference AF, two photodiodes A and B are provided in a pixel that corresponds to one pixel (enclosed by a solid line) in the pixel configuration not supporting image-plane phase-difference AF shown in FIG. 2(a). The pixel division method shown in FIG. 2(b) is merely an example; the pixel may be divided vertically or into two parts each horizontally and vertically (a total of four parts). Furthermore, the same image sensor may include multiple types of pixels divided by different division methods. The pixel arrangement is not limited to the Bayer array.

[0023] A CDS (Correlation Double Sampling) / AGC (Auto Gain Control) / AD (Analog to Digital) converter 202 performs correlated double sampling to remove reset noise, gain adjustment, and AD conversion on the AF signal and imaging signal read from the image sensor 201. The converter 202 outputs the imaging signal and AF signal that have been subjected to these processes to an image input controller 203 and an AF signal processor 204, respectively.

[0024] The image input controller 203 stores the imaging signal output from the converter 202 as an image signal in the SDRAM 209 via the bus 21. The image signal stored in the SDRAM 209 is read out by the display control unit 205 via the bus 21 and displayed on the display unit 206. In a recording mode in which the image signal is recorded, the image signal stored in the SDRAM 209 is recorded by the recording medium control unit 207 in a recording medium 208 such as a semiconductor memory.

[0025] The AF signal processing unit 204 performs correlation calculations on a pair of image signals, which are AF signals output from the converter 202, and calculates the amount of image shift between the pair of image signals. The AF signal processing unit 204 outputs the amount of image shift (focus detection amount) calculated in the ranging area and information on reliability to the camera control unit 30.

[0026] The ROM 210 stores control programs and processing programs executed by the camera control unit 30, as well as various data required for executing these programs. The flash ROM 211 stores various setting information related to the operation of the camera 20 set by the user.

[0027] The depth detection unit 301 in the camera control unit 30 collects the focus detection amounts for each image height from the AF signal processing unit 204 and stores them as a two-dimensional map of the focus detection amounts (corresponding to the depth in the depth direction) as shown in Fig. 4. This will be referred to as the depth map hereinafter.

[0028] The joint detection unit 302 detects and stores each joint and organ of a person based on the imaging signal input from the image input controller 203, as shown in FIGS. 5(d) to 5(i). The detection process performed by the joint detection unit 302 is realized by feature extraction processing using a CNN (convolutional neural network). A CNN is a CNN trained by machine learning, and corresponds to a trained model. The joint detection unit 302 may perform the detection process using a trained model trained by machine learning algorithms other than CNN (decision tree, support vector machine, etc.). Furthermore, the joint detection unit 302 may perform the object detection process using a trained model trained by machine learning using R-CNN (Regions with CNN features), etc.

[0029] The joint detection unit 302 detects a plurality of body parts of a person as a subject from an image. The body parts are sometimes called local regions, and are not limited to specific biological parts (such as the head or shoulders), but may be characteristic regions of a person.

[0030] The joint detection unit 302 may detect multiple body parts using an animal, a vehicle, or the like as a subject. The joint detection unit 302 may also detect representative points of a person's head, including the face, shoulders, torso, knees, etc. by estimating them, and output the detection results all at once. The joint detection unit 302 may also detect body parts other than the above-mentioned parts. The joint detection unit 302 may detect body parts of a person using rectangular areas or points. When a body part is detected using an area, the body part may be represented by a single point such as the center or center of gravity of the area. The joint detection unit 302 may also detect each body part from a predetermined area (region of interest) of the image. The joint detection unit 302 may detect body parts using a predetermined detection method for detecting body parts from a subject, rather than using a trained model based on machine learning.

[0031] The joint detection unit 302 may use CNN to output the coordinates of body parts in a clustered state for each person. In this case, the number of clusters output from the CNN can be considered as the number of people present in the frame. The joint detection unit 302 may estimate a person as a subject and multiple body parts of the person, and output the estimation results.

[0032] The camera control unit 30 controls each unit within the camera body 20 while exchanging information with them. Furthermore, in response to input from a camera operation unit 213 based on user operation, the camera control unit 30 performs various processes corresponding to user operation, such as turning the power on / off, changing various settings, capturing images, AF processing, and playback of recorded images. Furthermore, the camera control unit 30 transmits control commands for the lens unit 10 (lens control unit 106) and information about the camera body 20 to the lens control unit 106, and obtains information about the lens unit 10 from the lens control unit 106. The camera control unit 30 is configured by a microcomputer, and controls the entire camera system including the interchangeable lens 10 by executing a computer program stored in ROM 210.

[0033] The camera control unit 30 calculates the defocus amount using the image shift amount in the distance measurement area calculated by the AF signal processing unit 204, and controls the driving of the focus lens 103 via the lens control unit 106 based on the defocus amount.

[0034] The following describes the processing performed by the camera body 20. The camera control unit 30 performs the following processing in accordance with an imaging processing program, which is a computer program, or each unit within the camera performs the processing upon receiving instructions from the camera control unit 30. Figure 3 is a flowchart showing the procedure for the attitude estimation processing of the camera body 20. "S" indicates a step.

[0035] In step S301, the depth detection unit 301 acquires a depth map of 9 frames by 5 frames as shown in FIG. 4 using the method described above.

[0036] In S302, the joint detection unit 302 detects joints from the imaging signal using the method described above. The results of joint detection by the joint detection unit 302 will be described later with reference to FIG.

[0037] In S303, the camera control unit 30 repeats the processes from S305 to S306 for the number of joint connections in the human body. In this embodiment, as will be described later, the processes are repeated five times for the number of joint connections.

[0038] In S304, the camera control unit 30 connects the joints detected by the joint detection unit 302. The connection algorithm will be described later.

[0039] In S306, the camera control unit 30 checks whether all the people in the screen have been connected. For example, if two heads are detected, it is assumed that there are two people in the screen and the process is repeated twice. If there are people whose joints have not yet been connected, the process returns to S303; if not, the process ends.

[0040] FIG. 4 shows an example of a depth map. The image at the top of FIG. 4 is divided into 9×5 frames, each of which has depth information. In this embodiment, a 9×5 frame is used for the sake of understanding, but in actual image-plane phase-difference AF, it is possible to create a depth map with even higher resolution. The graph at the bottom of FIG. 4 shows the depth information for each frame in the row indicated by arrow 401 in the depth map calculated by the depth detection unit 301. For example, the depth of frame 402 is mid-distance 438, which will be used to explain the joint connection method described below.

[0041] Figure 5 shows how joints are detected from an image. In posture estimation, it is common to detect 14 points: head, neck, left and right shoulders, left and right elbows, left and right wrists, left and right hips, left and right knees, and left and right ankles, as shown in Figure 5(a). In this case, the number of joint connections is 13, and the method in the aforementioned paper 1 requires output from a neural network with 14 faces + 13 faces.

[0042] In this embodiment, to avoid the illustration becoming too complicated and difficult to see, six joints are shown as in Figure 5(b): the head, center of gravity, left and right hands, and left and right feet. However, in practical applications, 14 joints may be used. For example, suppose an image like that in Figure 5(c) is input to the joint detection unit 302. Then, the head's reaction appears as in Figure 5(d). Similarly, the center of gravity appears as in Figure 5(e), the right hand as in Figure 5(f), the left hand as in Figure 5(g), the right foot as in Figure 5(h), and the left foot as in Figure 5(i).

[0043] As shown in FIG. 5(b), the path connecting the joints requires five connections: head and center of gravity 501, right hand and center of gravity 502, left hand and center of gravity 503, right foot and center of gravity 504, and left foot and center of gravity 505. For example, if there is a two-point head response as shown in FIG. 5(d), there will be two people on the screen, so the loop from S303 to S306 in FIG. 3 will be processed twice. In the first loop, when connecting from head 511 to the center of gravity, there are two candidates, 512 and 513 in FIG. 5(e). The method for selecting this joint will be described later, but in this embodiment, selecting 512 completes one connection between the head and center of gravity. Similarly, in the case of five joints, connections between the five joints are repeated to complete the connection as shown in FIG. 5(b).

[0044] The joint connection method performed in S304 in the first embodiment will be described with reference to the flowchart in FIG. 6 and FIG.

[0045] The connection source and connection destination of the joints are specified in S303 in Fig. 3. For example, in Fig. 4, there are two people, people 410 and 420, in the image. Therefore, there are two sets of each joint in the part of the body that appears on the screen. Here, an example will be explained in which the center of gravity 411 of person 410 is the connection source and the left hand is the connection destination.

[0046] First, in S601, the camera control unit 30 stores depth information of the connection source. In this example, the depth of the center of gravity 411 of the connection source is calculated by the depth detection unit 301 and is set to short distance 432.

[0047] In S602, the camera control unit 30 searches for connection destinations one by one in the order they are found, stores the depth information in association with the coordinates, and also counts the number of connection destinations found.

[0048] In S603, the camera control unit 30 scans the entire screen and determines whether or not the search for joints has been completed. If the entire screen has been scanned, the process proceeds to S604. If there is still area remaining, the process proceeds to S602. In this example, the joint detection unit 302 detects two points 412 and 421 on the left hand, so the processes of S602 and S603 are repeated twice, and the number of joints is counted as two.

[0049] In S604, the camera control unit 30 starts a loop for the number of joints discovered.

[0050] In S605, the camera control unit 30 loops through the multiple connection destination joints in order, acquiring the depth of the joint coordinates using the depth detection unit 301. The depths of the connection source and connection destination are compared, and if the closest is found, the joint information, coordinates, and depth are overwritten and saved.

[0051] In S606, the camera control unit 30 closes the joint loop. In this example, it loops twice. In this example, the connection source 411 is at close range 432, the left hand 412 is at close range 434, and the left hand 421 is at medium range 438, so the left hand 412, which has the smallest distance difference, is saved.

[0052] In S607, the camera control unit 30 determines to connect the connection source with the connection destination that is closest in depth. In this example, the left hand 412, which is closer to the connection source 411, is selected and connected. In the loop of S303, by connecting all joints that are similarly close in depth, it is possible to distinguish between joints of different people that are at different distances and assemble a single person.

[0053] As described above, according to the first embodiment of the present invention, articulation of pose estimation can be performed with lower image plane phase difference AF and line integral amounts than with neural networks. Furthermore, even for ambiguous subjects, the accuracy of articulation can be improved by using depth differences, and articulation can be performed even with ranging information with low depth accuracy.

[0054] The image used by the depth detection unit 301 and the image used by the joint detection unit 302 may be captured at different times. In this case, the camera control unit 30 may apply the depth of the shifted position, taking into consideration that the position of the joint or organ detected by the joint detection unit 301 is shifted due to a time difference caused by the difference in the capture timing.

[0055] [Second embodiment] The method of joint connection performed in S304 in the second embodiment will be described using the flowchart in Fig. 7 and Fig. 4. Except for the sequence and processing flow described below, the same configuration and control as in the first embodiment will be performed.

[0056] The connection source and connection destination of the joint are specified in S303 of Fig. 3. As an example, an example will be described in which the center of gravity 411 of the person 410 in Fig. 4 is set as the connection source and the left hand as the connection destination.

[0057] 7, the camera control unit 30 stores the coordinates of the connection source, which in this example is the position of the center of gravity 411 of the connection source.

[0058] In S702, the camera control unit 30 searches for connection destinations one by one in the order they are found, stores the coordinates, and also counts the number of connection destinations found.

[0059] In S703, the camera control unit 30 determines whether the entire screen has been scanned to find joints, and if the entire screen has been scanned, the process proceeds to S704, and if there is still area remaining, the process proceeds to S702. In this example, the joint detection unit 302 detects two points 412 and 421 on the left hand, so the processes of S702 and S703 are repeated twice, and the number of joints is counted as two.

[0060] In S704, the camera control unit 30 starts a loop for the number of joints that have been discovered.

[0061] In S705, the camera control unit 30 calculates the variance (evaluation value) of depth information of each frame on the path on the depth map of the joint of the connection destination from the coordinates of the connection source using formula (1). The path selects a frame between the connection source and the connection destination using the Bresenham algorithm, and uses the depth of the corresponding point from the depth map.

[0062]

number

[0063] In the example, the route between source 411 and destination 412 in the first loop is 432 to 434, the depth is (10, 10, 10), and the variance is 0. In addition, in the second loop, the route between source 411 and destination 421 is 432 to 438, the depth is (10, 10, 10, 40, 20, 20, 20), and the variance is approximately 98.

[0064] In S706, the camera control unit 30 compares the currently saved variance of the connection destinations with the newly calculated variance, determines that the combination of connection destinations with the smaller variance is the combination of joints that should be connected for the same person, and overwrites and saves the joint information, coordinates, and depth that were saved previously. In this example, the coordinates of 412 and a variance of 0 are saved as the initial data in the first loop. In the second loop, the saved variance of 0 is compared with the variance of approximately 98 for connection destination 421, and since connection destination 421 is larger, it is not adopted, and the values ​​of connection destination 412 continue to be saved as is.

[0065] In S707, the camera control unit 30 closes the joint loop. In this example, the loop is performed twice.

[0066] In S708, the camera control unit 30 selects the connection destination with the smallest variance between the connection source and the connection destination route. In this example, the connection destination 412 is selected and connected.

[0067] In the S303 loop, all joints are connected together using paths with low variance. In the example shown in 435, when the background is placed between two people, the variance increases. Even if the people are close in depth, they can be distinguished as different people and assembled into a single person.

[0068] In this embodiment, whether a background is sandwiched between two people is determined by the variance of depth information using Equation 1, but other calculation methods may also be used.

[0069] As described above, according to the second embodiment of the present invention, articulation of pose estimation can be performed with lower image plane phase difference AF and line integral amounts than with neural networks. Furthermore, even for subjects with similar depths, the accuracy of articulation can be improved by using the variance of the paths, and articulation can be performed even with ranging information with low depth accuracy.

[0070] [Third embodiment] Figure 9 shows an example of a depth map. The image in the top of Figure 9 is divided into 9 x 5 boxes, each with its own depth information. The graph in the bottom of Figure 9 shows the depth information for each box in the row indicated by arrow 901 in the depth map calculated by the depth detection unit 301. For example, the depth of box 902 is far distance 938, which will be used in the explanation of the joint connection method described below.

[0071] The joint connection method performed in S304 in the second embodiment will be described using the flowchart in Fig. 8 and Fig. 9. Except for the sequence and processing flow described below, the same configuration and control as in the first or second embodiment will be performed.

[0072] The connection source and connection destination of the joint are specified in S303 of Fig. 3. As an example, an example will be described in which the center of gravity 911 of the person 910 in Fig. 4 is set as the connection source and the right hand as the connection destination.

[0073] 8, the camera control unit 30 stores the coordinates of the connection source, which in this example is the position of the center of gravity 411 of the connection source.

[0074] In S802, the camera control unit 30 searches for connection destinations one by one in the order they are found, stores the coordinates, and also counts the number of times they are found.

[0075] In S803, the camera control unit 30 determines whether the entire screen has been scanned to find joints, and if the entire screen has been scanned, the process proceeds to S804, and if there is still area remaining, the process proceeds to S802. In this example, the joint detection unit 302 detects two points 412 and 421 on the left hand, so the processes of S802 and S803 are repeated twice, and the number of joints is counted as two.

[0076] In S804, the camera control unit 30 starts a loop for the number of joints discovered earlier.

[0077] In S805, the camera control unit 30 acquires depth information for the coordinates of the connection source and the coordinates of the connection destination, and defines f(x) as a function of the depth between them as a linear line. Furthermore, the depth information for each frame of the path on the depth map from the coordinates of the connection source to the coordinates of the connection destination is calculated using equation (2). This equation has a lower value when there is a linear connection in the depth direction, which is considered here to be high continuity. For the path, a frame is selected between the connection source and destination using the Bresenham algorithm, and the depth of the corresponding point from the depth map is used.

[0078]

number

[0079] In the example, on the first loop, the route between source 911 and destination 912 is 931 to 933, the depth is (10, 20, 30), and the value of expression 2 is 0. On the second loop, the route between source 911 and destination 921 is 933 to 936, the depth is (30, 30, 30, 50), and the value of expression 2 is 5.

[0080] In S806, the camera control unit 30 compares the currently saved continuity of the connection destination with the continuity calculated this time, and overwrites and saves the joint information, coordinates, and depth of the connection destination with the higher (lower value) value. In the first loop of the example, the coordinates and continuity of 0 for 912 are saved as the initial data. In the second loop, the saved continuity of 0 is compared with the continuity of 5 for connection destination 921. In this case, the continuity of connection destination 921 is low (larger value), so it is not adopted, and the values ​​of connection destination 912 continue to be saved as is.

[0081] In S807, the camera control unit 30 closes the joint loop. In this example, the loop is performed twice.

[0082] In S808, the camera control unit 30 selects a connection destination with high continuity between the connection source and connection destination paths. In this example, connection destination 912 is selected and connected. In the loop of S303, joints with connection paths with low variance are connected to each other in the same way for all joints. In this example, the depth difference between 912 and 921 is 20, but in terms of continuity, 912 is more continuous, and it can be inferred that they are joints of the same person, and 921 can be distinguished as a joint of a different person, allowing one person to be assembled.

[0083] In this embodiment, Equation 2 is used to determine continuity, but other calculation methods may be used. For example, a method may be used in which the depth gradient is calculated using the depth and coordinates on the depth map and then compared.

[0084] As described above, according to the third embodiment of the present invention, articulation of pose estimation can be performed with lower image plane phase difference AF and line integral amounts than with neural networks. Furthermore, even for ambiguous subjects, the accuracy of articulation can be improved by using continuity, and articulation can be performed even with ranging information with low depth accuracy.

[0085] Although the preferred embodiments of the present invention have been described using the first to third embodiments, the present invention is not limited to these embodiments and various modifications and changes are possible within the scope of the gist of the present invention. [Explanation of symbols]

[0086] 10 Lens unit 101 Fixed Lens 102 aperture 103 Focus Lens 104 Aperture drive unit 105 Focus lens drive unit 106 Lens control unit 107 Lens operation section 20 Camera body 21 Bus 201 Image sensor 202 CDS / AGC / AD converter 203 Image Input Controller 204 AF signal processing unit 205 Display control unit 206 Display section 207 Recording medium control unit 208 Recording Media 209 SDRAM 210 ROM 211 Flash ROM 213 Camera control unit 214 Timing Generator 30 Camera control unit 301 Depth detection unit 302 Joint detection unit

Claims

1. a depth detection means for detecting a depth map in the depth direction from the image; joint detection means for detecting joints or organs from the image; a joint connection means for connecting the detected joint or organ; the joint detection means detects representative points of a plurality of joints or organs for each subject; When there are multiple representative points of connection destination joints or organs for a representative point of a connection source joint or organ, the joint connection means identifies a path between the connection source and connection destination representative points on the depth map for each connection destination, calculates a variance evaluation value of the depth of each point on the path, and determines and selects the connection destination with the smallest variance evaluation value as the joint or organ to be connected to of the same person as the connection source joint or organ, thereby repeating the process of connecting between multiple representative points as many times as the number of heads detected by the joint detection means.

2. 2. The posture estimation device according to claim 1, wherein the joint connection means compares a currently stored variance evaluation value of the connection destination with a newly calculated variance evaluation value, determines the combination of connection destinations with the smaller variance evaluation value as a combination of joints to be connected for the same person, and stores the joint information, coordinates, and depth.

3. 2. The posture estimation device according to claim 1, wherein the depth map is obtained by dividing the image into a plurality of frames and storing depth information for each frame as a two-dimensional map.

4. 2. The posture estimation device according to claim 1, wherein the detection process performed by the joint detection means is realized by a feature extraction process using a convolutional neural network.

5. 2. The posture estimation device according to claim 1, wherein the depth detection means calculates a phase difference from signals of different pupil divisions by correlation calculation.

6. The posture estimation device according to any one of claims 1 to 5, characterized in that the images used by the depth detection means and the images used by the joint detection means are taken at different times, and the depth of the shifted position is applied taking into account the fact that the positions of the joints or organs detected by the joint detection means are shifted due to the time difference.

7. an image sensor including a plurality of photodiodes that receive light beams from different pupil regions of an imaging optical system; An imaging device comprising: a posture estimation device according to any one of claims 1 to 6.

8. a depth detection step for detecting a depth map in the depth direction from the image; a joint detection step for detecting joints or organs from the image; and a joint connection step of connecting the detected joints or organs, In the joint connection step, when there are representative points of multiple connection destination joints or organs for a representative point of a connection source joint or organ, a path between the connection source and connection destination representative points is identified on the depth map for each connection destination, a variance evaluation value of the depth of each point on the path is calculated, and the connection destination with the smallest variance evaluation value is determined to be the joint or organ to be connected to of the same person as the connection source joint or organ, thereby repeating the process of connecting between multiple representative points as many times as the number of heads detected in the joint detection step.

9. A program that causes a computer to execute the posture estimation method according to claim 8.

Images