Information processing device, information processing method, and program

Abstract

To make it possible to shorten a processing time required for acquiring joint information from an image.SOLUTION: An information processing device includes: area detection means for detecting a subject area from a first image photographed by a first photographing unit; area estimation means for estimating a subject area for a second image photographed by a second photographing unit; joint point detection means for detecting joint information indicating a joint point of a subject appearing in the subject area detected by the area detection means, and joint information indicating a joint point of a subject appearing in the subject area estimated by the area estimation means; and distance detection means for detecting a distance from the first or second photographing unit to the subject on the basis of the joint information, and photographing parameters of the first and second photographing units. The area estimation means estimates a subject area for the second image of the current frame on the basis of the subject area and joint information acquired in the past frame for the current frame, the photographing parameters, and the subject area detected by the area detection means from the first image of the current frame.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing technique for acquiring joint information of a subject from an image. [Background technology]

[0002] Conventionally, methods have been proposed for acquiring joint information indicating the joint points of a subject from a camera image of the subject. An example of an application of joint information is a system (hereinafter referred to as a mixed reality system) that presents mixed reality using a head-mounted display (hereinafter abbreviated as HMD). In a mixed reality system, hand movement of the HMD wearer is detected using hand joint information estimated from an image of the hand captured by an HMD-mounted camera, and interaction can be realized by moving and rotating a computer graphics model (CG model) in accordance with the hand movement. Another application example is capturing an image of a person in front of the HMD wearer as the subject to acquire body joint information, and comparing distance information obtained from the person's joint information with distance information of the CG model to represent occlusion between the person and the CG model. Furthermore, Non-Patent Document 1 discloses a technology for estimating the two-dimensional positions of joint points on a subject's image and the relative depth values ​​of each joint point relative to a predetermined joint point from a monocular camera image. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] G. Moon, JY Chang, KM Lee, "Camera Distance-Aware Top-Down Approach for 3D Multi-Person Pose Estimation From a Single RGB Image", In ICCV, 2019. Summary of the Invention [Problem to be solved by the invention]

[0004] In order to present images generated by the above-mentioned mixed reality system to the HMD wearer without creating a sense of discomfort, it is desirable to acquire joint information of the subject (such as the hand of the HMD wearer or the body of a person in front of the user) with little delay. However, since it takes a certain amount of processing time to acquire joint information from a captured image, if the subject is moving, for example, a discrepancy may occur between the image displayed on the HMD based on the joint information acquired from the subject and the actual movement of the subject.

[0005] Therefore, an object of the present invention is to make it possible to reduce the processing time required to obtain joint information from an image. [Means for solving the problem]

[0006] An information processing device of the present invention comprises an area detection means that detects a subject area from a first image captured by a first imaging unit; an area estimation means that estimates a subject area for a second image captured by a second imaging unit; a joint point detection means that detects joint information indicating joint points of the subject that appear in the subject area detected by the area detection means and joint information indicating the joint points of the subject that appear in the subject area estimated by the area estimation means; and a distance detection means that detects a distance from the first imaging unit or the second imaging unit to the subject based on the joint information detected by the joint point detection means and imaging parameters of the first imaging unit and the second imaging unit, and is characterized in that the area estimation means estimates the subject area for the second image of the current frame based on the subject area and the joint information acquired in a frame previous to the current frame, the imaging parameters, and the subject area detected from the first image of the current frame by the area detection means. [Effects of the Invention]

[0007] According to the present invention, it is possible to reduce the processing time required to obtain joint information from an image. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a diagram illustrating an example of the configuration of an information processing device according to a first embodiment. [Figure 2] 10A and 10B are diagrams showing the detection result and estimation result of the subject region. [Figure 3] 10A and 10B are diagrams illustrating joint information detected in a joint point detection process. [Figure 4] FIG. 2 is a diagram illustrating a detailed configuration example of a region estimation unit according to the first embodiment. [Figure 5] FIG. 4 is an explanatory diagram of a position adjustment process according to the first embodiment. [Figure 6] 4 is a flowchart of information processing according to the first embodiment. [Figure 7] 10 is a flowchart of a region estimation process according to the first embodiment. [Figure 8] FIG. 10 is a diagram illustrating an example of the configuration of an information processing device according to a second embodiment. [Figure 9] FIG. 10 is a diagram illustrating a detailed configuration example of a region estimation unit according to the second embodiment. [Figure 10] 10 is a flowchart of a region estimation process according to the second embodiment. [Figure 11] FIG. 10 is a diagram illustrating an example of the configuration of an information processing device according to a third embodiment. [Figure 12] FIG. 13 is a diagram illustrating an example of the configuration of a region estimation unit according to the fourth embodiment. [Figure 13] FIG. 10 is an explanatory diagram of a color region detection process. [Figure 14] FIG. 2 illustrates an example of a hardware configuration. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of the present invention will be described with reference to the drawings. The following embodiments do not limit the present invention, and not all of the combinations of features described in the present embodiments are necessarily essential to the solution of the present invention. The configurations of the embodiments may be modified or changed as appropriate depending on the specifications of the device to which the present invention is applied and various conditions (such as usage conditions and usage environment). In the following embodiments, the same components will be described with the same reference symbols.

[0010] Before describing the configuration of the information processing device according to this embodiment, estimation of joint information and related processing will be described. The aforementioned Non-Patent Document 1 discloses a technique for estimating the depth value of each joint point based on a predetermined joint point. However, in a mixed reality system, for example, in a case where a computer graphics model (CG model) is moved based on hand joint information, the depth value Dx of the hand joint point Jx on the camera coordinate system is required. To calculate the depth value Dx of the hand joint point Jx on the camera coordinate system, it is necessary to calculate the depth value D0 of the reference joint point J0 on the camera coordinate system using equation (1), and further add the relative depth value Dd between the joint points J0 and Jx (already estimated in the results of Non-Patent Document 1).

[0011] Dx=Dd+D0 Formula (1)

[0012] Furthermore, there are a number of methods for determining the depth value D0 of the reference joint point J0, such as the following first depth value estimation method and second depth value estimation method. The first depth estimation method involves setting a distance between a reference joint point J0 and two of its adjacent joint points J1 and J2 in advance, and then solving a P3P problem using the resulting three-dimensional coordinate values ​​and the two-dimensional coordinates of each joint point detected on the image. By solving the P3P problem, the positions of the camera, reference joint point J0, and joint points J1 and J2 in three-dimensional space are calculated, allowing the depth value D0 between the camera and reference joint point J0 to be determined. This first depth estimation method allows the depth value D to be obtained from an image captured by a monocular camera.

[0013] The second depth estimation method uses a stereo camera to determine multiple joint points. The second depth estimation method uses the coordinate values ​​of each joint point J0 detected from the images of the left and right cameras that make up the stereo camera, and the imaging parameters of the stereo camera, to obtain a depth value D0 from the camera to the reference joint point J0 using a triangulation technique.

[0014] However, in the first depth value estimation method described above, when the posture of the subject changes and the area of ​​the triangle formed by the joint points J0, J1, and J2 on the camera image becomes relatively small, an error in the depth value D0 estimated in the P3P problem is likely to occur. If this error causes the estimated depth value J0 to vary widely between consecutive frames on the time axis (decreasing the stability of the depth estimation value), the HMD wearer may not be able to operate the CG model correctly.

[0015] In addition, with the second depth estimation method, depth can be measured if the reference joint point J0 of interest is detected in all stereo camera images. However, estimating joint information simultaneously from multiple stereo camera images requires a certain amount of processing time. For example, a camera updating images at a frame rate of 60 Hz must complete processing within 16.6 ms. This means that all stereo camera images must be processed sequentially. However, in a resource-limited computing environment, even if one image can be processed in 16.6 ms, the second image requires 33.3 ms for processing before the joint point information can be used. This processing delay can cause discomfort for the HMD wearer during interaction with a mixed reality system due to the delay in the displayed images, which can prevent correct operation of the CG model.

[0016] Therefore, the information processing device according to each of the following embodiments combines the first depth value estimation method and the second depth value estimation method described above, thereby improving the stability when estimating the depth value J0 in a stereo camera environment, and shortening the processing time to improve responsiveness.

[0017] First Embodiment First, as a first embodiment, an example using the joint information estimation algorithm disclosed in Non-Patent Document 1 will be described. In the first embodiment, an example of two-stage information processing will be given, including a first-stage process of detecting and cropping an area showing a subject from an input image captured by a camera, and a second-stage process of estimating joint information from the cropped area. Note that, since the image size that can be input to the second-stage joint information estimation process is fixed, the cropped area of ​​the subject needs to be resized in the first-stage cropping process.

[0018] Furthermore, the process of detecting the area in which the subject appears from the entire input image (hereinafter referred to as area detection process) in the first stage requires approximately 10 times the processing time of the process of estimating joint information in the second stage (hereinafter referred to as joint point detection process). In other words, when acquiring depth values ​​(distance values) using a stereo camera as in the second depth value estimation method, shortening the processing time required for area detection process is considered to be a very effective measure for improving responsiveness.

[0019] In the first embodiment, when detecting a subject area using two images from a stereo camera, normal area detection processing is performed on one image, and subject area estimation processing is performed on the other image using previously acquired subject area and joint information. That is, in the first embodiment, rather than performing processing that is heavy such as detecting a subject area from the entire input image, processing that is light such as estimating a subject area using previously acquired subject area and joint information is performed on the other image. This reduces the processing load on the two entire images from the stereo camera, enabling improved responsiveness by shortening processing time.

[0020] An information processing device 100 according to the first embodiment will be described below with reference to Fig. 1 to Fig. 7. Fig. 1 is a block diagram showing an example of the functional configuration of the information processing device 100 according to this embodiment. The imaging units 101R and 101L are left and right cameras that constitute a stereo camera, and their respective imaging ranges in real space partially overlap. In the example of FIG. 1, the imaging unit 101R is the right camera of the stereo camera, and the imaging unit 101L is the left camera. In this embodiment, the imaging units 101R and 101L are mounted on, for example, a head-mounted display (HMD) and capture an image of a subject. In this embodiment, an example of the subject is the raised hand of the HMD wearer.

[0021] The image capturing units 101R and 101L may be fixedly positioned in front of the eyes of the HMD wearer, or may be fixedly positioned in a location where they can capture the subject, the HMD wearer's hand. While the present embodiment uses a stereo camera with two cameras as an example, the present invention is not limited to this and may also use a multi-lens camera with three or more cameras. While the present embodiment uses the HMD wearer's hand as an example of the subject, the subject is not limited to the hand and may also be another person in front of the HMD wearer. In other words, when another person is the subject, the information processing device 100 of the present embodiment can also acquire joint information of the person's body. In this way, when the subject from which joint information is acquired is different, this can be handled by switching the dictionaries, etc. used in the area detection process and joint point detection process (described later) depending on the subject.

[0022] The image acquisition unit 110R receives an image captured by the imaging unit 101R and the imaging parameters of the imaging unit 101R, associates them with a frame ID (identification number), and stores them in the storage unit 190. Similarly, the image acquisition unit 110L receives an image captured by the imaging unit 101L and the imaging parameters, associates them with a frame ID (identification number), and stores them in the storage unit 190. In this embodiment, the imaging parameters are camera parameters such as the angle of view, focal length, principal point position, coordinate transformation matrix from the imaging unit 101R to the imaging unit 101L calculated by prior stereo camera calibration, and imaging date and time. In the following embodiments, an image acquired by the image acquisition unit 110R from the imaging unit 101R is referred to as a first image, and an image acquired by the image acquisition unit 110L from the imaging unit 101L is referred to as a second image. The image acquisition units 110R and 110L store the first and second images for each frame in the storage unit 190 together with the imaging parameters, individually linking them to the frame IDs for each frame.

[0023] Area detection unit 120 identifies an area in which a subject appears in the first image acquired by image acquisition unit 110R, and detects a rectangular area surrounding the subject (in this embodiment, this rectangular area is referred to as the subject area). Note that, although this embodiment provides an example of a configuration in which area detection unit 120 detects the subject area from the first image, it may also be configured to detect the subject area from the second image. In that case, however, the configuration subsequent to image acquisition unit 110R and image acquisition unit 110L in FIG. 1 will be swapped from the example in FIG. 1.

[0024] In this embodiment, the region detection unit 120 detects a subject region from an image by, for example, estimation processing using DetectNet as described in Non-Patent Document 1. Of course, this is just one example, and the region detection unit 120 may detect a subject region using a method other than the estimation processing using DetectNet as described in Non-Patent Document 1. For example, a method may be used in which color information of the subject captured by the imaging unit is registered and the subject region is detected based on a color region extracted from the captured image by referring to the registered color information. The region detection unit 120 associates the detected subject region with the frame ID of the image in which the subject region was detected and stores it in the storage unit 190. In other words, the storage unit 190 stores the subject region detected by the region detection unit 120 from the first image for each frame, associated with the frame ID.

[0025] Area estimation unit 140 performs processing to estimate a subject area for the second image of the current frame (present frame) acquired by image acquisition unit 110L. Area detection unit 120 described above detects a subject area from the first image. Area estimation unit 140, on the other hand, does not detect a subject area from the second image, but estimates a subject area based on previously acquired subject area and joint information, imaging parameters, and the subject area detected from the first image of the current frame by area detection unit 120. That is, area estimation unit 140 estimates a subject area in a shorter time and with less load than the area detection processing, based on the imaging parameters, the subject area and joint information acquired in a frame previous to the current frame, and the subject area detected from the first image of the current frame.

[0026] In this embodiment, the past frame with respect to the current frame is the frame immediately preceding the current frame (previous frame), but is not necessarily limited to the previous frame and may be a frame several frames prior. The region estimation unit 140 acquires the subject region and joint information of the previous frame by reading the subject region and joint information stored in the storage unit 190 in association with the frame ID of the previous frame. The subject region estimated by the region estimation unit 140 for the second image is acquired as a rectangular region, similar to the subject region detected from the first image by the region detection unit 120. The region estimation unit 140 then associates the estimated subject region with the frame ID of the second image from which the subject region was estimated, and stores the associated subject region in the storage unit 190. That is, the subject region estimated by the region estimation unit 140 for each frame of the second image is stored in the storage unit 190 in association with the frame ID. Details of the region estimation process performed by the region estimation unit 140 will be described later.

[0027] Image 200R in FIG. 2(b) shows an example of a first image, and image 200L in FIG. 2(a) shows an example of a second image. Assume that subject 105, which is the hand of the HMD wearer, is captured in first image 200R and second image 200L. Furthermore, subject region 210 in first image 200R indicates a rectangular subject region in which subject 105 is detected from image 200R by region detection unit 120. Furthermore, subject region 220 in second image 200L in FIG. 2(a) indicates a rectangular subject region estimated by region estimation unit 140.

[0028] Joint point detection unit 150R detects multiple joint points (i.e., hand joint points) of subject 105 from subject region 210 detected by region detection unit 120, and acquires joint information indicating the detected joint points. Then, joint point detection unit 150R associates the joint information with a frame ID and stores it in storage unit 190. That is, in storage unit 190, the joint information detected by joint point detection unit 150R from the first image for each frame is stored in association with the frame ID.

[0029] The joint point detection unit 150L detects a plurality of joint points of the subject 105 from the subject region estimated by the region estimation unit 140, and acquires joint information indicating the detected joint points. The joint point detection unit 150L then associates the joint information with a frame ID and stores it in the storage unit 190. That is, the joint information detected by the joint point detection unit 150L from the second image for each frame is stored in the storage unit 190, associated with the frame ID.

[0030] In this embodiment, joint point detection units 150R and 150L detect joint points of the subject using, for example, PoseNet, which is described in Non-Patent Document 1. Note that, in this embodiment, a configuration has been given in which joint point detection unit 150R detects joint points from subject region 210 of first image 200R, and joint point detection unit 150L detects joint points from subject region 220 of second image 200L, but the present invention is not limited to this. For example, a single joint point detection unit may be used to sequentially detect joint points from the subject regions of the first image and the second image.

[0031] Fig. 3(b) is a diagram showing a first image 300R in which a plurality of joint points 320 detected by joint point detection section 150R from the subject region 210 of first image 200R in Fig. 2(b) are displayed on subject 105. Similarly, Fig. 3(a) is a diagram showing a second image 300R in which a plurality of joint points 320 detected by joint point detection section 150L from the subject region 220 of second image 200L in Fig. 2(a) are displayed on subject 105.

[0032] 3(a) and 3(b) also show a CG model 310 that will be later synthesized to clearly illustrate the depth relationship between the subject 105 and the CG model generated by the synthesis unit 170 (described later). Details of the generation and synthesis of the CG model by the synthesis unit 170 will be described later. In this way, to synthesize the subject 105 and the CG model 310 so that the front-to-back relationship in the depth direction is correct, the joint point detection unit 150R acquires, for each joint point 320, image coordinates in the first image 300R and a depth value Dd based on the reference joint point J0. Similarly, the joint point detection unit 150R acquires, for each joint point 320, image coordinates in the second image 300L and a depth value Dd based on the reference joint point J0. The depth value Dd for each joint point 320 is a depth value relative to the depth value D0 of the reference joint point J0 in the camera coordinate system. In this embodiment, since the subject 105 is a hand, for example, a wrist joint point can be used as the reference joint point J0. Then, the joint point detection units 150R and 150L include the acquired position of each joint point and the depth value corresponding to that joint point together in joint information, and store the information in the storage unit 190 in association with the frame ID of the current frame.

[0033] Furthermore, the joint point detection units 150R and 150L check whether the distances between the multiple joints output by PoseNet are within a predetermined threshold, and if they are within the threshold, set an observation flag to 1. Conversely, if they exceed the threshold, set the observation flag to 0. The observation flag is a flag for setting whether to execute or not execute processing using joint points or joint information in the information processing device 100. Details of whether to execute or not execute processing depending on whether the observation flag is 1 or 0 will be described later. In this embodiment, by setting the observation flag, it is possible to determine, for example, when a part of the subject is cut off beyond the edge of the image and joint information cannot be correctly obtained, and it is possible to avoid executing processing using inappropriate joint information.

[0034] If the observation flag is 1, the distance detection unit 160 estimates a depth value D0 of a reference joint point J0 (e.g., a wrist joint point) from the imaging unit using the joint information detected by the joint point detection units 150L and 150R and the imaging parameters in the storage unit 190. As described above, the depth value D0 of the reference joint point J0 is the depth value of the reference joint point J0 in the camera coordinate system. Furthermore, the distance detection unit 160 performs a process of converting each joint point Jx of the hand in the camera coordinate system into a distance value Dx based on the camera coordinate system using the above-described equation (1). The distance detection unit 160 of this embodiment acquires the distance value Dx using the above-described second depth value estimation method. Then, the distance detection unit 160 stores the distance value Dx estimated for each joint point Jx in the joint information in the storage unit 190. On the other hand, if the observation flag is 0, the distance detection unit 160 does not execute the distance detection process using the joint information, and stores in the storage unit 190 information indicating that the depth value is indefinite.

[0035] The synthesis unit 170 moves and rotates the CG model 310 based on shape information of the CG model that has been prepared in advance and stored in the storage unit 190 and on joint information whose distances have been estimated by the distance detection unit 160, and determines the position and orientation of the CG model 310. The synthesis unit 170 further synthesizes a rendering image of the CG model 310 onto the first image and the second image, and outputs the synthesized image to the display unit 180 of the HMD. The display unit 180 is a display device mounted on the HMD, and presents a mixed reality to the HMD wearer by displaying an image synthesized at the frame rate (or an update interval close to the frame rate) of the image capturing units 101R and 101L. The displays of the display device mounted on the HMD are assumed to be left and right displays corresponding to both eyes, but may also be a single display. Although the storage unit 190 is shown in FIG. 1 as a storage device external to the information processing device 100, it may be a storage device built into the information processing device 100.

[0036] Fig. 4 is a diagram showing a detailed functional configuration of the region estimation unit 140. As shown in Fig. 4, the region estimation unit 140 has a determination unit 410, a position adjustment unit 420, and a size adjustment unit 430. Note that in Fig. 4, in order to make the input / output relationship with the region estimation unit 140 easier to understand, the region detection unit 120, image acquisition unit 110L, joint point detection unit 150L, ​​and storage unit 190 of Fig. 1 are also shown.

[0037] As described above, area estimation unit 140 estimates the object area in the second image of the current frame using previously acquired object area and joint information, imaging parameters, and the object area detected from the first image of the current frame by area detection unit 120. In addition, the joint information of the object area in the previous frame is information stored in storage unit 190 in association with the frame ID of the previous frame.

[0038] The determination unit 410 determines whether or not joint information linked to the frame ID of the previous frame is stored in the storage unit 190. If joint information linked to the frame ID of the previous frame is not stored in the storage unit 190, the determination unit 410 sets the observation flag to 0, but if the joint information is stored, the determination unit 410 sets the observation flag to 1 and stores the set observation flag in the storage unit 190.

[0039] The position adjustment unit 420 determines the position of the subject area relative to the second image. If the observation flag set by the determination unit 410 is 1, that is, if the joint information associated with the frame ID of the previous frame is stored in the storage unit 190, the position adjustment unit 420 executes the following process.

[0040] Image 200L in Fig. 5(a) is a diagram used to explain the operation of the position adjustment process by position adjustment unit 420. Note that image 200R in Fig. 5(b) is the same image as Fig. 2(b) described above. For example, when the observation flag is 1, position adjustment section 420 calculates a difference vector between the center of gravity position of the object region detected from the first image of the previous frame and the center of gravity position of object region 210 detected from the first image of the current frame by region detection section 120. Furthermore, position adjustment section 420 moves the position of object region 510 estimated for the second image of the previous frame by region estimation section 140 in accordance with the difference vector, and sets object region 220 after the movement as the object region in second image 200L of the current frame.

[0041] The difference vector acquired by the position adjustment unit 420 is not limited to the example obtained from the position difference of the object region detected from the first image of the previous frame and the current frame. For example, the position adjustment unit 420 may obtain the difference vector from the position difference of the object region estimated for the second image of the previous frame and the current frame. Alternatively, for example, an acceleration sensor may be attached to the subject, and position adjustment section 420 may determine the position of subject region 220 based on the output of the acceleration sensor. For example, position adjustment section 420 may determine the movement speed of the subject in three-dimensional space from the output of the acceleration sensor and the differential time between the previous frame and the current frame, and project the movement position predicted from the movement speed onto the second image to determine the position of subject region 220. Furthermore, for example, when calibration is performed to parallelize the visual axes of the first and second images of imaging units 101R and 101L, position adjustment unit 420 may use the vertical position of the subject region of region detection unit 120 as the vertical position of the subject region of region estimation unit 140. In other words, if the images of imaging units 101R and 101L are rectified, position adjustment unit 420 may use the vertical coordinate values ​​of subject region 210 in first image 200R as the vertical coordinate values ​​of subject region 220 in second image 200L.

[0042] Furthermore, for example, when the observation flag is 0 (i.e., when the joint information of the previous frame has not been recorded), the position adjustment unit 420 may perform the following process. In this case, a distance value with a high probability of occurrence is determined in advance as the distance between the subject and the imaging units 101R, 101L, and this is stored in the storage unit 190 as the initial value of the distance. Then, position adjustment unit 420 estimates the position of the subject area in the second image based on subject area 210 detected from the first image of the current frame by area detection unit 120 and the initial values ​​of the imaging parameters and distance stored in storage unit 190. That is, position adjustment unit 420 first acquires the three-dimensional position of the subject area based on imaging unit 101R, based on the subject area detected from the first image of the current frame by area detection unit 120 and the distance detected in a past frame by distance detection unit 160. Then, position adjustment unit 420 sets the position obtained by projecting the three-dimensional position onto the second image as the position of the subject area relative to the second image.

[0043] More specifically, the position adjustment unit 420 performs the following first to fourth adjustment processes. As the first adjustment process, the position adjustment unit 420 converts the image coordinates of the four vertices of the subject area 210 detected from the first image of the current frame into three-dimensional points on the camera coordinate system of the image capture unit 101R. For example, the position adjustment unit 420 acquires from the storage unit 190 the distance from the image capture unit 101R to the subject, which is set as the initial value of the distance, and the imaging parameters of the image capture unit 101R (focal length, principal point position), and outputs the three-dimensional position from the inverse matrix of the projection transformation matrix. Next, as a second adjustment process, the position adjustment unit 420 reads out a coordinate transformation matrix from the image capture unit 101R to the image capture unit 101L, which has been calculated in advance in stereo camera calibration and stored in the storage unit 190 as an imaging parameter. Next, as a third adjustment process, the position adjustment unit 420 converts the three-dimensional positions of the four vertices calculated in the camera coordinate system of the imaging unit 101R into positions in the camera coordinate system of the imaging unit 101L using the coordinate transformation matrix read out in the second adjustment process. Then, as the fourth adjustment process, the position adjustment unit 420 determines the position of the subject area relative to the second image by projecting the four vertices converted into the camera coordinate system of the imaging unit 101L onto the image coordinates using the imaging parameters of the imaging unit 101L.

[0044] The size adjustment unit 430 determines the size of the four vertices that represent the subject area estimated for the second image. In this embodiment, the size adjustment unit 430 adjusts the size of the subject area of ​​the second image using the size (pixel values ​​of vertical and horizontal width) of the subject area detected from the first image of the current frame by the area detection unit 120. For example, the size adjustment unit 430 determines the four vertices of the subject area of ​​the second image by applying the size of the subject area detected from the first image, centered on the center of gravity of the subject area whose position has been adjusted by the position adjustment unit 420. The size adjustment unit 430 then associates the subject area after the size adjustment with the frame ID and stores it in the storage unit 190.

[0045] FIG. 6 is a flowchart showing the flow of information processing executed by the information processing device 100 according to the first embodiment. First, in step S610, the image acquisition units 110R and 100L acquire images and imaging parameters from the corresponding first and second imaging units 101R and 101L, respectively. Then, the image acquisition units 110R and 100L associate the acquired first and second images and imaging parameters with a frame ID and store them in the storage unit 190. Next, in step S630, area detection section 120 detects the subject area from the first image as described above, and stores the detected subject area in storage section 190 in association with the frame ID.

[0046] Next, in step S640, determination unit 410 of area estimation unit 140 determines whether joint information of the previous frame is stored in storage unit 190. Furthermore, in the next step SS650, area estimation unit 140 executes the area estimation process described above to estimate the subject area of ​​the second image of the current frame. Then, area estimation unit 140 associates the estimated subject area with the frame ID and stores it in storage unit 190. The area estimation process of step S650 will be described later using the flowchart of FIG.

[0047] Next, in step S660, joint point detection units 150R and 150L execute the joint point detection process described above to estimate the joint points of the subject from the subject region and acquire joint information. Then, joint point detection units 150R and 150L associate the joint information with a frame ID and store it in storage unit 190.

[0048] Next, in step S670, distance detection unit 160 calculates the distance from the imaging unit to each joint point of the subject in the camera coordinate system based on the joint information detected by joint point detection unit 150R and joint point detection unit 150L and the imaging parameters stored in memory unit 190. Next, in step S680, composition unit 170 updates the position and orientation of CG model 310, generates an image by combining the first image and second image of the current frame with the CG model, and outputs the image to display unit 180. As a result, the composite image is displayed on the HMD.

[0049] Then, in step S690, information processing device 100 determines whether any unprocessed subject areas remain in the first image and the second image. If any unprocessed subject areas remain, information processing device 100 performs the processes from step S640 to step S680 on the unprocessed subject areas. On the other hand, if no unprocessed subject areas remain, information processing device 100 proceeds to step S695.

[0050] In step S695, the information processing device 100 determines whether an instruction to end the processing has been input by the HMD wearer. If an instruction to end the processing has not been input, the processing of the information processing device 100 returns to step S610, and the above-described processing is performed on the first and second images newly input from the first imaging units 101R and 101L. On the other hand, if an instruction to end the processing has been input, the information processing device 100 ends the processing of the flowchart in FIG. 6.

[0051] FIG. 7 is a flowchart showing details of the region estimation process performed in step S650 by the functional units of region estimation section 140 shown in FIG. First, in step S700, the determining unit 410 of the region estimation unit 140 executes the above-described determination process and a process of setting an observation flag according to the determination result. Next, in step S710, position adjustment section 420 of area estimation section 140 executes the above-described position adjustment process to determine the position of the subject area in the second image. Next, in step S720, size adjustment section 430 of region estimation section 140 executes the size adjustment process described above to determine the size of the subject region in the second image.

[0052] As described above, according to the information processing device 100 of the first embodiment, a subject area is estimated by performing area estimation processing with low load on either the first image or the second image. Therefore, in a mixed reality system to which the information processing device 100 of this embodiment is applied, highly responsive and stable image display corresponding to the movement of the subject can be realized, and the HMD wearer can operate the CG model with more natural interaction operations.

[0053] <Second embodiment> In the first embodiment described above, an example was described in which a region detection process with a heavy processing load is performed on a first image for each frame acquired by the imaging unit 101R, and a region estimation process with a light processing load is performed on a second image acquired by the imaging unit 101L. In the second embodiment, an example is described in which the area detection process for the first image is not executed every frame, but rather a restriction is imposed, such as once per second, thereby reducing the frequency of execution of the area detection process for the first image, thereby further reducing the processing load and improving responsiveness compared to the example of the first embodiment.

[0054] An information processing device according to a second embodiment that improves responsiveness by reducing the frequency of execution of area detection processing on a first image will be described below. Fig. 8 is a block diagram showing an example of the functional configuration of an information processing device 800 according to the second embodiment. In Fig. 8, the same functional units as those in the information processing device 100 according to the first embodiment are denoted by the same reference numerals as those in Fig. 1, and descriptions thereof will be omitted as appropriate.

[0055] Information processing device 800 of the second embodiment includes a region estimation unit 810R in addition to the functional units generally similar to those in FIG. 1 described above, and also includes a region estimation unit 810L instead of region estimation unit 140 in FIG. 1. Region estimation unit 810R is provided between image acquisition unit 110R and joint point detection unit 150R and in parallel with region detection unit 120. Region estimation unit 810L is provided between image acquisition unit 110L and joint point detection unit 150L, ​​similar to region estimation unit 140 in FIG. 1, but unlike region estimation unit 140 in FIG. 1, the subject region detection result is not input from region detection unit 120.

[0056] In the case of the information processing device 800 of the second embodiment, the area detection unit 120 executes the area detection process only when at least one of the following first and second execution conditions is satisfied. In this embodiment, the first execution condition is that the observation flag stored in the storage unit 190 is used as information indicating whether or not there is joint information in the previous frame, and the observation flag is 0 and there is no joint information in the previous frame. The second execution condition is that the area detection process has not been executed by the area detection unit 120 within a predetermined time period (for example, one second) from the current frame in the past.

[0057] In other words, if neither the first nor the second execution condition is satisfied, the area detection process by the area detection unit 120 is skipped, and the area estimation process is performed by the area estimation unit 810R. That is, if the observation flag is 1, there is joint information of the previous frame, and the area detection process has been performed by the area detection unit 120 within the past second, the area estimation process is performed by the area estimation unit 810R.

[0058] Area estimation unit 810R performs area estimation processing on the first image when area detection processing on the first image by area detection unit 120 is skipped. Area estimation unit 810R estimates the subject area of ​​the first image of the current frame using the imaging parameters stored in storage unit 190 and the subject area and joint information obtained from the first image or second image of the previous frame. Area estimation unit 810L performs area estimation processing on the second image when area detection processing on the first image by area detection unit 120 is skipped. Area estimation unit 810L estimates the subject area of ​​the second image of the current frame using the imaging parameters stored in storage unit 190 and the subject area and joint information of the first image or second image of the previous frame.

[0059] Since region estimation unit 810R and region estimation unit 810L have similar functional configurations, the detailed functional configuration of region estimation unit 810R will be described as a representative with reference to Fig. 9. Note that in order to make it easier to understand the input / output relationship with region estimation unit 810R, image acquisition unit 110R, joint point detection unit 150R, and storage unit 190 of Fig. 8 are also shown in Fig. 9. Note that in the case of region estimation unit 810L, image acquisition unit 110R of Fig. 9 becomes image acquisition unit 110L, and joint point detection unit 150R becomes joint point detection unit 150L.

[0060] 9, the region estimation unit 810R has a determination unit 910, a speed calculation unit 920, a position adjustment unit 930, and a size adjustment unit 940. Note that the determination unit 910 has substantially the same functions as the corresponding determination unit 410 in FIG. 4, and therefore a description thereof will be omitted where appropriate.

[0061] The velocity calculation unit 920 is provided, for example, between the determination unit 410 and the position adjustment unit 930. The velocity calculation unit 920 reads out the joint information from two frames before and the joint information from one frame before from the storage unit 190, and calculates the moving velocity of each joint point of the subject based on the elapsed time between those frames.

[0062] The position adjustment unit 930 assumes that each joint point of the subject is moving at a uniform velocity at the movement velocity of the joint point calculated by the velocity calculation unit 920, and predicts the position of each joint point in the current frame after each joint point in the previous frame has moved at the uniform velocity. Then, the position adjustment unit 930 stores the predicted position of each joint point in the storage unit 190 in association with the frame ID of the current frame.

[0063] The size adjustment unit 940 calculates a rectangular shape that contains the positions of each joint point predicted for the current frame by the position adjustment unit 930, and determines a rectangular area that adds a margin of, for example, 10% to the size of the rectangular shape as the subject area.The size adjustment unit 940 then stores the subject area after the size adjustment in the storage unit 190, linking it to the frame ID of the current frame.

[0064] The flow of information processing performed by information processing device 100 according to the second embodiment is generally similar to the flowchart of Fig. 6 described above, and therefore will not be illustrated or described again. In the case of the second embodiment, the region estimation process of step S650 in the flowchart of Fig. 6 is executed when the region detection process for the first image by region detection unit 120 is skipped. In the case of the second embodiment, the region estimation process of step S650 is performed by region estimation units 810R and 810L.

[0065] Fig. 10 is a flowchart showing details of the region estimation process performed in step S650 of the second embodiment by the functional units of region estimation unit 810R and region estimation unit 810L shown in Fig. 9. Note that, in the flowchart of Fig. 10 as well, region estimation unit 810R and region estimation unit 810L perform similar processes, so the process performed by region estimation unit 810R will be described as a representative example.

[0066] In the information processing device 800 of the second embodiment, first, in step S1000, the determination unit 910 of the region estimation unit 810R executes the same determination processing and observation flag setting processing as described above. Next, in step S1010, the velocity calculation unit 920 calculates the movement velocity of the joint point by the velocity calculation process described above. Next, in step S1020, the position adjustment unit 930 determines the position of each joint point in the current frame by the above-described position adjustment process. Next, in step S1030, the size adjustment unit 940 performs the size adjustment process described above to add a margin to the size of the rectangular shape that contains the positions of the joint points, thereby determining the subject area.

[0067] As described above, according to the information processing device 800 of the second embodiment, the frequency of execution of the area detection process by the area detection unit 120 is reduced, and instead area estimation process is executed by the area estimation units 810R and 810L. As a result, according to the second embodiment, the processing load can be reduced compared to the first embodiment, and joint information can be acquired in a shorter processing time. In other words, in a mixed reality system to which the information processing device 800 of the second embodiment is applied, the processing load can be reduced and image display that ensures responsiveness and stability to the movement of the subject can be realized, and the HMD wearer can operate the CG model with natural interaction operations.

[0068] <Third embodiment> In the first embodiment, joint point detection units 150R and 150L each detect joint points from the subject within the subject area, and determine whether or not each joint point is a correct joint point based on the distance between those joint points, thereby preventing the use of joint information of an incorrect joint point. That is, in the first embodiment, joint point detection units 150R and 150L always execute joint point detection processing, which imposes a heavy processing load, regardless of the state of the subject within the subject area.

[0069] In contrast, in the third embodiment, an example will be described in which it is determined whether a subject within a subject area satisfies a predetermined determination condition before executing joint point detection processing, and unnecessary processing is not executed based on the determination result, thereby further reducing the processing load. In the third embodiment, an example will be given in which the predetermined determination condition for a subject within a subject area is whether or not there is a subject in a state where no cut-off occurs, i.e., where part of the subject does not extend beyond the edge of the image. In the third embodiment, the processing load is reduced by executing joint point detection processing only when there is a subject with no cut-off. Furthermore, in the third embodiment, the joint point detection processing is executed using only subjects with no cut-off, thereby preventing erroneous joint information from being output.

[0070] Fig. 11 is a block diagram showing an example of the functional configuration of an information processing device 1100 according to the third embodiment. In Fig. 11, the same functional units as those in the information processing device 100 according to the first embodiment are denoted by the same reference numerals as in Fig. 1, and descriptions thereof will be omitted as appropriate. The information processing apparatus 1100 of the third embodiment includes an area determination unit 1110 in addition to the functional units generally similar to those in FIG.

[0071] Region determination unit 1110 determines whether a predetermined determination condition is met based on the region detection result by region detection unit 120 and the region estimation result by region estimation unit 140. In this embodiment, the predetermined determination condition is whether there is a subject in a state where no part of the subject extends beyond the edge of the image and no cut-off occurs. Region determination unit 1110 acquires from storage unit 190 the subject region detected by region detection unit 120 and the subject region estimated by region estimation unit 140, and determines whether there is a subject in these subject regions that is not cut-off. Region determination unit 1110 then sets an observation flag to 0 if all of the subjects in the subject region detected by region detection unit 120 and the subject region estimated by region estimation unit 140 are in a cut-off state, and sets the observation flag to 1 otherwise.

[0072] In the information processing device 1100 of the third embodiment, if the observation flag is 0, the information processing device 1100 does not execute the processing of the joint point detection units 150R and 150L and the distance detection unit 160. On the other hand, if the observation flag is 1, that is, if there is a subject that is not cut off in either the subject region detected by the region detection unit 120 or the subject region detected by the region estimation unit 140, the information processing device 1100 performs the joint point detection processing on the subject region of the subject that is not cut off. For example, if there is no cut off in the subject in the subject region detected by the region detection unit 120 but there is cut off in the subject in the subject region estimated by the region estimation unit 140, the information processing device 1100 executes the joint point detection processing by the joint point detection unit 150R. Furthermore, for example, if there is no cut off in the subject in the subject region estimated by the region estimation unit 140 but there is cut off in the subject in the subject region detected by the region detection unit 120, the information processing device 1100 executes the joint point detection processing by the joint point detection unit 150L.

[0073] As a result, distance detection unit 160 performs distance calculation processing using the joint information detected by joint point detection unit 150R or joint point detection unit 150L. That is, in the case of the third embodiment, distance detection unit 160 calculates distance value D based on the joint information detected from the image of either image capture unit 101R or image capture unit 101L, in other words, based on the joint information detected from the image captured by monocular imaging.

[0074] <Fourth embodiment> In the second embodiment described above, the speed calculation unit 920 in Fig. 9 calculates the movement speed of each joint point based on the position information of the joint point in the previous two frames and the previous one frame and the time between frames, and predicts the position of the object area in the current frame assuming that the movement speed is constant. However, predictions based on the assumption of constant speed can result in errors (prediction errors) with respect to the actual position of the object area, and an object area with a position or size that does not actually exist can be predicted.

[0075] In contrast, in the fourth embodiment, an example will be described in which the position and size are adjusted based on color information within the subject area. Note that the functional configuration of the information processing device of the fourth embodiment is the same as that of FIG. 8 described above, and therefore illustration and description thereof will be omitted. In the fourth embodiment, the processing performed in the area estimation unit is different from that of the second embodiment.

[0076] FIG. 12 is a diagram showing detailed functional configurations of region estimation units 810R and 810L of FIG. 8 in an information processing device of the fourth embodiment. In the fourth embodiment, region estimation unit 810R and region estimation unit 810L each have the same functional configuration, and therefore FIG. 12 shows the detailed functional configuration of region estimation unit 810R as a representative. In FIG. 12, the same functional units as those in FIG. 9 are assigned the same reference numerals. Also, as in the example of FIG. 9, FIG. 12 also shows image acquisition unit 110R, joint point detection unit 150R, and storage unit 190 of FIG. 8 to make the input / output relationship with region estimation unit 810R easier to understand. In the case of region estimation unit 810L, image acquisition unit 110R of FIG. 12 becomes image acquisition unit 110L, and joint point detection unit 150R becomes joint point detection unit 150L.

[0077] 12, the region estimation unit 810R of the fourth embodiment has a determination unit 910, a speed calculation unit 920, a position adjustment unit 1220, a size adjustment unit 1230, and further has a color region detection unit 1240. The determination unit 910 and the speed calculation unit 920 have the same functional units as those in the second embodiment, and therefore their description will be omitted.

[0078] In the fourth embodiment, storage unit 190 stores color information of an expected subject as registered color information. Color region detection unit 1240 acquires the registered color information from storage unit 190 and detects a color region corresponding to the registered color information from an image acquired by image acquisition unit 110R (first image 200R in this example). Color region detection unit 1240 then determines a rectangle (referred to as a color region rectangle) that contains the detected color region.

[0079] As in the second embodiment, the position adjustment unit 1220 and the size adjustment unit 1230 predict the position and size of the subject area based on the moving speed calculated by the speed calculation unit 920. Furthermore, the position adjustment unit 1220 and the size adjustment unit 1230 of the fourth embodiment adjust the position and size of the subject area predicted based on the moving speed based on the color area rectangle determined by the color area detection unit 1240.

[0080] For example, if the difference between the predicted position of the subject area and the position of the color area rectangle determined by the color area detection unit 1240 is within a predetermined position difference threshold, the position adjustment unit 1220 stores the position of the color area rectangle in the memory unit 190 as the position of the adjusted subject area. In addition, if the difference between the size of the predicted subject area and the size of the color area rectangle determined by the color area detection unit 1240 is within a predetermined size difference threshold, the size adjustment unit 1230 stores the size of the color area rectangle in the memory unit 190 as the size of the adjusted subject area.

[0081] Fig. 13(a) is a diagram showing a color region 1310 detected from the subject 105 in the first image 200R based on registered color information, and a subject region 210 predicted based on the moving speed. Fig. 13(a) is a diagram showing a color region rectangle 1320 determined by the color region detection unit 1240, and the subject 105 and subject region 210 in the first image 200R. Fig. 13(c) is a diagram showing an example after the position and size of the subject region have been adjusted to the position and size of the color region rectangle 1320 because the differences in position and size between the predicted subject region 210 and the color region rectangle 1320 shown in Fig. 13(b) are within thresholds.

[0082] In this embodiment, an example has been given in which, if the difference between the position and size of the subject area predicted based on the moving speed and the position and size of the color area rectangle is within a threshold, the position and size of the color area rectangle are used as the position and size of the subject area as they are. However, this is not limiting. For example, the adjustment amount of the position and size of the subject area may be weighted according to the amount of difference between the position and size of the subject area predicted based on the moving speed and the position and size of the color area rectangle. For example, if at least one of the differences between the position and size of the subject area predicted based on the movement speed and the position and size of the color area rectangle exceeds a threshold, the position and size predicted based on the movement speed may be used as the position and size of the subject area. Alternatively, if at least one of the differences between the position and size of the subject area predicted based on the moving speed and the position and size of the color area rectangle exceeds a threshold, the observation flag may be set to 0 so that the subsequent joint point detection process is not performed.

[0083] <Fifth embodiment> The area estimation units 810R and 810L of the second embodiment described above estimated the position of the subject area in the current frame based on the position information of the joint points between frames and the movement speed calculated from the time between frames, and detected the joint points of the subject. In contrast, in the fifth embodiment, an example will be described in which, for example, when the movement speed of a joint point is lower than a predetermined speed threshold and the movement amount of the joint point is not large, joint information detected from the subject region of the previous frame is used as is as joint information for the subject region of the current frame. Note that the functional configuration of the information processing device of the fifth embodiment is similar to that of Figures 8 and 9 described above, and therefore illustration and description thereof will be omitted. Also, in the fifth embodiment, as in the example of the second embodiment, the region estimation unit 810R and the region estimation unit 810L each have the same functional configuration, and therefore the example configuration of the region estimation unit 810R will be used as a representative example for the description.

[0084] In the fifth embodiment, the velocity calculation unit 920 of the region estimation unit 810R shown in FIG. 9 calculates the movement velocity of the joint point in the same manner as described above. In the fifth embodiment, if the calculated movement velocity is less than a predetermined velocity threshold, the velocity calculation unit 920 sets the observation flag corresponding to the frame ID of the current frame to 1 and stores it in the storage unit 190. Then, if the observation flag stored in the storage unit 190 is 1, the joint point detection unit 150R of the fifth embodiment may output the joint information of the previous frame directly to a subsequent component. Furthermore, if the observation flag stored in the storage unit 190 is 1, the distance detection unit 160 of the fifth embodiment may output information on the distance of the subject detected in the previous frame directly to a subsequent component.

[0085] <Hardware configuration example> FIG. 14 is a diagram showing an example of the hardware configuration of a personal computer or the like, which is an application example of the information processing apparatus of each of the above-described embodiments. In FIG. 14 , a CPU 1401 comprehensively controls each device connected via a bus 1410. The CPU 1401 reads and executes a control program stored in a ROM (read-only memory) 1402. It is assumed that an operating system (OS) and other information processing programs according to this embodiment are stored in a storage device 1406. These programs are read from the ROM 1402 or the storage device 1406, loaded into a RAM (random access memory) 1407, and appropriately executed by the CPU 1401. The information processing program according to this embodiment is read from the storage device 1406, loaded into the RAM 1407, and executed by the CPU 1401. This realizes the processing of each functional unit and each flowchart of each of the above-described embodiments. Note that, when configuring hardware as an alternative to software processing using the CPU 1401, it is sufficient to configure a calculation unit or circuit corresponding to each functional unit of each of the above-described embodiments. That is, each functional unit in the information processing device of each embodiment may be implemented entirely by hardware, or partly by software. When the program is executed by software, the software is executed by the CPU 1401 in FIG.

[0086] The imaging unit 101 is the imaging unit 101R and the imaging unit 101L mounted on the HMD described above. Similarly, the display unit 180 is a display device mounted on the HMD described above. The imaging unit 101 and the display unit 180 are connected to a bus 1410 via an I / F 1403. The monitor 1404 is a display device for the personal computer. The storage medium drive 1405 is a drive device provided in the personal computer for an optical disk or the like. The storage device 1406 is, for example, an HDD or SSD, and also includes the storage unit 190 described above. The mouse 1408 and keyboard 1409 are input devices used by the user to input instructions and the like into the personal computer.

[0087] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. The above-described embodiments are merely examples of specific implementations of the present invention, and should not be construed as limiting the technical scope of the present invention. In other words, the present invention can be implemented in various forms without departing from its technical concept or main features.

[0088] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. The above-described embodiments are merely examples of specific implementations of the present invention, and should not be construed as limiting the technical scope of the present invention. In other words, the present invention can be implemented in various forms without departing from its technical concept or main features.

[0089] The disclosure of this embodiment includes the following configuration, method, and program. (Configuration 1) an area detection means for detecting a subject area from a first image captured by the first imaging unit; an area estimation means for estimating a subject area for a second image captured by the second imaging unit; a joint point detection means for detecting joint information indicating joint points of the subject reflected in the subject area detected by the area detection means and joint information indicating joint points of the subject reflected in the subject area estimated by the area estimation means; a distance detection means for detecting a distance from the first imaging unit or the second imaging unit to the subject based on the joint information detected by the joint point detection means and imaging parameters of the first imaging unit and the second imaging unit; and the area estimation means estimates the subject area for the second image of the current frame based on the subject area and the joint information acquired in a frame past the current frame, the imaging parameters, and the subject area detected from the first image of the current frame by the area detection means. (Configuration 2) The information processing device described in configuration 1, characterized in that the area estimation means adjusts the position of the object area relative to the second image of the current frame based on the position of the object area detected by the area detection means for each frame of the first image. (Configuration 3) The information processing device described in configuration 2, wherein the area estimation means adjusts the position of the subject area based on a difference vector between the position of the subject area detected by the area detection means from the first image of the current frame and the position of the subject area detected from the first image of the past frame. (Configuration 4) The information processing device described in configuration 1, characterized in that the area estimation means adjusts the position of the subject area with respect to the second image of the current frame based on a difference in the position of the subject area estimated with respect to the second image in the frame two frames before and one frame before the current frame. (Configuration 5) The information processing device described in configuration 1, characterized in that the area estimation means acquires a movement speed of the subject in three-dimensional space based on the output of an acceleration sensor provided on the subject and a differential time between the current frame and the past frame, and determines a position where a movement position predicted from the movement speed is projected onto the second image as the position of the subject area relative to the second image of the current frame. (Configuration 6) The information processing device described in configuration 1, characterized in that when calibration is performed to parallelize the visual axes of the first image and the second image in the first imaging unit and the second imaging unit, the area estimation means uses the vertical position of the subject area detected by the area detection means as the vertical position of the subject area in the second image. (Configuration 7) The information processing device described in configuration 1, characterized in that the area estimation means acquires a three-dimensional position of the object area relative to the first imaging unit based on the object area detected by the area detection means and the distance detected in a past frame by the distance detection means, and sets the position of the object area relative to the second image as a position obtained by projecting the three-dimensional position onto the second image. (Configuration 8) The information processing device described in any one of configurations 1 to 7, characterized in that the area estimation means adjusts the size of the subject area for the second image of the current frame based on the size of the subject area detected from the first image of the current frame by the area detection means. (Configuration 9) an area determination means for determining whether the subject area detected by the area detection means and the subject area estimated by the area estimation means satisfy a predetermined determination condition; 9. The information processing device according to any one of configurations 1 to 8, wherein, when only one of the subject region detected by the region detection means and the subject region estimated by the region estimation means satisfies the predetermined determination condition, the distance detection means detects the distance using the joint information detected by the joint point detection means from the subject in the subject region that satisfies the predetermined determination condition. (Configuration 10) 10. The information processing device according to configuration 9, wherein the predetermined determination condition is that the subject area includes a subject that does not extend beyond an edge of the image. (Configuration 11) a second region estimation means for estimating the subject region for the first image; The information processing device described in any one of configurations 1 to 10, characterized in that the second area estimation means executes a process of estimating the subject area for the first image when the area detection means does not execute a process of detecting the subject area in accordance with a predetermined execution condition. (Configuration 12) the predetermined execution condition includes a first execution condition that the joint information has not been acquired in the past frame, and a second execution condition that a process of detecting the subject area from the first image has not been executed within a predetermined time period in the past from the current frame, The information processing device described in configuration 11, characterized in that the area detection means executes a process of detecting the subject area from the first image only when at least one of the first execution condition and the second execution condition is satisfied. (Configuration 13) The information processing device described in configuration 12, characterized in that the second area estimation means executes a process of estimating the subject area from the first image when neither the first execution condition nor the second execution condition is satisfied. (Configuration 14) The information processing device according to any one of configurations 11 to 13, wherein the second area estimation means acquires a movement speed of the joint point based on the joint information acquired for each frame by the joint point detection means, and estimates the subject area for the first image of the current frame in accordance with the movement speed. (Configuration 15) The information processing device described in configuration 14, characterized in that the second area estimation means adds a predetermined margin to the size of the estimated subject area depending on the moving speed, and adjusts the size of the subject area relative to the first image of the current frame. (Configuration 16) 14. The information processing device according to any one of configurations 11 to 13, wherein the second area estimation means estimates the subject area for the first image based on a registered color that has been registered in advance. (Configuration 17) The information processing device described in configuration 16, characterized in that the second area estimation means, when the size of the subject area estimated based on the registered color is within a predetermined threshold, sets the subject area estimated based on the registered color as the subject area for the first image. (Configuration 18) The information processing device described in configuration 17, wherein the second area estimation means weights the adjustment amount of the position and size of the subject area relative to the first image depending on the difference between the size of the subject area estimated based on the registered color and a predetermined threshold. (Configuration 19) the second area estimation means acquires a moving speed of the joint point based on the joint information acquired for each frame by the joint point detection means; 19. The information processing device according to any one of configurations 11 to 18, wherein the joint point detection means uses the joint information detected from the subject area of ​​the past frame as joint information of the subject area of ​​the current frame when the moving speed of the joint point is smaller than a predetermined speed threshold. (Configuration 20) the second area estimation means acquires a moving speed of the joint point based on the joint information acquired for each frame by the joint point detection means; 20. The information processing device according to any one of configurations 11 to 19, wherein the distance detection means uses the distance of the subject detected in the previous frame as the distance of the subject in the current frame when the moving speed of the joint point is smaller than a predetermined speed threshold. (Configuration 21) 21. The information processing device according to any one of configurations 1 to 20, further comprising a synthesis means for synthesizing a computer graphics model generated based on shape information of the computer graphics model and the distance to the subject onto the first image or the second image. (Method 1) a region detection step of detecting a subject region from a first image captured by the first imaging unit; a region estimation step of estimating a subject region for a second image captured by the second imaging unit; a joint point detection step of detecting joint information indicating joint points of the subject reflected in the subject area detected in the area detection step and joint information indicating joint points of the subject reflected in the subject area estimated in the area estimation step; a distance detection step of detecting a distance from the first imaging unit or the second imaging unit to the subject based on the joint information detected in the joint point detection step and imaging parameters of the first imaging unit and the second imaging unit; and an information processing method, characterized in that, in the area estimation step, the subject area for the second image of the current frame is estimated based on the subject area and the joint information acquired in a frame past the current frame, the imaging parameters, and the subject area detected from the first image of the current frame in the area detection step. (Program 1) A program for causing a computer to function as the information processing device according to any one of configurations 1 to 21. [Explanation of symbols]

[0090] 100: Information processing device, 120: Area detection unit, 140: Area estimation unit, 150R, 150L: Joint point detection units, 160: Distance detection unit, 170: Synthesis unit, 190: Storage unit

Claims

1. an area detection means for detecting a subject area from a first image captured by the first imaging unit; an area estimation means for estimating a subject area for a second image captured by the second imaging unit; a joint point detection means for detecting joint information indicating joint points of the subject reflected in the subject area detected by the area detection means and joint information indicating joint points of the subject reflected in the subject area estimated by the area estimation means; a distance detection means for detecting a distance from the first imaging unit or the second imaging unit to the subject based on the joint information detected by the joint point detection means and imaging parameters of the first imaging unit and the second imaging unit; and the area estimation means estimates the subject area for the second image of the current frame based on the subject area and the joint information acquired in a frame past the current frame, the imaging parameters, and the subject area detected from the first image of the current frame by the area detection means.

2. 2. The information processing device according to claim 1, wherein the area estimation means adjusts the position of the subject area relative to the second image of the current frame based on the position of the subject area detected by the area detection means for each frame of the first image.

3. 3. The information processing device according to claim 2, wherein the area estimation means adjusts the position of the subject area based on a difference vector between the position of the subject area detected by the area detection means from the first image of the current frame and the position of the subject area detected from the first image of the past frame.

4. The information processing device according to claim 1, characterized in that the area estimation means adjusts the position of the subject area relative to the second image of the current frame based on a difference in the position of the subject area estimated for the second image in the frame two frames before and one frame before the current frame.

5. The information processing device described in claim 1, characterized in that the area estimation means acquires the movement speed of the subject in three-dimensional space based on the output of an acceleration sensor provided on the subject and the differential time between the current frame and the past frame, and determines the position of the movement position predicted from the movement speed, projected onto the second image, as the position of the subject area relative to the second image of the current frame.

6. The information processing device according to claim 1, characterized in that, when calibration is performed to parallelize the visual axes of the first image and the second image in the first imaging unit and the second imaging unit, the area estimation means uses the vertical position of the subject area detected by the area detection means as the vertical position of the subject area in the second image.

7. The information processing device described in claim 1, characterized in that the area estimation means acquires a three-dimensional position of the subject area relative to the first imaging unit based on the subject area detected by the area detection means and the distance detected in a past frame by the distance detection means, and sets the position of the subject area relative to the second image as the position of the subject area relative to the second image.

8. 2. The information processing device according to claim 1, wherein the area estimation means adjusts the size of the subject area for the second image of the current frame based on the size of the subject area detected from the first image of the current frame by the area detection means.

9. an area determination means for determining whether the subject area detected by the area detection means and the subject area estimated by the area estimation means satisfy a predetermined determination condition; 2. The information processing device according to claim 1, wherein, when only one of the subject region detected by the region detection means and the subject region estimated by the region estimation means satisfies the predetermined determination condition, the distance detection means detects the distance using the joint information detected by the joint point detection means from the subject in the subject region that satisfies the predetermined determination condition.

10. 10. The information processing apparatus according to claim 9, wherein the predetermined determination condition is that the subject area includes a subject that does not extend beyond an edge of the image.

11. a second region estimation means for estimating the subject region for the first image; 2. The information processing device according to claim 1, wherein the second area estimation means executes a process of estimating the subject area for the first image when the area detection means does not execute a process of detecting the subject area in accordance with a predetermined execution condition.

12. the predetermined execution condition includes a first execution condition that the joint information has not been acquired in the past frame, and a second execution condition that a process of detecting the subject area from the first image has not been executed within a predetermined time period in the past from the current frame, 12. The information processing apparatus according to claim 11, wherein the area detection means executes the process of detecting the subject area from the first image only when at least one of the first execution condition and the second execution condition is satisfied.

13. 13. The information processing apparatus according to claim 12, wherein the second area estimation means executes a process of estimating the subject area from the first image when neither the first execution condition nor the second execution condition is satisfied.

14. 12. The information processing apparatus according to claim 11, wherein the second area estimation means acquires a movement speed of the joint point based on the joint information acquired for each frame by the joint point detection means, and estimates the subject area for the first image of the current frame in accordance with the movement speed.

15. 15. The information processing apparatus according to claim 14, wherein the second area estimation means adjusts the size of the subject area relative to the first image of the current frame by adding a predetermined margin to the size of the estimated subject area in accordance with the moving speed.

16. 12. The information processing apparatus according to claim 11, wherein the second area estimation means estimates the subject area for the first image based on a registered color that is registered in advance.

17. 17. The information processing device according to claim 16, wherein the second area estimation means, when the size of the subject area estimated based on the registered color is within a predetermined threshold, sets the subject area estimated based on the registered color as the subject area for the first image.

18. 18. The information processing device according to claim 17, wherein the second area estimation means weights an adjustment amount of the position and size of the subject area relative to the first image in accordance with a difference between the size of the subject area estimated based on the registered color and a predetermined threshold value.

19. the second area estimation means acquires a moving speed of the joint point based on the joint information acquired for each frame by the joint point detection means; 12. The information processing device according to claim 11, wherein the joint point detection means uses the joint information detected from the subject area of ​​the past frame as the joint information of the subject area of ​​the current frame when the movement speed of the joint point is smaller than a predetermined speed threshold.

20. the second area estimation means acquires a moving speed of the joint point based on the joint information acquired for each frame by the joint point detection means; 12. The information processing device according to claim 11, wherein the distance detection means uses the distance of the subject detected in the previous frame as the distance of the subject in the current frame when the moving speed of the joint point is smaller than a predetermined speed threshold.

21. 21. The information processing apparatus according to claim 1, further comprising a synthesis unit that synthesizes a computer graphics model generated based on shape information of the computer graphics model and the distance to the subject onto the first image or the second image.

22. a region detection step of detecting a subject region from a first image captured by the first imaging unit; a region estimation step of estimating a subject region for a second image captured by the second imaging unit; a joint point detection step of detecting joint information indicating joint points of the subject reflected in the subject area detected in the area detection step and joint information indicating joint points of the subject reflected in the subject area estimated in the area estimation step; a distance detection step of detecting a distance from the first imaging unit or the second imaging unit to the subject based on the joint information detected in the joint point detection step and imaging parameters of the first imaging unit and the second imaging unit; and an information processing method, characterized in that, in the area estimation step, the subject area for the second image of the current frame is estimated based on the subject area and the joint information acquired in a frame past the current frame, the imaging parameters, and the subject area detected from the first image of the current frame in the area detection step.

23. Computer, an area detection means for detecting a subject area from a first image captured by the first imaging unit; an area estimation means for estimating a subject area for a second image captured by the second imaging unit; a joint point detection means for detecting joint information indicating joint points of the subject reflected in the subject area detected by the area detection means and joint information indicating joint points of the subject reflected in the subject area estimated by the area estimation means; a distance detection means for detecting a distance from the first imaging unit or the second imaging unit to the subject based on the joint information detected by the joint point detection means and imaging parameters of the first imaging unit and the second imaging unit; and the area estimation means is a program that functions as an information processing device that estimates the subject area for the second image of the current frame based on the subject area and the joint information acquired in a frame past the current frame, the imaging parameters, and the subject area detected from the first image of the current frame by the area detection means.

Images