Image processing apparatus, head-mounted type display device, image processing method and program

Abstract

To control a plurality of objects that exist in different depth distances so as to be displayed in respective actual depth distances.SOLUTION: An image processing apparatus comprises: image acquisition means that acquires a first image for a right eye and a second image for a left eye; distance acquisition means that acquires a depth distance from a user of an object which is detected in the first image and the second image; and display control means that controls a display position of a first object in display means on the basis of a first depth distance which is the depth distance of the first object detected in the first image and the second image, and controls a display position of a second object in the display means on the basis of a second depth distance which is the depth distance of the second object detected in the first image and the second image.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to an image processing device, a head-mounted display device, an image processing method, and a program. [Background technology]

[0002] With the recent spread of head mounted displays (HMDs), users can experience a space with a sense of depth within an image by viewing a left eye image and a right eye image with the left and right eyes, respectively. In this case, since the positions of the user's left and right eyes are different from the positions of the imaging devices that capture the left eye image and the right eye image, there may be a difference between the actual depth and the depth when the user views the image with the HMD. Patent Document 1 discloses a technology for correcting the difference in depth by shifting the left eye image and the right eye image as a whole. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] US Patent Application Publication No. 2021 / 0043170 Summary of the Invention [Problem to be solved by the invention]

[0004] When the left-eye image and the right-eye image are shifted overall, the sense of distance of an object at a certain depth distance can be corrected to match the actual depth distance, but it is difficult to correct multiple objects at different depth distances so that they are each displayed at their actual depth distance.

[0005] An object of the present invention is to provide an image processing device that can control a plurality of objects that exist at different depth distances so that they are displayed at their actual depth distances. [Means for solving the problem]

[0006] A first aspect of the present invention is an image processing device characterized by having an image acquisition means for acquiring a first image for the right eye and a second image for the left eye, a distance acquisition means for acquiring a depth distance from a user of an object detected in the first image and the second image, and a display control means for controlling a display position of the first object on a display means based on a first depth distance that is the depth distance of the first object detected in the first image and the second image, and for controlling the display position of the second object on the display means based on a second depth distance that is the depth distance of a second object detected in the first image and the second image.

[0007] A second aspect of the present invention is an image processing method comprising the steps of: acquiring a first image for the right eye and a second image for the left eye; acquiring a depth distance from a user of an object detected in the first image and the second image; controlling a display position of the first object on a display means based on a first depth distance that is the depth distance of a first object detected in the first image and the second image; and controlling a display position of the second object on the display means based on a second depth distance that is the depth distance of a second object detected in the first image and the second image.

[0008] A third aspect of the present invention is a program for causing a computer to function as each of the means of the image processing device described above. Effect of the Invention

[0009] According to the present invention, a plurality of objects existing at different depth distances can be controlled so as to be displayed at their respective actual depth distances. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is an external view of an HMD. [Diagram 2] FIG. 2 is a block diagram illustrating the configuration of an HMD. [Diagram 3] 11 is a flowchart illustrating a process of correcting a display position of an object. [Figure 4] FIG. 13 is a diagram illustrating a situation in which multiple objects with different depths are detected. [Diagram 5] 1 is a diagram illustrating an example of a left-eye image and a right-eye image. [Figure 6] 11A and 11B are diagrams illustrating calculation of a depth distance of an object. [Figure 7] 13A and 13B are diagrams illustrating a method for calculating a correction amount for the display position of the dog. [Figure 8] 11A and 11B are diagrams illustrating a method for calculating a correction amount for a display position of a person. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] <Embodiment> Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings. Figures 1(A) and 1(B) are external views of a head mounted display device (HMD) 100 as an image processing device according to the present invention. The HMD 100 includes a left image capture device 108a for the left eye, a right image capture device 108b for the right eye, a left eyepiece unit 102a for the left eye, and a right eyepiece unit 102b for the right eye.

[0012] 1A is a front perspective view of the HMD 100. The HMD 100 is connected to a headband 200. The headband 200 fixes the HMD 100 to the user's head. Hereinafter, a user wearing the HMD 100 will also be simply referred to as a user. The left imaging device 108a and the right imaging device 108b capture images of the outside in order to generate a display image for the left eye (left eye image) and a display image for the right eye (right eye image), respectively.

[0013] 1B is a perspective view of the back (rear) of the HMD 100. The left eyepiece 102a and the right eyepiece 102b are members with which the user aligns the left eye and the right eye, respectively. The user can view a left eye image and a right eye image through the left eyepiece 102a and the right eyepiece 102b.

[0014] 2 is a block diagram illustrating the configuration of the HMD 100. A left eye 101a and a right eye 101b represent the left and right eyeballs of a user. The HMD 100 includes a left eyepiece 102a for the left eye, a right eyepiece 102b for the right eye, a left eyepiece 103a for the left eye, a right eyepiece 103b for the right eye, a left light splitter 104a for the left eye, and a right light splitter 104b for the right eye. The HMD 100 also includes a left light receiving lens 105a for the left eye, a right light receiving lens 105b for the right eye, a left image sensor 106a for the left eye, and a right image sensor 106b for the right eye. The HMD 100 also includes a left display 107a for the left eye, a right display 107b for the right eye, a left image sensor 108a, a right image sensor 108b, and a bus 109. The HMD 100 also includes a central processing unit (CPU) 110, a read only memory (ROM) 111, a random access memory (RAM) 112, an object detection unit 113, and a distance acquisition unit 114. Each block included in the HMD 100 can transmit and receive data to and from each other via a bus 109.

[0015] 2 shows an example in which the image processing device according to the present invention is configured as a part of the HMD 100. The present invention is not limited to this, and the image processing device may be configured as a device separate from the HMD 100. In this case, the image processing device includes a CPU 110, a ROM 111, a RAM 112, an object detection unit 113, and a distance acquisition unit 114.

[0016] The left eyepiece 102a and the right eyepiece 102b are used by the user to view the left eye image and the right eye image. The left eyepiece 103a and the right eyepiece 103b are used to focus the image viewed by the user. The left light splitter 104a and the right light splitter 104b are light splitters for splitting image information from the left eye 101a and the right eye 101b into a line-of-sight detection system. The light split by the left light splitter 104a and the right light splitter 104b is sent to the left image sensor 106a and the right image sensor 106b through the left light receiving lens 105a and the right light receiving lens 105b. The image information from the left eye 101a and the right eye 101b formed by the left image sensor 106a and the right image sensor 106b, respectively, is sent to other blocks through the bus 109.

[0017] The left display 107a and the right display 107b display images captured by the left imaging device 108a and the right imaging device 108b, respectively, and superimpose and display images of virtual objects. The left imaging device 108a and the right imaging device 108b are disposed at a distance from each other, and capture two images with parallax (an image for the left eye and an image for the right eye).

[0018] The CPU 110 executes various image processing and various calculation processing using values ​​calculated in other blocks, thereby controlling the entire HMD 100. For example, the CPU 110 executes processing to detect the gaze position from the eye images captured by the left imaging element 106a and the right imaging element 106b, and processing to extract various information from the images captured by the left imaging device 108a and the right imaging device 108b.

[0019] The ROM 111 stores a program for implementing the processes executed by the CPU 110, and data used in the processes executed by the CPU 110. The RAM 112 stores images captured by the left imaging element 106a, the right imaging element 106b, the left imaging device 108a, and the right imaging device 108b as digital data. The RAM 112 is also used as a work memory for temporarily storing data used when the CPU 110 executes processes.

[0020] The object detection unit 113 acquires images captured by the left imaging device 108a and the right imaging device 108b, and detects objects present in the acquired images. The object detection unit 113 can acquire information such as the position, type, and size of the detected object.

[0021] The distance acquisition unit 114 acquires, from difference information between the two images captured by the left imaging device 108a and the right imaging device 108b, how far an object detected in the two images is physically away from the HMD 100. That is, the distance acquisition unit 114 can acquire the actual distance (hereinafter also referred to as depth distance) from the user wearing the HMD 100 to the object. The distance acquisition unit 114 is not limited to the object detected in the two images, but can also acquire how far each pixel in the entire image is physically away from the HMD 100.

[0022] Fig. 3 is a flowchart illustrating an example of a process for correcting the display position of an object. The flow of the process shown in Fig. 3 will be described below with reference to Figs.

[0023] In step S301, the CPU 110 detects an object from the left eye image and the right eye image captured by the left imaging device 108a and the right imaging device 108b, respectively, using the object detection unit 113. The left eye image and the right eye image may be recorded images. The object detection unit 113 (image acquisition means) can acquire the recorded left eye image and the right eye image stored in a storage unit such as the ROM 111, for example. The object detection unit 113 may also acquire the left eye image and the right eye image from an external device.

[0024] The CPU 110 acquires information on the position and type of each detected object. The position of the object is represented by coordinates on the two-dimensional image. The type of the object is a type indicating the characteristics of the object, such as a person or a dog, and can be determined in advance.

[0025] 4(A) and 4(B) are diagrams illustrating a situation in which multiple objects with different depths are detected. Fig. 4(A) shows a situation in which a dog 401 and a person 402 are present as subjects at different distances from the user in real space.

[0026] Fig. 4(B) is a schematic diagram of the situation in Fig. 4(A) viewed from above, including a user 403 wearing the HMD 100. The distance between the left eye 101a and the right eye 101b is D, and the distance between the imaging devices, the left imaging device 108a and the right imaging device 108b, is E. The distance from the user 403 to the dog 401 is F, and the distance from the user 403 to the person 402 is G.

[0027] CPU 110 executes the processes from step S302 to step S307 for each object detected in step S301. In the example of Fig. 4A, CPU 110 executes the processes from step S302 to step S307 for two objects, a dog 401 and a person 402.

[0028] In steps S302 and S304, CPU 110 determines whether the object detected in step S301 is a target for correcting the display position. CPU 110 corrects the display position of the object if the detected object satisfies a predetermined condition. The predetermined condition determined in step S302 is that the type of the object is a type that is predetermined as a correction target. The predetermined condition determined in step S304 is that the depth distance of the object is shorter than a predetermined threshold.

[0029] The predetermined condition is not limited to the condition regarding the type of object and the depth distance of the object. The predetermined condition may be, for example, a condition regarding the size of the object, the position of the object in the captured image, etc. Furthermore, the CPU 110 may change a threshold value for determining whether or not the predetermined condition is satisfied, for example, according to the number of objects detected in the image.

[0030] 3 shows an example of correcting the display position of an object when the object satisfies a predetermined condition for both the type and the depth distance, but is not limited to this. CPU 110 may correct the display position of an object when the object satisfies a predetermined condition for either the type or the depth distance. CPU 110 may correct both the display positions of the objects detected in the left and right captured images without determining the predetermined condition.

[0031] In step S302, the CPU 110 determines whether the type of the object is a type to be corrected. The type to be corrected is a type that is determined in advance as a type to be subjected to the display position correction process. In the example of FIG. 4(A), dogs and people are the types to be corrected.

[0032] If the type of the object is to be corrected, the process proceeds to step S303. If the type of the object is not to be corrected, the process proceeds to step S308. In step S302, the CPU 110 narrows down the objects to be corrected by determining the type of the object. By narrowing down the objects to be corrected to a predetermined type of object, the CPU 110 can reduce the processing load. The predetermined type can be, for example, a type of object that can achieve the effect of correction in actual use, or a type of object that the user focuses on. If the object is a dog 401, the type of dog 401, which is a dog, is to be corrected, so the process proceeds to step S303.

[0033] In step S303, the CPU 110 acquires the depth distance of the object by the distance acquisition unit 114. When the object is the dog 401 in Fig. 4(A), the distance acquisition unit 114 can calculate the actual depth distance from the difference in the positions of the dog 401 in the images captured by the left imaging device 108a and the right imaging device 108b.

[0034] 5(A) and 5(B) show examples of images captured by the left imaging device 108a and the right imaging device 108b, respectively. In the left eye image of FIG. 5(A), a dog 503 (dog 401) is located at a horizontal position 502. In the right eye image of FIG. 5(B), a dog 507 (corresponding to the dog 503 in the left eye image) is located at a horizontal position 506. The distance acquisition unit 114 can calculate the depth distance F from the HMD 100 to the dog 503 (dog 507) using the distance (position difference) between the position 502 and the position 506.

[0035] 6(A) and 6(B) are diagrams for explaining calculation of depth distance. In the example of Fig. 5(A) and 5(B), the depth distance F is calculated from the difference between the positions (e.g., the position 303a of the dog 503 and the position 507a of the dog 507) representing the objects (dog 503, dog 507) in the left-eye image and the right-eye image. In contrast, as shown in Fig. 6(A), an object that exists in a close distance, such as a smartphone 601 placed at hand, has a larger difference in depth distance compared to the dog 503 in Fig. 5(A) even if it is the same object.

[0036] 6B, distance A is the distance from the HMD 100 worn by the user 602 to the top end of the smartphone 601, and distance C is the distance from the HMD 100 to the bottom end of the smartphone 601. Distance B is the distance from the HMD 100 to the midpoint of the smartphone 601.

[0037] When there is a difference in the depth distance, the distance acquisition unit 114 calculates the distance from the user to the center of the object as the depth distance. The center of the object may be, for example, the midpoint between the top and bottom ends of the object, or the center of gravity of the object. The distance to the center of the object may be the average value of the depth distance of the top end of the object and the depth distance of the bottom end of the object.

[0038] In step S304 of Fig. 3, CPU 110 determines whether the depth distance of the object calculated in step S303 is shorter than a predetermined threshold. If the depth distance of the object is shorter than the predetermined threshold, the process proceeds to step S305. If the depth distance of the object is equal to or greater than the predetermined threshold, the process proceeds to step S308. In the example of Fig. 4(B), it is determined that the depth distance F of dog 401 is shorter than the predetermined threshold, and the process proceeds to step S305.

[0039] In step S304, the CPU 110 narrows down the objects to be processed by determining the depth distance of the object. By narrowing down the objects to be processed to those whose depth distance is shorter than a predetermined threshold, the CPU 110 can reduce the processing load. The reason for narrowing down the objects to those whose depth distance is shorter than the threshold is that the shorter the depth distance of the object, the more effective it is in suppressing the shift in the sense of depth caused by the correction of the display position. The threshold for determining the depth distance may be predetermined according to, for example, the range that the user intends to capture. The threshold may also be set or changed so that the number of objects to be corrected corresponds to the processing capacity of the HMD 100.

[0040] In step S305, the CPU 110 acquires a correction amount for the display position of the object. The CPU 110 can acquire the correction amount for the display position of the object based on the depth distance of the object acquired in step S303, the distance D between the left eye 101a and the right eye 101b, and the distance E between the imaging devices, the left imaging device 108a and the right imaging device 108b.

[0041] The distance D between the left eye 101a and the right eye 101b may be measured and stored in advance for each user, or may be obtained from eye images captured by the left imaging element 106a and the right imaging element 106b. The distance E between the left and right imaging devices is stored in advance as a design value.

[0042] The process of step S305 for dog 401 will be specifically described with reference to FIGS. 7(A) to 7(C). 7A is a diagram for explaining the angles at which a dog 401 is captured by each of the left eye 101a, the right eye 101b, the left imaging device 108a, and the right imaging device 108b.

[0043] The center line 700 is a straight line extending from the midpoint between the left and right eyes in the direction in which the dog 401 is located. The left imaging device 108a captures the dog 401 with a left imaging device line of sight 702a that forms an angle θ1 with the center line 700. The right imaging device 108b captures the dog 401 with a right imaging device line of sight 702b that forms an angle θ1 with the center line 700. The angle θ1 is expressed by the following formula 1 using the distance E between the left and right imaging devices and the depth distance F of the dog 401.

number

[0044] The left eye 101a captures the dog 401 with a left eye line of sight 701a that forms an angle θ2 with the center line 700. The right eye 101b captures the dog 401 with a right eye line of sight 701b that forms an angle θ2 with the center line 700. The angle θ2 is expressed by the following formula 2 using the distance D between the left eye 101a and the right eye 101b and the depth distance F of the dog 401.

number

[0045] As shown in FIG. 7B, the user 403 is actually viewing images captured by the left and right imaging devices 108a and 108b with his / her left and right eyes 101a and 101b, respectively. In the state of FIG. 7B, the left eye 101a captures the dog 401 on the left imaging device line of sight 702a, which forms an angle θ1 with the center line 700, and the right eye 101b captures the dog 401 on the right imaging device line of sight 702b, which forms an angle θ1 with the center line 700. In this case, the user 403 recognizes that the dog 401 is present at the position 703 of the intersection of the left imaging device line of sight 702a and the right imaging device line of sight 702b. The depth distance H from the user 403 to the position 703 is expressed by the following formula 3. The depth distance H is shorter than the actual depth distance F from the user 403 to the dog 401.

number

[0046] If the left eye 101a can capture the dog 401 along the left eye line of sight 701a and the right eye 101b can capture the dog 401 along the right eye line of sight 701b, the user 403 can capture the dog 401 with a sense of actual distance and recognize that the dog 401 is located at a position F from the user 403. At a depth distance H, it is sufficient that the left eye 101a can capture the dog 401 at a position 704 along the left eye line of sight 701a and the right eye 101b can capture the dog 401 at a position 705 along the right eye line of sight 701b. The CPU 110 can correct the display positions of the dog 401 in the left eye image and the right eye image so that the left eye 101a and the right eye 101b capture the dog 401 at positions 704 and 705.

[0047] 7C is a diagram for explaining the correction amount of the display position of dog 401 in the left eye image and the right eye image. The distance from user 403 to left display 107a and right display 107b is I. Below, a method for calculating the correction amount in the left eye image will be explained, but the correction amount in the right eye image can be calculated in the same way as in the case of the left eye image.

[0048] In a state in which the dog 401 is captured at position 703 on the left imaging device line of sight 702a in Fig. 7(B), the dog 401 is displayed at position 706 on the left display 107a in Fig. 7(C). By displaying the dog 401 at position 707 where the left eye line of sight 701a and the left display 107a intersect, the left eye 101a of the user 403 can capture the dog 401 at position 704 on the left eye line of sight 701a in Fig. 7(B). The distance J between position 706 on the left imaging device line of sight 702a and position 707 on the left eye line of sight 701a is the correction amount by which the dog 401 should be moved on the left display 107a. The correction amount, distance J, is expressed by the following equation 4.

number

[0049] Similarly, by moving the dog 401 on the right display 107b a distance J away from the center line 700, the right eye 101b of the user 403 can capture the dog 401 at position 705 on the right eye line of sight 701b in Figure 7(B).

[0050] 3, CPU 110 controls the display position of the object in the left-eye image and the right-eye image. CPU 110 performs display control of the object by correcting a predetermined display position of the object with a correction amount based on the depth distance acquired in step S305. Specifically, CPU 110 cuts out an area of ​​the object (dog 401) in the left-eye image and the right-eye image, and moves the cut-out area of ​​the object by the correction amount acquired in step S305.

[0051] CPU 110 cuts out the image area of ​​dog 503 located at position 502 on the left-eye image shown in Fig. 5(A), and moves it by the correction amount (distance J) acquired in step S305. Dog 511 shown in Fig. 5(C) is dog 503 from Fig. 5(A) that has been moved to position 510. In Fig. 5(A), of the area where dog 503 was displayed before correction, the area where dog 511 after the correction is not displayed (area that does not overlap with dog 511 after correction) becomes occlusion area 512 where no image exists.

[0052] Similarly, CPU 110 cuts out the image area of ​​dog 507 located at position 506 on the right-eye image shown in Fig. 5(B) and moves it by the correction amount (distance J) acquired in step S305. Dog 517 shown in Fig. 5(D) is dog 507 in Fig. 5(B) that has been moved to position 516. In Fig. 5(B), among the areas where dog 507 was displayed before correction, an area where dog 517 is not displayed after the correction (an area not overlapping with dog 517 after correction) becomes occlusion area 518 where no image exists.

[0053] In step S307, the CPU 110 estimates and generates a background for the occlusion region generated in step S306. Figures 5(E) and 5(F) show images in which backgrounds are estimated and generated for the occlusion regions 512 and 518 in Figures 5(C) and 5(D) and the generated images are fitted. The CPU 110 estimates and generates images for each occlusion region from the surrounding images of the occlusion region 512 in the left eye image and the occlusion region 518 in the right eye image. Figures 5(E) and 5(F) show a state in which backgrounds are fitted into the occlusion regions 512 and 518 in Figures 5(C) and 5(D) and the dogs 503 and 507 in Figures 5(A) and 5(B) are moved to the positions of the dogs 511 and 517.

[0054] In step S308, CPU 110 determines whether or not there is an unprocessed object for which the display position correction process has not been performed among the objects detected in step S301. If there is an unprocessed object, CPU 110 returns to step S302 and executes the processes of steps S302 to S307 for the unprocessed object. If there is no unprocessed object, the process shown in FIG. 3 ends.

[0055] In the example shown in FIG. 4(A), a dog 401 and a person 402 are detected as objects, so after processing the dog 401, the CPU 110 executes the processes of steps S302 to S307 on the person 402, which is an unprocessed object.

[0056] 5(A) to 5(F), 8(A) to 8, and Equations 5 to 8, the processing of steps S302 to S307 for the person 402 will be described. Since the person 402 and the dog 401 have different depth distances, the correction amounts in the left-eye image and the right-eye image differ between the person 402 and the dog 401. Detailed description of the processing that is the same as the processing for the dog 401 will be omitted.

[0057] In step S302, the CPU 110 determines that the type of the person 402 is a correction target, and proceeds to step S303. In step S303, the CPU 110 acquires the depth distance of the person 402.

[0058] A person 505 (person 402) in the left-eye image in FIG. 5A is located at a horizontal position 504. A person 509 (corresponding to the person 505 in the left-eye image) in the right-eye image in FIG. 5B is located at a horizontal position 508. The distance acquisition unit 114 can calculate the depth distance G from the HMD 100 to the person 505 (person 509) using the distance (positional difference) between the position 504 and the position 508. In the example of FIG. 4B,

[0059] In step S304, the CPU 110 determines whether or not the depth distance G of the person 402 is shorter than a predetermined threshold value. In the example of FIG. 4B, it is determined that the depth distance G of the person 402 is shorter than the predetermined threshold value, and the process proceeds to step S305.

[0060] In step S305, the CPU 110 acquires a correction amount for the display position of the person 402. The process of step S305 for the person 402 will be specifically described with reference to Figures 8(A) to 8(C). Figure 8(A) is a diagram for explaining the angles at which the person 402 is captured by each of the left eye 101a, the right eye 101b, the left imaging device 108a, and the right imaging device 108b.

[0061] The center line 800 is a straight line extending from the midpoint between the left and right eyes in the direction toward the person 402. The left imaging device 108a captures the person 402 with a left imaging device line of sight 802a that forms an angle θ3 with the center line 800. The right imaging device 108b captures the person 402 with a right imaging device line of sight 802b that forms an angle θ3 with the center line 800. The angle θ3 is expressed by the following formula 5 using the distance E between the left and right imaging devices and the depth distance G of the person 402.

number

[0062] The left eye 101a captures the person 402 with a left eye line of sight 801a that forms an angle θ4 with the center line 800. The right eye 101b captures the person 402 with a right eye line of sight 801b that forms an angle θ4 with the center line 800. The angle θ4 is expressed by the following equation 6 using the distance D between the left eye 101a and the right eye 101b and the depth distance G of the person 402.

number

[0063] As shown in FIG. 8B, the user 403 actually has his / her left eye 101a and right eye 101b. In this state, the left eye 101a and the right eye 101b are viewing images captured by the left and right imaging devices 108a and 108b, respectively. In the state of FIG. 8B, the left eye 101a captures the person 402 on the left imaging device line of sight 802a, which forms an angle θ3 with the center line 800, and the right eye 101b captures the person 402 on the right imaging device line of sight 802b, which forms an angle θ3 with the center line 800. In this case, the user 403 recognizes that the person 402 is present at a position 803, which is the intersection point between the left imaging device line of sight 802a and the right imaging device line of sight 802b. The depth distance G from the user 403 to the position 803 is expressed by the following formula 7. The depth distance K is a distance shorter than the actual depth distance G from the user 403 to the person 402.

number

[0064] If the left eye 101a can capture the person 402 along the left eye line of sight 801a and the right eye 101b can capture the person 402 along the right eye line of sight 801b, the user 403 can capture the person 402 with an actual sense of distance and recognize that the person 402 is present at a position G from the user 403. At a depth distance K, it is sufficient that the left eye 101a can capture the person 402 at a position 804 along the left eye line of sight 801a and the right eye 101b can capture the person 402 at a position 805 along the right eye line of sight 801b. The CPU 110 can correct the display position of the person 402 in the left eye image and the right eye image so that the left eye 101a and the right eye 101b capture the person 402 at the positions 804 and 805.

[0065] 8C is a diagram for explaining the correction amount of the display position of person 402 in the left eye image and the right eye image. The distance from user 403 to left display 107a and right display 107b is I. Below, a method for calculating the correction amount in the left eye image will be explained, but the correction amount in the right eye image can be calculated in the same way as for the left eye image.

[0066] In a state in which the person 402 is captured at position 803 on the left imaging device line of sight 802a in Fig. 8(B), the person 402 is displayed at position 806 on the left display 107a in Fig. 8(C). By displaying the person 402 at position 807 where the left eye line of sight 801a and the left display 107a intersect, the left eye 101a of the user 403 can capture the person 402 at position 804 on the left eye line of sight 801a in Fig. 8(B). The distance L between position 806 on the left imaging device line of sight 802a and position 807 on the left eye line of sight 801a is the correction amount by which the person 402 should be moved on the left display 107a. The correction amount, distance L, is expressed by the following equation 8.

number

[0067] Similarly, by moving person 402 on right display 107b a distance L away from center line 800, right eye 101b of user 403 can capture person 402 at position 805 on the right eye line of sight 801b in Figure 8(B).

[0068] In step S306 of FIG. 3, the CPU 110 cuts out the area of ​​the person 402 in the left-eye image and the right-eye image, and moves the area of ​​the person 402 cut out by the correction amount acquired in step S305.

[0069] CPU 110 cuts out the image area of ​​person 505 located at position 504 on the left-eye image shown in FIG. 5(A), and moves it by the correction amount (distance L) acquired in step S305. Person 514 shown in FIG. 5(C) is person 505 from FIG. 5(A) who has been moved to position 513. In FIG. 5(A), among the areas where person 505 was displayed before correction, the areas where person 514 is not displayed after correction (areas that do not overlap with person 514 after correction) are areas where no image exists. This results in an inclusion region 515.

[0070] Similarly, CPU 110 cuts out the image area of ​​person 509 located at position 508 on the right-eye image shown in Fig. 5(B) and moves it by the correction amount (distance L) acquired in step S305. Person 520 shown in Fig. 5(D) is person 509 in Fig. 5(B) who has been moved to position 519. In Fig. 5(B), within the area where person 509 was displayed before correction, the area where person 520 after the correction is not displayed (area not overlapping with person 520 after correction) becomes occlusion area 521 where no image exists.

[0071] In step S307, the CPU 110 estimates and generates a background for the occlusion region generated in step S306. Figures 5(E) and 5(F) show images in which backgrounds are estimated and generated for the occlusion regions 515 and 521 in Figures 5(C) and 5(D) and the generated images are fitted. The CPU 110 estimates and generates images for each occlusion region from the surrounding images of the occlusion region 515 in the left eye image and the occlusion region 521 in the right eye image. Figures 5(E) and 5(F) show a state in which backgrounds are fitted into the occlusion regions 515 and 521 in Figures 5(C) and 5(D) and the people 505 and 509 in Figures 5(A) and 5(B) are moved to the positions of the people 514 and 520.

[0072] The HMD 100 moves the display positions of the dog 401 and the person 402 by the respective correction amounts J and L. In this way, the HMD 100 can correct the display positions of multiple objects detected from the left and right captured images by the correction amounts based on the respective depth distances. The correction amounts of multiple objects detected from the left and right captured images differ depending on the depth distances of each object, and can be calculated by the same calculation method as equations 1 to 8 using the algorithm described in the flowchart of FIG. 3.

[0073] In the example of FIG. 4(A), there are two objects detected from the left and right captured images: a dog 401 and a person 402. When the processing up to step S307 for each of the objects is completed, it is determined in step S308 that there are no unprocessed objects, and the processing shown in FIG. 3 ends.

[0074] According to the above embodiment, the HMD 100 corrects the display positions of multiple objects detected in the left eye image and the right eye image with a correction amount based on the respective depth distances. By correcting the display positions in the left eye image and the right eye image, the HMD 100 can control the display so that multiple objects that exist at different depth distances are displayed at their actual depth distances. Therefore, a user wearing the HMD 100 can observe multiple objects at different depth distances with a sense of actual distance.

[0075] In the above embodiment, the HMD 100 is configured to include a left imaging device 108a and a right imaging device 108b as an imaging unit, and a left display 107a and a right display 107b as a display unit. Alternatively, the imaging unit may be provided in an external device, and the HMD 100 may acquire a left eye image and a right eye image transferred from the external device. The HMD 100 can execute a series of correction processes described in FIG. 3 on the image acquired from the external device, and display the image after the series of correction processes on the left display 107a and the right display 107b. The HMD 100 can display the corrected image after the series of correction processes by outputting the corrected left eye image and the corrected right eye image to a storage unit such as the ROM 111.

[0076] The left eye image and the right eye image may be recorded images input via an input unit of the HMD 100. The HMD 100 executes the series of correction processes described in Fig. 3 on the recorded image data, and can display the images after the series of correction processes have been applied on the left display 107a and the right display 107b.

[0077] The above-described embodiment (including the modified examples) is merely an example, and the present invention also includes configurations obtained by appropriately modifying or changing the configurations of the above-described embodiments within the scope of the gist of the present invention. The present invention also includes configurations obtained by appropriately combining the configurations of the above-described embodiments.

[0078] <Other embodiments> The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0079] The disclosure of this embodiment includes the following configuration, method, and program. (Configuration 1) an image acquisition means for acquiring a first image for a right eye and a second image for a left eye; a distance acquisition means for acquiring a depth distance from a user to an object detected in the first image and the second image; a display control means for controlling a display position of the first object on a display means based on a first depth distance which is the depth distance of a first object detected in the first image and the second image, and for controlling a display position of the second object on the display means based on a second depth distance which is the depth distance of a second object detected in the first image and the second image; 13. An image processing device comprising: (Configuration 2) the display control means corrects a predetermined display position of the first object by a first correction amount based on the first depth distance, and corrects a predetermined display position of the second object by a second correction amount based on the second depth distance; 2. The image processing device according to claim 1, (Configuration 3) The display control means acquiring the first correction amount based on the first depth distance, a distance between imaging devices that captured the first image and the second image, and a distance between the left eye and the right eye of the user; acquiring the second correction amount based on the second depth distance, a distance between imaging devices that captured the first image and the second image, and a distance between the left eye and the right eye of the user; 3. The image processing device according to configuration 2. (Configuration 4) The display control means, in each of the first image and the second image, generating an image of a region in which the first object is not displayed after the correction from an image of a periphery of the first object, among a region in which the first object was displayed before the correction; generating an image of a region in which the second object is not displayed after the correction from an image of the periphery of the second object, out of a region in which the second object was displayed before the correction; 4. The image processing device according to configuration 2 or 3. (Configuration 5) the display control means corrects a display position of the object when the object detected in the first image and the second image satisfies a predetermined condition. 5. The image processing device according to any one of configurations 2 to 4. (Configuration 6) the predetermined condition includes a condition that the type of the object is a predetermined type; 6. The image processing device according to configuration 5. (Configuration 7) The predetermined condition includes a condition that the depth distance of the object is shorter than a predetermined threshold value. 7. The image processing device according to configuration 5 or 6. (Configuration 8) the display control means outputs the corrected first image and the corrected second image to a storage unit. 8. The image processing device according to any one of configurations 2 to 7, (Configuration 9) the image acquisition means acquires the first image and the second image from an external device; 9. The image processing device according to any one of configurations 1 to 8. (Configuration 10) the first image and the second image are recorded images; 10. The image processing device according to any one of configurations 1 to 9. (Configuration 11) The depth distance of the object detected in the first image and the second image is a distance from the user to a center of the object. 11. The image processing device according to any one of configurations 1 to 10. (Configuration 12) An image processing device according to any one of configurations 1 to 11, an imaging means for capturing the first image and the second image; The display means; A head-mounted display device comprising: (method) obtaining a first image for a right eye and a second image for a left eye; obtaining a depth distance from a user of an object detected in the first image and the second image; controlling a display position of the first object on a display means based on a first depth distance that is the depth distance of a first object detected in the first image and the second image, and controlling a display position of the second object on the display means based on a second depth distance that is the depth distance of a second object detected in the first image and the second image; 13. An image processing method comprising: (program) 12. A program for causing a computer to function as each of the means of the image processing device according to any one of configurations 1 to 11. [Explanation of symbols]

[0080] 100: HMD, 110: CPU, 113: object detection unit, 114: distance acquisition unit

Claims

1. an image acquisition means for acquiring a first image for a right eye and a second image for a left eye; a distance acquisition means for acquiring a depth distance from a user to an object detected in the first image and the second image; a display control means for controlling a display position of the first object on a display means based on a first depth distance which is the depth distance of a first object detected in the first image and the second image, and for controlling a display position of the second object on the display means based on a second depth distance which is the depth distance of a second object detected in the first image and the second image; 13. An image processing device comprising:

2. the display control means corrects a predetermined display position of the first object by a first correction amount based on the first depth distance, and corrects a predetermined display position of the second object by a second correction amount based on the second depth distance; 2. The image processing device according to claim 1,

3. The display control means acquiring the first correction amount based on the first depth distance, a distance between imaging devices that captured the first image and the second image, and a distance between the left eye and the right eye of the user; acquiring the second correction amount based on the second depth distance, a distance between imaging devices that respectively captured the first image and the second image, and a distance between the left eye and the right eye of the user; 3. The image processing device according to claim 2.

4. The display control means, in each of the first image and the second image, generating an image of a region in which the first object is not displayed after the correction from an image of a periphery of the first object, out of a region in which the first object was displayed before the correction; generating an image of a region in which the second object is not displayed after the correction from an image of the periphery of the second object, out of a region in which the second object was displayed before the correction; 3. The image processing device according to claim 2.

5. the display control means corrects a display position of the object when the object detected in the first image and the second image satisfies a predetermined condition.

3. The image processing device according to claim 2.

6. the predetermined condition includes a condition that the type of the object is a predetermined type; 6. The image processing device according to claim 5,

7. The predetermined condition includes a condition that the depth distance of the object is shorter than a predetermined threshold value.

6. The image processing device according to claim 5,

8. the display control means outputs the corrected first image and the corrected second image to a storage unit.

3. The image processing device according to claim 2.

9. the image acquisition means acquires the first image and the second image from an external device; 2. The image processing device according to claim 1,

10. the first image and the second image are recorded images; 2. The image processing device according to claim 1,

11. The depth distance of the object detected in the first image and the second image is a distance from the user to a center of the object.

2. The image processing device according to claim 1,

12. An image processing device according to any one of claims 1 to 11, an imaging means for capturing the first image and the second image; The display means; A head-mounted display device comprising:

13. obtaining a first image for a right eye and a second image for a left eye; obtaining a depth distance from a user of an object detected in the first image and the second image; controlling a display position of the first object on a display means based on a first depth distance that is the depth distance of a first object detected in the first image and the second image, and controlling a display position of the second object on the display means based on a second depth distance that is the depth distance of a second object detected in the first image and the second image; 13. An image processing method comprising:

14. A program for causing a computer to function as each of the means of the image processing device according to any one of claims 1 to 11.

Images