Image processing device, image processing method, and program

Abstract

To make an HMD usable without anxiety by making an image of an actual world with a head-mounted display (HMD) being mounted on a user subsequently confirmable.SOLUTION: According to the present disclosure, an image processing device for controlling a display device wearable on the head of a user, includes acquisition means for acquiring an actual image being an image obtained by imaging an actual space around the user, and storage means for storing the actual image acquired by the acquisition means in a state in which the display device is mounted on the user.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present disclosure relates to an image processing system including a head-mounted display. [Background technology]

[0002] Head-mounted displays (hereinafter referred to as HMDs) are used as a form of display device for viewing images that combine virtual and real worlds. HMDs are display devices that can be worn on a user's head and display images of a virtual world, primarily composed of computer graphics, according to the user's position and posture, providing the user with the sensation of being immersed in a "non-real" space. In recent years, there have been an increasing number of opportunities to experience HMD images at event venues, stores, and other locations. However, while viewing images on an HMD, both of the user's eyes are covered, making the user unable to see their surroundings and leaving them vulnerable. This has made it difficult for users to enjoy images safely in places where many people gather, such as the event venues mentioned above.

[0003] Incidentally, for the purpose of crime prevention, video recording may be performed using cameras installed at fixed locations. As a technology related to recording with an HMD, Patent Document 1 proposes a technology for recording VR video displayed on an HMD. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2017-146578 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the technology of Patent Document 1 is intended to record the video displayed on the HMD, and does not take into consideration the safety of the user who is using the HMD. [Means for solving the problem]

[0006] The image processing device of the present disclosure is an image processing device that controls a display device that can be worn on a user's head, and is equipped with an acquisition means that acquires a real image, which is an image captured of the real space around the user, and a storage means that stores the real image acquired by the acquisition means while the user is wearing the display device. [Effects of the Invention]

[0007] According to the present disclosure, the user can later check images of the real world that were displayed while wearing the HMD, allowing the user to use the HMD with peace of mind. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating the overall configuration of an image processing system including an HMD. [Figure 2] FIG. 2 is a diagram illustrating an outline of the internal configuration of an HMD. [Figure 3] FIG. 1 is a diagram illustrating a hardware configuration of an image processing apparatus. [Figure 4] FIG. 1 is a diagram illustrating a functional configuration of a first embodiment. [Figure 5] 1 is a flowchart showing the overall flow of processing in the first embodiment. [Figure 6] 10 is a flowchart showing the flow of an image saving process. [Figure 7] FIG. 10 is a diagram illustrating a functional configuration of a second embodiment. [Figure 8] 10 is a flowchart showing the overall flow of processing according to the second embodiment. [Figure 9] FIG. 10 is a diagram illustrating a functional configuration of a third embodiment. [Figure 10] 10 is a flowchart showing the overall flow of processing according to the third embodiment. [Figure 11] FIG. 10 is a diagram illustrating a functional configuration of a fourth embodiment. [Figure 12] 10 is a flowchart showing the overall flow of processing according to the fourth embodiment. [Figure 13] FIG. 13 is a diagram illustrating a functional configuration of a fifth embodiment. [Figure 14] 13 is a flowchart showing the flow of image saving processing in the fifth embodiment. [Figure 15] FIG. 13 is a diagram illustrating a functional configuration of a sixth embodiment. [Figure 16] 13 is a flowchart showing the overall flow of processing according to the sixth embodiment. [Figure 17] 13 is a flowchart showing the flow of a second image saving process in the sixth embodiment. [Figure 18] FIG. 20 is a diagram illustrating a saved image in the sixth embodiment. [Figure 19] FIG. 13 is a diagram illustrating a functional configuration of a seventh embodiment. [Figure 20] 13 is a flowchart showing the overall flow of processing according to the seventh embodiment. [Figure 21] 13 is a flowchart showing the flow of image saving processing in a modified example of the seventh embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. Note that the following embodiments do not limit the present disclosure, and not all of the combinations of features described in the embodiments are necessarily essential to the solutions of the present disclosure.

[0010] First Embodiment In the first embodiment, while a user is wearing an HMD (head mounted display), an image processing device stores a real image, which is an image of the real space around the user.

[0011] (Image processing system configuration) 1 shows the configuration of an HMD system 1 as an example of an image processing system including an image processing device according to the present disclosure. The HMD system 1 includes an HMD 101 and an image processing device 102, which are communicatively connected via a transmission path 103 to communicate image data, control signals, and the like. The transmission path 103 includes a video signal line such as an HDMI (registered trademark) cable and a data signal line such as a USB cable. In addition, input devices such as a controller and a keyboard are communicatively connected to the image processing device 102 to receive input from a user. The communication connection between the HMD 101 and the image processing device 102 and the communication connection between the image processing device 102 and the input devices may be a wired connection such as a USB cable, or a wireless connection such as Bluetooth (registered trademark).

[0012] The HMD 101 is worn on the user's head, and allows the user's left and right eyes to observe a left-eye display image and a right-eye display image (enlarged virtual image). While the HMD 101 shown in FIG. 1 is a goggle-type headset worn on the user's head by a band 104, the HMD is not limited to this and may be a sunglasses-type or other type. As shown in FIG. 1, the HMD of this embodiment has stereo cameras 201a and 201b and stereo cameras 205a and 205b arranged on the left and right at a predetermined interval. While a total of four cameras are provided in FIG. 1, the number of cameras is not limited to this and may be at least one. The installation location is also not limited to the example shown in FIG. 1.

[0013] 2 is a diagram showing the internal configuration of the HMD 101. The HMD 101 includes multiple cameras 201a, 201b, 205a, and 205b, a proximity sensor 202, displays 203a and 203b, eyepieces 204a and 204b, and a distance sensor 206. The HMD 101 may also include an IMU (inertial measurement unit) for realizing position tracking, a speaker for outputting audio, a microphone for inputting audio, a vibrator for generating vibrations, and an LED lamp for indicating the status of the device.

[0014] 2, eyepiece 204a, display 203a, and camera 201a are arranged in this order from the closest to the eye so as to face the user's left eye, while eyepiece 204b, display 203b, and camera 201b are arranged in this order from the closest to the eye so as to face the user's right eye.

[0015] The cameras 201a and 201b are RGB cameras that capture images of the real space around the user. The images captured by the cameras 201a and 201b are sequentially sent to the image processing device 102 and used to generate a display image. The cameras 201a and 201b are so-called stereo cameras, with two cameras arranged left and right at a known distance.

[0016] The cameras 205a and 205b are cameras for positioning (position tracking) and capture images of the real space around the user. The images captured by the cameras 205a and 205b are sequentially transmitted to the image processing device 102 and used for self-position estimation by Visual SLAM, generation of an environment map, and the like. The cameras 205a and 205b are also so-called stereo cameras, in which two cameras are arranged on the left and right at a known distance. Although the cameras 205a and 205b are shown in FIG. 1 as being located on the left and right ends of the housing of the HMD 101 relative to the RGB cameras 201a and 201b, respectively, this is not limiting. For example, they may be located below or above. Furthermore, in addition to the cameras 205a and 205b, cameras capable of capturing images behind, to the left and right of the user may be provided. Furthermore, an omnidirectional camera capable of generating an image of a 360° range by image processing from images captured by multiple cameras may be used.

[0017] In the following description, cameras 201a and 201b used to generate display images are referred to as display image generating cameras. When the multiple cameras 201a and 201b for generating display images are not to be distinguished from one another, they are given the symbol 201. When the multiple cameras 205a and 205b for positioning (position tracking) are referred to as positioning cameras, they are given the symbol 205 when they are not to be distinguished from one another.

[0018] The timings at which the display image generating camera 201 and the alignment camera 205 start and end capturing images are controlled by the CPU 301 of the image processing device 102. For example, the timing at which the alignment camera 205 starts capturing images may be the timing at which the HMD 101 is started, the timing at which the HMD 101 is attached to the user's head, or the timing at which the user or operator inputs an instruction to start alignment processing. The timing at which the alignment camera 205 ends capturing images may be the timing at which the user removes the HMD 101 from the user's head, or the timing at which the user or operator inputs an instruction to end the display processing. Similarly, the timing at which the display image generating camera 201 starts capturing images may be the timing at which the HMD 101 is started, the timing at which the HMD 101 is attached to the user's head, or the timing at which the user or operator inputs an instruction to start display processing. The timing at which the display image generating camera 201 ends capturing images may be the timing at which the user removes the HMD 101 from the user's head, or the timing at which the user or operator inputs an instruction to end the display processing.

[0019] Since the display image generating camera 201 is used to generate a display image, it is preferable that it has a higher resolution and is capable of capturing color images compared to the alignment camera 205. On the other hand, it is preferable that the alignment camera 205 does not prioritize image quality and has a wider angle of view compared to the display image generating camera 201. Furthermore, the alignment camera 205 may have a low frame rate, low resolution, and capture monochrome images in order to reduce the processing load on the CPU.

[0020] Although the present embodiment illustrates a configuration example in which the display image generating camera 201 and the positioning camera 205 are provided, the display image generating camera 201 may also serve as the positioning camera 205. That is, not only may a display image be generated using a real image captured by the display image generating camera 201, but also a self-position may be estimated and an environment map may be generated. The self-position is expressed, for example, in 6DoF (degrees of freedom). Specifically, it is expressed by forward / back, up / down, left / right, pitch, yaw, and roll. Note that the method for estimating the self-position is not limited to Visual SLAM using multiple cameras, but may also be performed using a distance sensor 206 such as LiDAR or an IMU.

[0021] The proximity sensor 202 is a sensor that is provided on, for example, the surface of the housing of the HMD 101 that comes into contact with the user's head and detects that the HMD 101 is being worn by the user. The proximity sensor 202 outputs a signal indicating that the HMD 101 is being worn when the distance between the user's head and the proximity sensor 202 is smaller than a predetermined threshold.

[0022] Displays 203a and 203b are configured with display panels such as liquid crystal panels or organic EL panels. Furthermore, eyepieces 204a and 204b corresponding to the left and right eyes are arranged in front of displays 203a and 203b. A user of HMD 101 can observe enlarged virtual images of the images displayed on displays 203a and 203b through these eyepieces 204a and 204b.

[0023] The image processing device 102 performs processing to generate a display image for the left eye and a display image for the right eye, and displays these images on the displays 203a and 203b of the HMD 101. At this time, by providing an appropriate parallax between the display image for the left eye and the display image for the right eye, it is possible to give the user a visual perception with a sense of depth.

[0024] The HMD system 1 of this embodiment will be described as a system configuration in which the image processing device 102 and the HMD 101 are configured separately, but may also be configured as an integrated HMD system in which the image processing device 102 is included inside the HMD 101, for example.

[0025] 3 is a diagram showing an example of the configuration of an image processing device 102 according to the present disclosure. The image processing device 102 includes a CPU 301, a GPU 302, a RAM 303, a ROM 304, an HDD 305, a general-purpose interface (I / F) 306, an output I / F 307, a network I / F 308, and an input I / F 309. These components are connected via a system bus 310.

[0026] The CPU 301 comprehensively controls the entire HMD system 1. The CPU 301 is a processor that controls the entire system by reading and executing a system program stored in the ROM 301 or HDD 305. The CPU 301 also realizes the operation of this embodiment by reading and executing an application program stored in the ROM 301 or HDD 305. Although FIG. 3 shows one CPU, the system may be configured with multiple CPUs.

[0027] The GPU 302 is a processor that performs image processing upon receiving instructions from the CPU 301. The GPU 302, for example, performs rendering of CG (computer graphics) and generates a display image to be displayed on the display of the HMD 101. The display image may be CG only, or may be a virtual reality image in which CG, which is a virtual object, is superimposed on a real image acquired from the RGB camera 201 of the HMD 101. The display image will be described later. Although FIG. 3 shows one GPU, the system may be configured with multiple GPUs.

[0028] The RAM 303 is a general-purpose RAM and is used as a work memory for temporarily storing various pieces of information when, for example, the CPU 301 executes a program. The ROM 104 is a general-purpose ROM and stores, for example, programs executed by the CPU 301 and the GPU 302. The HDD (hard disk) 305 is a storage medium (storage unit) for storing image data, results of various processes, other data, and various programs executed by the CPU 301 and the GPU 302. The HDD 305 may be an SSD or flash memory.

[0029] The general-purpose I / F 306 is a serial bus interface such as USB or IEEE 1394, and is used to connect peripheral devices. The general-purpose I / F 306 is also used to acquire images input from the RGB camera 201 and alignment camera of the HMD 101, and to acquire signals input from the sensors of the HMD 101. The output I / F 307 is an interface such as HDMI or display port, and is used to display images on the display 203 of the HMD 101 and to output audio to a speaker (not shown).

[0030] The network I / F 308 is an interface for establishing a communication connection to a LAN or the Internet under the control of the CPU 301. The image processing device 102 can establish a communication connection with an HMD system 1 used by another person via the network, or with an external content distribution server or the like. The system bus 310 controls the overall data flow of the image processing device 102. Note that the image processing device 102 may include components other than those described above. The input I / F 309 is a serial bus interface such as USB or IEEE1394, and connects input devices 311 such as a keyboard, mouse, touch panel, and controller.

[0031] (HMD functional configuration) Fig. 4 is a diagram showing the functional configuration of the image processing device 102 in the first embodiment. The image processing device 102 has a real image acquisition unit 401, a display image generation unit 402, a display unit 403, an attachment determination unit 404, and an image storage unit 405. Program modules corresponding to the components shown in Fig. 4 are included in an application program. The CPU 301 executes each program module, causing the CPU 301 to function as each component shown in Fig. 4. The same applies to each embodiment described later in this specification.

[0032] The real image acquisition unit 401 acquires a real image captured by an imaging unit (cameras 201, 205) provided in the HMD 101. A real image is an image of real space, and may be either a still image or a video, but in this embodiment, a video will be assumed for explanation. The real image acquired from the display image generation camera 201 is input to the display image generation unit 402. The real image acquired from the alignment camera 205 is input to the image storage unit 405.

[0033] The display image generation unit 402 generates a display image to be displayed on the display 203 of the HMD 101. The display unit 403 outputs the display image generated by the display image generation unit 402 to the HMD 101 and displays it on the display. The display image may be either a still image or a moving image, but the description in this embodiment will be given assuming a moving image.

[0034] The wearing determination unit 404 determines whether the HMD 101 is worn by the user. The wearing determination unit 404 determines whether the HMD 101 is worn by the user based on a signal input from the proximity sensor 202. When the wearing determination unit 404 acquires a signal indicating that wearing has been detected from the proximity sensor 202, the wearing determination unit 404 determines that the HMD 101 is worn by the user. The wearing determination unit 404 outputs the determination result to the image storage unit 405.

[0035] When the image storage unit 405 acquires a determination result indicating that the HMD 101 is worn by the user from the wearing determination unit 404, that is, when the user is wearing the HMD 101, the image storage unit 405 stores the real image acquired by the real image acquisition unit 401 in the HDD 305. In this embodiment, the image storage unit 405 stores the real image acquired from the alignment camera 205 in the HDD 305.

[0036] (Processing performed by the image processing device) FIG. 5 is a flowchart showing the overall processing flow of the first embodiment. The overall processing flow executed by the image processing device 102 will be described using FIG. 5. The flowchart shown in FIG. 5 is realized, for example, by the CPU 301 reading a program stored in the HDD 305 into the RAM 303 and executing it. The processing of the first embodiment will be described with reference to FIG. 5. Note that the symbol "S" in the description of each process indicates a step in the flowchart. When the flowchart starts, it is assumed that real images have been captured by the alignment camera 201 and the display image generation camera 205 provided in the HMD 101, and that the real images have been input sequentially to the image processing device 102. The same applies hereinafter in this specification.

[0037] In S501, the display image generation unit 402 generates a display image. Examples of display images include VR (virtual reality) images, AR (augmented reality) images, and MR (mixed reality) images. VR images are images that are primarily composed of CG (computer graphics), and are entirely virtual, representing a non-real CG space. AR images are images in which various information (virtual objects, etc.) is added to real images in real time and displayed. MR images are images that further extend AR images, representing a mixed reality space in which virtual objects and virtual spaces that do not actually exist are superimposed on the real world. Using the HMD 101, the user can view these images from any position or angle. Note that when generating an image including a real image as a display image, a real image acquired from the display image generation camera 201 in FIG. 2 is used. As described above, the display image generation unit 402 preferably generates a left-eye display image and a right-eye display image having appropriate parallax.

[0038] In S502, the display unit 403 displays the left eye display image generated in S501 on the left eye display 203a of the HMD 101, and displays the right eye display image on the right eye display 203b.

[0039] During the display process of S501 to S502, in S503, the wearing determination unit 404 determines whether the user is wearing the HMD 101. This determination is made, for example, using the proximity sensor 202 provided in the HMD 101. Note that the method of determining whether the user is wearing the HMD 101 is not limited to using the proximity sensor 202. If a signal indicating that wearing has been detected is acquired from the proximity sensor 202, the process proceeds to S504. If a signal indicating that wearing has been detected is not acquired from the proximity sensor 202, the process proceeds to S505.

[0040] In S504, the image storage unit 405 performs image storage processing, the details of which will be described later (FIG. 6).

[0041] In S505, the image storage unit 405 determines whether or not to end the display processing. If the display processing is not to be ended, the process returns to S501. If the display processing is to be ended, for example, due to a stop instruction input by the user, the process of this flowchart ends.

[0042] (Details of image saving process) 6 is a flowchart showing the flow of the image saving process in S504 of the first embodiment. The flow of the image saving process of the first embodiment executed by the CPU 301 of the image processing apparatus 102 will be described with reference to FIG.

[0043] In S601 , the real image acquisition unit 401 acquires a real image input from the alignment camera 205 connected to the HMD 101 . In S602, the image storage unit 405 stores the real image acquired in S601 in the HDD 305.

[0044] As described above, the processing of the first embodiment makes it possible to save an image of the real world when the user is wearing the HMD 101. Specifically, it is possible to save (record) an image of the real space around the user when the user is wearing the HMD 101 and viewing a virtual space image (referred to as a VR image) or the like. This allows the user to later check the state of the real world while viewing the VR image, making it possible to give the user a sense of security while using the HMD 101.

[0045] In the above description, the images of real space to be saved are images acquired from the positioning camera 205, but this is not limiting. Images acquired from the display image generating camera 201 may also be saved. When saving images acquired from the positioning camera 205, reducing the amount of data can be prioritized over image quality, making it possible to record for a long period of time. On the other hand, when saving images acquired from the display image generating camera 201, good image quality can be prioritized over data volume.

[0046] In the above-described flowchart, the step of determining whether the HMD 102 is being worn is provided after the step of generating a display image and the step of displaying the display image, but the order is not limited to this. For example, the step of determining whether the HMD 102 is being worn may be provided before the step of generating a display image and the step of displaying the display image.

[0047] <Second embodiment> The image processing device 102A of the second embodiment notifies people around the user that a real image is being stored.

[0048] (HMD system configuration) The system configuration and hardware configuration of the HMD system of the second embodiment are the same as those of the first embodiment, and therefore a description thereof will be omitted.

[0049] (Functional configuration of image processing device) 7 is a diagram showing the functional configuration of an image processing device 102A according to the second embodiment. The image processing device 102A includes a real image acquisition unit 401, a display image generation unit 402, a display unit 403, an attachment determination unit 404, an image storage unit 405, and a recording status notification unit 701.

[0050] The real image acquisition unit 401, display image generation unit 402, display unit 403, wearing determination unit 404, and image storage unit 405 are the same as those in the first embodiment, so their explanations will be omitted and the same reference numerals will be used in the following explanation.

[0051] The difference from the first embodiment is that the image processing device 102A in the second embodiment includes a recording status notification unit 701. The recording status notification unit 701 notifies those around the user that the image storage unit 405 is executing image storage processing.

[0052] (Processing performed by the image processing device) Fig. 8 is a flowchart showing the processing flow of the second embodiment. The processing flow of the second embodiment executed by the image processing device 102A will be described with reference to Fig. 8. Note that S501 to S505 in the flowchart shown in Fig. 8 are the same as the processing of S501 to S505 (Fig. 5) in the first embodiment. Compared to the flowchart in Fig. 5, the flowchart in Fig. 8 adds processing of S801. Image saving processing is started in S504, and S801 is executed while the image saving processing is being executed.

[0053] In S801, the recording status notification unit 701 notifies those around the user that the image saving process is being executed. One way to notify is to flash an LED provided on the HMD 101. Note that well-known methods are not limited to this. For example, the notification may be made by playing a sound from a speaker or voicing a message.

[0054] In S505, the image storage unit 405 determines whether or not to end the display processing. If the display processing is not to be ended, the process returns to S501. If the display processing is to be ended, the process of this flowchart ends.

[0055] As described above, while the user is wearing the HMD 101, it is possible to notify people around the user that real-world images of the user's surroundings are being recorded. This increases the user's sense of security while watching VR. Furthermore, by informing people around that recording is being performed, it is expected that pranks and crimes will be prevented.

[0056] <Modification> In the second embodiment, an example of notifying people around the user is described. However, the user may be notified that the real image has been saved. For example, when it is determined in step S505 that the display has ended, the CPU 301 of the image processing device 102A notifies the user that the real image has been saved. The notification may be, for example, by displaying text or an icon indicating that the real image has been saved on the display of the HMD 101, or by outputting an audio announcement from the speaker. Furthermore, the notification that the real image of the surroundings will be recorded may be made before the display process is started, not just after the display process has ended. Furthermore, an icon indicating that the real image of the surroundings is being recorded may be displayed on the display screen during the display process. This increases the user's sense of security while wearing the HMD 101.

[0057] <Third embodiment> The image processing device 102B of the third embodiment further includes a switching unit 902 that switches whether or not the image storage unit 405 stores a real image. In the third embodiment, the switching unit 902 switches whether or not the image storage unit 405 stores a real image, based on the content displayed on the HMD 101.

[0058] (HMD system configuration) The system configuration and hardware configuration of the HMD system of the third embodiment are the same as those of the first embodiment, and therefore a description thereof will be omitted.

[0059] (Functional configuration of image processing device) 9 is a diagram showing the functional configuration of an image processing device 102B according to the third embodiment. The image processing device 102B includes a real image acquisition unit 401, a display image generation unit 402, a display unit 403, an attachment determination unit 404, an image storage unit 405, a display mode setting unit 901, and a switching unit 902.

[0060] The real image acquisition unit 401, the display image generation unit 402, the display unit 403, the wearing determination unit 404, and the image storage unit 405 are the same as those in the first embodiment, so descriptions thereof will be omitted and the same reference numerals as those in the first embodiment will be used. The difference from the first embodiment is that a display mode setting unit 901 and a switching unit 902 are provided.

[0061] The display mode setting unit 901 sets a display mode, which is the type of image to be displayed on the HMD 101. The display modes include a VR mode and a see-through mode. The VR mode is a mode in which VR video is displayed. In the VR mode, the real image acquired by the real image acquisition unit 401 is not included in the display content. Therefore, the user cannot see the surroundings. On the other hand, the see-through mode is a mode in which the above-mentioned MR video or AR video is displayed. In the see-through mode, the real image acquired by the real image acquisition unit 401 is reflected in the display content in real time. Therefore, in the see-through mode, the user can see the surroundings. The mode in which the image processing device 102B is to operate can be specified by a user operation.

[0062] The switching unit 902 switches whether to save the real image acquired by the real image acquisition unit 401. Specifically, even if the wearing determination unit 404 detects that the HMD 101 is being worn, if the content displayed on the HMD 101 includes a real image acquired by the real image acquisition unit 401, the switching unit 902 switches so that the image storage unit 405 does not save the real image. Furthermore, if wearing of the HMD 101 is detected and the content displayed on the HMD 101 does not include a real image acquired by the real image acquisition unit 401, the switching unit 902 switches so that the image storage unit 405 saves the real image. When the display mode can be set as in this embodiment, the switching unit 902 switches whether to save the real image by the image storage unit 405, depending on the display mode set by the display mode setting unit 901.

[0063] When the wearing determination unit 404 determines that the HMD 101 is worn by the user and when a display mode other than the see-through mode is set by the display mode setting unit 901, the switching unit 902 instructs the image storage unit 405 to store the real image. In the example of the present embodiment, when the VR mode is set, the switching unit 902 instructs the image storage unit 405 to store the real image. On the other hand, when the wearing determination unit 404 determines that the HMD 101 is not worn by the user or when the see-through mode is set by the display mode setting unit 901, the switching unit 902 instructs the image storage unit 405 not to store the real image.

[0064] In this way, the amount of data recorded can be reduced by not saving the real image when the user can see the surroundings from the displayed content, and by saving the real image when the user cannot see the surroundings from the displayed content.

[0065] (Processing performed by the image processing device) Fig. 10 is a flowchart showing the processing flow of the third embodiment. The processing flow of the third embodiment executed by the image processing device 102B will be described with reference to Fig. 10. Note that S501 to S505 in the flowchart shown in Fig. 10 are the same as S501 to S505 in the processing of the first embodiment. Compared to the flowchart in Fig. 5, S1001 and S1002 have been added to the flowchart in Fig. 10.

[0066] In S1001, the display mode setting unit 901 sets the display mode. This setting is performed based on a user operation. The image processing device 102B may display a UI screen for setting the display mode on the display 203, the HMD 101 connected to the image processing device 102B, or another display device, and may accept input from the input device 311 on the UI screen. As described above, the display modes include, for example, the VR mode and the see-through mode.

[0067] In S501, the display image generation unit 402 generates a display image according to the display mode. That is, in the see-through mode, an MR image or an AR image in which a real image and a virtual image are superimposed is generated, and in the VR mode, an image that does not include a real image, such as computer graphics or a movie, is generated.

[0068] In S502, the display unit 403 displays the left eye display image generated in S502 on the left eye display 203a of the HMD 101, and displays the right eye display image on the right eye display 203b. In S503, the wearing determination unit 404 determines whether the user is wearing the HMD 101. This determination is the same as in the first embodiment. If it is determined that the user is wearing the HMD 101, the process proceeds to S1002. If not, S1002 and S504 are skipped and the process proceeds to S505.

[0069] In S1002, the switching unit 902 checks the display mode set in S1001. If the display mode is not the see-through mode, the process proceeds to S504, where image saving processing is executed. If the display mode is the see-through mode, the process skips S504 and proceeds to S505.

[0070] The processing of S504 and S505 is the same as in the first embodiment. That is, in S504, the image storage unit 405 performs image storage processing, and in S505, the image storage unit 405 determines whether or not to end the display processing. If the display processing is not to be ended, the process returns to S501, and if the display processing is to be ended, the process of this flowchart ends.

[0071] As described above, the processing of the third embodiment allows an image of the real space around the user to be saved (recorded) when the user wears the HMD 101 on their head and the mode is set to a mode other than the see-through mode. Therefore, an image of the real space around the user can be saved (recorded) when watching a video that does not include a real image, such as a VR video. In the see-through mode, a real image is displayed on the HMD 101, so the user can check the state of the real world without recording. By not saving a real image in the see-through mode, the amount of data to be recorded can be reduced.

[0072] <Fourth embodiment> The image processing device 102C of the fourth embodiment further includes a switching unit 1102 that switches whether or not the image storage unit 405 stores a real image. In the fourth embodiment, the switching unit 1102 switches whether or not to store a real image based on the position and orientation of the user wearing the HMD 101.

[0073] (HMD system configuration) The system configuration and hardware configuration of the HMD system of the fourth embodiment are the same as those of the first embodiment, and therefore a description thereof will be omitted.

[0074] (Functional configuration of image processing device) 11 is a diagram showing the functional configuration of an image processing device 102C according to the fourth embodiment. The image processing device 102C includes a real image acquisition unit 401, a display image generation unit 402, a display unit 403, an attachment determination unit 404, an image storage unit 405, a position and orientation acquisition unit 1101, and a switching unit 1102.

[0075] The real image acquisition unit 401, the display image generation unit 402, the display unit 403, the wearing determination unit 404, and the image storage unit 405 are the same as those in the first embodiment, so descriptions thereof will be omitted and the same reference numerals as those in the first embodiment will be used. The difference from the first embodiment is that a position and orientation acquisition unit 1101 and a switching unit 1102 are provided.

[0076] The position and orientation acquisition unit 1101 acquires information about the position and orientation of the HMD 101 worn by the user. The position and orientation of the HMD 101 can be estimated by a self-position estimation process using, for example, Visual SLAM technology. Visual SLAM technology is a technology that acquires surrounding information (three-dimensional information) from image data obtained by a camera or image sensor, and simultaneously estimates the self-position and creates an environmental map from the surrounding three-dimensional information. Note that the method by which the position and orientation acquisition unit 1101 acquires the position and orientation information of the HMD 101 is not limited to Visual SLAM technology, and may also use, for example, Lidar SLAM technology, which acquires surrounding information (three-dimensional information) using a distance sensor such as Lidar. As described above, the position and orientation information of the HMD 101 is expressed, for example, in 6DoF.

[0077] The switching unit 1102 switches whether or not the image storage unit 405 stores a real image, based on the determination result of the wearing determination unit 404 and the position and orientation information of the HMD 101 acquired by the position and orientation acquisition unit 1101. Specifically, when the wearing determination unit 404 determines that the HMD 101 is worn by the user and the position and orientation information of the HMD 101 acquired by the position and orientation acquisition unit 1101 approximately matches a predetermined reference position and orientation, the switching unit 1102 switches so as to store the real image. When the position and orientation information approximately matches the predetermined reference position and orientation, the case where the position and orientation indicated by the position and orientation information completely matches the reference position and orientation includes not only a case where the position and orientation indicated by the position and orientation information completely matches the reference position and orientation, but also a case where the position and orientation is within a predetermined range from the reference position and orientation.

[0078] On the other hand, if the wearing determination unit 404 determines that the HMD 101 is not being worn by the user, or if the position and orientation information of the HMD 101 acquired by the position and orientation acquisition unit 1101 is not within a predetermined range from a predetermined reference position and orientation, the switching unit 1102 switches so as not to save the real image.

[0079] (Processing performed by image processing device 102C) Fig. 12 is a flowchart showing the overall processing flow of the fourth embodiment. The processing flow of the fourth embodiment executed by the image processing device 102C will be described with reference to Fig. 12. Note that S501 to S505 in the flowchart shown in Fig. 12 are the same as S501 to S505 in the processing of the first embodiment. Compared to the flowchart in Fig. 5, S1201, S1202, S1203, and S1204 have been added to the flowchart in Fig. 12.

[0080] When the flowchart of FIG. 12 starts, first, in S1201, the position and orientation acquisition unit 1101 determines information indicating the position and orientation of the HMD 101. When using the above-mentioned Visual SLAM technology, the position and orientation acquisition unit 1101 acquires real images from the alignment camera 205 of the HMD 101, estimates the distances to multiple specific objects (feature points), and performs relative self-location estimation using this information. For example, in the case of a stereo camera, the distance to the feature points can be estimated using parallax information. Then, as the HMD moves, the distance to the feature points changes, and this difference is compared using multiple feature points to estimate the relative positions of the objects. Note that acquisition of the position and orientation information of the HMD 101 is not limited to Visual SLAM technology, and estimation can also be performed using any other method.

[0081] In S1202, the position and orientation acquisition unit 1101 sets the position and orientation acquired in S1201 as the reference position and orientation.

[0082] Next, the CPU 301 of the image processing device 102C executes the processes of S501 to S503. That is, in S501, the display image generation unit 402 generates a display image. In S502, the display unit 403 displays the left eye display image generated in S502 on the left eye display 203a of the HMD 101, and displays the right eye display image on the right eye display 203b. In S503, the wearing determination unit 404 determines whether the user is wearing the HMD 101. The determination method is the same as in the first embodiment. If it is determined that the user is wearing the HMD 101, the process proceeds to S1203. If not, S1203, S1204, and S504 are skipped, and the process proceeds to S505.

[0083] In S1203, the position and orientation acquisition unit 1101 acquires the current position and orientation of the HMD 101. The method for acquiring the position and orientation is the same as in S1201.

[0084] In S1204, the image storage unit 405 determines whether the position and orientation acquired in S1203 match the reference position and orientation set in S1202. Note that even if the position and orientation do not match perfectly with the reference position and orientation, it may be determined that the position and orientation match as long as there is a certain degree of deviation. In this case, a margin indicating the range of deviation is defined in advance. The image storage unit 405 compares the position and orientation acquired in S1203 with the reference position and orientation set in S1202 and determines that the position and orientation match the reference position and orientation if the margin is within that range.

[0085] If the position and orientation acquired in S1203 is within a predetermined range from the reference position and orientation set in S1202, the process proceeds to S504, where image saving processing is executed. If the position and orientation acquired in S1203 is not within a predetermined range from the reference position and orientation set in S1202, S504 is skipped, and the process proceeds to S505.

[0086] The processing of S504 and S505 is the same as in the first embodiment. That is, in S504, the image storage unit 405 performs image storage processing, and in S505, the image storage unit 405 determines whether or not to end the display processing. If the display processing is not to be ended, the process returns to S501, and if the display processing is to be ended, the process of this flowchart ends.

[0087] In the image saving process of S504 shown in the fourth embodiment, the user's position and orientation at the time when the user puts on the HMD 101 and starts using it (generating and displaying a display image) is set as the reference position and orientation. When the user's position and orientation match the reference position and orientation (+ margin), the image saving unit 405 can save a real image of the user's surroundings. This allows the real world to be recorded with emphasis on any area the user was facing at the start of the process, allowing for later review. For example, if a baggage is placed in front of the user, by starting the process with a position and orientation that allows the baggage to be seen, an image of the area including the baggage can be recorded even while the HMD is in use. This increases the user's sense of security while using the HMD. Furthermore, since the real image is saved with little change in the user's position and orientation, there is little blurring of the image, which can prevent motion sickness when the user later reviews the saved real world.

[0088] <Modification 1 of the Fourth Embodiment> In the fourth embodiment, an example has been shown in which the position and orientation of the user at the time when the user wears the HMD 101 and starts using it (generating and displaying a display image) is set as the reference position and orientation, but this is not limiting. For example, before starting the processing of FIG. 12, the CPU 301 of the image processing device 101 may display a UI screen including a real image on the display 203 of the HMD 100 and instruct the user to turn toward an area to be focused on. In this case, the CPU 301 may accept in S1201 the position and orientation in a state in which the user has determined (specified) the orientation or position in accordance with the instruction on the UI screen, and set it as the reference position and orientation.

[0089] <Modification 2 of the Fourth Embodiment> The method for specifying the reference position and orientation is not limited to this. For example, the user may specify the reference position and orientation via an input device such as a keyboard or a mouse. In this case, the CPU 301 of the image processing device 102C may display the real image acquired by the HMD 101 on the display 203 in real time, and may specify an area on the screen that the user wants to focus on.

[0090] <Fifth embodiment> In the first to fourth embodiments, the image processing device 102D stores the acquired real image without processing it, but this is not limiting. In the fifth embodiment, the image processing device 102D may generate a wide-angle image based on the real image and store the wide-angle image. The wide-angle image will be described later.

[0091] (HMD system configuration) The system configuration and hardware configuration of the HMD system of the fifth embodiment are the same as those of the first embodiment, and therefore a description thereof will be omitted.

[0092] (HMD functional configuration) 13 is a diagram showing the functional configuration of an image processing device 102D according to the fifth embodiment. The image processing device 102D includes a real image acquisition unit 401, a display image generation unit 402, a display unit 403, an attachment determination unit 404, an image storage unit 405, and a storage image generation unit 1301.

[0093] The real image acquisition unit 401, display image generation unit 402, display unit 403, wearing determination unit 404, and image storage unit 405 are the same as those in the first embodiment, so their explanations will be omitted and they will be given the same reference numerals as those in the first embodiment.

[0094] The saved image generating unit 1301 generates a saved image to be saved in the image saving unit 405, based on the real image acquired by the real image acquiring unit 401. In the fifth embodiment, the saved image generating unit 1301 generates a wide-angle image.

[0095] (Processing performed by image processing device 102D) The overall flow of processing executed by the image processing device 102D of the fifth embodiment is similar to the processing of the first embodiment shown in Fig. 5, and therefore description thereof will be omitted. Below, the image saving processing of S504, which is different from the processing of the first embodiment, will be described.

[0096] (Details of Image Saving Process in Fifth Embodiment) 14 is a flowchart showing the flow of image saving processing according to the fifth embodiment. The flow of image saving processing according to the fifth embodiment, which is executed by the image processing device 102D, will be described with reference to FIG.

[0097] In S1401, the real image acquisition unit 401 acquires real images from the left and right alignment cameras 205a and 205b connected to the HMD 101. Note that the images acquired here are not limited to images captured by the alignment cameras 205a and 205b. For example, if RGB cameras 201a and 201b for generating display images or other cameras are connected, real images captured by these multiple cameras may be acquired.

[0098] In S1402, the saved image generation unit 1301 generates a wide-angle image using the multiple real images acquired in S1401. A wide-angle image is an image with a wider angle of view than the imaging range (angle of view) of a single camera that captures the real images. Because the alignment cameras 205a and 205b are positioned a predetermined distance apart, the imaging ranges they capture partially overlap and partially differ. A method for generating a wide-angle image can be, for example, a stitching technique. In the stitching technique, the CPU 301 (or the GPU 302) extracts feature points from each of the multiple real images, matches the multiple real images based on the feature points, and synthesizes the images based on the matching results. As a result, an image obtained by connecting images acquired from both alignment cameras 205a and 205b so that the feature points match can be acquired as the wide-angle image. Note that the method for generating a wide-angle image is not limited to stitching technology, and any method may be used.

[0099] In S1403, the image storage unit 505 stores the wide-angle image generated in S1402 in the HDD 305.

[0100] As described above, the image processing device 102D of the fifth embodiment can generate and store an image with a wider angle of view than that of one of the cameras provided in the HMD 102, for example, the position information acquisition camera 205. This allows the user to later check a wider range of the real world compared to the first embodiment. This can increase the user's sense of security while viewing VR content. Note that the images used to generate the wide-angle image are not limited to real images acquired from the position information acquisition cameras 205a and 205b. For example, multiple real images acquired from the display image generation cameras 201a and 201b may be used. In this case, it is sufficient to generate an image with a wider angle of view than that of one of the display image generation cameras 201. Alternatively, if cameras are provided behind or on the left and right of the user, multiple images acquired from these cameras may be combined to generate an image with a wider angle of view than that of any one of the cameras.

[0101] Sixth Embodiment When a virtual image (CG image) and a real image are superimposed, as in MR video or AR video, a region of the real image behind the virtual object (CG object) is hidden by the virtual object. In this case, even when a video is displayed in see-through mode, for example, a part of the real image may be hidden by the virtual object and not be visible to the user. Therefore, in the sixth embodiment, the image processing device 102E saves an image of the region of the real image that is hidden by the virtual object in the displayed image.

[0102] (HMD system configuration) The system configuration and hardware configuration of the image processing device 102E of the sixth embodiment are the same as those of the first embodiment, and therefore a description thereof will be omitted.

[0103] (Functional configuration of image processing device) 15 is a diagram showing the functional configuration of an image processing device 102E according to the sixth embodiment. The image processing device 102E includes a real image acquisition unit 401, a display image generation unit 402, a display unit 403, an attachment determination unit 404, an image storage unit 405, a display mode setting unit 1501, a switching unit 1502, and a storage image generation unit 1503.

[0104] The real image acquisition unit 401, the display image generation unit 402, the display unit 403, the wearing determination unit 404, and the image storage unit 405 are the same as those in the first embodiment, so descriptions thereof will be omitted and the same reference numerals as those in the first embodiment will be used. What differs from the first embodiment is that a display mode setting unit 1501, a switching unit 1502, and a storage image generation unit 1503 are provided.

[0105] A display mode setting unit 1501 sets a display mode, which is the type of image to be displayed on the HMD 101. In this embodiment, as in the third embodiment, the display mode can be set to either the VR mode or the see-through mode by a user operation.

[0106] The switching unit 1502 switches the image saving process executed by the image saving unit 405 according to the determination result of the wearing determination unit 404 and the display mode set by the display mode setting unit 1501. Note that in the above-described third embodiment, images are not saved in a display mode (see-through mode) in which the display content includes a real image. That is, when the see-through mode is set, the display content includes a real image, so the image is not saved. In the sixth embodiment, even if the display content includes a real image, if there is an area hidden by CG, the real image of that area is saved.

[0107] Specifically, when the wearing determination unit 404 determines that the HMD 101 is worn by the user and the see-through mode is set by the display mode setting unit 1501, the switching unit 1502 switches to execute the second image saving process. In the second image saving process, unlike the image saving process described in the first embodiment (hereinafter referred to as the first image saving process), the image saving unit 405 saves the image for saving generated by the saved image generating unit 1503. Note that when the wearing determination unit 404 determines that the HMD 101 is worn by the user and the display mode setting unit 1501 has set a mode other than the see-through mode (VR mode), the image saving unit 405 executes the first image saving process.

[0108] In the second image saving process, the save image generating unit 1503 generates an image for saving. The image for saving is an image that includes an area hidden by CG from the real image included in the display image. The process of generating an image for saving will be described later (FIGS. 17 and 18).

[0109] (Processing executed by image processing device 102E) Fig. 16 is a flowchart showing the processing flow of the sixth embodiment. The processing flow of the sixth embodiment executed by the image processing device 102E will be described with reference to Fig. 16. Note that S501 to S505 in the flowchart shown in Fig. 16 are the same as S501 to S505 in the processing of the first embodiment. Compared to the flowchart in Fig. 5, S1601, S1602, and S1603 have been added to the flowchart in Fig. 16.

[0110] In S1601, the display mode setting unit 1501 sets the display mode. This process is similar to S1001 in the third embodiment and is executed based on a user operation. The display mode is set to either the VR mode or the see-through mode.

[0111] Next, in S501, the display image generation unit 402 generates a display image according to the display mode. In S502, the display unit 403 displays the left eye display image generated in S502 on the left eye display 203a of the HMD 101, and displays the right eye display image on the right eye display 203b. In S503, the wearing determination unit 404 determines whether the user is wearing the HMD 101. This determination is the same as in the first embodiment. That is, if it is determined that the user is wearing the HMD 101, the process proceeds to S1602. If not, the process proceeds to S505.

[0112] In S1602, the image storage unit 405 determines the display mode set in S1601. If the display mode is set to a mode other than the see-through mode, i.e., the VR mode, the process proceeds to S504, and if the display mode is set to the see-through mode, the process proceeds to S1603.

[0113] In S504, the image storage unit 405 performs the first image storage process, which is the same image storage process (FIG. 6) as in the first embodiment.

[0114] In S1603, the image saving unit 405 executes a second image saving process. The second image saving process has different processing content from the first image saving process. The second image saving process will be described in detail later.

[0115] While the first image saving process in S504 and the second image saving process in S1603 are being executed, the image saving unit 405 determines whether to end the display process in S505. If the display process is not to be ended, the process returns to S501, and if the display process is to be ended, the process of this flowchart ends.

[0116] (Details of the second image saving process) Fig. 17 is a flowchart showing the flow of the second image saving process executed in S1603 of Fig. 16. Fig. 18 is a diagram showing an outline of the second image saving process. The flow of the second image saving process executed by the image processing device 102E of the sixth embodiment will be described with reference to Figs. 17 and 18.

[0117] In S1701, the real image acquisition unit 401 acquires a real image from the camera of the HMD 101. In the sixth embodiment, the real image acquisition unit 401 acquires a real image from the display image generation camera 201. A real image 1801 as shown in FIG. 18(a) is acquired.

[0118] In S1702, the saved image generation unit 1503 acquires the CG superimposition area from the display image generation unit 402. This will be described with reference to FIG. 18. Assume that the display image generation unit 402 renders a CG image 1811 shown in FIG. 18(b) and superimposes it on the real image 1801 shown in FIG. 18(a) to generate a display image 1821 shown in FIG. 18(c). One technique for obtaining a superimposed image by superimposing two images is alpha blending. Alpha blending is a process of combining two images using a coefficient α (α is a real number between 0 and 1) according to the following (Equation 1):

[0119] dst(x,y)=src1(x,y)×α+src2(x,y)×(1-α) …(Formula 1)

[0120] This (Equation 1) means that the pixel values ​​of the two input images src1 and src2 are blended at a ratio of α:1-α to generate the pixel values ​​of the output image dst. Since addition is performed, the two input images must be the same size. The real image 1801 in FIG. 18(a) corresponds to src2, the CG image 1811 in FIG. 18(b) corresponds to src1, and the display image 1821 in FIG. 18(c) corresponds to dst. The value of α can be determined for each pixel. The display image 1821 is generated by setting α=1 for pixels in the CG image 1811 where the virtual object 1812 exists and α=0 for other pixels. The pixels in the CG image 1811 where the virtual object 1812 exists can be determined, for example, by acquiring a depth image corresponding to the CG image 1811 and binarizing it using the upper depth limit value of the rendering range of the virtual object 1812. As shown in FIG. 18(d), the white area 1832 is the area where the virtual object 1812 exists and is blended at α=1. The black area is an area where the virtual object 1812 does not exist, and is blended with α=0.

[0121] The area where the virtual object 1812 is superimposed on (foreground) the real image 1801 as seen by the user is the area 1832 (area where α=1) shown in white in FIG. 18(d). That is, the area shown in white in FIG. 18(d) is the area where the virtual object 1812 and the real image overlap (superimposition area), and the area shown in black is the area where only the real image is displayed. Hereinafter, the area where the virtual object 1812 is superimposed on (foreground) the real image 1801 will be referred to as the superimposition area.

[0122] In S1703, the saved image generation unit 1503 determines an area from which to cut out the real image 1801. Hereinafter, the area to be cut out will be referred to as the cut-out area. The cut-out area is determined so as to include at least the overlapping area 1832 acquired in S1702. For example, the saved image generation unit 1503 sets the smallest rectangular frame 1842 that circumscribes the overlapping area 1832. Using FIG. 18 as an example, the inside of the rectangular frame 1842 in the image 1841 corresponding to the display image 1821 represents the cut-out area. Note that the method for determining the cut-out area is not limited to this. As long as it includes at least the overlapping area 1832, it may be a polygon, a circle, or the overlapping area 1832 itself, instead of a rectangular frame.

[0123] In S1704, the saved image generation unit 1503 cuts out a cutout area from the real image acquired in S1701 based on the cutout area determined in S1703. Taking Fig. 18 as an example, a cutout area corresponding to a rectangular frame 1842 shown in Fig. 18(d) is cut out from the real image 1801, and a saved image 1851 as shown in Fig. 18(f) is generated.

[0124] In S1705, the image storage unit 405 stores the saved image 1851 extracted in S1704 in the HDD 305. Note that although the example in Fig. 18 shows one frame of a display image made up of multiple frames, the saved image generation unit 1503 generates saved images for the other frames in the same manner.

[0125] As described above, in the sixth embodiment, in a mode in which a superimposed image in which a real image and a CG (virtual object) are superimposed is displayed on the HMD 101, the saved image generation unit 1503 cuts out and saves the real image that serves as the background of the CG (virtual object). That is, the saved image generation unit 1503 identifies an area in the superimposed image in which a virtual object is superimposed on the foreground of the real image, and cuts out an area corresponding to the identified superimposed area from the real image to generate a saved image. The image storage unit 405 stores the saved image generated by the saved image generation unit 1503 in the HDD 305.

[0126] This allows the user to save a real image of an area hidden behind a CG (virtual object) while viewing an MR image or an AR image using the HMD 101. This allows the user to later check the state of the real world that was hidden behind the CG (virtual object). This increases the sense of security of the user while using the HMD 102. In addition, the size of the saved image is cropped to a size smaller than the size of the real image captured by the camera, reducing the amount of data saved.

[0127] In the above-described process for generating a saved image, an example has been shown in which a superimposed area is cut out from the real image acquired from the camera 201 for generating a displayed image to generate a saved image, but the present invention is not limited to this. An area of ​​the real image corresponding to the cut-out area determined in S1703 may be cut out from the camera 205 for positioning, and a saved image may be generated and saved.

[0128] Seventh Embodiment In the second image saving process shown in the sixth embodiment, the amount of data to be saved is reduced by cutting out and saving a part of the image, but the amount of saved data may also be reduced in the time axis direction. In the seventh embodiment, the image processing device 102F saves a real image when motion is detected in the real world, and does not save the real image when motion is not detected in the real world.

[0129] (HMD system configuration) The system configuration and hardware configuration of the image processing device 102F of the seventh embodiment are the same as those of the first embodiment, and therefore a description thereof will be omitted.

[0130] (Functional configuration of image processing device) 19 is a diagram showing the functional configuration of an image processing device 102F according to the seventh embodiment. The image processing device 102F includes a real image acquisition unit 401, a display image generation unit 402, a display unit 403, an attachment determination unit 404, an image storage unit 405, a display mode setting unit 1901, a position and orientation acquisition unit 1902, a moving object detection unit 1903, a switching unit 1904, and a storage image generation unit 1905.

[0131] The real image acquisition unit 401, display image generation unit 402, display unit 403, wearing determination unit 404, and image storage unit 405 are the same as those in the first embodiment. The display mode setting unit 1901 is the same as in the third embodiment and sets the display mode to either VR mode or see-through mode. The position and orientation acquisition unit 1902 acquires information about the position and orientation of the HMD 101 worn by the user, similar to the position and orientation acquisition unit 1101 described in the fourth embodiment.

[0132] The moving object detection unit 1903 detects a moving object from the real image acquired by the real image acquisition unit 401. Details of the moving object detection will be described later.

[0133] The switching unit 1904 determines whether to save the real image based on the determination result of the wearing determination unit 404, the display mode set by the display mode setting unit 1901, and the detection result of the moving object detection unit 1903. In the seventh embodiment, when the HMD 101 is worn by the user, the display mode is set to a mode other than the see-through mode, and a moving object is detected in the real image, the switching unit 1904 instructs the image saving unit 405 to save the real image. Otherwise, the switching unit 1904 does not issue an instruction to save the real image.

[0134] The saved image generating unit 1905 generates a saved image by clipping the image so as to include the area where the moving object is detected in the real image. The process of generating a saved image will be described later.

[0135] (Processing executed by image processing device 102F) Fig. 20 is a flowchart showing the overall processing flow of the seventh embodiment. The overall processing flow of the seventh embodiment executed by the image processing device 102F will be described with reference to Fig. 20. Note that S501 to S503, S504, and S505 in the flowchart shown in Fig. 20 are the same as S501 to S503, S504, and S505 in the processing of the first embodiment. Compared to the flowchart in Fig. 5, S2001, S2002, S2003, S2004, and S2005 have been added to the flowchart in Fig. 20.

[0136] In S2001, the display mode setting unit 1901 sets the display mode. This process is similar to S1001 in the third embodiment and is executed based on a user operation. As described above, the display mode is set to either the VR mode or the see-through mode.

[0137] In S501, the display image generation unit 402 generates a display image according to the display mode. In S502, the display unit 403 displays the left eye display image generated in S502 on the left eye display 203a of the HMD 101, and displays the right eye display image on the right eye display 203b. In S503, the wearing determination unit 404 determines whether the user is wearing the HMD 101. This determination is the same as in the first embodiment. If it is determined that the user is wearing the HMD 101, the process proceeds to S2002. If not, the process proceeds to S505.

[0138] In S2002, the switching unit 1904 checks the display mode set in S2001. If the display mode is other than the see-through mode (VR mode), the process proceeds to S2003, and if the display mode is the see-through mode, the process proceeds to S505.

[0139] In S2003, the real image acquisition unit 401 acquires a real image from the camera of the HMD 101. Furthermore, the position and orientation acquisition unit 1902 acquires position and orientation information of the HMD 101. In the seventh embodiment, an image captured by the positioning camera 205 is acquired as the real image. Note that an image captured by the display image generation camera 201 may also be acquired as the real image.

[0140] In S2004, the moving object detection unit 1903 detects a moving object from the real image based on the real image and the position and orientation information acquired in S2003. The moving object can be detected by estimating optical flow using, for example, the Lucas-Kanade method. The moving object detection unit 1903 detects an area whose optical flow is different from the surroundings as a moving object. Note that the method of moving object detection is not limited to this.

[0141] In S2005, the switching unit 1904 determines whether or not a moving object was detected in the real image in S2004. If a moving object was detected, the process proceeds to S504, and if a moving object was not detected, the process proceeds to S505.

[0142] In S504, the image saving unit 405 executes image saving processing. The image saving processing is the same as the image saving processing in the first embodiment. During the image saving processing in S504, the image saving unit 405 determines in S505 whether or not to end the display processing. If the display processing is not to be ended, the process returns to S501, and if the display processing is to be ended, the process of this flowchart ends.

[0143] When the display process is continued, the moving object detection process is also continued. Therefore, during the period when the display process is being executed, the real image is saved during the period when a moving object is detected in the real image, and the real image is not saved during the period when a moving object is not detected. Therefore, the saved image may be an intermittent image.

[0144] As described above, the processing of the seventh embodiment allows a real image to be saved when a movement is detected in the real world. This allows the user to later check the state of the real world during a period when a change occurred while viewing with an HMD, thereby increasing the user's sense of security while viewing VR content. Because the period during which a real image is saved is limited to the period during which a movement is detected, the recording time is shorter and the amount of data saved can be reduced compared to the first embodiment.

[0145] <Modification 1 of the Seventh Embodiment> In the seventh embodiment described above, an example was shown in which a real image is stored when a movement is detected in the real world, but the image to be stored does not have to be the entire real image. For example, the image storage unit 405 may cut out and store an area in which a moving object is detected from the real image.

[0146] (Details of Image Saving Process in Modification 1 of Seventh Embodiment) 21 is a flowchart showing the flow of image saving processing according to Modification 1 of the seventh embodiment. This flowchart is executed in S504 of FIG.

[0147] In S2101, the saved image generation unit 1905 determines the region where a moving object was detected as a clipping region as a result of the moving object detection process in S2004 described above. In S2102, the saved image generation unit 1905 clips the clipping region determined in S2101 from the real image. In S2103, the image storage unit 405 stores the image of the clipping region clipped in S2102 in the HDD 305, and this flowchart ends.

[0148] By the above processing, in the seventh embodiment, it is possible to further reduce the storage area within one frame of an image, and reduce the amount of data to be stored.

[0149] <Modification 2 of the Seventh Embodiment> In the seventh embodiment, the image processing device 102F stores a real image when motion is detected in the acquired real image (when a moving object is detected). In addition, the image processing device 102F may notify the user when motion is detected. For example, when a moving object is detected in the real image in S2004 described above, the image processing device 102F may notify the user by displaying text or an icon indicating that a moving object has been detected on the display 203 of the HMD 101 or by outputting a voice announcement. This allows the user to decide whether to continue or stop using the HMD 104, thereby further improving the user's safety.

[0150] <Modification 3 of the Seventh Embodiment> In the seventh embodiment, the image processing device 102F performs moving object detection on the entire acquired real image, but the area in which moving object detection is performed is not limited to this. For example, a setting of a region of interest in the real image may be received from the user in advance, and moving object detection may be performed on the received region of interest. For example, before starting the processing of FIG. 20 , the CPU 301 of the image processing device 102F may receive a designation of a region of interest from the user via an input device such as a keyboard or mouse. In this case, the CPU 301 of the image processing device 102F may display a real image acquired by the HMD 101 on a display screen in real time and receive a designation of a region of interest from the user on the screen. Alternatively, the CPU 301 may display a UI screen and instruct the user to face the region of interest. In this case, the CPU 301 may set the region of interest to an area captured while the user is facing in accordance with the instructions on the UI screen.

[0151] Although preferred embodiments of the present disclosure have been described above with reference to the accompanying drawings, the present disclosure is not limited to such examples. For example, it is preferable that the images stored by the image storage unit 405 have a smaller amount of data than the acquired real image. For example, as shown in the third, fourth, and seventh embodiments, an image with reduced data in the time axis direction may be stored by limiting the storage period of the acquired real image. Alternatively, as shown in the sixth embodiment, an image with a partial area of ​​the acquired real image cut out may be stored to reduce the amount of data. Alternatively, these may be combined to reduce the amount of data in the time axis direction or within a frame. Furthermore, a low-frame-rate image obtained by thinning out a predetermined percentage of frames from the real image or an image with reduced resolution may be stored as an image for storage. Furthermore, the present disclosure is not limited to the examples described in each embodiment, and the elements and concepts described in each embodiment may be combined for implementation. It is clear that a person skilled in the art can conceive of various modifications and alterations within the scope of the technical ideas disclosed herein, and it is understood that such modifications and alterations naturally fall within the technical scope of the present invention.

[0152] <Other embodiments> The present disclosure can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0153] The disclosure of the above-described embodiment includes the following configurations.

[0154] (Configuration 1) An image processing device that controls a display device that can be worn on a user's head, an acquisition means for acquiring a real image that is an image of a real space around the user; a storage unit for storing the real image acquired by the acquisition unit while the user is wearing the display device; An image processing device comprising:

[0155] (Configuration 2) 2. The image processing device according to configuration 1, wherein the real image acquired by the acquisition means is captured by an imaging means included in the display device.

[0156] (Configuration 3) 3. The image processing device according to configuration 1 or 2, further comprising a notification unit that notifies people around the user that the storage unit is storing the real image.

[0157] (Configuration 4) 4. The image processing device according to any one of configurations 1 to 3, further comprising a switching unit that switches whether or not the saving unit saves the real image.

[0158] (Configuration 5) 5. The image processing device according to configuration 4, wherein the switching means switches whether or not to save the real image based on the content displayed on the display device.

[0159] (Configuration 6) The image processing device according to configuration 4, wherein the switching means switches so as not to save the real image when the real image acquired by the acquisition means is included in the content displayed on the display device.

[0160] (Configuration 7) The image processing device further includes a setting means for setting the image processing device to operate in either a first mode in which an image in which the real image acquired by the acquisition means and a virtual image are superimposed on the display device, or a second mode in which an image not including the real image is displayed, 5. The image processing device according to configuration 4, wherein the switching means switches the image processing device to save the real image when the second mode is set by the setting means.

[0161] (Configuration 8) the acquiring means further acquires position and orientation information of the display device in a state where the display device is worn by the user; The image processing device according to any one of configurations 4 to 7, wherein the switching means further switches whether or not to save the real image acquired by the acquisition means based on the position and orientation information acquired by the acquisition means.

[0162] (Configuration 9) The image processing device according to configuration 8, wherein the switching means switches to save the real image when the position and orientation of the display device indicated by the position and orientation information acquired by the acquisition means is within a predetermined range from a predetermined reference position and orientation.

[0163] (Configuration 10) 10. The image processing device according to configuration 9, wherein the reference position and orientation is the position and orientation of the user at the time when the user starts using the display device.

[0164] (Configuration 11) a receiving unit configured to receive a designation of the reference position and orientation from the user; The image processing device according to configuration 9 or 10, wherein the switching means switches to save the real image when the position and orientation of the display device indicated by the position and orientation information acquired by the acquisition means is within a predetermined range from the reference position and orientation accepted by the acceptance means.

[0165] (Configuration 12) The image processing device further includes a detection unit that detects a moving object from the real image acquired by the acquisition unit, 12. The image processing device according to any one of configurations 4 to 11, wherein the switching means switches to save the real image when the detecting means detects the moving object.

[0166] (Configuration 13) 13. The image processing device according to configuration 12, wherein the detection means detects the moving object from the real image acquired by the acquisition means for a predetermined region of interest in the real space.

[0167] (Configuration 14) a generating unit that generates an image for storage based on the real image acquired by the acquiring unit; 14. The image processing device according to any one of configurations 1 to 13, wherein the saving means saves the image for saving generated by the generating means as the real image.

[0168] (Configuration 15) 15. The image processing device according to configuration 14, wherein the image for storage is an image having a smaller amount of data than the actual image acquired by the acquisition means.

[0169] (Configuration 16) 15. The image processing device according to configuration 14, wherein the image for storage is an image in which data is reduced in the time axis direction compared to the actual image acquired by the acquisition means.

[0170] (Configuration 17) 15. The image processing device according to configuration 14, wherein the image for storage is an image obtained by cutting out a partial area of ​​the real image acquired by the acquisition means.

[0171] (Configuration 18) the acquisition means acquires the real images captured by a plurality of imaging units each having an imaging range of a predetermined angle of view, 15. The image processing device according to configuration 14, wherein the generating means generates an image having a field of view wider than the predetermined field of view as the image to be saved based on the plurality of real images acquired by the acquiring means.

[0172] (Configuration 19) 15. The image processing device according to claim 14, wherein when a superimposed image in which the real image acquired by the acquisition means and a virtual image including a virtual object are superimposed is displayed on the display device, the generation means generates the image for storage so that the superimposed image includes at least an area in the real image that is hidden by the virtual object.

[0173] (Configuration 20) The image processing device further includes a detection unit that detects a moving object from the real image acquired by the acquisition unit, 15. The image processing device according to configuration 14, wherein the generating means generates, as the image to be saved, an image that is cut out so as to include the area in the real image where the moving object is detected.

[0174] (Configuration 21) 21. The image processing device according to any one of configurations 1 to 20, wherein the image processing device is configured integrally with the display device.

[0175] (Configuration 22) 22. The image processing device according to any one of configurations 1 to 21, wherein the image processing device is configured separately from the display device and is communicatively connected to the display device via a transmission path.

[0176] (Configuration 23) 1. An image processing method executed by a computer that controls a display device that can be worn on a user's head, comprising: an acquisition step of acquiring a real image that is an image of a real space around the user; a step of storing the real image acquired in the acquiring step while the user is wearing the display device; An image processing method comprising:

[0177] (Configuration 24) A program to be executed by a computer that controls a display device that can be worn on a user's head, an acquisition step of acquiring a real image that is an image of a real space around the user; a step of storing the real image acquired in the acquiring step while the user is wearing the display device; Programs including.

Claims

1. An image processing device that controls a display device that can be worn on a user's head, an acquisition means for acquiring a real image that is an image of a real space around the user; a storage unit for storing the real image acquired by the acquisition unit while the user is wearing the display device; An image processing device comprising:

2. 2. The image processing device according to claim 1, wherein the real image acquired by the acquisition means is captured by an imaging means provided in the display device.

3. 2. The image processing apparatus according to claim 1, further comprising: a notification unit that notifies people around the user that the storage unit is storing the real image.

4. 2. The image processing apparatus according to claim 1, further comprising a switching unit for switching whether or not the saving unit saves the real image.

5. 5. The image processing apparatus according to claim 4, wherein the switching means switches whether or not to save the real image based on the content displayed on the display device.

6. The image processing device according to claim 4 , wherein the switching means switches the display device so that the real image acquired by the acquisition means is not saved when the content displayed on the display device includes the real image acquired by the acquisition means.

7. The image processing device may further include a setting unit that sets the image processing device to operate in either a first mode in which the display device displays an image in which the real image acquired by the acquisition unit is superimposed on a virtual image, or a second mode in which the display device displays an image that does not include the real image, 5. The image processing apparatus according to claim 4, wherein the switching means switches the image processing apparatus so as to store the real image when the second mode is set by the setting means.

8. the acquiring means further acquires position and orientation information of the display device in a state where the display device is worn by the user; 5. The image processing apparatus according to claim 4, wherein the switching means further switches whether or not to save the real image acquired by the acquisition means based on the position and orientation information acquired by the acquisition means.

9. 9. The image processing device according to claim 8, wherein the switching means switches to store the real image when the position and orientation of the display device indicated by the position and orientation information acquired by the acquisition means is within a predetermined range from a predetermined reference position and orientation.

10. The image processing device according to claim 9 , wherein the reference position and orientation is the position and orientation of the user at the time when the user starts using the display device.

11. a receiving unit configured to receive a designation of the reference position and orientation from the user; 10. The image processing device according to claim 9, wherein the switching means switches to store the real image when the position and orientation of the display device indicated by the position and orientation information acquired by the acquisition means is within a predetermined range from the reference position and orientation accepted by the acceptance means.

12. The image processing device further includes a detection unit that detects a moving object from the real image acquired by the acquisition unit, 5. The image processing apparatus according to claim 4, wherein the switching means switches to store the real image when the detecting means detects the moving object.

13. 13. The image processing apparatus according to claim 12, wherein the detecting means detects the moving object from the real image acquired by the acquiring means for a predetermined region of interest in the real space.

14. a generating unit that generates an image for storage based on the real image acquired by the acquiring unit; 2. The image processing apparatus according to claim 1, wherein the storage means stores the image for storage generated by the generation means as the real image.

15. 15. The image processing apparatus according to claim 14, wherein the image for storage has a smaller amount of data than the actual image acquired by the acquisition means.

16. 15. The image processing apparatus according to claim 14, wherein the image for storage is an image in which data is reduced in a time axis direction compared to the actual image acquired by the acquisition means.

17. 15. The image processing apparatus according to claim 14, wherein the image for storage is an image obtained by cutting out a partial area of ​​the real image acquired by the acquisition means.

18. the acquisition means acquires the real images captured by a plurality of imaging units each having an imaging range of a predetermined angle of view, 15. The image processing apparatus according to claim 14, wherein the generating means generates, as the image to be saved, an image having a field of view wider than the predetermined field of view based on the plurality of real images acquired by the acquiring means.

19. 15. The image processing device according to claim 14, wherein when a superimposed image in which the real image acquired by the acquisition means and a virtual image including a virtual object are superimposed is displayed on the display device, the generation means generates the image for storage so as to include at least an area in the real image that is hidden by the virtual object in the superimposed image.

20. The image processing device further includes a detection unit that detects a moving object from the real image acquired by the acquisition unit, 15. The image processing apparatus according to claim 14, wherein the generating means generates, as the image to be saved, an image that is cut out so as to include the area in the real image where the moving object is detected.

21. 2. The image processing device according to claim 1, wherein the image processing device is integrally configured with the display device.

22. 2. The image processing apparatus according to claim 1, wherein the image processing apparatus is configured separately from the display device and is communicatively connected to the display device via a transmission path.

23. 1. An image processing method executed by a computer that controls a display device that can be worn on a user's head, comprising: an acquisition step of acquiring a real image that is an image of a real space around the user; a step of storing the real image acquired in the acquiring step while the user is wearing the display device; An image processing method comprising:

24. A program to be executed by a computer that controls a display device that can be worn on a user's head, an acquisition step of acquiring a real image that is an image of a real space around the user; a step of storing the real image acquired in the acquiring step while the user is wearing the display device; Programs including.

Images