Image processing device, processing method therefor, and program

Abstract

To provide an easy-to-use mechanism by controlling blur correction of a composite image depending on the state of a user.SOLUTION: An image processing device includes generation means for generating a composite image by combining images of virtual objects displayed according to the position and orientation related to the viewpoint of a user, first output means for outputting the composite image generated by the generation means to a first display device used by the user, and second output means for outputting a composite image corrected for blur using the composite image to a second display device, and the second output means outputs the composite image generated by the generation means to the second display device when a condition related to blur correction is satisfied on the basis of the position of the user.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a technique for correcting blur in a composite image. [Background technology]

[0002] In recent years, mixed reality (MR) technology has become widespread. By using mixed reality technology, it is possible to provide a user wearing a head mounted display (HMD) with a mixed reality image in which a CG model is placed on a real image, allowing the user to experience a mixed reality world (mixed reality space) that combines reality and virtuality. To generate a mixed reality image, a method is used in which the position of the HMD and the position of the CG model are specified using sensors or two-dimensional markers.

[0003] Patent Document 2 discloses a technique for synthesizing an image including a person experiencing an experience, taken by an objective camera, with a CG image that is projected while the person is experiencing the experience, and displaying the synthesized image on a display.

[0004] In this way, it is common not only to simply display an image to the viewer, but also to project the image that the viewer is looking at or the state of the viewer's experience. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2003-308514 A [Patent Document 2] JP 2015-170232 A Summary of the Invention [Problem to be solved by the invention]

[0006] As mentioned above, in MR, the image seen by the person experiencing the experience is displayed on an external display (displaying the same image as that displayed on the person's HMD), so that someone who is not experiencing the experience can check it from outside.

[0007] In this case, because the participant is moving, the real image (video) captured by the HMD becomes very blurred (a lot of blurring occurs). The participant feels no discomfort even if the display is blurred because the MR image generated from the real image is displayed on the HMD in accordance with the participant's movements. However, people who are not experiencing the experience may feel sick (prone to motion sickness) because they are viewing an image (blurred image) on an external display that is unrelated to their movements.

[0008] In particular, external displays tend to be large, which can make the images appear more blurry, and there is the problem that the images displayed on the HMD are not suitable for people who have not experienced it.

[0009] On the other hand, even if a person has not experienced the experience, there are cases where they need to see the actual working state of the person experiencing the experience (such as in the case of verification work), in which they need to see the image displayed on the person experiencing the experience as is, so uniformly controlling blurring is problematic.

[0010] Therefore, an object of the present invention is to provide an easy-to-use mechanism for controlling blur correction of a synthetic image depending on the state of the user. [Means for solving the problem]

[0011] An image processing device for achieving the object of the present invention comprises a generation means for generating a composite image by combining images of virtual objects displayed according to the position and orientation related to the viewpoint of an experiencer, a first output means for outputting the composite image generated by the generation means to a first display device used by the experiencer, and a second output means for outputting a composite image corrected for blur using the composite image to a second display device, wherein the second output means outputs the composite image generated by the generation means to the second display device when conditions related to blur correction are satisfied based on the position of the experiencer. Effect of the Invention

[0012] According to the present invention, it is possible to provide a user-friendly mechanism for controlling blur correction of a composite image according to the state of the user. [Brief description of the drawings]

[0013] [Figure 1] FIG. 1 is a diagram showing an overview of the system. [Diagram 2] FIG. 2 is a block diagram showing an example of the hardware configuration of each of the HMD 101 and the PC 100. [Diagram 3] FIG. 2 is a block diagram showing an example of the functional configuration of the PC 100. [Figure 4] FIG. 2 is a block diagram showing an example of a module configuration of the PC100. [Diagram 5] 4 is a flowchart of processing performed by the HMD 101, the PC 100, and the external display 160. [Figure 6] 11 is a flowchart of the process in step S510. [Figure 7] FIG. 2 is a diagram showing an example of the configuration of information held in the external memory 211. [Figure 8] 4A to 4C are diagrams showing examples of displays on the external display 160 during shake correction. [Figure 9] 11A and 11B are diagrams showing an example of a display on the external display 160 when image stabilization is released. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Hereinafter, an embodiment of the present invention will be described with reference to the accompanying drawings. Note that the embodiment described below shows an example of a specific implementation of the present invention, and is one of the specific examples of the configuration described in the claims.

[0015] The system according to this embodiment is a mixed reality presentation system that presents a mixed reality space, which is a composite space of a real space and a virtual space, to a head-mounted display device such as an HMD or a display device provided separately from the head-mounted display device. First, an overview of the system according to this embodiment will be described with reference to FIG.

[0016] An HMD 101 is worn on the user's head, and an image of the mixed reality space output from a PC 100 serving as an image processing device is displayed on the display screen of the HMD 101. The user experiences the mixed reality space by observing the image of the mixed reality space displayed on the display screen of the HMD 101.

[0017] In addition, in this embodiment, it is assumed that there are multiple users experiencing the game (experiencer A, experiencer B).

[0018] The HMD 101 is fitted with an optical marker 103 that can be detected (measured) by an optical sensor 104, and the measurement result of the optical marker 103 by the optical sensor 104 is sent to the PC 100. In FIG. 1, the number of optical sensors 104 is set to one for the sake of simplicity, but in reality, a plurality of optical sensors 104 are disposed. The PC 100 obtains the position and orientation of the optical marker 103 based on the measurement result of the optical marker 103 by the optical sensor 104. A technique for obtaining the position and orientation of the optical marker 103 based on the measurement result of the optical marker 103 by the optical sensor 104 is well known, and therefore a detailed description of the technique will be omitted. Then, when a camera provided in the HMD 101 is set as the user's viewpoint (user viewpoint), for example, the PC 100 converts the position and orientation of the optical marker 103 into the position and orientation of the user's viewpoint by using a previously obtained "relative position and orientation relationship between the user's viewpoint and the optical marker 103."

[0019] The method of acquiring the position and orientation of the user's viewpoint is not limited to a specific method. For example, the position and orientation measurement sensor used to obtain the position and orientation of the user's viewpoint is not limited to the optical sensor 104, and other types of position and orientation measurement sensors such as a magnetic sensor and an ultrasonic sensor may be used to acquire the position and orientation of the user's viewpoint. Also, a two-dimensional marker 102 arranged in the real space may be captured by a camera provided in the HMD 101, and the position and orientation of the user's viewpoint may be obtained using the captured image. Also, the position and orientation measurement sensor and the two-dimensional marker may be used in combination to acquire the position and orientation of the user's viewpoint.

[0020] The PC 100 then generates a virtual object such as a CG model 130, places it in a virtual space, and generates an image of the virtual object as seen from the user's viewpoint. The PC 100 then generates a composite image by combining the generated image of the virtual object with an image of the real space captured by a camera of the HMD 101 as an image of a mixed reality space, and transmits the generated image of the mixed reality space to the HMD 101. This allows the user to observe an image of the mixed reality space in front of his or her eyes that corresponds to the position and orientation of his or her viewpoint.

[0021] In addition, an external display 160 is provided in the real space to display an image of the user experiencing the mixed reality space. This external display 160 is used by users who are not experiencing the space to check the image of the person experiencing the space.

[0022] In this embodiment, it is assumed that an external display 160 is connected to the PC 100 of each HMD 101. Images displayed on each HMD 101 are displayed on each external display 160. An image of the HMD 101 of experiencer A is displayed on the external display 160 for experiencer A. An image of the HMD 101 of experiencer B is displayed on the external display 160 for experiencer B.

[0023] The display control on the external display 160 is an example of a feature of this embodiment.

[0024] Next, an example of the hardware configuration of each of the HMD 101 and the PC 100 will be described with reference to the block diagram of Fig. 2. Note that the configuration shown in Fig. 2 is an example of a hardware configuration applicable to the HMD 101 and the PC 100, and is not limited to the configuration in Fig. 2.

[0025] First, a description will be given of the HMD 101. The right-eye / left-eye video camera 221 has a right-eye video camera for capturing an image of real space to be provided to the right eye of the user, and a left-eye video camera for capturing an image of real space to be provided to the left eye of the user. The captured images (images of real space) by each video camera are sent to the PC 100.

[0026] The right-eye / left-eye display 222 has a right-eye display for providing an image (including images and text) to the right eye of the user, and a left-eye display for providing an image (including images and text) to the left eye of the user. The image for the right eye sent from the PC 100 is displayed on the right-eye display, and the image for the left eye is displayed on the left-eye display.

[0027] Next, a description will be given of the PC 100. The general-purpose bus 212 functions as an interface for receiving images in real space sent from the right-eye / left-eye video camera 221, and is constituted by, for example, an external input terminal (for example, an IEEE1394 terminal).

[0028] The CPU 201 executes various processes using computer programs and data stored in the ROM 202 and the RAM 203. In this way, the CPU 201 controls the overall operation of the PC 100, and also executes or controls each process to be performed by the PC 100, which will be described later.

[0029] The ROM 202 stores computer programs and data that do not need to be rewritten, for example, a BIOS (Basic Input / Output System) that is a control program for the CPU 201.

[0030] The RAM 203 has an area for storing computer programs and data loaded from the ROM 202 or the external memory 211. The RAM 203 also has an area for storing images of real space received from the right-eye / left-eye video camera 221 via the general-purpose bus 212. The RAM 203 also has an area for storing measurement results received from the optical sensor 104 via the communication I / F controller 208. The RAM 203 also has an area for storing images of real space received from the objective camera 106 via the communication I / F controller 208. The RAM 203 also has a work area used when the CPU 201 executes various processes. In this way, the RAM 203 can provide various areas as needed.

[0031] The video controller 206 is for performing display control of the right-eye / left-eye display 222, the display 210, and the external display 160. For example, an external output terminal (e.g., a Digital Visual Interface) is used to output a video signal (a signal for displaying images and characters) to the right-eye / left-eye display 222. The display 210 is configured with a CRT or an LCD screen, and can display the results of processing by the CPU 201 as images and characters. The display 210 can be used to input and display information such as various system settings.

[0032] The input controller 205 is for controlling the operation of the input device 209. The input device 209 functions as a user interface such as a mouse or a keyboard, and can input various instructions to the CPU 201 by the user operating it. For example, if the input device 209 is a touch panel, the user can input various instructions by pressing (touching with a finger or the like) an icon, cursor, or button displayed on the touch panel. Note that the touch panel may be a touch panel capable of detecting positions touched by multiple fingers, such as a multi-touch screen.

[0033] The memory controller 207 controls the reading and writing of information from and to the external memory 211. The external memory 211 is a memory device such as a hard disk drive, a flexible disk, or a card-type memory connected to a PCMCIA card slot via an adapter. The external memory 211 stores an OS (operating system), as well as computer programs and data for causing the CPU 201 to execute or control each process to be described later as being performed by the PC 100.

[0034] The computer programs stored in the external memory 211 include computer programs for causing the CPU 201 to execute processes in accordance with each flowchart described below. The data stored in the external memory 211 also includes information that is treated as known information in each of the following embodiments including this embodiment, and data (drawing data) required to draw various virtual objects including the CG model 130. The computer programs and data stored in the external memory 211 are loaded into the RAM 203 as appropriate under the control of the CPU 201, and become targets for processing by the CPU 201. The external memory 211 may be a memory device such as a USB memory or an SD card.

[0035] The communication I / F controller 208 functions as an interface for receiving measurement results and images of real space from the optical sensor 104. The communication I / F controller 208 also functions as an interface for performing data communication with an external device via a network such as a LAN or the Internet. For example, the communication I / F controller 208 is capable of Internet communication using TCP / IP. The communication I / F controller 208 also controls communication with the optical sensor 104 via Gigabit Ethernet (registered trademark) or the like. Note that, in FIG. 1, data communication between the optical sensor 104 and the external display 160 and the PC 100 is wired, but may be wireless. For example, the PC 100 can perform data communication between the optical sensor 104 and the external display 160 through various communication forms such as a LAN cable, a USB cable, and wireless communication.

[0036] The general-purpose bus 212 , CPU 201 , ROM 202 , RAM 203 , video controller 206 , input controller 205 , memory controller 207 , and communication I / F controller 208 are all connected to a system bus 204 .

[0037] In this embodiment, the HMD 101 and the PC 100 are separate devices, but the HMD 101 and the PC 100 may be integrated together. Also, the PC 100 may be treated as a server device.

[0038] Next, an example of the module configuration of the PC 100 will be described with reference to the block diagram of Fig. 4. The operating system 401 controls input / output with the HMD 101, and passes images of the real space obtained from the right-eye / left-eye video cameras 221 via the general-purpose bus 212 to the mixed reality platform 403. The operating system 401 also outputs images of the mixed reality space drawn by the graphic engine 402 to the right-eye / left-eye displays 222 via the video controller 206.

[0039] The graphic engine 402 generates an image of a virtual object using rendering data of the virtual object stored in the external memory 211, and generates an image in a mixed reality space by superimposing the generated image of the virtual object on an image captured by the right-eye / left-eye video camera 221. The engine used for generating (drawing) the image of the virtual object may be, for example, a widely used graphic engine such as OpenGL or DirectX, or a uniquely developed graphic engine. In this embodiment, OpenGL is used as the graphic library.

[0040] The mixed reality platform (also called MR platform) 403 obtains the position and orientation of the user's viewpoint as described above, and aligns the real space with the virtual space. These techniques can be realized using existing techniques disclosed in, for example, Japanese Patent Application Laid-Open No. 2002-32784, Japanese Patent Application Laid-Open No. 2006-072903, Japanese Patent Application Laid-Open No. 2007-166427, etc.

[0041] A mixed reality application (also called an MR application or viewer application) 404 receives information required for drawing virtual objects, such as the position and orientation of the viewpoint, color and shape information of the virtual object, and the position and orientation of the virtual object, from the mixed reality platform 403, and issues a drawing command for an image of the virtual object to the graphic engine 402. At this time, a command is issued using the OpenGL API, setting the identification information and position and orientation information of the virtual object to be drawn.

[0042] Furthermore, when the mixed reality platform 403 receives a captured image from the objective camera 106, it passes the captured image to the mixed reality application 404. A correction module 405 performs a process of correcting blur of the image drawn by the graphic engine 402. The mixed reality platform 403 and the mixed reality application 404 control whether or not to perform the blur correction process.

[0043] The correction module 405 can be realized by using an electronic image stabilization technique. For example, a captured image is read into a memory once, and compared with the next captured image, and the image misalignment is corrected by shifting pixels. There are various techniques for image stabilization, and any of them may be used. The image to be corrected is the image to be output to the HMD 101.

[0044] The image that has been subjected to blur correction by the correction module 405 or the image that has not been subjected to blur correction is output to the external display 160.

[0045] Next, a process performed by the HMD 101 and the PC 100 to generate one image (one frame) of the mixed reality space and display it on the HMD 101 or the external display 160 will be described with reference to the flowchart of FIG.

[0046] In step S500, correction processing settings related to blur correction are performed by the PC 100. The correction processing settings are set in the HMD information 700 and the CG model information 720. The setting information is input from a setting screen displayed on the display 210 of the PC 100.

[0047] Here, Fig. 7 will be described. Fig. 7 is a diagram showing an example of setting information used in this embodiment. The setting information is assumed to be stored in the external memory 211.

[0048] The setting information includes HMD information 700 and CG model information 720. The HMD information 700 stores the settings of each HMD 101.

[0049] The CG model information 720 stores the settings of each CG model in the virtual space.

[0050] The HMD information 700 includes information on an HMD ID 701 , a position 702 , an attitude 703 , whether or not an external display is displayed 704 , a target 705 , and a setting value 706 .

[0051] The HMDID 701 stores an ID for each HMD 101. The position 702 and orientation 703 store the position and orientation information of the HMD 101 calculated by the mixed reality platform 403. This information is constantly updated.

[0052] External display display availability 704 is information (flag) that sets whether or not the image to be output to the HMD 101 is to be output to an external display. When it is ON, it is set to output to the external display 160, and is a setting for performing shake correction processing. When it is OFF, it is set not to output to the external display 160, and is a setting for not performing shake correction processing. When it is ON, a target 705 and a setting value 706 are set.

[0053] Note that even if external display display availability 704 is ON, if target 705 and setting value 706 are not set, the image may be output to external display 160 but correction processing may not be performed.

[0054] The target 705 and the setting value 706 are settings when performing the shake correction process, and the standard for performing the shake correction process is set.

[0055] The target 705 can be set to cancel the blur correction process depending on the relationship with the HMD 101 of another user, or to cancel the blur correction process depending on the relationship with the CG. If the blur correction process is to be canceled when approaching the HMD 101, the ID of the target HMD is set in the target 705, and distance information is stored in the setting value 706.

[0056] Moreover, in the case where the correction process is to be cancelled when approaching the CG, the target CG is set in the target 705 and distance information is stored in the setting value 706 .

[0057] A CG model ID may be set in the target 705, but in this embodiment, it is assumed that the CG model information 720 is used to set whether the CG is to undergo correction cancellation processing.

[0058] The CG model information 720 includes information on a model ID 721 , a model name 722 , a file path 723 , a position 724 , an orientation 725 , and a correction control release target 726 .

[0059] The model ID 721 stores the ID of the CG model. The model name 722 stores the file name of the CG model itself. The file path 723 stores the location where the file that becomes the CG model is saved.

[0060] The position 724 and orientation 725 have values ​​of the position and orientation in the virtual space to be displayed in the real space in which the CG model is displayed.

[0061] The correction control cancellation target 726 stores information for setting whether or not to cancel the correction process. For a CG model for which the correction process (correction control) is to be cancelled, ON is set. For a CG model for which the correction process (correction control) is not to be cancelled, OFF is set.

[0062] In other words, when the correction control cancellation target 726 is ON and the target 705 is CG, the correction process is canceled when the HMD 101 is in the specified position, and an image with the same blur as the image displayed on the HMD 101 is displayed on the external display 160.

[0063] 7 is loaded into memory, and the HMD 101 is started. When the HMD 101 is started, the process proceeds to step S502.

[0064] In step S502, capturing of a real image is started using the function of the right-eye / left-eye video camera 221.

[0065] In step S503, the right-eye / left-eye video camera 221 outputs captured images of real space (images captured by the right-eye camera and images captured by the left-eye camera) to the PC 100. Note that capturing and transmitting are repeated, and MR images are sequentially received from the PC 100 and displayed on the right-eye / left-eye displays 222 of the HMD 101.

[0066] In step S504, the CPU 201 receives an image of real space output from the right-eye / left-eye video camera 221 via the general-purpose bus 212, and in step S505, the CPU 201 stores the received image of real space in the RAM 203 or the external memory 211.

[0067] In step S506, the CPU 201 acquires the measurement result of the optical marker 103 by the optical sensor 104 via the communication I / F controller 208, and obtains the position and orientation of the user's viewpoint (the position and orientation of the HMD 101) based on the acquired measurement result as described above. The positions and orientations of the user's viewpoints obtained in step S506 are the position and orientation of the right viewpoint (right-eye video camera) and the left viewpoint (left-eye video camera). As for the right viewpoint, similar to the above, it can be obtained by converting the position and orientation of the optical marker 103 using the previously obtained "relative position and orientation relationship between the right viewpoint and the optical marker 103". Similarly, as for the left viewpoint, similar to the above, it can be obtained by converting the position and orientation of the optical marker 103 using the previously obtained "relative position and orientation relationship between the left viewpoint and the optical marker 103".

[0068] In step S507, the CPU 201 associates the position and orientation of the viewpoint calculated in step S506 with the identification information (ID) of the HMD 101 and stores the same in the external memory 211. For example, as shown in HMD information 700 in Fig. 7, the ID, position, and orientation are associated and managed for each HMD.

[0069] In step S508, the CPU 201 reads rendering data of the virtual object stored in the external memory 211 into the RAM 203 based on the CG model information 720 in Fig. 7, generates a virtual object based on the rendering data read out, and places the generated virtual object in a virtual space. The CPU 201 then generates an image of the virtual object seen from the viewpoint based on the position and orientation of the viewpoint determined in step S506. The CPU 201 then generates an image of the mixed reality space (MR image) by superimposing the image of the virtual object on the image of the real space received in step S504. Strictly speaking, the CPU 201 generates an image of the mixed reality space for the right eye by superimposing the image of the virtual object generated based on the position and orientation of the right viewpoint on the image captured by the right eye camera (image of the real space), and generates an image of the mixed reality space for the left eye by superimposing the image of the virtual object generated based on the position and orientation of the left viewpoint on the image captured by the left eye camera (image of the real space).

[0070] Step S508 shows an example of a generation process for generating a composite image by combining images of virtual objects displayed according to the position and orientation related to the viewpoint of the user.

[0071] In step S509, the CPU 201 determines whether or not to display the MR image generated in step S508 on the external display based on the setting of the external display display / non-display 704. If it is ON, the process proceeds to step S510, and if it is OFF, the process proceeds to step S514.

[0072] In step S510, CPU 201 performs correction control when outputting to external display 160. Details of the correction control process will be described in detail with reference to FIG.

[0073] In step S511, the CPU 201 outputs the MR image to the external display 160 via the video controller 206. This MR image may be an image that has been subjected to blur correction processing, or an image that has not been subjected to blur correction processing (cancelled).

[0074] Step S511 shows an example of the second output process in which a composite image corrected for blur using the composite image is output to the second display device. Also, this shows an example of the second output process in which the composite image generated in step S508 is output to the second display device when the condition related to blur correction is satisfied based on the position of the user. Furthermore, when the distance to the object is not satisfied, the blur-corrected composite image is output to the second display device, and when the distance to the object is not satisfied, the blur correction is released and the composite image generated in step S508 is output to the second display device.

[0075] In step S512, the external display 160 receives the MR images from the PC 100, and in step S513, displays the received MR images.

[0076] 8 and 9 show examples of what is displayed on the external display 160 in step S513.

[0077] FIG. 8 shows a state in which user A is looking at a CG 802 for which correction control is to be released, but since the user A is outside the distance of the set value 706 (the condition is not satisfied), the image is displayed with blur correction processing.

[0078] In Fig. 9, user A is looking at CG 902 for which correction control is released, and is within the distance of set value 706 (satisfying the criterion), so the blur correction process is released and displayed. The dotted CG of 902 shows the model displayed due to blur. Note that the dotted line display of 903 is for the purpose of explaining the blur for the sake of convenience, and is not limited to the dotted line display.

[0079] Also, as shown in 901, in order to alert users other than the experiencer who are watching the external display 160, the message "The same image as the experiencer is being displayed. There is no blur correction processing, so you may feel sick. Please be careful." is displayed. This display is information that is output from the PC 100 when the correction control is released. Also, the display of 901 can be arbitrarily set by the setting information (not shown) in FIG. 7.

[0080] In step S514, the CPU 201 outputs the MR images (the right eye image of the mixed reality space and the left eye image of the mixed reality space) generated in step S508 to the HMD 101 via the video controller 206.

[0081] Step S514 shows an example of a first output process in which the composite image generated in step S508 is output to the first display device used by the experiencer.

[0082] In step S515, the right-eye / left-eye display 222 receives the image of the right-eye mixed reality space and the image of the left-eye mixed reality space transmitted in step S514 from the PC 100. Then, in step S516, the right-eye / left-eye display 222 displays the image of the right-eye mixed reality space on the right-eye display.

[0083] Next, the correction control process in step S510 will be described with reference to the flowchart in FIG.

[0084] In step S601, the CPU 201 identifies the HMD 101 of the other experiencer or the target CG based on the setting information in Fig. 7. In the case of the HMD 101 of the other experiencer, it is desirable that the other experiencer is included in the captured image. In other words, when the other experiencer is looking at the other experiencer and conditions such as setting values ​​are satisfied, the blur correction control is released.

[0085] In step S602, the CPU 201 calculates and acquires the distance between the HMD 101 of the other experiencer or the target CG. The distance can be calculated based on the position of the HMD 101 of the target experiencer and the position of the HMD of the other experiencer, or the position of the HMD 101 of the target experiencer and the position of the CG.

[0086] In step S603, the CPU 201 determines whether the acquired distance satisfies the distance of the set value 706. Specifically, it determines whether the HMD 101 of the target experiencer is located at a position farther than the distance of the set value 706. If it is located farther, the condition is not satisfied and the process proceeds to step S607. If it is located close, the condition is satisfied and the process proceeds to step S604. In other words, the above-mentioned condition is a condition related to the shake correction and a condition for canceling the shake correction. It is also the distance between the position of the experiencer and the target object.

[0087] In step S604, CPU 201 determines whether or not blur correction has been canceled. That is, if the blur correction state has changed to a state in which blur correction is not performed (canceled state), the process proceeds to step S605. If the blur correction state has already been canceled, the process proceeds to step S606.

[0088] In step S605, since shake correction is normally performed, when shake correction is canceled, in order to alert users other than the experiencer that the correction process has been canceled, CPU 201 outputs (notifies) information that shake correction is not being performed to external display 160. An example of the alert is 901 in FIG. 9.

[0089] In this embodiment, control is performed so that a warning is issued when blur correction is canceled, but it is also possible to output warning information (output at a predetermined interval) while blur correction is canceled.

[0090] Furthermore, it goes without saying that when a warning message is displayed and it is time to perform blur correction, the warning message is controlled to be cancelled.

[0091] Step S605 shows an example of a notification process that notifies the user that blur correction has not been performed when the conditions for blur correction are met based on the user's position and the composite image generated in step S508 is output to the second display device.

[0092] In step S606, the CPU 201 acquires the MR image (uncorrected MR image) generated in step S508.

[0093] In step S607, the correction module 405 performs blur correction using the MR image generated in step S508.

[0094] In step S608, the CPU 201 acquires the MR image corrected in step S607.

[0095] The MR image acquired in step S606 or step S607 is output in step S511.

[0096] As described above, according to this embodiment, the shake correction process is switched on and off to display on the external display depending on the conditions of the HMD 101, which is a user-friendly mechanism. In particular, it is possible to reduce the burden of viewing a blurred image for a long period of time and prevent motion sickness.

[0097] As another embodiment, the following form is also conceivable. In the above embodiment, the shake correction process is cancelled when the conditions are satisfied, but in another embodiment, the shake correction process is executed when the conditions are satisfied (for example, the condition that the HMD 101 is outside the set distance is satisfied) and the shake correction process is cancelled when the conditions are not satisfied (for example, the condition that the HMD 101 is within the set distance is satisfied).

[0098] In this way, it is only necessary to be able to switch between execution and cancellation of the blur correction process, and the criterion for satisfying the condition (whether the criterion is within the distance or outside the distance) may be either.

[0099] Finally, an example of the functional configuration of the PC 100 will be described with reference to the block diagram of FIG. 3. Each functional unit in FIG. 3 may be implemented by hardware or software (computer program). In the former case, for example, by mounting such hardware on the HMD 101, the HMD 101 can execute each of the above-mentioned processes as those performed by the PC 100. In the latter case, such a computer program is stored in the external memory 211, and the CPU 201 appropriately reads out and executes this computer program into the RAM 203, thereby realizing the function of the corresponding functional unit. In addition, the PC 100 can also be referred to as an image processing device.

[0100] The generation unit 301 is a functional unit that generates a composite image by combining images of virtual objects displayed according to the position and orientation related to the viewpoint of the user. It is preferable that the image be an image of a mixed reality space in which a captured real image is combined with CG.

[0101] The first output unit 302 is a functional unit that outputs the composite image generated by the generation unit 301 to a first display device (for example, the HMD 101) used by the user.

[0102] The second output unit 303 is a functional unit that outputs a composite image corrected for blur using the composite image to a second display device (for example, the external display 160). The second output unit 303 is also a functional unit that outputs the composite image generated by the generation unit 301 to the second display device when a condition related to blur correction (specifically, a condition for canceling blur correction) is satisfied based on the position and orientation of the user.

[0103] The notification unit 304 is a functional unit that notifies the user that blur correction has not been performed when the condition for canceling blur correction is met based on the position and orientation of the user and the composite image generated by the generation unit 301 is output to the second display device. An example of the notification is 901.

[0104] The condition for canceling the blur correction is the distance between the position and posture of the user and the target object.

[0105] In addition, the second output unit 303 outputs the blur-corrected composite image to the second display device when the distance to the object is not satisfied, and cancels blur correction and outputs the composite image generated by the generation unit 301 to the second display device when the distance to the object is not satisfied.

[0106] That is, the image processing device according to this embodiment can improve convenience in the form of verification, etc., by outputting a composite image without blur correction depending on the position of the user when displaying a composite image on an external display. In particular, blur correction is performed for actions not required for verification, and is controlled not to be performed for actions necessary for verification (requiring accurate display), so that verification from the outside can be facilitated while reducing physical discomfort such as sickness of the user checking from the outside.

[0107] The external display may be any display device that can be used by people other than the experiencer.

[0108] In the above-described embodiment, the HMD 101 and the PC 100 are separate devices, but the HMD 101 and the PC 100 may be integrated, and the HMD 101 may perform not only the above-described processes that it is its own device to perform, but also the above-described processes that it is the PC 100 to perform.

[0109] In the above embodiment, the HMD 101 has been described as a video see-through head-mounted display device, but an optical see-through head-mounted display device may be used as the HMD 101.

[0110] In addition to or instead of the HMD 101 worn on the user's head, an image of the mixed reality space based on the user's viewpoint may be output to another display device, for example, a mobile terminal device such as a tablet terminal device or a smartphone. In this case, the mobile terminal device can acquire the position and orientation of the mobile terminal device as the user's viewpoint by using a sensor or application provided in the mobile terminal device. The image of the mixed reality space based on the user's viewpoint may be generated by the PC 100 or may be generated on the mobile terminal device side.

[0111] In the above embodiment, a mixed reality presentation system (MR system) that presents a mixed reality space that combines a real space and a virtual space to a user has been described as an example, but the present invention may be applied to a so-called VR system that presents only a virtual space to a user. In this case, processing related to capturing an image of the real space and combining it with an image of the real space is not required.

[0112] Further, in the above embodiment, for the sake of simplicity, one HMD 101 is connected to one PC 100, but two or more HMDs 101 may be connected to one PC 100.

[0113] Although the embodiment of the present invention has been described above, the present invention can be embodied, for example, as a system, an apparatus, a method, a program, a recording medium, etc. Specifically, the present invention may be applied to a system composed of multiple devices, or may be applied to an apparatus composed of a single device.

[0114] Furthermore, the program of the present invention is a program that enables a computer to execute the processing method of the flowchart, and the storage medium of the present invention stores the program that enables a computer to execute the processing method.

[0115] As described above, it goes without saying that the object of the present invention can be achieved by supplying a recording medium on which a program that realizes the functions of the above-mentioned embodiments is recorded to a system or device, and having the computer (or CPU or MPU) of that system or device read and execute the program stored on the recording medium.

[0116] In this case, the program itself read out from the recording medium will realize the novel functions of the present invention, and the recording medium storing the program will constitute the present invention.

[0117] Examples of recording media for supplying the program include flexible disks, hard disks, optical disks, magneto-optical disks, CD-ROMs, CD-Rs, DVD-ROMs, magnetic tapes, non-volatile memory cards, ROMs, EEPROMs, silicon disks, solid state drives, etc.

[0118] Furthermore, it goes without saying that the functions of the above-mentioned embodiments are not only realized by the computer executing a program it has read, but also includes cases where an OS (operating system) running on the computer performs some or all of the actual processing based on the instructions of the program, and the functions of the above-mentioned embodiments are realized through that processing.

[0119] Furthermore, it goes without saying that this also includes cases where a program read from a recording medium is written into a memory provided on a function expansion board inserted into a computer or a function expansion unit connected to a computer, and then a CPU or the like provided on the function expansion board or function expansion unit performs some or all of the actual processing based on the instructions of the program code, thereby realizing the functions of the above-mentioned embodiments.

[0120] The present invention may be applied to a system consisting of multiple devices, or to an apparatus consisting of a single device. Needless to say, the present invention can also be applied to a case where the object is achieved by supplying a program to a system or apparatus. In this case, the effect of the present invention can be enjoyed by the system or apparatus by reading a recording medium storing a program for achieving the present invention into the system or apparatus.

[0121] Furthermore, by downloading and reading a program for achieving the present invention from a server, database, etc. on a communication line using a communication program, the system or device can enjoy the effects of the present invention.

[0122] In addition, some or all of the embodiments described above may be used in appropriate combination, and some or all of the embodiments described above may be used selectively. [Explanation of symbols]

[0123] 100 PC 101 HMD 103 Optical marker 102 Two 3D marker 104 Optical sensor 130 CG model 160 External data Display

Claims

1. An image processing device comprising: a generation means for generating a composite image by combining images of virtual objects displayed in accordance with a position and orientation related to a viewpoint of an experiencer; a first output means for outputting the composite image generated by the generation means to a first display device used by the experiencer; and a second output means for outputting a composite image that has been subjected to blur correction using the composite image to a second display device, wherein the second output means outputs the composite image generated by the generation means to the second display device when a condition related to blur correction is satisfied based on the position of the experiencer.

2. 2. The image processing device according to claim 1, further comprising a notification unit that notifies a user that blur correction has not been performed when a condition related to blur correction is satisfied based on the position of the user and the composite image generated by the generation unit is output to a second display device.

3. 3. The image processing apparatus according to claim 1, wherein the condition relating to the shake correction is a condition for canceling the shake correction, and is a distance between a position of the user and the target object.

4. 4. The image processing device according to claim 3, wherein the second output means outputs a blur-corrected composite image to the second display device when a distance to the object is not satisfied, and cancels blur correction and outputs a composite image generated by the generation means to the second display device when the distance to the object is not satisfied.

5. A processing method for an image processing device, comprising: a generation step of generating a composite image by combining images of virtual objects displayed in accordance with the position and orientation related to the viewpoint of an experiencer; a first output step of outputting the composite image generated by the generation step to a first display device used by the experiencer; and a second output step of outputting a composite image that has been blur-corrected using the composite image to a second display device, wherein the second output step outputs the composite image generated by the generation step to the second display device when a condition related to blur correction is satisfied based on the position of the experiencer.

6. A program for making an image processing device function as: a generation means for generating a composite image by synthesizing images of virtual objects displayed in accordance with the position and orientation related to the viewpoint of an experiencer; a first output means for outputting the composite image generated by the generation means to a first display device used by the experiencer; and a second output means for outputting a composite image corrected for blur using the composite image to a second display device, the program causing the second output means to output the composite image generated by the generation means to the second display device when a condition related to blur correction is satisfied based on the position of the experiencer.

Images