Processing apparatus, processing method, and program

Abstract

To improve convenience of a display unit that allows a user to grasp the surrounding situation by seeing a picked-up image.SOLUTION: When a user wearing a display unit 110 is not moving, the display unit 110 displays a composite video obtained by composing a CG video and a background video with a relatively large display delay with each other. When the user wearing the display unit 110 is moving, the display unit 110 displays a composite video obtained by composing a CG video and a background video with a relatively small display delay with each other.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to a processing device, a processing method, and a program. [Background technology]

[0002] There are display devices that allow a user to understand the surrounding situation by viewing a captured image. One example of such a display device is a head-mounted display that displays images using video see-through. Patent Document 1 discloses a technique for displaying captured images and CG (Computer Graphics) on such a display device. Patent Document 1 discloses that a process to be performed by an imaging device that captures real space and a process to be performed by an image processing device capable of data communication with the imaging device are determined through negotiation between the imaging device and the image processing device. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2007-286851 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the technology described in Patent Document 1 aims to share the load between an imaging device and an image processing device, and does not take into consideration improving the convenience of a display device that allows a user to understand the surrounding situation by viewing an image captured by the display device.

[0005] Therefore, an object of the present disclosure is to improve the convenience of a display device that allows a user to understand the surrounding situation by viewing a captured image. [Means for solving the problem]

[0006] The processing device of the present disclosure is a processing device that performs processing to display on a display means worn by a user, and has a display processing means that performs processing to cause the display means to display a composite image formed by combining multiple images including captured images captured from a viewpoint corresponding to the user's viewpoint, and is characterized in that the display processing means displays, on the display means, a first composite image formed by combining captured images with a relatively long delay time from capture to display, or a second composite image formed by combining captured images with a relatively short delay time, depending on the movement state of the display means. [Effects of the Invention]

[0007] According to the present disclosure, it is possible to improve the convenience of a display device that is used in a manner that allows a user to understand the surrounding situation by viewing a captured image. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an example of a configuration of a display device and an information processing device. [Figure 2] FIG. 2 is a diagram illustrating a connection configuration of a display device and an information processing device. [Figure 3] FIG. 10 is a diagram illustrating a first video display process. [Figure 4] FIG. 10 is a diagram illustrating a second video display process. [Figure 5] 10 is a flowchart showing a first example of the overall flow of the operation of the display device. [Figure 6] 10 is a flowchart showing a second example of the overall operation flow of the display device. [Figure 7A] FIG. 10 is a diagram showing a first example of the display order of a background image and a CG image. [Figure 7B] FIG. 10 is a diagram showing a second example of the display order of the background video and the CG video. [Figure 8] FIG. 10 is a diagram illustrating an example of a first video display switching process. [Figure 9] FIG. 10 is a diagram illustrating an example of a second video display switching process. [Figure 10]10 is a flowchart showing a third example of the overall operation flow of the display device. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments will be described with reference to the drawings. (First embodiment) First, the first embodiment will be described. In this embodiment, a case where the display device worn by a user is a head-mounted display is exemplified. The head-mounted display has a camera and a display. An image of the surroundings of the head-mounted display captured by the camera is displayed on the display. A user wearing the head-mounted display can understand the surrounding situation by viewing the image. In the following description, the captured image captured by the camera is also referred to as a background image. Furthermore, in this embodiment, a case where an image to be combined with the background image is CG is exemplified. In the following description, the image to be combined with the background image is also referred to as a CG image. The CG image is combined with the background image as an image of a virtual space corresponding to the real space displayed in the background image. Furthermore, an image obtained by combining the background image and the CG image is also referred to as a composite image. Note that in this embodiment, a case where the captured image and the composite image are moving images is exemplified.

[0010] Furthermore, in this embodiment, a case where the first image display process is performed when the user wearing the head-mounted display is not moving (in other words, when the head-mounted display is not moving) is exemplified. When the first image display process is performed, there is a large delay in the display of the background image, but a composite image with high composite accuracy between the background image and the CG image is displayed. The display delay of the background image occurs, for example, due to the time difference between the timing when the background image is captured and the timing when the background image (composite image using the background image) is displayed. This time difference or a time proportional to this time difference is the delay time of the background image.

[0011] In addition, in this embodiment, a case where the second image display process is performed when the user wearing the head mounted display is moving (in other words, when the head mounted display is moving) is exemplified. When the second image display process is performed, a composite image is displayed in which the precision of combining the user background image and the CG image is low but the display delay of the background image is small. In this embodiment, a case will be exemplified in which the information processing system can dynamically switch between the first video display process and the second video display process.

[0012] 1 is a diagram showing an example of the configuration of a display device 110 and an information processing device 120. As described above, this embodiment illustrates a case where the display device 110 is a head-mounted display. Also, this embodiment illustrates a case where the information processing device 120 is a terminal device such as a smartphone.

[0013] In this embodiment, FIG. 1 illustrates a case where a display device 110 includes a control unit 111, a storage unit 112, a memory 113, an input unit 114, an output unit 115, a sensor 116, and a communication unit 117. The control unit 111 controls each unit of the display device 110. The control unit 111 has, for example, a CPU (Central Processing Unit). Furthermore, the control unit 111 may have one or more processors different from the CPU (for example, a GPU (Graphics Processing Unit)) in addition to or instead of the CPU. Furthermore, instead of the control unit 111 controlling the entire display device 110, the entire display device 110 may be controlled by a plurality of hardware devices sharing the processing. Furthermore, at least some of the processing executed by the control unit 111 may be executed by using dedicated hardware. The dedicated hardware is, for example, an ASIC (Application Specific Integrated Circuit) and an FPGA (Field Programmable Gate Array).

[0014] The storage unit 112 stores programs executed by the control unit 111, databases, various data (e.g., user setting data), etc. The storage unit 112 has, for example, a non-volatile storage medium that can electrically erase and record the programs, databases, and various data. The non-volatile storage medium is, for example, an SSD (Solid State Drive) or flash memory.

[0015] The memory 113 is used as a buffer memory that temporarily stores various types of data and as a work area for the control unit 111. The memory 113 is, for example, a RAM (Random Access Memory). Among the processes in the flowcharts described below, the processes of the display device 110 are realized by, for example, the control unit 111 executing a program stored in the storage unit 112 using the memory 113 as a work area.

[0016] The input unit 114 is used to input information such as instructions to the display device 110. The input unit 114 has, for example, a power button for instructing the display device 110 to be powered on and off, and operation buttons for instructing screen transitions, etc. Note that the input unit 114 does not necessarily have to be built into the display device 110. In this case, for example, information indicating instructions, etc. to the display device 110 may be input to the display device 110 via the communication unit 117, which will be described later.

[0017] The output unit 115 outputs various types of information. The output unit 115 has, for example, at least one of a computer display, a light-emitting device, and a sound-producing device. The computer display of the output unit 115 may display a GUI (Graphical User Interface) that allows the user to perform interactive operations with the display device 110. In this case, the computer display of the output unit 115 may have, for example, a touch panel. The light-emitting device is, for example, an LED (Light Emitting Diode). The sound-producing device is, for example, a speaker. The output unit 115 may have a configuration for outputting various types of information via the communication unit 117, which will be described later.

[0018] The sensor 116 performs various measurements. The sensor 116 includes, for example, an imaging sensor (camera) that captures video of the surroundings of the display device 110, a sensor that measures the surrounding conditions of the display device 110, and a sensor that measures the attitude and position of the display device 110. The sensor that measures the surrounding conditions of the display device 110 is, for example, a Light Detection and Ranging (LiDAR) sensor and a Time of Flight (ToF) sensor. In this case, the surrounding conditions of the display device 110 include, for example, at least one of the distance from the display device 110 to an object around the display device 110 and the position of the object around the display device 110. The sensor that measures the attitude and position of the display device 110 is, for example, an Inertial Measurement Unit (IMU) and a geomagnetic sensor. Note that at least one of the sensors 116 does not necessarily need to be built into the display device 110. In this case, information measured by the sensor may be input to the display device 110 via, for example, a communication unit 117, which will be described later.

[0019] The communication unit 117 communicates with an external device. The communication may be wired communication, wireless communication, or communication via a network. The communication unit 117 has a communication device that performs at least one of these types of communication. The communication unit 117 may have, for example, a USB (Universal Serial Bus) connector for connecting the display device 110 to an external device so that the display device 110 can communicate with the external device. The communication unit 117 may also have a connector (such as an RJ45 connector) for connecting to Ethernet. The communication unit 117 may also have a NIC (Network Interface Card) with a built-in communication IC or the like. The communication unit 117 may also have a communication unit for wireless communication with a controller or an external device based on Wi-Fi (registered trademark), Bluetooth (registered trademark), or the like.

[0020] Here, an example of the configuration of the display device 110 (terminal device) has been described assuming that the display device 110, which is an example of a processing device, is a head-mounted display. However, the above-described configuration of the display device 110 is only an example. The display device 110 does not have to be a head-mounted display. For example, the display device 110 may be a wearable display such as smart glasses. Furthermore, the configuration of the display device 110 may be different from the configuration exemplified in FIG. 1.

[0021] In this embodiment, FIG. 1 illustrates an example in which an information processing device 120 includes a control unit 121, a storage unit 122, a memory 123, an input unit 124, an output unit 125, a sensor 126, and a communication unit 127.

[0022] The control unit 121 controls each unit of the information processing device 120. The control unit 121 has, for example, a CPU. Furthermore, the control unit 121 may have one or more processors different from the CPU in addition to or instead of the CPU. Furthermore, instead of the control unit 121 controlling the entire information processing device 120, the entire information processing device 120 may be controlled by a plurality of hardware components sharing the processing. Furthermore, at least some of the processing performed by the control unit 121 may be performed by using dedicated hardware. The dedicated hardware is, for example, an ASIC and an FPGA.

[0023] The storage unit 122 stores programs executed by the control unit 111, databases, various data (e.g., user setting data), etc. The storage unit 122 includes, for example, a non-volatile storage medium. The non-volatile storage medium is, for example, an SSD or a flash memory.

[0024] The memory 123 is used as a buffer memory that temporarily stores various types of data and as a work area for the control unit 111. The memory 123 is, for example, a RAM. Among the processes in the flowcharts described below, the processes of the information processing device 120 are realized by, for example, the control unit 121 executing a program stored in the storage unit 122 using the memory 123 as a work area.

[0025] The input unit 124 is used to input information such as instructions to the information processing device 120. The input unit 124 has, for example, a power button for instructing the information processing device 120 to be powered on and off, and operation buttons for instructing screen transitions, etc. Note that the input unit 124 does not necessarily have to be built into the information processing device 120. In this case, for example, information indicating instructions, etc. to the information processing device 120 may be input to the information processing device 120 via a communication unit 127, which will be described later.

[0026] The output unit 125 outputs various types of information. The output unit 125 has, for example, at least one of a computer display, a light-emitting device, and a sound-producing device. The computer display of the output unit 125 may display a GUI that allows the user to perform interactive operations with the information processing device 120. In this case, the computer display of the output unit 125 may have, for example, a touch panel. The light-emitting device is, for example, an LED. The sound-producing device is, for example, a speaker. The output unit 125 may have a configuration for outputting various types of information via the communication unit 127, which will be described later.

[0027] The sensor 126 performs various measurements. The sensor 126 includes, for example, an imaging sensor (camera) that captures video of the surroundings of the information processing device 120, a sensor that measures the situation around the information processing device 120, and a sensor that measures the attitude and position of the information processing device 120. The sensor that measures the situation around the information processing device 120 is, for example, a LiDAR sensor and a ToF sensor. The sensor that measures the attitude and position of the information processing device 120 is, for example, an IMU and a geomagnetic sensor. The sensor 126 may also include a pedometer. Note that at least one of the sensors 126 does not necessarily need to be built into the information processing device 120. In this case, for example, information measured by the sensor may be input to the information processing device 120 via a communication unit 127, which will be described later.

[0028] The communication unit 127 communicates with an external device. The form of communication may be wired communication, wireless communication, or communication via a network. The communication unit 127 has a communication device that performs at least one of these types of communication. The communication unit 127 may have, for example, a USB connector for connecting the information processing device 120 to an external device so that the information processing device 120 can communicate with the external device. The communication unit 127 may also have a connector for connecting to Ethernet. The communication unit 127 may also have an NIC with a built-in communication IC or the like. The communication unit 127 may also have a communication unit for wireless communication with a controller or an external device based on Wi-Fi (registered trademark), Bluetooth (registered trademark), or the like.

[0029] Here, an example of the configuration of a terminal device has been described assuming that the information processing device 120 is a terminal device such as a smartphone. However, the above-described configuration of the information processing device 120 is just an example. The information processing device 120 does not have to be a smartphone. For example, the information processing device 120 may be a tablet terminal, a laptop computer, or a desktop personal computer. Furthermore, the configuration of the information processing device 120 may be different from the configuration exemplified in FIG. 2.

[0030] An example of a connection configuration between the display device 110 and the information processing device 120 will be described below with reference to Fig. 2. Fig. 2 is a diagram showing an example of a connection configuration between the display device 110 and the information processing device 120. 2, the present embodiment illustrates a case where the information processing system includes a display device 110 and an information processing device 120. Here, the description will be given assuming that the display device 110 and the information processing device 120 have the same hardware configuration as the display device 110 and the information processing device 120 shown in FIG.

[0031] 2 illustrates an example in which the display device 110 is a head-mounted display equipped with a Wi-Fi communication unit and can connect to a local network via a router. Also, FIG. 2 illustrates an example in which the information processing device 120 is a terminal such as a smartphone equipped with a Wi-Fi communication unit and can connect to a local network via a router like the display device 110.

[0032] In this case, the display device 110 and the information processing device 120 communicate with each other via a local network. The display device 110 and the information processing device 120 can bidirectionally communicate all kinds of data, such as data measured by the sensor 116 of the display device 110 and CG images drawn by the information processing device 120. The above-described connection method and connection configuration of the display device 110 and the information processing device 120 are merely examples, and the above-described connection method and connection configuration may be different.

[0033] Hereinafter, an example of a method for determining whether a user wearing the display device 110 is moving in the information processing system of this embodiment will be described. Note that determining whether a user wearing the display device 110 is moving is equivalent to determining whether the display device 110 is moving. Therefore, in the description of each embodiment, the movement of the user can be read as the movement of the display device 110.

[0034] There are various methods for determining whether or not a user is moving. In this embodiment, a case where whether or not a user is moving is determined using values ​​obtained by a camera mounted on a head-mounted display will be exemplified.

[0035] For example, there are head-mounted displays with 6DoF (Six Degrees of Freedom) functionality. Many of these head-mounted displays can estimate their own position using a technology called SLAM (Simultaneous Localization and Mapping). SLAM is a technology that uses sensors to measure the situation around the user, recognizes feature points of objects around the user based on the measurement results, and simultaneously creates a map of the user's surroundings (environmental mapping) and estimates the user's own position based on the feature points. The sensors in question are, for example, LiDAR sensors, cameras, and ToF sensors. By using this function, the head-mounted display can estimate the user's current position and posture based on a certain point.

[0036] In this embodiment, a case is exemplified in which the display device 110 estimates its own position using a camera mounted on the display device 110, and calculates the user's movement amount per unit time as the user's movement speed based on the result of the self-position estimation. In this embodiment, a case is exemplified in which the display device 110 determines that the user is moving when the user's movement speed is equal to or greater than a certain value, and determines that the user is not moving when the user's movement speed is less than the certain value. In this manner, even if the user is moving, if a predetermined condition is satisfied, the user may be considered not to be moving, and the display device 110 may determine whether the user is moving. For example, the display device 110 may determine that the user is moving when the user is moving so much that there is a risk of contacting an object around the user wearing the display device 110, and may otherwise determine that the user is not moving even if the user is actually moving.

[0037] As described above, the case where the sensor 116 mounted on the display device 110 (head-mounted display) is used when estimating the self-position of the head-mounted display has been exemplified. However, this is not necessarily the case. For example, the display device 110 may determine the position and posture of the user based on information measured by sensors arranged around the display device 110. Another method for determining whether the user is moving is to determine whether the user is moving by using a pedometer function mounted on the information processing device 120 (smartphone). Another method for determining whether the user is moving is to determine whether the user is moving by estimating the user's motion from the movement of each joint in the user's body. In this way, the method for detecting the moving state of the display device 110 is not limited.

[0038] Also, here, the case where the determination of whether the user is moving is performed by the display device 110 has been exemplified. However, this is not necessarily the case. For example, the determination of whether the user is moving may be performed by the information processing device 120. The above-described method for determining whether the user is moving is merely an example, and a method different from the above-described method may be used.

[0039] 3 and 4, an example of the operation of the display device 110 and the information processing device 120 will be outlined below. In this embodiment, a case where the display device 110 and the information processing device 120 perform a first video display process and a second video display process will be illustrated. 3 is a sequence diagram showing an example of the first video display process. The first video display process includes a process of combining a background video and a CG video by the information processing device 120. Here, an example of the first video display process will be described assuming that the display device 110 and the information processing device 120 have the same hardware configuration as in FIG.

[0040] In step S301, the display device 110 captures an image of the surroundings of the display device 110 as a background image using a camera mounted on the display device 110. Also, in this step S301, the display device 110 caches the background image captured by the camera in the memory 113.

[0041] In this embodiment, the display device 110 is attached to a user (human). In this case, the display device 110 is typically attached to the user so that the camera's viewpoint and the user's viewpoint are close to (or coincide with) each other. The user may also be a moving object other than a human. For example, the display device 110 may be attached to an industrial robot. In this case, for example, a camera mounted on the display device 110 may be used as a camera corresponding to the eye function of the moving object (such as an industrial robot). In this case, the viewpoint of the camera coincides with the viewpoint of the moving object (such as an industrial robot). Furthermore, a camera other than the camera mounted on the display device 110 may be mounted as a camera corresponding to the eye function of the moving object (such as an industrial robot). In this case, the viewpoint of the camera does not coincide with the viewpoint of the moving object. However, as described above, it is preferable that the viewpoint of the camera and the viewpoint of the moving object are close to each other. It is preferable that the viewpoint of the camera mounted on the display device 110 and the viewpoint of the moving object are linked.

[0042] Next, in step S302, the display device 110 estimates its own position based on the background image captured in step S301. Based on the result of the self-position estimation, information on the position and orientation of the display device 110 is obtained. Next, in step S303, the display device 110 transmits the background image captured in step S301 and the position and orientation information of the display device 110 obtained in step S302 to the information processing device 120 via the communication unit 117. The information processing device 120 receives the background image and the position and orientation information of the display device 110.

[0043] Next, in step S304, the information processing device 120 generates CG video based on the position and orientation information of the display device 110 received in step S303. The time required for the processing of step S304 depends, for example, on the performance of the information processing device 120 and the implementation of the application that generates the CG video. However, drawing CG video generally requires a large number of calculations. Therefore, the processing time of step S304 generally accounts for a large proportion of the overall time for the video display processing.

[0044] Next, in step S305, the information processing device 120 performs a process of synthesizing the background video received in step S303 and the CG video generated in step S304. In this way, Fig. 3 (first video display process) illustrates a case where the information processing device 120 generates a synthesized video in which the background video and the CG video are synthesized.

[0045] Next, in step S306, the information processing device 120 transmits the composite video generated in step S305 to the display device 110 via the communication unit 127. The display device 110 receives the composite video. Finally, in step S307, the display device 110 displays the composite video received in step S306 on the computer display.

[0046] As described above, in the first image display process, in step S305, the background image and the CG image are composited by the information processing device 120. In the first image display process, the process of step S305 is executed after the process of step S304 is completed, and then the processes of steps S306 and S307 are executed in this order. Therefore, if the process of step S304 for generating the CG image takes a long time, there is a risk of a large delay between the capture of the background image used in the composite image and the display of the composite image using the background image on the display device 110 (head-mounted display). However, when the first image display process is executed, the time lag between the background image and the CG image is small (preferably does not occur). Therefore, when the first image display process is executed, a composite image with high composite accuracy between the background image and the CG image is generated. Note that the time lag between the background image and the CG image occurs based on, for example, the time difference between the capture of the background image and the generation of the CG image to be composited with the background image.

[0047] 4 is a sequence diagram showing an example of the second image display process. The second image display process includes a process of combining a background image and a CG image by the display device 110. Here, an example of the second image display process will be described assuming that the display device 110 and the information processing device 120 have the same hardware configuration as in FIG. In the description of FIG. 4, the same reference numerals as those used in FIG. 3 are used, and detailed description of the same processes as those described in the first video display process will be omitted.

[0048] In step S401, the display device 110 transmits information about the position and orientation of the display device 110 to the information processing device 120 via the communication unit 117. The information about the position and orientation of the display device 110 is obtained, for example, from the result of the self-position estimation performed in step S302. The information processing device 120 receives the information about the position and orientation of the display device 110.

[0049] In step S402, the information processing device 120 transmits the CG image generated in step S304 to the display device 110 via the communication unit 127. In the second image display process, a process of compositing the background image and the CG image is performed in step S403, which will be described later. Therefore, in step S402, the information processing device 120 preferably includes or accompanies alpha channel information or chromakey information in the CG image. This allows the display device 110 to identify the position and orientation of the CG image relative to the background image. Also, in step S402, the display device 110 receives the CG image and caches it in memory 113.

[0050] Next, in step S403, display device 110 performs a process of combining the background image captured in step S301 with the CG image. In this way, FIG. 4 (second image display process) illustrates a case where display device 110 generates a combined image by combining the background image and the CG image. At this time, display device 110 does not wait until the CG image is transmitted from information processing device 120 in step S402, but instead combines the background image captured in step S301 with the CG image that has been received at that time. In this way, the processes of steps S304 and S402 may be performed after step S403.

[0051] As described above, in the second video display process, the display device 110 executes the processes of steps S403 and S307 (generation and display of a composite video) without waiting for the process of step S402 (transmission of a CG video by the information processing device 120). Therefore, even if the process of step S304 (generation of a CG video) takes time, a background video with less display delay than in the first video display process can be displayed on the display device 110 (head-mounted display).

[0052] However, in the second video display process, a background video with a small display delay is combined with a CG video that takes time to generate, which may result in a time lag or a misalignment in the display position between the background video and the CG video, which may reduce the accuracy of combining the background video and the CG video.

[0053] As described above, the first video display process and the second video display process have two contradictory characteristics. Therefore, in this embodiment, the display device 110 performs processing according to the movement state of the display device 110 (user usage status) as follows, thereby improving user convenience.

[0054] Next, an example of the detailed operation of the display device 110 in this embodiment will be described with reference to Fig. 5. Fig. 5 is a flowchart showing an example of the overall flow of the operation of the display device 110. Below, an example of the operation of each device will be described, taking as an example a case where the user moves around while wearing the display device 110.

[0055] The flowchart shown in FIG. 5 starts, for example, when the display device 110 is started up while the display device 110 is worn by the user. In step S501, control unit 111 determines whether or not to end the process based on the state of each unit of display device 110. For example, control unit 111 ends the process of this flowchart when an operation to stop the operation of display device 110 is performed via input unit 114. If the result of the determination in step S501 is to continue the process, control unit 111 transitions the process to step S502.

[0056] In step S502, control unit 111 instructs the camera included in sensor 116 to capture an image (background image) of the surroundings of display device 110. The camera captures the background image based on the instruction. Control unit 111 performs processing, including image processing, on the background image captured by the camera. This processing includes, for example, processing to develop RAW data and processing to adjust image quality. Control unit 111 caches the processed background image in memory 113. Then, control unit 111 proceeds to processing of step S503. The processing of step S502 corresponds to the processing of step S301 in FIG. 3.

[0057] Next, in step S503, the control unit 111 performs self-position estimation based on the background video cached in the memory 113 in step S502. The process of step S502 corresponds to the process of step S302 in FIG.

[0058] Next, in step S504, the control unit 111 calculates the moving speed of the display device 110 (user) based on the information on the position of the display device 110 obtained from the self-position estimation executed in step S503. Next, in step S505, the control unit 111 determines whether the display device 110 is moving or not based on the moving speed of the display device 110 (user) calculated in step S504. For example, the control unit 111 determines that the display device 110 is moving if the moving speed of the display device 110 is equal to or greater than a certain value, and determines that the display device 110 is not moving if the moving speed of the display device 110 is not equal to or greater than the certain value. The certain value may be 0 or a positive number (constant value > 0).

[0059] If the result of the determination in step S505 is that the display device 110 (user) has not moved, the control unit 111 shifts the process to the process in step S506. On the other hand, if the result of the determination in step S505 is that the display device 110 (user) has moved, the control unit 111 shifts the process to the process in step S507.

[0060] In step S506, control unit 111 performs a first video display process to display an image (composite image) obtained by combining the background image and the CG image on output unit 115 (computer display). The process of step S506 corresponds to the processes of steps S303 to S307 in Fig. 3. After step S506 ends, control unit 111 shifts the process to the process of step S501 described above.

[0061] In step S507, control unit 111 performs second image display processing to display an image (composite image) obtained by combining the background image and the CG image on output unit 115 (computer display). The processing in step S507 corresponds to the processing in steps S401 to S307 in Fig. 4. After step S507 is completed, control unit 111 shifts the processing to step S501 described above.

[0062] As described above, in this embodiment, when a user wearing the display device 110 is not moving, the display device 110 displays, as an example of a first composite image, a composite image obtained by combining a background image with a relatively large display delay and a CG image. This composite image is a composite image in which the composition accuracy of the background image and the CG image is relatively high. Furthermore, when a user wearing the display device 110 is moving, the display device 110 displays, as an example of a second composite image, a composite image obtained by combining a background image with a relatively small display delay and a CG image. The background image in this composite image is a background image in which the delay from the timing of imaging is relatively small. Therefore, for example, when a user is moving, it is possible to show the user a background image with a small time lag from the real space, with an emphasis on improving user safety. On the other hand, when a user is not moving, it is possible to show the user a composite image with a relatively high composition accuracy of the background image and the CG image, with an emphasis on improving the quality of the user experience. This improves the convenience of a display device in a usage mode in which the user can grasp the surrounding situation by viewing captured images.

[0063] In a display device 110 worn by a user, such as a head-mounted display, it may take some time from when a background image is captured until the composite image seen by the user is displayed on the display. In this case, the user will see an image with a delay compared to when the user sees the surroundings directly without using a display. Therefore, if the user moves around while wearing the display device 110, the delay may cause inconvenience such as the user coming into contact with surrounding objects.

[0064] The aforementioned delay can be reduced by displaying a background image captured by a camera on a display without waiting for other processing. However, if a composite image is generated by combining such a background image with an image generated separately from the background image and displayed on a display, the accuracy of the composite image may be reduced. For example, in a head-mounted display used for MR (Mixed Reality) applications, as exemplified in this embodiment, a CG image is composited with the background image. Therefore, when a background image is displayed without waiting for the generation of the CG image, for example, a CG image generated based on a background image captured at a timing different from the timing of capturing the background image is composited with the background image. Therefore, there is a risk of a time lag and a display position lag between the background image and the CG image used to generate the composite image. This may result in a decrease in the accuracy of the combination of the background image and the CG image, thereby degrading the quality of the user experience.

[0065] In contrast, as described above, in the present embodiment, by performing display processes (first image display process and second image display process) according to the operating state of the display device 110, it is possible to dynamically switch the composite image displayed on the display device 110. For example, when the user is not moving, the display device 110 displays a composite image synthesized with a background image having a relatively large display delay. This allows a CG image to be generated based on the background image, thereby enabling a composite image with high synthesis accuracy to be generated. This makes it possible to prevent a decrease in the quality of the user experience. On the other hand, when the user is moving, the display device 110 displays a composite image synthesized with a background image having a relatively small display delay. This makes it possible to display a situation that is close to the real space at the current time on the display. This makes it possible to prevent contact between the user and surrounding objects, etc. From the above, in the present embodiment, it is possible to increase the safety of a user wearing the display device 110 while limiting a decrease in the quality of the user experience.

[0066] (Second embodiment) Next, a second embodiment will be described. In the first embodiment, an example was given in which the image display process is switched depending on whether the user is moving. However, even when the user is moving, it may be preferable to combine the background image and the CG image with high accuracy. For example, the user may want to view the CG image from various angles while moving. In such a case, if the image display process is immediately switched from the first image display process to the second image display process while the user is moving, there is a risk that a composite image with low composition accuracy will be displayed. Therefore, there is a risk that the quality of the user experience will be reduced.

[0067] In contrast, for example, if a user is located near a position corresponding to a CG image, a composite image with high synthesis accuracy can be displayed even if the user is moving, thereby preventing a decrease in the quality of the user experience. Therefore, in this embodiment, a case where the image display process is dynamically switched depending on the movement state and position of the display device 110 is illustrated. As described above, the main difference between this embodiment and the first embodiment is the criteria for switching the image display process. Therefore, in the description of this embodiment, the same components as those in the first embodiment will be denoted by the same reference numerals as those used in FIGS. 1 to 5, and detailed descriptions thereof will be omitted. For example, the hardware configurations of the display device 110 and the information processing device 120 are the same as those of the display device 110 and the information processing device 120 in FIG. 1 described in the first embodiment. Therefore, detailed descriptions of the hardware configurations of the display device 110 and the information processing device 120 will be omitted.

[0068] In this embodiment, when the user is in a CG viewing area where the user is expected to view the CG video (when the display device 110 is located), the first video display process is executed even if the user is moving. An example of the CG viewing area will be described below.

[0069] The CG viewing area is determined, for example, based on the position of the CG image. The position of the CG image may be, for example, a position in a virtual space where the CG image is displayed, a position in real space corresponding to the position in the virtual space, or a position in a synthetic image. For example, the CG viewing area may often be within a certain distance from the position of the CG image. However, depending on the content, there may be CG images that are limited in the distance and line of sight (angle) that the user can view. In this case, a range determined according to the distance and line of sight that the user can view from the position of the CG image may be set as the CG viewing area. Furthermore, in an MR head-mounted display that displays a composite image of a background image and a CG image, the CG viewing area may be set taking into consideration the environment of the real space.

[0070] The method for setting the CG viewing area is not limited. For example, the CG viewing area may be set by the user of display device 110. Furthermore, the CG video or data attached to the CG video may contain information that allows identification of the CG viewing area. Furthermore, the CG viewing area may be dynamically determined by an application. Note that the CG viewing area does not have to be set. For example, instead of a CG viewing area, an area based on the position of an obstacle that actually exists in real space may be set.

[0071] In this embodiment, a case is illustrated in which a range a certain distance from the position of the CG image is set as the CG viewing area, regardless of the CG image or the environment in real space. Note that the CG viewing area and the method for setting the CG viewing area described above are merely examples, and the CG viewing area and the method for setting the CG viewing area may be different from those described above.

[0072] Hereinafter, with reference to FIG. 6, a detailed operation of the display device 110 in this embodiment will be described. Fig. 6 is a flowchart showing an example of the overall flow of the operation of the display device 110. Note that by using the same reference numerals as those used in Fig. 5, detailed description of parts showing the same content as in the operation of the display device 110 described in the first embodiment will be omitted.

[0073] In step S601, control unit 111 determines whether the user is within the CG viewing area based on the position information of display device 110 obtained from the self-position estimation executed in step S503. Note that determining whether the user is within the CG viewing area is equivalent to determining whether display device 110 is located within the CG viewing area. Therefore, in the description of each embodiment, the user being within the CG viewing area can be interpreted as display device 110 being located within the CG viewing area. Similarly, in the description of each embodiment, the user not being within the CG viewing area can be interpreted as display device 110 not being located within the CG viewing area.

[0074] If the result of the determination in step S601 is that the user is within the CG viewing area, control unit 111 shifts the process to the process in step S506. On the other hand, if the result of the determination in step S601 is that the user is not within the CG viewing area, control unit 111 shifts the process to the process in step S504.

[0075] As described above, in the information processing system of this embodiment, when a user wearing the display device 110 is within a CG viewing area set as an area for viewing CG images, the user can view composite images with high synthesis accuracy regardless of whether the user is moving. Furthermore, when the user leaves the CG viewing area, the system operates in the same manner as in the first embodiment, so that it is possible to provide image display processing according to whether the user is moving. This makes it possible to further limit the degradation of the quality of the user experience while improving the safety of the user when wearing the display device 110.

[0076] (Third embodiment) Next, a third embodiment will be described. In the first and second embodiments, a case has been exemplified in which the safety of a user wearing the display device 110 is increased while limiting degradation in the quality of the user experience by switching between two image display processes, a first image display process and a second image display process. However, there is a risk that a temporary degradation in the safety of a user wearing the display device 110 and a temporary degradation in the quality of the user experience may occur at the timing when the image display process is switched.

[0077] 7A and 7B are diagrams illustrating an example of the display order of frame images constituting a background video and a CG video. FIGS. 7A and 7B are diagrams illustrating an example of a third embodiment. In FIGS. 7A and 7B, one of the multiple rectangles lined up next to "Background" represents one frame image of the background video. Also, one of the multiple rectangles lined up next to "CG" represents one frame image of the CG video. The numbers shown in each rectangle indicate the display order. The initial value of the display order is 0. Numbers without parentheses indicate the display order of the frame images of the background video. Numbers in parentheses indicate the display order of the frame images of the CG video.

[0078] For example, suppose there is a difference in display delay of n frames (n is a positive integer) between a first video display process that uses a background video with a relatively large display delay and a second video display process that uses a background video with a relatively small display delay.

[0079] In this case, the background image (composite image) displayed by the second image display process is displayed n frames ahead of the background image (composite image) displayed by the first image display process. The time for one frame is determined based on the reciprocal of the frame rate. Therefore, when the image display process switches from the first image display process to the second image display process, n frames of the background image (frame images) are not displayed at the timing of the switch of the image display process.

[0080] FIG. 7A illustrates a case where the difference in display delay between the first and second video display processes is four frames (when n=4) as shown in "Second Video Display Process (Low Latency)" and "First Video Display Process (High Precision)" (see timings t1 and t5). In this case, when the video display process switches from the first video display process to the second video display process at timing t8 of "First to Second Video Display Process (First and Second Forms)," four frame images are not displayed as background video frame images. Specifically, frame images 3, 4, 5, and 6 in display order are not displayed as background video frame images. FIG. 7B illustrates a case where the difference in display delay between the first and second video display processes is five frames (when n=5) (see timings t1 and t6). In this case, when the video display process switches from the first video display process to the second video display process at timing t9 of the "first to second video display process (first and second forms)," five frame images are not displayed as background video frame images. Specifically, frame images 3, 4, 5, 6, and 7 in the display order are not displayed as background video frame images.

[0081] Furthermore, it takes n frames to display the next frame image of the background image (composite image) last displayed in the second image display process as the background image. Therefore, when the image display process switches from the second image display process to the first image display process, the background image is not updated during that period and remains frozen.

[0082] As mentioned above, FIG. 7A illustrates the case where n=4. In this case, from timing t4 to t8 of the "second-to-first video display process (first and second forms)," when the video display process switches from the second video display process to the first video display process, the same frame image is displayed continuously as a frame image of the background video. Specifically, the frame image with the display order of 2 is displayed continuously for a period of four frames. FIG. 7B illustrates the case where n=5. In this case, from timing t4 to t9 of the "second-to-first video display process (first and second forms)," when the video display process switches from the second video display process to the first video display process, the same frame image is displayed continuously as a frame image of the background video. Specifically, the frame image with the display order of 2 is displayed continuously for a period of five frames.

[0083] Furthermore, differences in display delay may occur not only in background images but also in CG images. In Fig. 7A, when the video display process switches from the first video display process to the second video display process, as at timing t8 in the "first to second video display process (first and second forms)," two frame images are not displayed as CG frame images. Specifically, frame images with display orders 3 and 4 are not displayed. Furthermore, when the video display process switches from the second video display process to the first video display process, as at timings t4 to t8 in the "second to first video display process (first and second forms)," the same frame image is displayed consecutively as CG frame images. Specifically, the frame image with display order 0 is displayed consecutively for a period of four frames. In FIG. 7B, when the video display process switches from the first video display process to the second video display process, as at timing t9 in the "first to second video display process (first and second forms)," three frame images are not displayed as CG video frame images. Specifically, frame images 3, 4, and 5 in the display order are not displayed. Also, when the video display process switches from the second video display process to the first video display process, as at timings t4 to t9 in the "second to first video display process (first and second forms)," the same frame image is displayed consecutively as CG video frame images. Specifically, the frame image with the display order of 0 is displayed consecutively for a period of five frames.

[0084] Comparing the processes of steps S306 and 402 described with reference to FIGS. 3 and 4, in both processes, video data is transmitted from information processing device 120 to display device 110. However, in step S306, a composite video in which a background video and a CG video are combined is transmitted, whereas in step S402, only the CG video is transmitted. This difference in the amount of transmitted data results in a difference in the time required for the processes of steps S306 and S402. If this time difference is one frame or more, problems such as CG video not being displayed or the display of the CG video stopping, similar to the background video, can occur when switching between the two video display processes, the first video display process and the second video display process.

[0085] In this way, when the first video display process is performed, a display delay may occur in each of the background video and the CG video relative to the second video display process. In Fig. 7A, the difference between the display delay of the background image in the first video display process and the display delay of the background image in the second video display process is four frames. In Fig. 7B, the difference between the display delay of the background image in the first video display process and the display delay of the background image in the second video display process is five frames.

[0086] In Fig. 7A, the difference between the display delay of the CG image in the first video display process and the display delay of the CG image in the second video display process is two frames. In Fig. 7B, the difference between the display delay of the CG image in the first video display process and the display delay of the CG image in the second video display process is three frames.

[0087] 7A and 7B illustrate a case where the display delay of the background image is greater than the display delay of the CG image (see, for example, the four-frame period from timing t1 to t5 and the two-frame period from timing t3 to t5 in FIG. 7A). The difference between the display delay of the background image and the display delay of the CG image is based on the difference in the amount of transmission data (i.e., communication time) described above. Note that in the second image display process, there may be no display delay in at least one of the background image and the CG image. The amount of display delay of the background image and the amount of display delay of the CG image may be a predetermined estimated value or a calculated value. The information processing system (display device 110 and information processing device 120) may calculate the amount of display delay of the background image and the amount of display delay of the CG image based on, for example, the amount of image data and the processing speed of each process (amount of data processed per unit time). Therefore, the amount of transmission data (i.e., communication time) described above may also be a predetermined estimated value or a calculated value. The information processing system (display device 110 and information processing device 120) may calculate the aforementioned amount of transmitted data (in other words, communication time) based on, for example, the amount of video data and the processing speed of each process (amount of data processed per unit time).

[0088] Therefore, this embodiment illustrates a case in which the effects of problems that occur when switching between two image display processes, a first image display process and a second image display process, are reduced. As described above, this embodiment differs from the first and second embodiments mainly in the process of switching between the first and second image display processes. Therefore, in the description of this embodiment, the same components as those in the first and second embodiments will be designated by the same reference numerals as those used in FIGS. 1 to 6, and detailed descriptions thereof will be omitted. For example, the hardware configurations of the display device 110 and the information processing device 120 are the same as those of the display device 110 and the information processing device 120 in FIG. 1 described in the first embodiment. Therefore, detailed descriptions of the hardware configurations of the display device 110 and the information processing device 120 will be omitted.

[0089] First, an example of a method for reducing the influence of a problem that occurs when switching the video display process from the first video display process to the second video display process will be described. A problem that occurs when switching from the first video display process to the second video display process is that the background video and CG video are not displayed by an amount corresponding to the difference in display delay between the first video display process and the second video display process. If this difference in display delay is small, the time during which the background video and CG video are not displayed when switching from the first video display process to the second video display process is also short, so the impact of the background video and CG video not being displayed is small. On the other hand, if this difference in display delay is large, the impact of the problem of the background video and CG video not being displayed is significant.

[0090] Therefore, in this embodiment, instead of switching the video display process within one frame, a case where the video display process is gradually switched over multiple frames is illustrated as an example. This makes it possible to prevent frame images with a large time difference from being displayed consecutively as background video and CG video (in other words, preventing large jumps in video) when switching the video display process.

[0091] For example, if there is a difference of n frames between the display delay of the background image in the first video display process and the display delay of the background image in the second video display process, the video display process is switched over a time period equivalent to n frames. As described above, FIG. 7A illustrates a case where n=4 (see timing t1 of the "second video display process (low delay)" and timing t5 of the "first video display process (high accuracy)"). In this case, as shown at timings t8 to t12 of the "first to second video display process (third form)" in FIG. 7A, the video display process is switched over a time period equivalent to four frames. FIG. 7B illustrates a case where n=5 (see timing t1 of the "second video display process (low delay)" and timing t6 of the "first video display process (high accuracy)"). In this case, as shown at timings t8 to t12 of the "first to second video display process (third form)" in FIG. 7B, the video display process is switched over a time period equivalent to five frames. In this embodiment, the process of switching the image display process from the first image display process to the second image display process is also referred to as a first image display switching process.

[0092] For example, the background video is skipped one frame at a time for each frame (thinning out the frame images one frame at a time). In this way, the difference in display delay between the two video display processes (first video display process and second video display process) can be bridged without significant skipping of the background video. In the "first to second video display process (third form)" of FIGS. 7A and 7B, the number of frame images that can be displayed in the first video display process is four. In this case, an example is shown in which the background video is skipped one frame at a time for each frame. Specifically, FIG. 7A illustrates an example in which the background videos with display orders of 5, 7, and 9 are skipped and displayed. FIG. 7B illustrates an example in which the background videos with display orders of 5, 7, 9, and 11 are skipped and displayed. Note that in FIGS. 7A and 7B, the background video is skipped one frame at a time in the first video switching process. Furthermore, skipping the background video by one frame may be determined based on the difference (=n) between the display delay of the background video in the first video display process and the display delay of the background video in the second video display process. For example, if the number of frame images that can be displayed in the first video display process is two, the background video may be displayed by skipping two frames for each frame.

[0093] Furthermore, when the difference in display delay between the CG images in the two video display processes is m frames (m≧1), the CG images are displayed by skipping one frame every n÷m frames. This prevents a decrease in the accuracy of combining the background image and the CG image when switching between the two video display processes. As described above, FIG. 7A illustrates a case where m=2 (see timing t3 of the "second video display process (low delay)" and timing t5 of the "first video display process (high accuracy)"). As described above, FIG. 7A also illustrates a case where n=4. Therefore, the "first → second video display process (third form)" in FIG. 7A illustrates a case where the CG images are displayed by skipping one frame every 2 (=4÷2) frames. Specifically, FIG. 7A illustrates a case where the CG images with display orders 4 and 7 are displayed by skipping one frame. FIG. 7B also illustrates the case where m=3 (see timing t3 in the "second video display process (low delay)" and timing t6 in the "first video display process (high accuracy)"). As mentioned above, FIG. 7B also illustrates the case where n=5. In this case, n÷m (=5÷3) is not an integer. Therefore, for example, the CG video may be alternately displayed by skipping one frame for every frame with the smallest integer greater than n÷m, and the CG video may be alternately displayed by skipping one frame for every frame with the largest integer smaller than n÷m. The "first-to-second video display process (third form)" in FIG. 7B illustrates the case where the CG video is alternately displayed by skipping one frame for every two frames, and then the CG video is alternately displayed by skipping one frame for every other frame. Specifically, FIG. 7B illustrates the case where the CG video images with the display order of 4, 6, and 9 are alternately displayed. In Fig. 7A, the fact that the CG video is displayed every two frames in the first video switching process is indicated by a bold frame. In Fig. 7B, the fact that the CG video is displayed every two frames in the first video switching process and every one frame in the first video switching process are alternately indicated by a bold frame.

[0094] Hereinafter, an example of the operation of the display device 110 and the information processing device 120 when switching the video display process from the first video display process to the second video display process will be outlined with reference to Fig. 8. Fig. 8 is a sequence diagram showing an example of the first video display switching process. Note that, here, an example of the process when switching the video display process from the first video display process to the second video display process will be described, assuming that the display device 110 and the information processing device 120 have the same hardware configuration as Fig. 1. Also, here, the same reference numerals as those used in Figs. 3 and 4 are used, and detailed description of the same processes as those described in the first video display process and the second video display process will be omitted.

[0095] In step S801, the display device 110 acquires the background video cached in step S301 from the memory 113. In this embodiment, an example is shown in which, in step S801, the display device 110 acquires the background video two frames after the background video displayed one frame before.

[0096] Next, in step S802, the display device 110 acquires the CG video cached in step S402 from the memory 113. In this embodiment, an example is shown in which, in step S802, the display device 110 acquires the CG video one frame after the CG video displayed in the previous frame, or the CG video two frames after the CG video displayed in the previous frame.

[0097] Next, in step S803, the display device 110 combines the background image acquired in step S801 with the CG image acquired in step S802. In the present embodiment (step S801 described above), an example has been given of a case where the background video is skipped every frame. Also, in the present embodiment (step S802 described above), an example has been given of a case where the CG video is skipped every two (=4÷2) frames. However, these are merely examples, and a method for solving the problem that occurs when switching the video display process from the first video display process to the second video display process described above may be a method different from the method described above.

[0098] Next, an example of a method for reducing the influence of a problem that occurs when switching the video display process from the second video display process to the first video display process will be described. A problem that occurs when switching the video display process from the second video display process to the first video display process is that the display of the background video and CG video stops (the display does not change) by an amount corresponding to the difference in display delay between the first video display process and the second video display process. If this difference in display delay is small, the period during which the display of the background video and CG video stops when the video display process switches from the second video display process to the first video display process is also short, and the impact of the stoppage of the display of the background video and CG video is small. However, if this difference in display delay is large, the impact of the problem of the stoppage of the display of the background video and CG video becomes significant.

[0099] Therefore, in this embodiment, the video display process is not switched within one frame, but is switched gradually over multiple frames, thereby preventing the background video and CG video from stopping for a long period of time when the video display process is switched.

[0100] For example, if there is a difference of n frames between the display delay of the background image in the first video display process and the display delay of the background image in the second video display process, the video display process is switched over a time period of 2n frames. As described above, FIG. 7A illustrates the case where n=4. In this case, as shown at timings t4 to t12 of the "second to first video display process (third form)" in FIG. 7A, the video display process is switched over a time period of 8 (=2×4) frames. FIG. 7B illustrates the case where n=5. In this case, as shown at timings t4 to t14 of the "second to first video display process (third form)" in FIG. 7B, the video display process is switched over a time period of 10 (=2×5) frames. In this embodiment, the process of switching the video display process from the second video display process to the first video display process is also referred to as the "second video display switching process."

[0101] For example, the same background video as the previous frame is displayed every two frames (in other words, the background video of the same frame is displayed for two frames each). This makes it possible to compensate for the difference in display delay between the two video display processes (first video display process and second video display process) without stopping the background video for a long period of time. The "second-to-first video display process (third form)" in FIG. 7A illustrates an example in which the background videos with the display order of 3, 4, 5, and 6 are displayed for two frames each. The "second-to-first video display process (third form)" in FIG. 7B illustrates an example in which the background videos with the display order of 3, 4, 5, 6, and 7 are displayed for two frames each. Note that in FIGS. 7A and 7B, the background video is displayed for two frames each in the second video switching display process, as indicated by a bold frame.

[0102] Furthermore, if the difference in display delay between the CG images in the two video display processes is m frames (m≧1), the same CG image as the previous frame is displayed every 2n÷m frames. This prevents a decrease in the accuracy of combining the background image and the CG image when switching between the two video display processes. The "second to first video display process (third form)" in FIG. 7A illustrates a case in which the same CG image as the previous frame is displayed every four (=2×4÷2) frames. Specifically, FIG. 7A illustrates a case in which CG images 3 and 6 in the display order are displayed for two frames. Also, in FIG. 7B, 2n÷m (2×5÷3) is not an integer. In this case, for example, the same CG image as the previous frame may be displayed alternately every frame with the largest integer smaller than 2n÷m, and every frame with the smallest integer larger than 2n÷m. The "second to first video display process (third form)" in Fig. 7B illustrates an example in which the same CG video as the previous frame is displayed every three frames, and the same CG video as the previous frame is displayed every four frames, alternating in this order. Specifically, Fig. 7B illustrates an example in which CG videos with display orders of 2, 5, and 7 are displayed for two frames. Note that Fig. 7A shows, with bold frames, that in the second video switching display process, the same CG video as the previous frame is displayed every three frames, and the same CG video as the previous frame is displayed every four frames, alternating in this order.

[0103] Hereinafter, an overview of an example of the operation of the display device 110 and the information processing device 120 when switching the video display process from the second video display process to the first video display process will be described with reference to Fig. 9. Fig. 9 is a sequence diagram showing an example of the second video display switching process. Note that, here, an example of the process when switching the video display process from the second video display process to the first video display process will be described, assuming that the display device 110 and the information processing device 120 have the same hardware configuration as Fig. 1. Also, here, the same reference numerals as those used in Figs. 3 and 4 are used, and detailed description of the same processes as those described in the first video display process and the second video display process will be omitted.

[0104] In step S901, the display device 110 acquires the background video cached in step S301 from the memory 113. In this embodiment, an example is shown in which, in step S901, the display device 110 acquires the same background video as the background video displayed one frame before, or the background video displayed one frame after the background video displayed one frame before.

[0105] Next, in step S902, the display device 110 acquires the CG video cached in step S402 from the memory 113. In this embodiment, an example is shown in which, in step S902, the display device 110 acquires the same CG video as the CG video displayed one frame before, or the CG video displayed one frame after the CG video displayed one frame before.

[0106] Next, in step S903, the display device 110 combines the background image acquired in step S901 with the CG image acquired in step S902.

[0107] In this embodiment (step S901 described above), an example has been given of a case where the same background image as the previous frame is displayed every two frames. Also, in this embodiment (step S902 described above), an example has been given of a case where the same CG image as the previous frame is displayed every 2n÷m frames. However, these are merely examples, and a method for solving the problem that occurs when switching the image display process from the second image display process described above to the first image display process may be a method different from the method described above.

[0108] Next, an example of the detailed operation of the display device 110 in this embodiment will be described with reference to Fig. 10. Fig. 9 is a flowchart illustrating an example of the overall flow of the operation of the display device 110. Note that the same reference numerals as those used in Fig. 5 are used here, and detailed description of the same processes as those of the display device 110 described in the first embodiment will be omitted.

[0109] If the result of the determination in step S505 is that the display device 110 (user) has not moved, the control unit 111 shifts the process to the process in step S1001. On the other hand, if the result of the determination in step S505 is that the display device 110 (user) has moved, the control unit 111 shifts the process to the process in step S1006, which will be described later.

[0110] In step S1001, the control unit 111 determines whether the user's movement state has changed from the user's movement state in the previous frame. The movement state indicates whether the user is determined to be moving and can be either a "not moving state" or a "moving state." In step S1001, the control unit 111 determines whether the user's movement state has changed from a "moving state" to a "not moving state." If the result of the determination in step S1001 is that the user's movement state has changed from a "moving state" to a "not moving state," the control unit 111 proceeds to processing in step S1002. On the other hand, if the result of the determination in step S1001 is that the user's movement state has not changed from a "moving state" to a "not moving state" (if it remains in a "not moving state"), the control unit 111 proceeds to processing in step S506. In this case, the processing in steps S1002 to S1005 is not performed. The processing in step S506 is the first video display processing described above.

[0111] In step S1002, the control unit 111 sets the value of the variable i, which identifies the order of the composite image (frame) to be displayed in the process of switching the image display process from the second image display process to the first image display process (second image display switching process), to an initial value (=0).

[0112] Next, in step S1003, control unit 111 performs second video display switching processing. The second video display switching processing is processing for switching the video display processing from the second video display processing to the first video display processing. In one execution of step S1003, a composite video (frame) to be displayed (i+1)th when switching the video display processing from the second video display processing to the first video display processing is generated, and the composite video is displayed. The processing of step S1003 corresponds to the processing of steps S301 to S307 in FIG. 9. Furthermore, the processing of step S1003 is repeatedly executed a number of frames equal to twice the difference (=n) between the display delay of the background video in the first video display processing and the display delay of the background video in the second video display processing.

[0113] Next, in step S1004, control unit 111 determines whether the value of variable i is 2n. If the result of the determination in step S1004 is that the value of variable i is not 2n, control unit 111 shifts the process to step S1005. In step S1005, control unit 111 increments variable i. Then, control unit 111 shifts the process to the process of step S1003 described above. In step S1003, control unit 111 generates a composite video (frame) to be displayed as the (i+1)th composite video after the increment, and displays the composite video. The processes of steps S1003 to S1005 are repeatedly executed for the number of frames equal to twice the difference (=n) between the display delay of the background video in the first video display process and the display delay of the background video in the second video display process.

[0114] If it is determined in step S1004 that the value of the variable i is 2n, the control unit 111 advances the process to step S506 (first video display process).

[0115] In step S1006, the control unit 111 determines whether the user's movement state has changed from the user's movement state in the previous frame. In step S1006, the control unit 111 determines whether the user's movement state has changed from a "not moving state" to a "moving state."

[0116] If the result of the determination in step S1006 is that the user's movement state has changed from "not moving" to "moving," control unit 111 shifts the process to step S1002. On the other hand, if the result of the determination in step S1006 is that the user's movement state has not changed from "not moving" to "moving" (if it remains "moving"), control unit 111 shifts the process to step S507. In this case, the processes of steps S1007 to S1009 are not performed. The process of step S507 is the second video display process described above.

[0117] In step S1007, the control unit 111 sets the value of the variable j, which identifies the order of the composite image (frame) to be displayed in the process of switching the image display process from the first image display process to the second image display process (first image display switching process), to an initial value (=0).

[0118] Next, in step S1008, control unit 111 performs a first video display switching process. The first video display switching process is a process for switching the video display process from the first video display process to the second video display process. In one execution of step S1008, a composite video (frame) to be displayed as the (j+1)th video when switching the video display process from the first video display process to the second video display process is generated, and the composite video is displayed. The process of step S1008 corresponds to the processes of steps S301 to S307 in FIG. 7. Furthermore, the process of step S1008 is repeatedly executed a number of times equal to the difference (=n) between the display delay of the background video in the first video display process and the display delay of the background video in the second video display process.

[0119] Next, in step S1009, control unit 111 determines whether the value of variable j is n. If the result of the determination in step S1009 is that the value of variable j is not n, control unit 111 proceeds to the processing of step S1010. In step S1010, control unit 111 increments variable j. Then, control unit 111 proceeds to the processing of step S1008 described above. In step S1008, control unit 111 generates a composite video (frame) to be displayed as the j+1th variable after the increment, and displays the composite video. The processing of steps S1008 to S1010 is repeatedly executed the number of frames equal to the difference (=n) between the display delay of the background video in the first video display processing and the display delay of the background video in the second video display processing.

[0120] If it is determined in step S1009 that the value of the variable j is n, the control unit 111 advances the process to step S507 (second video display process).

[0121] As described above, when switching between two video display processes, the information processing system of this embodiment generates a composite image having a display delay smaller than that of the first video display process and larger than that of the second video display process as an example of a third composite image and displays it on the display device 110. Therefore, rather than switching between the two video display processes within one frame, the two video display processes can be switched gradually over a period of multiple frames. This makes it possible to prevent the background image and CG image from skipping significantly or stopping for a long period of time when switching between the two video display processes.

[0122] This reduces the effects of problems that occur when switching between the first image display process and the second image display process, further suppressing degradation in the quality of the user experience and further improving the safety of the user wearing the display device 110. In this embodiment, an example of a method for reducing the influence of each problem that occurs when switching between two video display processes has been described based on the first embodiment. However, the method described in this embodiment can also be applied to the second embodiment.

[0123] Furthermore, in the first to third embodiments, the image display process is switched to simultaneously improve the safety of the user when wearing the display device 110 (head-mounted display) and the quality of the user experience. In addition to this, for example, the information processing system (e.g., the display device 110) may output a composite image to an external device (such as a terminal device). Also, for example, the information processing system (e.g., the display device 110) may record the composite image using a function such as screen capture. Regardless of the image display process used for display on the display device 110, the information processing system (e.g., the display device 110) may always use the composite image generated by the first image display process for output and recording as the composite image for external output and recording. In this way, a composite image with high composite accuracy between the background image and the CG image can always be used for output and recording. This reduces the impact on the output image and the recorded image caused by switching between the two image display processes, the first image display process and the second image display process.

[0124] (Other Examples) The present disclosure can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. Furthermore, the above-described embodiments of the present disclosure are merely examples of specific embodiments for carrying out the present disclosure, and the technical scope of the present disclosure should not be interpreted as being limited by these. In other words, the present disclosure can be embodied in various forms without departing from its technical concept or main features.

[0125] The disclosure of the present embodiment also includes, for example, the following configurations, methods, and programs. (Configuration 1) A processing device that performs processing for displaying information on a display means worn by a user, a display processing means for performing processing to display on the display means a composite image obtained by combining a plurality of images including an image captured at a viewpoint corresponding to the user's viewpoint; The display processing means causes the display means to display, depending on the movement state of the display means, a first composite image formed by combining captured images having a relatively long delay time from capture to display, or a second composite image formed by combining captured images having a relatively short delay time. (Configuration 2) The processing device according to configuration 1, characterized in that the display processing means displays the first composite image on the display means when the display means is not moving, and displays the second composite image on the display means when the display means is moving. (Configuration 3) 3. The processing device according to configuration 1 or 2, further comprising an acquisition unit for acquiring the first composite image from an external device. (Configuration 4) 4. The processing device according to configuration 3, wherein the acquisition means acquires the first composite image from the external device when the display means is not moving. (Configuration 5) 5. The processing device according to any one of configurations 1 to 4, wherein the processing device comprises a generating means for generating the second composite image. (Configuration 6) 6. The processing device according to configuration 5, wherein the generating means generates the second composite image when the display means is moving. (Configuration 7) 7. The processing device according to any one of configurations 1 to 6, further comprising a detection means for detecting a moving state of the display means. (Configuration 8) The processing device according to any one of configurations 1 to 7, wherein the display processing means displays the first composite image or the second composite image on the display means depending on the movement state of the display means and the position of the display means. (Configuration 9) The processing device according to configuration 8, wherein the display processing means causes the display means to display the first composite image when the display means is moving and is located within an area determined based on the position of an image to be composited with the captured image, and causes the display means to display the second composite image when the display means is moving and is not located within the area. (Configuration 10) the composite image includes the first composite image, the second composite image, and a third composite image that is displayed when display of the first composite image and the second composite image is switched, The processing device according to any one of configurations 1 to 9, wherein the third composite image is a composite image obtained by combining captured images captured at different times relative to the captured images used to generate the first composite image and the second composite image. (Configuration 11) The processing device according to configuration 10, characterized in that the timing of capturing the captured image used to generate the third composite image is later than the timing of capturing the captured image used to generate the first composite image to be displayed, out of the first composite image and the second composite image, and is earlier than the timing of capturing the captured image used to generate the second composite image to be displayed, out of the first composite image and the second composite image. (Configuration 12) The processing device according to configuration 10 or 11, characterized in that the display processing means causes the display means to display the third composite image for a period based on a difference between a delay time from capturing the captured image used for the first composite image to displaying it and a delay time from capturing the captured image used for the second composite image to displaying it. (Configuration 13) 13. The processing device according to any one of configurations 10 to 12, further comprising a determination means for determining an image to be used in generating the third composite image. (Configuration 14) The processing device described in configuration 13, characterized in that the determination means determines the captured image to be used to generate the third composite image based on a difference between a delay time from capture to display of the captured image used for the first composite image and a delay time from capture to display of the captured image used for the second composite image. (Configuration 15) The processing device according to configuration 13 or 14, characterized in that the determination means determines an image to be composited with the captured image in the third composite image based on a difference between a delay time from capturing to displaying the captured image used in the first composite image and a delay time from capturing to displaying the captured image used in the second composite image, and a difference between a delay time from generating to displaying an image to be composited with the captured image in the first composite image and a delay time from generating to displaying an image to be composited with the captured image in the second composite image. (Configuration 16) a second generating means for generating the composite image corresponding to the composite image displayed on the display means; 16. The processing device according to any one of configurations 1 to 15, wherein the second generating means generates the first composite image regardless of the movement state of the display means. (Configuration 17) 17. The processing device according to any one of configurations 1 to 16, wherein the plurality of images include the captured images captured by an imaging means and CG (Computer Graphics). (Configuration 18) 18. The processing device according to any one of configurations 1 to 17, comprising at least one of an imaging means for capturing the captured image and the display means. (Configuration 19) The processing device according to any one of configurations 1 to 18, characterized in that the time difference between the timing of capturing the captured image in the first composite image and the timing of generating the image to be composited with the captured image is smaller than the time difference between the timing of capturing the captured image in the second composite image and the timing of generating the image to be composited with the captured image. (Configuration 20) An information processing system including the processing device according to any one of configurations 1 to 19 and an information processing device, the information processing device has a third generation means for generating the first composite image, The information processing system is characterized in that the processing device comprises an acquisition means for acquiring the first composite image generated by the third generation means. (Method 1) A processing method for performing processing for displaying on a display means worn by a user, a display processing step of performing processing for displaying on the display means a composite image obtained by combining a plurality of images including an image captured at a viewpoint corresponding to the user's viewpoint; The display processing step is a processing method characterized in that the display means displays, depending on the movement state of the display means, a first composite image formed by combining captured images having a relatively long delay time from capture to display, or a second composite image formed by combining captured images having a relatively short delay time. (Program 1) 20. A program for causing a computer to function as the display processing means of the processing device according to any one of configurations 1 to 19. [Explanation of symbols]

[0126] 110: Display device, 120: Information processing device

Claims

1. A processing device that performs processing for displaying information on a display means worn by a user, a display processing means for performing processing to display on the display means a composite image obtained by combining a plurality of images including an image captured at a viewpoint corresponding to the user's viewpoint; The display processing means displays, on the display means, a first composite image formed by combining captured images having a relatively long delay time from capture to display, or a second composite image formed by combining captured images having a relatively short delay time, depending on the movement state of the display means.

2. 2. The processing device according to claim 1, wherein the display processing means displays the first composite image on the display means when the display means is not moving, and displays the second composite image on the display means when the display means is moving.

3. 3. The processing device according to claim 1, further comprising: an acquisition unit for acquiring the first composite image from an external device.

4. 4. The processing device according to claim 3, wherein the acquisition means acquires the first composite image from the external device when the display means is not moving.

5. 3. The processing device according to claim 1, further comprising a generating unit for generating the second composite image.

6. 6. The processing device according to claim 5, wherein said generating means generates said second composite image when said display means is moving.

7. 3. The processing device according to claim 1, further comprising a detection means for detecting a movement state of said display means.

8. 3. The processing device according to claim 1, wherein the display processing means displays the first composite image or the second composite image on the display means in accordance with a movement state of the display means and a position of the display means.

9. The processing device according to claim 8, characterized in that the display processing means displays the first composite image on the display means when the display means is moving and is located within an area determined based on the position of the image to be composited with the captured image, and displays the second composite image on the display means when the display means is moving and is not located within the area.

10. the composite image includes the first composite image, the second composite image, and a third composite image that is displayed when display of the first composite image and the second composite image is switched, 3. The processing device according to claim 1, wherein the third composite image is a composite image obtained by combining captured images captured at different times relative to the captured images used to generate the first composite image and the second composite image.

11. 11. The processing device according to claim 10, wherein the timing of capturing the captured image used to generate the third composite image is later than the timing of capturing the captured image used to generate the first composite image to be displayed, of the first composite image and the second composite image, and earlier than the timing of capturing the captured image used to generate the second composite image to be displayed, of the first composite image and the second composite image.

12. The processing device according to claim 10, characterized in that the display processing means causes the display means to display the third composite image for a period based on a difference between a delay time from capturing the captured image used for the first composite image to displaying it and a delay time from capturing the captured image used for the second composite image to displaying it.

13. 11. The processing device of claim 10, further comprising: determining means for determining which images are to be used to generate the third composite image.

14. The processing device according to claim 13, characterized in that the determination means determines the captured image to be used to generate the third composite image based on a difference between a delay time from capture to display of the captured image used for the first composite image and a delay time from capture to display of the captured image used for the second composite image.

15. The processing device according to claim 13, characterized in that the determination means determines the image to be combined with the captured image in the third composite image based on a difference between a delay time from capturing to displaying the captured image used in the first composite image and a delay time from capturing to displaying the captured image used in the second composite image, and a difference between a delay time from generating to displaying the image to be combined with the captured image in the first composite image and a delay time from generating to displaying the image to be combined with the captured image in the second composite image.

16. a second generating means for generating the composite image corresponding to the composite image displayed on the display means; 3. The processing device according to claim 1, wherein the second generating means generates the first composite image regardless of the movement state of the display means.

17. 3. The processing device according to claim 1, wherein the plurality of images include the captured image captured by an imaging unit and CG (Computer Graphics).

18. 3. The processing device according to claim 1, further comprising at least one of an imaging unit for capturing the captured image and the display unit.

19. 3. The processing device according to claim 1, wherein a time difference between the timing of capturing the captured image in the first composite image and the timing of generating the image to be composited with the captured image is smaller than a time difference between the timing of capturing the captured image in the second composite image and the timing of generating the image to be composited with the captured image.

20. 3. An information processing system comprising the processing device according to claim 1 or 2 and an information processing device, the information processing device has a third generation means for generating the first composite image, The information processing system is characterized in that the processing device includes an acquisition means for acquiring the first composite image generated by the third generation means.

21. A processing method for performing processing for displaying on a display means worn by a user, a display processing step of performing processing for displaying on the display means a composite image obtained by combining a plurality of images including an image captured at a viewpoint corresponding to the user's viewpoint; The display processing step is a processing method characterized in that, depending on the movement state of the display means, the display means displays a first composite image formed by combining captured images having a relatively long delay time from capture to display, or a second composite image formed by combining captured images having a relatively short delay time.

22. 3. A program for causing a computer to function as the display processing means of the processing device according to claim 1.

Images