Electronic device, display device, method for controlling electronic device, and program

Abstract

This technology provides the ability to send and receive data of virtual objects using common communication methods. [Solution] The electronic device of the present invention is characterized by comprising: a first acquisition means for acquiring position and orientation information of a display device; a generation means for generating a single image data including data of a virtual object representing a virtual object, transparency data of the virtual object image, and depth data of the virtual object, based on the information acquired by the first acquisition means; and a transmission means for transmitting the single image data generated by the generation means to an external party.

Description

[Technical Field]

[0001] The present invention relates to an electronic device, a display device, a control method for an electronic device, and a program. [Background Art]

[0002] As technologies for fusing the real world and the virtual world in real time and seamlessly, Mixed Reality (MR) technology and Augmented Reality (AR) technology are known. These are expected to be applied to various fields, such as assembly support that superimposes and displays work procedures and wiring conditions during assembly work, and surgical support that superimposes and displays the internal state of the body on the patient's body surface. Furthermore, Virtual Reality (VR) technology is known as a technology that uses computer technology to provide a virtual environment and experience different from reality. A user can experience a virtual reality space using a Head Mounted Display (HMD) or a controller, thereby obtaining a sensation as if they are actually at that place and in that situation.

[0003] To make a user feel as if a virtual object actually exists, a video see-through type device is sometimes used. The video see-through type device captures an image of the real space with a camera, and displays a composite image obtained by superimposing an image of the virtual object on an image of the real space (background image) on a display unit such as a display in real time. As such a device, a portable information terminal having a camera on the back surface (for example, a tablet terminal) or a video see-through type HMD equipped with a camera is used.

[0004] In video-see-through HMDs, images acquired from the mounted camera, along with information regarding the camera's position and orientation, may be transmitted to an information processing unit. In this case, the information processing unit performs a composite process, overlaying an image of a virtual object onto the background image, and transmits the composite image to the HMD. The HMD then displays the received composite image to the user. Typically, the large data size of the composite image leads to a large amount of communication between the information processing unit and the HMD, which can cause delays in displaying the composite image.

[0005] Patent Document 1 discloses a technique for reducing the amount of data transmitted by performing frame decimation on composite images. [Prior art documents] [Patent Documents]

[0006] [Patent Document 1] Japanese Patent Publication No. 2019-062397 [Patent Document 2] Japanese Patent Publication No. 2013-134706 [Overview of the project] [Problems that the invention aims to solve]

[0007] However, the technology disclosed in Patent Document 1 results in a decrease in frame rate. Therefore, a configuration in which data of the virtual object before synthesis is transmitted to the HMD and the synthesis process is performed on the HMD can also be considered. With this configuration, the amount of data of the virtual object is smaller than the amount of data of the synthesized image, so the amount of communication between the information processing device and the HMD can be reduced. However, in order to correctly represent occlusion, the virtual object data requires data of the virtual object image, transparency information of the virtual object image, and depth information of the virtual object, and a special communication standard (communication method) must be used to adopt the above configuration.

[0008] The present invention aims to provide a technology that enables the transmission and reception of virtual object data using a general communication standard. [Means for solving the problem]

[0009] The electronic device of the present invention is characterized by comprising: a first acquisition means for acquiring position and orientation information of a display device; a generation means for generating a single image data including virtual object image data representing a virtual object, transparency data of the virtual object image, and depth data of the virtual object, based on the information acquired by the first acquisition means; and a transmission means for transmitting the single image data generated by the generation means to an external party. [Effects of the Invention]

[0010] According to the present invention, data of virtual objects can be transmitted and received using a general communication method. [Brief explanation of the drawing]

[0011] [Figure 1] This is a block diagram of the system in Embodiment 1. [Figure 2] This is a block diagram of the information processing device in Embodiment 1. [Figure 3] This is a block diagram of the HMD in Embodiment 1. [Figure 4] This is a flowchart of the information processing device in Embodiment 1. [Figure 5] This is a flowchart of the HMD in Embodiment 1. [Figure 6] This is a schematic diagram showing the data of a virtual object in Embodiment 1. [Figure 7] This is a flowchart of the information processing device in Embodiment 2. [Figure 8] This is a schematic diagram showing the data of a virtual object in Embodiment 2. [Figure 9] This is a sequence diagram of the system in Embodiment 3. [Figure 10] This is a schematic diagram of the screen in Embodiment 3. [Figure 11] This is a sequence diagram of the system in Embodiment 4. [Figure 12] It is a schematic diagram of a screen according to the fourth embodiment. Mode for Carrying Out the Invention

[0012] <Embodiment 1> FIG. 1 is a block diagram showing the configuration of a mixed reality system according to Embodiment 1. An HMD 100 (display device) includes an imaging unit 101, a depth information generation unit 102, a composite image generation unit 103, and a display unit 104. An information processing apparatus 108 (electronic device) includes a position and orientation information generation unit 105, a rendering unit 106, and an output image generation unit 107.

[0013] The HMD 100 is an example of a video see-through type display device. The video see-through type display device only needs to be configured by the imaging unit 101, the depth information generation unit 102, the composite image generation unit 103, and the display unit 104. The video see-through type display device may be, for example, a portable information terminal such as a tablet or a smartphone, or a display device such as a handheld display (HHD). The HMD 100 transmits a background image (described later) to the position and orientation information generation unit 105 of the information processing apparatus 108. In Embodiment 1, it is assumed that the HMD 100 and the information processing apparatus 108 are connected by wire, but the HMD 100 and the information processing apparatus 108 may be connected wirelessly.

[0014] A user uses the HMD 100 by wearing it on their head. Although the HMD 100 has an information processing function, instead of the HMD 100, an HMD specialized for an imaging function and a display function, and an information processing apparatus specialized for an information processing function may be used.

[0015] The imaging unit 101 includes a camera (imaging apparatus) that continuously captures images of a real space. In Embodiment 1, an image of the real space (real space image) captured by the camera is used as the background image. However, the background image is not limited to a real-time real space image, for example, a pre-stored storage medium It may be a moving image or a still image which is a real-space image stored therein. The background image may be a moving image or a still image representing a virtual space, or the like. The background image captured by the imaging unit 101 is input to a depth information generation unit 102 and a position and orientation information generation unit 105 of an information processing apparatus 108. For example, the imaging unit 101 includes a USB interface board, and outputs the background image to the outside from the USB interface board. In the first embodiment, it is assumed that the camera included in the imaging unit 101 is a stereo camera including two cameras corresponding to the user's left and right eyes. However, the camera included in the imaging unit 101 is not particularly limited, and may be, for example, a monocular camera.

[0016] The depth information generation unit 102 acquires depth information from the image captured by the imaging unit 101. For example, depth information of a user's hand may be acquired, or depth information of an object held by the hand may be acquired. The depth information is used to correctly represent occlusion when generating a composite image (described later). Various known methods can be used for this purpose. The method described in Patent Document 2 may be used for hand detection. In the method described in Patent Document 2, it is necessary to register color information in advance. Therefore, color information may be registered in advance in a storage medium provided in the HMD 100 or the information processing apparatus 108, and the depth information generation unit 102 may read out the color information. Alternatively, depth information may be acquired using a stereo camera. The method is not limited to these, as long as depth information can be acquired.

[0017] A composite image generation unit 103 generates a composite image (composite image data) by compositing (superimposing) a virtual object image (an image representing a virtual object) received from the information processing apparatus 108 onto the background image captured by the imaging unit 101. In the composite image, pixels other than the region of the virtual object are pixels of the background image (real-space image).

[0018] The display unit 104 is a display element provided in the HMD 100 and displays video based on the composite image generated by the composite image generation unit 103. The display element used in the display unit 104 can be an EL (Electro Luminescence) panel, an LCD, or the like, but is not limited to these. The display unit 104 can take any form as long as it can display the composite image.

[0019] The position and orientation information generation unit 105 generates position and orientation information (position and orientation information) of the camera (imaging device) included in the imaging unit 101. The position and orientation information of the imaging device may be generated based on a background image or based on an optical sensor. This position and orientation information may be interpreted as position and orientation information of the HMD 100. When using an optical sensor, it is necessary to determine in advance the relative position and orientation of the HMD 100 (imaging device) with respect to the optical sensor. In Embodiment 1, position and orientation information is estimated (acquired) from the background image. The position and orientation information generation unit 105 detects features from the background image repeatedly captured by the imaging unit 101. In feature detection, for example, feature points having a brightness gradient are detected. Here, the position and orientation information generation unit 105 detects a predetermined number or fewer feature points, or selects and uses a predetermined number or fewer feature points from the detected feature points. Details of feature detection will be described later.

[0020] The rendering unit 106 renders virtual objects to be superimposed on the background image captured by the imaging unit 101.

[0021] The output image generation unit 107 generates a single image data (output image) based on the rendering results from the rendering unit 106, which includes data for a virtual object image representing a virtual object, transparency data for the virtual object image, and depth data for the virtual object. The transparency data and depth data are used to generate a composite image (described later). The transparency data is used to correctly represent light transmission, etc., and the depth data is used to correctly represent occlusion. Various known methods can be used for using this data. Cut.

[0022] Figure 2 is a block diagram showing the hardware configuration of the information processing device 108. The CPU 200 comprehensively controls each component of the information processing device 108, which is connected via the bus 207. The CPU 200 realizes the functions of the position and orientation information generation unit 105, the rendering unit 106, and the output image generation unit 107 by reading and executing programs stored in the ROM 202 (read-only memory). The ROM 202 stores the operating system (OS), various programs related to Embodiment 1, device drivers, etc. The programs stored in the ROM 202 are temporarily stored in the RAM 201 (random access memory) and executed by the CPU 200. The keyboard 203 and mouse 204, which are connected as input interfaces, accept user operations on the information processing device 108. The interface 205 receives input signals from external devices (such as display devices) in a format that the information processing device 108 can process, and transmits output signals to external devices in a format that the device can process. For example, I / F205 receives image data and other data to be used in processing by the information processing device 108 from the HMD100 in a format that the information processing device 108 can process. I / F205 also transmits image data to be displayed on the HMD100 in a format that the HMD100 can process. In Embodiment 1, the DisplayPort standard is used for communication via I / F205, but the communication standard used is not limited to this. For example, the HDMI® standard may be used.

[0023] Figure 3 is a block diagram showing the hardware configuration of the HMD100. The CPU 300 comprehensively controls each component of the HMD100, which is connected via the bus 305. The CPU 300 realizes the functions of the depth information generation unit 102 and the composite image generation unit 103 by reading and executing programs stored in the ROM 302 (read-only memory). The ROM 302 stores the operating system (OS), various processing programs related to Embodiment 1, device drivers, etc. The programs stored in the ROM 302 are temporarily stored in the RAM 201 (random access memory) and executed by the CPU 300. The I / F 304 receives input signals from external devices in a format that the HMD100 can process, and transmits output signals to external devices in a format that the device can process. For example, the I / F 304 receives image data for display on the HMD100 from the information processing device 108 in a format that the HMD100 can process. Furthermore, the I / F304 transmits image data and other data used for processing by the information processing device 108 to the information processing device 108 in a format that the information processing device 108 can process. In Embodiment 1, the DisplayPort standard is used for communication via the I / F304, but the communication standard used is not limited to this.

[0024] Figure 4 is a flowchart of the information processing device 108. Each process shown in Figure 4 is realized by the CPU 200 loading the program stored in ROM 202 into RAM 201 and executing it. For example, the operation shown in Figure 4 may be initiated when the user performs a predetermined operation, such as launching an application for experiencing a mixed reality (MR) space using HMD 100.

[0025] In S401, the CPU 200 (position and orientation information generation unit 105) uses the background image to acquire (estimate) the position and orientation information of the HMD 100 (imaging device). In Embodiment 1, the position and orientation information consists of a set of six parameters: three parameters representing the position and three parameters representing its orientation (direction).

[0026] First, the position and orientation information generation unit 105 receives the background image captured by the imaging unit 101 from the HMD 100. The position and orientation information generation unit 105 performs feature detection on the received background image. Feature detection refers to detecting the coordinates of feature points in the image. Here, we will explain the process of detecting feature points from an image. For example, if the brightness gradient between neighboring pixels is greater than or equal to a threshold... Points are defined as feature points. The luminance gradient is the change in density between adjacent pixels in the image. Luminance gradient detection is performed using known edge detection operators such as the Sobel operator or the Prewitt operator. For each pixel, the edge detection operator is applied to the horizontal and vertical directions of the image. Then, the edge intensity (luminance gradient) is calculated based on the output values. For a given pixel, the horizontal output value of the edge detection operator is f x The vertical output value is f y In this case, the edge intensity I at the pixel is calculated using equation (1).

number

[0027] The position and orientation information generation unit 105 then estimates the position and orientation of the HMD 100 based on the detected feature points and generates position and orientation information. Any existing method can be used to estimate the position and orientation. In Embodiment 1, the position and orientation at the time of imaging are estimated using prediction such as EKF (Extended Kalman Filter). The position and orientation information generation unit 105 uses the difference Δt from the timing t-1 when the previous image was captured to the timing t when the current image was captured, and the post-internal state of the EKF at timing t-1 to determine the pre-internal state of the EKF at timing t. Note that the method is not particularly limited as long as it is possible to estimate the position and orientation at the time of imaging.

[0028] In S402, the CPU 200 (rendering unit 106) renders the virtual object. To determine the position and orientation of the virtual object, the rendering unit 106 uses the position and orientation information estimated in S401. The 3D engine used by the rendering unit 106 to render the virtual object is not particularly limited. As a result of rendering, the rendering unit 106 generates RGBA data containing color information and transparency information, and Z data containing depth information. RGBA is a color model that adds Alpha (transparency) to the three primary colors of Red, Green, and Blue. By specifying the intensity and transparency of each color, it is possible to represent a wide range of colors, including transparency. Hereafter, color information (data of the virtual object image) will be referred to as RGB data, transparency information (transparency data) as A data, and depth information (depth data) as Z data. However, the color model of the color information is not limited to RGB, and may be another color model such as GBR. Figure 6A is a schematic diagram showing an example of RGBA data and Z data. Since the HMD100 of Embodiment 1 has a camera for the right eye and a camera for the left eye, two sets of RGBA data are generated: RGBA data for the right eye (R) and RGBA data for the left eye (L). Similarly, two sets of Z data are generated: Z data for the right eye (R) and Z data for the left eye (L).

[0029] In S403, the CPU 200 (output image generation unit 107) splits the RGBA data generated in S402 into RGB data and A data. Figure 6B is a schematic diagram showing an example of RGB data, A data, and Z data.

[0030] In S404, the CPU 200 (output image generation unit 107) generates AZ data representing transparency and depth by combining the A data obtained in S403 and the Z data obtained in S402. The processing in S403 and S404 yields RGB data and AZ data. Figure 6C is a schematic diagram showing an example of RGB data and AZ data.

[0031] In S405, the CPU 200 (output image generation unit 107) generates an output image containing RGB data, A data, and Z data. Here, the output image is generated by arranging the RGB data obtained in S403 and the AZ data obtained in S404. Figure 6D is a schematic diagram showing an example of an output image with RGB data and AZ data arranged side by side. The resolution (image size) of the output image generated in S405 may be arbitrarily selected by the user, or The resolution may be a fixed resolution predetermined by the manufacturer.

[0032] In S406, the CPU 200 (output image generation unit 107) determines whether the resolution (image size) of the output image generated in S405 is a resolution that can be transmitted via the I / F 205. This determination can be interpreted as determining whether the resolution of the output image is below a threshold. In Embodiment 1, the DisplayPort standard is used for communication via the I / F 205. Depending on the image resolution, it may not be possible to transmit using the DisplayPort standard. If the CPU 200 determines that the resolution (image size) of the output image is a resolution that can be transmitted via the I / F 205 (a resolution that can be transmitted using the DisplayPort standard) (i.e., the resolution of the output image is below a threshold), the CPU 200 proceeds to S408. Otherwise, the CPU 200 proceeds to S407.

[0033] In S407, the CPU200 (output image generation unit 107) performs a reduction process on the output image (RGB data, AZ data, or both). The output image generation unit 107 reduces the resolution of the output image to a resolution that can be transmitted via the I / F205 (a resolution below the threshold mentioned above). The method of reduction is not particularly limited; for example, the reduction process can be performed using a graphics API. Figure 6E is a schematic diagram showing an example of a reduced output image. In Figure 6E, the horizontal size (horizontal dimensions) of the output image in Figure 6D has been reduced (compressed) to half its original size.

[0034] In S408, the CPU200 (output image generation unit 107) transmits the output image to the HMD100 via the I / F205. If the processing in S407 is performed, the reduced output image is transmitted.

[0035] When using the DisplayPort standard, the threshold for S406 may be the resolution defined in the DisplayPort standard. Since the upper limit resolution of the DisplayPort 1.0 standard is 3840 x 2160, the threshold for S406 may be set to a resolution of 3840 x 2160 or lower, and the resolution of the output image may be reduced to a resolution of 3840 x 2160 or lower in S407. By doing so, transmission to the HMD100 will be possible regardless of which generation of the DisplayPort standard is used.

[0036] Furthermore, while we have explained an example of arranging RGB data and AZ data horizontally, the arrangement of RGB data and AZ data is not particularly limited. Figure 6F is a schematic diagram showing an example of an output image in which RGB data and AZ data are arranged vertically.

[0037] Figure 5 is a flowchart of the HMD100. Each process shown in Figure 5 is realized by the CPU300 loading the program stored in ROM302 into RAM301 and executing it.

[0038] In S501, the CPU 300 (synthetic image generation unit 103) receives the output image generated by the output image generation unit 107 from the information processing device 108 via the I / F 304.

[0039] In S502, the CPU 300 (composite image generation unit 103) processes the output image received from the information processing device 108 back to its original state (the state immediately after rendering). The composite image generation unit 103 divides the output image into RGBA data and Z data. If a reduced output image is received, the composite image generation unit 103 performs an enlargement process on the output image and divides the enlarged output image into RGBA data and Z data. The method of enlargement is not particularly limited; for example, pixel data interpolation using a shader may be performed as part of the enlargement process. The process in S502 is the reverse of the process in S403 to S407 in Figure 4.

[0040] In S503, the CPU300 (synthetic image generation unit 103) processes the RG generated in S502. A composite image is generated using BA data, Z data, a background image captured by the imaging unit 101, and depth information generated by the depth information generation unit 102. The generated composite image is displayed on the display unit 104. This allows the user to be presented with a composite image in which a virtual object image is superimposed on the background image, enabling the user to experience the MR space.

[0041] As described above, in Embodiment 1, data of virtual objects can be transmitted and received using a general (general-purpose) communication standard. Therefore, highly versatile devices (such as HMDs and information processing devices) can be used in the system, and the system can be realized at a low cost. Furthermore, a system with distributed processing load can be realized. Although an example of applying the present invention to a mixed reality system has been described, the present invention can also be applied to augmented reality systems, virtual reality systems, and the like.

[0042] <Embodiment 2> In Embodiment 1, AZ data was generated by combining A data and Z data, and the RGB data and AZ data were arranged side by side. However, in Embodiment 2, the RGB data, A data, and Z data are arranged side by side without combining the A data and Z data. In the description of Embodiment 2, explanations similar to those in Embodiment 1 will be omitted.

[0043] Figure 7 is a flowchart of the information processing device 108 in Embodiment 2. In Figure 7, the processing of S404 in Figure 4 is omitted, and the processing of S701 is performed instead of the processing of S405. In S701, the CPU 200 (output image generation unit 107) generates an output image that includes RGB data, A data, and Z data. Here, the output image is generated by arranging the RGB data, A data, and Z data obtained by the processing of S402 and S403. Figure 8 is a schematic diagram showing an example of an output image in which RGB data, A data, and Z data are arranged. In Figure 8, an example is shown in which the RGB data, A data, and Z data are arranged horizontally, but the arrangement of the RGB data, A data, and Z data is not particularly limited.

[0044] As described above, in Embodiment 2, similar to Embodiment 1, data of virtual objects can be sent and received using a general (general-purpose) communication standard.

[0045] <Embodiment 3> In Embodiment 1, an output image with a set resolution (for example, a resolution arbitrarily selected by the user) was generated. However, the HMD100 may not be able to receive or process the output image with the set resolution (the HMD100 may not be able to display a video (composite image) based on the output image). In Embodiment 3, the user is prompted to change the resolution of the output image so that the HMD100 can display a video (composite image) based on the output image.

[0046] Figure 9 is a sequence diagram of the HMD 100 and the information processing device 108 in Embodiment 3.

[0047] In the S901, the CPU 200 (output image generation unit 107) sets the resolution of the output image according to user operation. The resolution is not particularly limited. For example, a pull-down menu or an input box for directly entering a resolution value may be displayed on the display connected to the information processing device 108. In that case, the user can specify (set) the resolution using the keyboard 203 and mouse 204. Setting the resolution of the output image may also be interpreted as setting the resolution of a virtual object.

[0048] In the S902, CPU200 (rendering unit 106) renders a virtual object for testing. Dulling is performed. The test virtual image is used to allow the user to confirm whether or not the virtual object is displayed on the HMD100. Then, the CPU200 (output image generation unit 107) generates an output image including the image of the test virtual object at the set resolution and sends it to the HMD100.

[0049] In the S903, the CPU 300 (synthetic image generation unit 103) displays the image (video) on the display unit 104. If the synthetic image generation unit 103 can receive and process the output image output from the information processing device 108, it displays a synthetic image based on the output image on the display unit 104. However, depending on the resolution of the output image, the synthetic image based on the output image may not be displayed on the display unit 104.

[0050] In S904, the CPU 200 controls the system to display a predetermined notification prompting the user to confirm whether the composite image is displayed on the display unit 104 (whether the virtual object is displayed on the display unit 104). For example, the CPU 200 controls the system to display a predetermined message on the display connected to the information processing device 108. Figure 10 is a schematic diagram of the confirmation screen displayed on the display connected to the information processing device 108. The user looks at the confirmation screen in Figure 10 to confirm whether the composite image is displayed (whether the virtual object is displayed). If the composite image is not displayed, the user changes the resolution of the output image and repeats the display confirmation and setting change until the composite image is displayed on the HMD 100. The setting screen in Figure 10 includes the message "Please change the resolution setting," but it may also include a message indicating the direction of the resolution change, such as "Please lower the resolution."

[0051] As explained above, in Embodiment 3, as a way to deal with the situation when the HMD 100 cannot display the video (composite image) based on the output image, the user is notified to change the resolution of the output image. By doing so, the user can change the resolution of the output image so that the virtual object is displayed on the HMD 100 (display unit 104).

[0052] <Embodiment 4> In Embodiment 3, the user sets the resolution of the output image on the information processing device 108, visually checks whether the virtual object is displayed, and changes the resolution if the virtual object is not displayed. In Embodiment 4, when the HMD 100 receives the output image, it performs a data check of the output image and notifies the information processing device 108 of the check result. If the HMD 100 is unable to display the virtual object, the information processing device 108 notifies the user to that effect.

[0053] Figure 11 is a sequence diagram of the HMD 100 and the information processing device 108 in Embodiment 3.

[0054] In S1101, similar to S901 in Figure 9, the CPU 200 (output image generation unit 107) sets the resolution of the output image according to the user's operation.

[0055] In S1102, similar to S902, the CPU200 (rendering unit 106) renders a virtual object for testing. Then, the CPU200 (output image generation unit 107) generates an output image, including the image of the virtual object for testing, at the set resolution and sends it to the HMD100.

[0056] In S1103, the CPU 300 performs a data check on the output image received from the information processing device 108. If the data check in S1103 indicates that the HMD 100 cannot display the virtual object, in S1104, the CPU 300 sends a predetermined signal (error signal) to the information processing device 108. When the information processing device 108 receives the error signal, in S1105, C The PU200 controls the system to send predetermined notifications (error notifications) to the user.

[0057] If the HMD100 is unable to display a virtual object, it may be because, for example, the output image (output image data) received by the HMD100 is corrupted, or the output image data format is not the data format expected by the HMD100. If the output image (output image data) received by the HMD100 is corrupted, an error signal indicating data corruption is sent from the HMD100 to the information processing device 108, and the user is notified of the data corruption. Upon realizing the data corruption, the user attempts to reconnect the HMD100 and the information processing device 108, or resend the output image.

[0058] If the data format of the output image is not the data format expected by the HMD100, an error signal indicating a data format mismatch is sent from the HMD100 to the information processing device 108, and a notification prompting the user to change the data format (e.g., resolution) is issued. For example, the error screen shown in Figure 12 is displayed on a display connected to the information processing device 108. Alternatively, the CPU 200 may automatically reduce the resolution in response to the receipt of the error signal without displaying the error screen shown in Figure 12.

[0059] As described above, in Embodiment 4, if the HMD 100 is unable to display the video (composite image) based on the output image, a predetermined notification is given to the user. This allows the user to change the resolution of the output image, etc., without having to check the HMD 100.

[0060] Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of its gist.

[0061] The various controls described above may or may not be performed by a single piece of hardware (e.g., a processor or circuit). Multiple pieces of hardware (e.g., multiple processors, multiple circuits, or a combination of one or more processors and one or more circuits) may share the processing to control the entire device.

[0062] Furthermore, the above-mentioned processors are processors in a broad sense, including general-purpose processors and specialized processors. General-purpose processors include, for example, CPUs (Central Processing Units), MPUs (Micro Processing Units), and DSPs (Digital Signal Processors). Specialized processors include, for example, GPUs (Graphics Processing Units), ASICs (Application Specific Integrated Circuits), and PLDs (Programmable Logic Devices). Programmable logic devices include, for example, FPGAs (Field Programmable Gate Arrays) and CPLDs (Complex Programmable Logic Devices).

[0063] Furthermore, although embodiments of the present invention have been described in detail, the present invention is not limited to these specific embodiments, and various forms that do not depart from the spirit of the invention are also included in the present invention. Moreover, each of the embodiments described above is merely one embodiment of the present invention, and it is possible to combine each embodiment as appropriate.

[0064] <Other Embodiments> The present invention can also be realized by supplying a program that implements one or more of the functions of the above embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. Furthermore, it can also be achieved by a circuit that performs one or more functions.

[0065] This embodiment includes the following configurations, methods, and programs. (Composition 1) A first acquisition means for acquiring position and orientation information of the display device, A generation means generates a single image data that includes data of a virtual object image representing a virtual object, data of the transparency of the virtual object image, and data of the depth of the virtual object, based on the information acquired by the first acquisition means. A transmission means for transmitting the one image data generated by the generation means to an external party. An electronic device characterized by having the following features. (Configuration 2) The generating means is Based on the information acquired by the first acquisition means, the virtual object is rendered. Based on the rendering results, generate the one image data. The electronic device according to configuration 1, characterized by the features described above. (Composition 3) The rendering result includes a first data representing the virtual object image and the transparency, and a second data which is the depth data. The generation means divides the first data into the virtual object image data and the transparency data. The electronic device according to configuration 2, characterized by the features described above. (Composition 4) The generation means generates one image data representing an image in which the virtual object image and the images representing transparency and depth are placed side by side. An electronic device according to any one of configurations 1 to 3 characterized by the above. (Composition 5) The generation means generates one image data representing an image in which the virtual object image, the image representing transparency, and the image representing depth are arranged side by side. An electronic device according to any one of configurations 1 to 3 characterized by the above. (Composition 6) If the image size of one image data generated by the generation means is greater than a threshold, a reduction means reduces the image size of the one image data to the image size of the threshold. It further possesses, If the image size of the one image data generated by the generation means is greater than the threshold, the transmission means transmits the reduced image data from the reduction means to the outside. An electronic device according to any one of configurations 1 to 5, characterized by the above. (Composition 7) The threshold is the image size defined by the DisplayPort standard. The electronic device according to configuration 6, characterized by the features described therein. (Composition 8) The aforementioned threshold is for image sizes of 3840 x 2160 or less. The electronic device according to configuration 6 or 7, characterized by the above. (Composition 9) The system further includes a setting means for setting the image size specified by the user as the image size of the image data generated by the generation means. An electronic device according to any one of configurations 6 to 8, characterized by the features described above. (Composition 10) The one image data transmitted by the transmission means is input to the display device. The aforementioned electronic device is If the display device is unable to display a video based on the aforementioned one image data, the first receiving means receives a predetermined signal. It further possesses An electronic device according to any one of configurations 6 to 8, characterized by the features described above. (Composition 11) A control means that controls the first receiving means to provide a predetermined notification to the user when the predetermined signal is received by the first receiving means, Setting means for setting the image size specified by the user as the image size of the image data generated by the generation means. It further possesses The electronic device according to configuration 10, characterized by the above. (Composition 12) When the first receiving means receives the predetermined signal, a reduction means reduces the image size of the image data generated by the generating means. It further possesses The electronic device according to configuration 10, characterized by the above. (Composition 13) The transmission means transmits the one image data in accordance with the DisplayPort standard. An electronic device according to any one of configurations 1 to 12, characterized by the features described herein. (Composition 14) A second receiving means for receiving the one image data transmitted from the electronic device described in any one of configurations 1 to 13, A second acquisition means for acquiring data of a real-space image representing the real space and data of the depth of the said real space, A synthesis means that generates composite image data in which the virtual object is superimposed on the real space, based on the one image data received by the second receiving means and the real space image data and depth data acquired by the second acquisition means, Display means for displaying video based on the aforementioned composite image data A display device characterized by having the following features. (method) An acquisition step to obtain information on the position and orientation of the display device, A generation step generates a single image data set that includes data of a virtual object image representing a virtual object, data of the transparency of the virtual object image, and data of the depth of the virtual object, based on the information acquired in the acquisition step. A method for controlling an electronic device, characterized by comprising a transmission step of transmitting the one image data generated in the generation step to an external party. (program) A program for causing a computer to function as one of the electronic devices described in any one of configurations 1 to 13. [Explanation of symbols]

[0066] 100: HMD 200: CPU 105: Position and orientation information generation unit 106: Rendering unit 107: Output image generation unit

Claims

1. A first acquisition means for acquiring position and orientation information of the display device, A generation means generates a single image data that includes data of a virtual object image representing a virtual object, data of the transparency of the virtual object image, and data of the depth of the virtual object, based on the information acquired by the first acquisition means. A transmission means for transmitting the one image data generated by the generation means to an external party. An electronic device characterized by having the following features.

2. The generating means is Based on the information acquired by the first acquisition means, the virtual object is rendered. Based on the rendering results, generate the one image data. The electronic device according to feature 1.

3. The rendering result includes first data representing the virtual object image and transparency, and second data which is depth data. The generation means divides the first data into the virtual object image data and the transparency data. The electronic device according to feature 2.

4. The generation means generates one image data representing an image in which the virtual object image and the images representing transparency and depth are placed side by side. The electronic device according to feature 1.

5. The generation means generates one image data representing an image in which the virtual object image, the image representing transparency, and the image representing depth are arranged side by side. The electronic device according to feature 1.

6. If the image size of one image data generated by the generation means is greater than a threshold, a reduction means reduces the image size of the one image data to the image size of the threshold. It further possesses, If the image size of the one image data generated by the generation means is greater than the threshold, the transmission means transmits the reduced image data from the reduction means to the outside. The electronic device according to feature 1.

7. The threshold is the image size defined in the DisplayPort standard. The electronic device according to feature 6.

8. The threshold is an image size of 3840 x 2160 or less. The electronic device according to feature 6.

9. The system further includes a setting means for setting the image size specified by the user as the image size of the image data generated by the generation means. The electronic device according to feature 6.

10. The one image data transmitted by the transmission means is input to the display device. The aforementioned electronic device is If the display device is unable to display the video based on the aforementioned single image data, the first receiving means receives a predetermined signal. It further possesses The electronic device according to feature 6.

11. A control means that controls the first receiving means to provide a predetermined notification to the user when the predetermined signal is received by the first receiving means, Setting means for setting the image size specified by the user as the image size of the image data generated by the generation means. It further possesses The electronic device according to feature 10.

12. When the first receiving means receives the predetermined signal, a reduction means reduces the image size of the image data generated by the generating means. It further possesses The electronic device according to feature 10.

13. The transmission means transmits the one image data in accordance with the DisplayPort standard. The electronic device according to feature 1.

14. A second receiving means for receiving the one image data transmitted from the electronic device described in any one of claims 1 to 13, A second acquisition means for acquiring data of a real-space image representing the real space and data of the depth of the said real space, A synthesis means that generates composite image data in which the virtual object is superimposed on the real space, based on the one image data received by the second receiving means and the real space image data and depth data acquired by the second acquisition means, Display means for displaying video based on the aforementioned composite image data A display device characterized by having the following features.

15. An acquisition step to obtain information on the position and orientation of the display device, A generation step generates a single image data set that includes data of a virtual object image representing a virtual object, data of the transparency of the virtual object image, and data of the depth of the virtual object, based on the information acquired in the acquisition step. A method for controlling an electronic device, characterized by comprising a transmission step of transmitting the one image data generated in the generation step to an external party.

16. A program for causing a computer to function as one of the means of an electronic device according to any one of claims 1 to 13.

Images